A cross tenant metadata driven processing framework for Azure Data Factory and Azure Synapse Analytics achieved by coupling orchestration pipelines with a SQL database and a set of Azure Functions.
Acts as the entry point and basis for all interactions between the framework and the worker pipelines.
The services concept is used to decouple interactions from the intended orchestrator type and allow future orchestrator types to be included by implementing a new set of service methods. In all cases the below abstract methods will be used for the type provided.
Namespace: mrpaulandrew.azure.procfwk.Services.
public static PipelineService GetServiceForRequest
(
PipelineRequest pr,
ILogger logger
)
{
if (pr.OrchestratorType == PipelineServiceType.ADF)
return new AzureDataFactoryService(pr, logger);
if (pr.OrchestratorType == PipelineServiceType.SYN)
return new AzureSynapseService(pr, logger);
throw new InvalidRequestException (
"Unsupported orchestrator type: " +
(pr.OrchestratorType?.ToString() ?? "<null>"));
}
Properties:
Property Name | Type |
---|---|
PipelineExists | string |
PipelineName | string |
PipelineId | string |
PipelineType | string |
ActivityCount | int |
Properties:
Property Name | Type |
---|---|
PipelineName | string |
RunId | string |
ActualStatus | string |
SimpleStatus | string |
The simple status value is resolved internally from the actual status in the return class using a private method and switch statement.
string simpleStatus = actualStatus switch
{
"Queued" => Running,
"InProgress" => Running,
"Canceling" => Running, //microsoft typo
"Cancelling" => Running,
_ => Complete,
};
return simpleStatus;
Inherits PipelineRunStatus.
The constructor instantiates the FailedActivity list below.
Properties:
Property Name | Type |
---|---|
ResponseCount | int |
ResponseErrorCount | int |
FailedActivity | list |
Internally set properties:
Property Name | Type |
---|---|
ActivityRunId | string |
ActivityName | string |
ActivityType | string |
ErrorCode | string |
ErrorType | string |
ErrorMessage | string |