A cross tenant metadata driven processing framework for Azure Data Factory and Azure Synapse Analytics achieved by coupling orchestration pipelines with a SQL database and a set of Azure Functions.
Called as part of the infant pipeline this Function performs a soft check against the target orchestrator to ensure the worker pipeline exists before continuing and attempting to execute the worker.
The soft check ensures the metadata is left in a controlled state instead of the proceeding execute pipeline function throwing an execption.
If the worker is available, its resource ID value will be returned with a PipelineExists value of true.
If the worker isn’t available in the target orchestrator instance the specific internal function exception is caught and supressed allowing the framework pipeline to deal with the problem and update the metadata. A PipelineExists value of false will be returned.
Other unexpected exceptions will still be thrown in the usual way.
Reasons for the validation check to fail include:
Namespace: mrpaulandrew.azure.procfwk.
GET, POST
{
"tenantId": "123-123-123-123-1234567",
"applicationId": "123-123-123-123-1234567",
"authenticationKey": "Passw0rd123!",
"subscriptionId": "123-123-123-123-1234567",
"resourceGroupName": "ADF.procfwk",
"orchestratorName": "FrameworkFactory",
"orchestratorType": "ADF",
"pipelineName": "Intentional Error"
}
See Services return classes.
Worker exists:
{
"PipelineExists": "True",
"PipelineName": "Wait 1",
"PipelineId": "/subscriptions/123-123-123-123-1234567/resourceGroups/ADF.procfwk/providers/Microsoft.DataFactory/factories/WorkersFactory/pipelines/Wait 1",
"PipelineType": "Microsoft.DataFactory/factories/pipelines",
"ActivityCount": "1"
}