Orchestrate
.procfwk
A cross tenant metadata driven processing framework for Azure Data Factory and Azure Synapse Analytics achieved by coupling orchestration pipelines with a SQL database and a set of Azure Functions.
- Overview
- Contents
Functions (Azure)
- Execute Pipeline
- Check Pipeline Status
- Validate Pipeline
- Get Error Details
- Cancel Pipeline
- Send Email
Azure Functions provide the middle ware in the processing framework allowing the orchestration pipelines to interact with the worker pipelines or access external resources.
The highly scalable, cheap, serverless compute provides the required customisations to extend the framework and acheive a level of decoupling between orchestration resources as well as supporting its own authenticate to Key Vault when needed.
All the Functions within the processing framework are created using:
Services
To support the interchangeable use of orchestrators in the processing framework (Azure Data Factor or Azure Synapse Analytics) the Function App’s internal classes are setup using a services architecture.
At runtime the abstract class PipelineService is called which inspects the request for the orchestrator type. Given the enumerable value the required orchestrator service class is then instantiated.
See services for more details.
Helpers
Alongside the primary function methods a set of helper classes are used to define return types and support the function requests.
See helpers for more details.
Managed Identity (MSI)
Enabling the Azure Function App managed identity is optional if you want to store worker pipeline SPN details in Azure Key Vault.
The MSI could also be used to authenticate against the worker pipelines directly removing the need for the SPN metadata. Although this option is understood, it is not implemented or supported by the framework as it would underpin the cross Azure tenant execution support.