A cross tenant metadata driven processing framework for Azure Data Factory and Azure Synapse Analytics achieved by coupling orchestration pipelines with a SQL database and a set of Azure Functions.

- Overview
- Contents

View the Project on GitHub mrpaulandrew/procfwk

Cross Tenant & Subscription Execution

« Contents

During a run of the processing framework worker pipelines are triggered for execution. These worker pipelines can be deployed to any orchestrator resource, anywhere within the Microsoft Azure platform. In addition, a single set of metadata can call out to worker pipelines in multiple orchestrator instances. As long as the framework is provided with the following details to authenticate against the target orchestrator instance where the worker pipeline can be triggered.

To further clarify, within the metadata database these authentication details are connected to a worker pipeline. Meaning, granular authentication to a worker pipeline wherever it is deployed.

For example, the processing framework could be setup in the following ways depending on your requirements:

These are just example scenarios, any other combination of Tenant/Subscription/orchestrator is possible. See service principal handling for more details on setup and authentication storage options within the framework.

Azure Data Factory and Azure Synapse Analytics are also interchangeable in any of the above statements given the processing frameworks support for different orchestrator.

A demonstration of using cross tenant worker pipelines is available on YouTube here for Azure Data Factory:

YouTube Demo Video