A cross tenant metadata driven processing framework for Azure Data Factory and Azure Synapse Analytics achieved by coupling orchestration pipelines with a SQL database and a set of Azure Functions.

- Overview
- Contents

View the Project on GitHub mrpaulandrew/procfwk

Cross Tenant & Subscription Execution

« Contents

During execution of the processing framework worker pipelines are triggered for execution. These worker pipelines can be deployed to any Data Factory resource, anywhere within the Microsoft Azure platform. In addition, a single set of metadata can call out to worker pipelines in multiple Data Factory instances. As long as the framework is provided with the following details to authenticate against the taraget Data Factory instance the worker pipeline can be triggered.

To further clarify, within the metadata database these authentication details are connected to a worker pipeline. Meaning, granular authentication to a worker pipeline wherever it is deployed.

For example, the processing framework could be setup in the following ways depending on your requirements:

These are just example scenarios, any other combination of Tenant/Subscription/Data Factory is possible. See service principal handling for more details on setup and authentication storage options within the framework.

A demonstration of using cross tenant worker pipelines is available on YouTube here:

YouTube Demo Video