A cross tenant metadata driven processing framework for Azure Data Factory and Azure Synapse Analytics achieved by coupling orchestration pipelines with a SQL database and a set of Azure Functions.

- Overview
- Contents

View the Project on GitHub mrpaulandrew/procfwk

Welcome to the Orchestrate .procfwk Home Page

This site provides details on the latest version of the processing framework (procfwk) code project, available on GitHub here, as a single source of all information needed to use and support this solution.

Please use the Contents page, also available in the side bar, to navigate.

Code Project Overview

What is procfwk?

This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is made possible by coupling ADF with a SQL Database that houses execution batches, execution stages and pipeline information that is later called using an Azure Functions App. The execution batch, stage and worker pipeline metadata structure allows; concurrent overall batch handling with inner stages of process dependencies executed in sequence. Then at the lowest level, all worker pipelines within a stage to be executed in parallel offering scaled out control flows where no inter-dependencies exist.

The framework is designed to integrate with any existing Data Factory solution by making the lowest level executor a stand alone worker pipeline that is wrapped in a higher level of controlled (sequential) dependencies. This level of abstraction means operationally nothing about the monitoring of the orchestration processes is hidden in multiple levels of dynamic activity calls. Instead, everything from the processing pipeline doing the work (the Worker) can be inspected using out-of-the-box ADF features.

This framework can also be used in any Azure Tenant and allows the creation of complex control flows across multiple Data Factory resources and even across Azure Tenant/Subscriptions as well by connecting Service Principal details through metadata to targeted Tenants > Subscriptions > Resource Groups > Data Factory and Pipelines, this offers very granular administration over any data processing components in a given environment from a single point of orchestration.

Why use procfwk?

To answer the question of why use a metadata driven framework please watch the following YouTube video.

YouTube Video

Framework Capabilities

Deployment Steps

For details on how to deploy the processing framework to your Azure Tenant see Deploying ProcFwk.

Complete Data Factory Activity Chain

The following offers a view of all pipeline activities at every level within the processing framework if flattened out.