Orchestrate
.procfwk

Logo

A cross tenant metadata driven processing framework for Azure Data Factory and Azure Synapse Analytics achieved by coupling orchestration pipelines with a SQL database and a set of Azure Functions.


- Overview
- Contents


View the Project on GitHub mrpaulandrew/procfwk

Properties


¬ę¬†Contents / Database


The properties table within the metadata database controls configuration to support different framework processing behaviour. The table supports value history and should be updated using the stored procedure [procfwkHelpers].[AddProperty].

Properties:


OverideRestart

Default Value: 0

Role: Should processing not be restarted from the point of failure or should a new execution will be created regardless. 1 = Start New, 0 = Restart. See framework restarts.


PipelineStatusCheckDuration

Default Value: 30

Role: Duration applied to the Wait activity within the Infant pipeline Until iterations.


UnknownWorkerResultBlocks

Default Value: 1

Role: If a worker pipeline returns an unknown status. Should this block and fail downstream pipeline? 1 = Yes, 0 = No.


CancelledWorkerResultBlocks

Default Value: 1

Role: If a worker pipeline returns an cancelled status. Should this block and fail downstream pipeline? 1 = Yes, 0 = No.


UseFrameworkEmailAlerting

Default Value: 1

Role: Do you want the framework to handle pipeline email alerts via the database metadata? 1 = Yes, 0 = No.


EmailAlertBodyTemplate

Default Value:

<hr/>
<strong>Pipeline Name: </strong>##PipelineName###<br/>
<strong>Status: </strong>##Status###<br/><br/>
<strong>Execution ID: </strong>##ExecId###<br/>
<strong>Run ID: </strong>##RunId###<br/><br/>
<strong>Start Date Time: </strong>##StartDateTime###<br/>
<strong>End Date Time: </strong>##EndDateTime###<br/>
<strong>Duration (Minutes): </strong>##Duration###<br/><br/>
<strong>Called by Data Factory: </strong>##CalledByADF###<br/>
<strong>Executed by Data Factory: </strong>##ExecutedByADF###<br/>
<hr/>	

Role: Basic HTML template of execution information used as the eventual body in email alerts sent.


FailureHandling

Accepted Values: None, Simple, DependencyChain

Role: Controls processing bahaviour in the event of Worker failures. See failure handling.


SPNHandlingMethod

Accepted Values: StoreInDatabase, StoreInKeyVault.

Role: Controls how service principal values are stored within the framework. See SPN Handling.


ExecutionPrecursorProc

Example Value: [dbo].[ExampleCustomExecutionPrecursor]

Role: This procedure will be called first in the parent pipeline and can be used to perform/update any required custom behaviour in the framework execution. For example, enable/disable Worker pipelines given a certain run time/day. Invalid proc name values will be ignored.