Orchestrate
.procfwk

Logo

A cross tenant metadata driven processing framework for Azure Data Factory and Azure Synapse Analytics achieved by coupling orchestration pipelines with a SQL database and a set of Azure Functions.


- Overview
- Contents


View the Project on GitHub mrpaulandrew/procfwk

Properties


« Contents / Database


The properties table within the metadata database controls configuration to support different framework processing behaviour. The table supports value history and should be updated using the stored procedure [procfwkHelpers].[AddProperty].

Properties:


OverideRestart

Default Value: 0

Role: Should processing not be restarted from the point of failure or should a new execution will be created regardless. 1 = Start New, 0 = Restart. See framework restarts.


PipelineStatusCheckDuration

Default Value: 30

Role: Duration applied to the Wait activity within the Infant pipeline Until iterations.


UnknownWorkerResultBlocks

Default Value: 1

Role: If a worker pipeline returns an unknown status. Should this block and fail downstream pipeline? 1 = Yes, 0 = No.


CancelledWorkerResultBlocks

Default Value: 1

Role: If a worker pipeline returns an cancelled status. Should this block and fail downstream pipeline? 1 = Yes, 0 = No.


UseFrameworkEmailAlerting

Default Value: 1

Role: Do you want the framework to handle pipeline email alerts via the database metadata? 1 = Yes, 0 = No.


EmailAlertBodyTemplate

Default Value:

<hr/>
<strong>Pipeline Name: </strong>##PipelineName###<br/>
<strong>Status: </strong>##Status###<br/><br/>
<strong>Execution ID: </strong>##ExecId###<br/>
<strong>Run ID: </strong>##RunId###<br/><br/>
<strong>Start Date Time: </strong>##StartDateTime###<br/>
<strong>End Date Time: </strong>##EndDateTime###<br/>
<strong>Duration (Minutes): </strong>##Duration###<br/><br/>
<strong>Called by Orchestrator: </strong>##CalledByOrc###<br/>
<strong>Executed by Orchestrator Type: </strong>##ExecutedByOrcType###<br/>
<strong>Executed by Orchestrator: </strong>##ExecutedByOrc###<br/>
<hr/>

Role: Basic HTML template of execution information used as the eventual body in email alerts sent.


FailureHandling

Accepted Values: None, Simple, DependencyChain

Role: Controls processing bahaviour in the event of Worker failures. See failure handling.


SPNHandlingMethod

Accepted Values: StoreInDatabase, StoreInKeyVault.

Role: Controls how service principal values are stored within the framework. See SPN Handling.


ExecutionPrecursorProc

Example Value: [dbo].[ExampleCustomExecutionPrecursor]

Role: This procedure will be called first in the parent pipeline and can be used to perform/update any required custom behaviour in the framework execution. For example, enable/disable Worker pipelines given a certain run time/day. Invalid proc name values will be ignored.


UseExecutionBatches

Example Values: 0/1

Role: Establishes if execution batches are used as a level above execution stages within the framework processing.


PreviousPipelineRunsQueryRange

Example Value: -1

Role: Used as a date range, today +- this value, when checking for if an execution for a given pipeline is already running. The primary use for the property is within the utility pipeline to check if the parent is already running. Here the value is wrapped in the following orchestrator expression as a request body passed to the Azure Management API:

{
  "lastUpdatedAfter": "@{adddays(utcnow(),int(activity('Get Query Run Days Value').output.firstRow.PropertyValue))}",
  "lastUpdatedBefore": "@{utcnow()}",
  "filters": [
    {
      "operand": "PipelineName",
      "operator": "Equals",
      "values": [
        "@{pipeline().parameters.PipelineName}"
      ]
    }
  ]
}