A cross tenant metadata driven processing framework for Azure Data Factory and Azure Synapse Analytics achieved by coupling orchestration pipelines with a SQL database and a set of Azure Functions.
The properties table within the metadata database controls configuration to support different framework processing behaviour. The table supports value history and should be updated using the stored procedure [procfwkHelpers].[AddProperty].
Default Value: 0
Role: Should processing not be restarted from the point of failure or should a new execution will be created regardless. 1 = Start New, 0 = Restart. See framework restarts.
Default Value: 30
Role: Duration applied to the Wait activity within the Infant pipeline Until iterations.
Default Value: 1
Role: If a worker pipeline returns an unknown status. Should this block and fail downstream pipeline? 1 = Yes, 0 = No.
Default Value: 1
Role: If a worker pipeline returns an cancelled status. Should this block and fail downstream pipeline? 1 = Yes, 0 = No.
Default Value: 1
Role: Do you want the framework to handle pipeline email alerts via the database metadata? 1 = Yes, 0 = No.
Default Value:
<hr/>
<strong>Pipeline Name: </strong>##PipelineName###<br/>
<strong>Status: </strong>##Status###<br/><br/>
<strong>Execution ID: </strong>##ExecId###<br/>
<strong>Run ID: </strong>##RunId###<br/><br/>
<strong>Start Date Time: </strong>##StartDateTime###<br/>
<strong>End Date Time: </strong>##EndDateTime###<br/>
<strong>Duration (Minutes): </strong>##Duration###<br/><br/>
<strong>Called by Orchestrator: </strong>##CalledByOrc###<br/>
<strong>Executed by Orchestrator Type: </strong>##ExecutedByOrcType###<br/>
<strong>Executed by Orchestrator: </strong>##ExecutedByOrc###<br/>
<hr/>
Role: Basic HTML template of execution information used as the eventual body in email alerts sent.
Accepted Values: None, Simple, DependencyChain
Role: Controls processing bahaviour in the event of Worker failures. See failure handling.
Accepted Values: StoreInDatabase, StoreInKeyVault.
Role: Controls how service principal values are stored within the framework. See SPN Handling.
Example Value: [dbo].[ExampleCustomExecutionPrecursor]
Role: This procedure will be called first in the parent pipeline and can be used to perform/update any required custom behaviour in the framework execution. For example, enable/disable Worker pipelines given a certain run time/day. Invalid proc name values will be ignored.
Example Values: 0/1
Role: Establishes if execution batches are used as a level above execution stages within the framework processing.
Example Value: -1
Role: Used as a date range, today +- this value, when checking for if an execution for a given pipeline is already running. The primary use for the property is within the utility pipeline to check if the parent is already running. Here the value is wrapped in the following orchestrator expression as a request body passed to the Azure Management API:
{
"lastUpdatedAfter": "@{adddays(utcnow(),int(activity('Get Query Run Days Value').output.firstRow.PropertyValue))}",
"lastUpdatedBefore": "@{utcnow()}",
"filters": [
{
"operand": "PipelineName",
"operator": "Equals",
"values": [
"@{pipeline().parameters.PipelineName}"
]
}
]
}