A cross tenant metadata driven processing framework for Azure Data Factory and Azure Synapse Analytics achieved by coupling orchestration pipelines with a SQL database and a set of Azure Functions.
Email alerting can be delivered within the processing framework and using existing metadata to drive behaviour offering easy control, flexible and granular ways to subscribe to the status of a given worker pipeline outcome.
The following statements have been met in terms of alerting capabilities and design:
Using email alerting in as part of the processing framework is set by the property UseFrameworkEmailAlerting.
As stated above, the alerting features of the framework are completely optional. You may already have a different way of producing alerts for your pipelines. If so, simply set this property value to ‘0’ and alerting will be disabled for your processing framework.
The following helper stored procedure can be used to add and remove email recipients from the metadata database tables.
[procfwk].[AddRecipientPipelineAlerts] - Assuming a recipient email address record has already been added this procedure creates the alert link between the recipient record and the Worker pipeline(s). Default parameters can be used to add the link for either a specific pipeline or all pipelines. Then either all pipeline status values or just a subset. A comma separated string can be used to translate the named values into the correct bitmask value.
The following code snippets provide example of adding alerts for three different recipients for different worker pipelines and different combinations are outcomes.
EXEC [procfwkHelpers].[AddRecipientPipelineAlerts]
@RecipientName = N'Test User 1',
@AlertForStatus = 'All';
EXEC [procfwkHelpers].[AddRecipientPipelineAlerts]
@RecipientName = N'Test User 2',
@PipelineName = 'Intentional Error',
@AlertForStatus = 'Failed';
EXEC [procfwkHelpers].[AddRecipientPipelineAlerts]
@RecipientName = N'Test User 3',
@PipelineName = 'Wait 1',
@AlertForStatus = 'Success, Failed, Cancelled';
Once the metadata has been returned from the database to the orchestrators infant pipeline emails are sent by the framework using the Send Email Azure Function.
For re-use outside of the processing framework the email sender is also wrapped up as a utilities pipeline that exposes email content as pipeline level parameters.