A cross tenant metadata driven processing framework for Azure Data Factory and Azure Synapse Analytics achieved by coupling orchestration pipelines with a SQL database and a set of Azure Functions.

- Overview
- Contents

View the Project on GitHub mrpaulandrew/procfwk

Metadata Integrity Checks


For every new execution of the processing framework the database stored procedure [procfwk].[CheckMetadataIntegrity] is called to inspect and check the content of the metadata tables.

This procedure is intended as a set pass/fail checks to prevent any runtime failures during a given framework execution run. The stored procedure is called near the very start of the parent pipeline and includes the following metadata checks:

In the event any check fails an exception will be raised stopping the parent pipeline and framework execution run from proceeding.

If an exception is thrown, the stored procedure can be used in debug mode, seen below, where details of the failed check(s) can be return as a result set via a SQL database client.

EXEC [procfwk].[CheckMetadataIntegrity] 
    @DebugMode = 1;

--OR, with batch executions enabled:

EXEC [procfwk].[CheckMetadataIntegrity] 
    @BatchName = 'Daily',
    @DebugMode = 1;