Replacing a Rogue Azure Function with ADF Orchestration

After an Azure Function wiped every D365 Gold table (see: A Shadow Azure Function Wiped Every Gold Table at 2am), I needed to replace its only useful feature — running Extract, Transform, and HRIS History in sequence — with something that used the existing ADF pipelines and their built-in safety mechanisms.

---

The Problem

The three ADF pipelines existed independently:

• ExtractDynamics365 — D365 OData to staging tables
• TransformDynamics365 — staging to Gold tables
• hrisHistory — snapshot vw_hris_detailed into SCD2 history table

None of them called the next. There was no ADF trigger that chained them. The only automation that ran all three in sequence was the Azure Function — which had just proven itself unsafe.

Stopping the Azure Function’s trigger and starting individual triggers for each pipeline would leave them on independent schedules with no dependency guarantees. Transform could fire before Extract finished. hrisHistory could snapshot stale data.

---

The Solution: ExecutePipeline Chaining

ADF has an ExecutePipeline activity type that calls one pipeline from another, with waitOnCompletion: true to block until the child finishes. I created OrchestrateD365 with three activities chained by success dependencies:

Extract (ExtractDynamics365)
  └─ on success ─→ Transform (TransformDynamics365)
                      └─ on success ─→ HrisHistory (hrisHistory)

Extract and Transform share the same four parameters (Environment, SrcSystem, RunBacklog, Pipelineid), passed through from the parent. hrisHistory takes no parameters.

If Extract fails, Transform never starts. If Transform fails, hrisHistory never starts. Each child pipeline retains its own retry logic, logging, and error handling — the orchestrator controls only sequencing.

Deployed via CLI:

az datafactory pipeline create \
  --resource-group sql-dataplatform-rg \
  --factory-name DataManagementServices \
  --name OrchestrateD365 \
  --pipeline @pipeline/OrchestrateD365.json

---

Trigger Consolidation

The old trigger landscape:

Trigger	State	Pipeline	Schedule
Trigger_Dynamics365	Started	TriggerAzureFunctionELT	Hourly
Trigger_hris_history	Started	hrisHistory	Hourly

Two independent hourly triggers. The first called the Azure Function, which ran its own Extract + Transform + PDD. The second ran hrisHistory standalone. They were not coordinated — hrisHistory could fire before the function’s Transform step finished, snapshotting incomplete data.

The new trigger landscape:

Trigger	State	Pipeline	Schedule
Trigger_Dynamics365	Stopped	TriggerAzureFunctionELT	—
Trigger_hris_history	Stopped	hrisHistory	—
Trigger_OrchestrateD365	Started	OrchestrateD365	Every 3 hours

One trigger, one pipeline, three stages in guaranteed order. I changed the interval from hourly to every 3 hours — HR data in D365 does not change frequently enough to justify hourly refreshes, and reducing frequency gives D365’s OData rate limiter more headroom.

# Stop old triggers
az datafactory trigger stop --name Trigger_Dynamics365 ...
az datafactory trigger stop --name Trigger_hris_history ...

# Create and start the new trigger
az datafactory trigger create --name Trigger_OrchestrateD365 \
  --properties '{
    "type": "ScheduleTrigger",
    "pipelines": [{
      "pipelineReference": {
        "referenceName": "OrchestrateD365",
        "type": "PipelineReference"
      },
      "parameters": {
        "Environment": "DEV",
        "SrcSystem": "D365",
        "RunBacklog": 0,
        "Pipelineid": -999
      }
    }],
    "typeProperties": {
      "recurrence": {
        "frequency": "Hour",
        "interval": 3,
        "startTime": "2026-05-06T03:00:00",
        "timeZone": "E. Africa Standard Time"
      }
    }
  }'

az datafactory trigger start --name Trigger_OrchestrateD365 ...

---

Manual Override

The orchestration pipeline accepts the same parameters as Extract and Transform, so running a single table manually still works:

# Full refresh (all active entities)
az datafactory pipeline create-run \
  --name OrchestrateD365 \
  --parameters '{"Environment":"DEV","SrcSystem":"D365",
                 "RunBacklog":0,"Pipelineid":-999}'

# Single table (e.g., config ID 9)
az datafactory pipeline create-run \
  --name OrchestrateD365 \
  --parameters '{"Environment":"DEV","SrcSystem":"D365",
                 "RunBacklog":0,"Pipelineid":9}'

The individual pipelines can still be triggered independently if needed — the orchestrator does not lock them.

---

What Remains

eltd365 is still deployed. Its trigger is stopped, but the function endpoint is live and could be called manually or by another system. It should be decommissioned once the new orchestration pipeline has proven stable over a few cycles.

TriggerAzureFunctionELT is still in ADF. It serves no purpose now, but I left it in place rather than deleting it — removing a pipeline that might be referenced elsewhere without a full dependency audit is not worth the risk at 4am.