Your data pipeline doesn't have an uptime problem. It has a trust problem.

Uptime tells you the server was on. It says nothing about whether Monday's board pack was right. Here is how we design Azure data pipelines around freshness, completeness and trust — with SLAs the business actually understands.

Ask most teams whether their data pipeline is reliable and they will quote you an uptime figure. Ninety-nine point something. The servers were on, the jobs ran, nothing went red. And yet the finance director still opened Monday’s board pack, spotted a number that looked wrong, and quietly rebuilt the figure in a spreadsheet before the meeting.

That is the gap. Uptime measures the plumbing. Trust measures whether the business acts on what the plumbing delivers. A pipeline can be up all week and still ship stale, incomplete or subtly wrong data into a report someone is about to make a decision on. Reliability, done properly, is a promise about the data — not the infrastructure.

Here is how we design that promise into Azure data pipelines for mid-market teams.

Reliability is a business promise, not a green tick

Start by defining reliability in terms the business already cares about:

• Freshness — how recently the data reflects reality. “Yesterday’s sales by 07:00” is a freshness commitment. A green pipeline that delivered at 11:00 has failed it.
• Completeness — did every expected source and record arrive? A feed that silently drops one region is worse than a feed that fails loudly, because nobody goes looking.
• Accuracy — do the numbers reconcile to the source system and to each other?
• Stakeholder trust — the lagging indicator. It is earned when freshness, completeness and accuracy hold for long enough that people stop rebuilding numbers by hand.

Measure these, publish them, and reliability stops being an engineering abstraction.

Set SLAs the business understands

Not every dataset deserves the same effort, and pretending otherwise wastes money. We tier data by how much a wrong or late number actually costs.

• Tier 1 — board, finance and regulatory reporting. Target availability around 99.9%, delivered and reconciled before the working day starts, freshness measured in minutes to a couple of hours, alerting that pages a human, and a documented incident response with a named owner.
• Tier 2 — operational dashboards teams run their week on. High availability, refreshed several times a day, alerting into a shared channel, next-business-day resolution.
• Tier 3 — exploratory and long-tail datasets. Best-effort, refreshed daily or weekly, monitored but not paged.

The tier sets the SLA. The SLA sets the alerting and the incident-response expectation. Everyone knows, in advance, what “broken” means for each dataset and who moves when it breaks.

Build for failure, because failure is the normal case

Pipelines do not fail because engineers are careless. They fail because a source API times out, a file lands late, a schema changes without warning, or a run collides with another. Resilient pipelines assume all of that and keep their promises anyway:

• Retries with backoff on anything that touches a network, so a transient blip self-heals instead of paging someone at 3am.
• Restart-safe, idempotent processing — re-running a failed step produces the same result rather than duplicating rows. This is the single most valuable property a pipeline can have.
• Checkpoints so a job resumes from where it stopped instead of reprocessing everything.
• Modular pipeline isolation so a failure in one feed does not take down the other nine.
• Centralised monitoring — one place to see every pipeline’s state, freshness and last-good run, rather than logs scattered across services.

Structure the data so trust is designed in: the medallion pattern

How you lay the data out matters as much as how you move it. We use the medallion architecture — bronze, silver, gold — because it makes each stage’s job explicit.

• Bronze is raw, immutable landing. You keep the data exactly as it arrived, along with its ingestion metadata — where it came from, when it landed, which run produced it. Bronze is what makes a pipeline replayable: if a downstream bug is found, you can reprocess history from source without re-extracting.
• Silver is where raw becomes trustworthy: deduplication, schema standardisation, type enforcement, validation rules and quality checks. Bad records are quarantined here, not silently dropped.
• Gold is business-modelled and analytics-ready — conformed dimensions, agreed measures, the shapes your semantic model and reports consume.

Because each layer has one responsibility, you get auditability for free: you can trace any number in a gold report back through silver to the exact bronze record it came from. That lineage is what lets you answer “why did this figure change?” with a fact instead of a shrug.

How we implement it on Azure

The pattern maps cleanly onto modular layers — ingestion, storage, transformation, serving and orchestration — that you can build and test independently. In practice, on the Microsoft stack that usually means:

• Ingestion and orchestration in Azure Data Factory or Fabric Data Factory pipelines — linked services to each source, parameterised datasets, schedule and event triggers, and pipeline-level retry policies and timeouts set deliberately rather than left on defaults.
• Storage as a lakehouse over OneLake or ADLS Gen2, with the bronze, silver and gold zones physically separated.
• Transformation in the tool that fits the workload — dataflows or notebooks for heavier logic, SQL for set-based work.
• Monitoring wired into Azure Monitor or the Fabric monitoring hub, with alerts tied to the SLA tier, not to every warning.
• CI/CD so pipelines are deployed from source control through dev, test and production, not clicked together by hand in a live workspace.

On platform choice, the honest guidance is that it depends on where you already are. Azure Data Factory and Synapse remain the right call for teams with an established Azure estate and heavy, cost-sensitive orchestration. Microsoft Fabric is increasingly the right call when you want ingestion, storage, transformation and Power BI in one governed SaaS environment and you value simplicity over granular control. Most mid-market teams do not need a multi-cloud lakehouse platform, and paying for one is a common and expensive mistake.

Automate the last mile

A pipeline that lands perfect gold data and then waits eight hours for a manual dataset refresh has not finished the job. Close the loop: trigger Power BI or Fabric semantic-model refreshes from the pipeline itself, so reports update the moment new data is certified — event-driven refreshes rather than fixed 6am schedules that fire whether or not the data actually arrived. Refresh becomes one more governed, monitored step in the flow, not a separate thing someone remembers to press.

What good looks like

A reliable pipeline is not one that never fails. It is one that fails in isolation, recovers on its own, tells you honestly when it cannot, and — because the data is structured for it — lets you prove exactly what happened and replay it cleanly. Do that consistently and the spreadsheets by the finance director’s elbow quietly disappear. That is the real metric.

If your reporting still gets rebuilt by hand before the important meetings, that is a trust problem worth fixing — and exactly the kind of work our Analytics Acceleration Programme is built for.