A green tick is not reliability. How to make Azure and Fabric data pipelines reliable: freshness, completeness and validation checks, alerts and SLAs.
Updated 27 October 2026 with the Fabric monitoring tools, a diagram of the reliability loop, and new questions.
You make Azure data pipelines reliable by instrumenting them to answer three questions on every run, is the data fresh, is it complete and is it valid, and by treating a failed answer as an alert to a named person rather than a line in a log. The run succeeding is not the test; the numbers being right is. Reliability for business reporting is a promise the business can read (sales updated by 7am, within half a percent of the ledger), loads that are safe to replay, and a report that shows when it was last loaded. In Fabric the tools are pipeline run history, workspace monitoring, Activator and the capacity metrics app; in Azure Data Factory they are pipeline monitoring with Azure Monitor and Log Analytics. The practice is the same in both, and this post sets it out.
We have written before that a pipeline does not have an uptime problem, it has a trust problem. This post is the engineering answer to that. If the last one was about why freshness and completeness matter to the business, this one is about how you actually build an Azure pipeline that tells you when it is wrong, before the board pack does.
Most Azure Data Factory and Fabric pipelines are monitored the way a smoke alarm monitors a house: they go off when something has already caught fire. The pipeline either completed or it failed. Green tick, move on. The problem is that the most expensive data failures do not fail the pipeline at all. A source system silently sends half the rows. A currency field arrives as text. A join quietly drops orphaned records. The pipeline runs green, the data is wrong, and nobody knows until a number looks off in a meeting three weeks later.
Observability is the discipline of instrumenting the pipeline so it can tell you the data is wrong even when the run succeeded. Here is what that looks like in practice on Azure.
The three questions observability has to answer
Good pipeline observability answers three questions at every run, not just did it finish.
Is the data fresh? When did each table last update, and is that within the window the business expects? A sales table that should refresh nightly and last updated 40 hours ago is a freshness failure, even if every run since was green, because the source stopped sending.
Is the data complete? Did roughly the expected volume of rows arrive? A feed that normally lands 50,000 rows and today landed 12,000 has not failed, it has been truncated upstream, and the pipeline has no idea unless you told it what normal looks like.
Is the data valid? Do the values obey the rules they must obey: no nulls in a key column, currencies in range, dates not in the future, referential integrity intact? These are the failures that pass every technical check and produce confidently wrong reports.
Uptime answers none of these. That is why uptime is a comfortable metric and a misleading one.
Building it on Azure: where the checks actually live
You do not need a separate observability product to do this well on Azure. You need to instrument the platform you already have.
Freshness and completeness at the silver boundary. As data is promoted from raw (bronze) to conformed (silver), write a small metadata record for every table: row count, load timestamp, source watermark. Compare each run against a rolling baseline, the average of the last N runs, and raise an alert when volume deviates beyond a threshold you set per table. In Fabric this is a lightweight notebook or dataflow step; in ADF it is a stored-procedure or Azure Function activity writing to a control table. The mechanism is boring. The discipline of doing it on every table is what separates a trustworthy platform from a hopeful one.
Validation as explicit tests, not assumptions. Every business rule that matters should be an assertion that runs as part of the pipeline and fails loudly when broken. Not-null on keys, uniqueness on identifiers, accepted ranges on measures, referential integrity between fact and dimension. Tools like dbt formalise these as data tests; you can achieve the same with SQL assertions in an ADF or Fabric pipeline. The point is that the rule is written down and checked automatically, not held in an engineer’s head.
Lineage so you can answer what else is affected. When a silver table is found to be wrong, the immediate question is which gold tables, which semantic models and which reports depend on it. Fabric surfaces lineage across the workspace; if you are on ADF and Synapse, Microsoft Purview gives you the same map. Without lineage, every data incident becomes a manual hunt, and the business loses trust faster from the response than from the original error.
Alerting to a human, with context. An alert that says pipeline failed is nearly useless. An alert that says customer_orders landed 76% below its 7-day average at 02:14, downstream: Sales gold mart, Exec semantic model tells someone what broke, how badly, and what is now at risk. Route these to where the team already looks, Teams, email, or an incident channel, not to a monitoring console nobody opens.

Reliability is a design choice, not a hope
Observability tells you when something is wrong. Reliability engineering reduces how often it happens and how badly it hurts when it does.
Idempotent, replayable loads. Every load should be safe to run twice. If a pipeline fails halfway and you re-run it, it must not double-count or half-populate. This is why bronze exists as an immutable landing zone: you replay from bronze rather than re-pulling from a production source that has moved on. A pipeline you cannot safely re-run is a pipeline you are afraid of, and fear is not an operating model.
High availability that matches the actual need. Azure gives you availability zones, geo-redundant storage and regional failover, but not every pipeline needs all of it, and paying for five-nines on a monthly finance load is waste. The right question is: what is the business cost of this data being late or unavailable for an hour, a day, a week? Match the redundancy to that answer. HA is a spectrum you buy deliberately, not a box you tick.
Security and access as part of the design. Who can read bronze versus gold, whose credentials the pipeline runs under, whether secrets sit in Key Vault or in plain text in a linked service: these are reliability concerns as much as security ones, because a pipeline that breaks when a rotated credential expires is an availability incident. Managed identities and Key Vault are not optional extras; they are how the pipeline keeps running when a password changes.
The SLA the business can actually read
All of this rolls up into a service level the business understands, and it is not 99.9% uptime. It is a promise in their language: the sales figures are complete and reconciled by 7am, and if they are not, you will be told before you present them. Freshness, completeness, and a commitment to warn, not a percentage about a server. That is the SLA worth writing, because it is the one the business can hold you to and the one that actually protects them.
Hopton Analytics builds Azure data pipelines that are observable by design, so your team finds out about a bad load before your board does. If your pipelines are green but you still do not fully trust the numbers, book a free audit.
Build for failure, because failure is normal
Pipelines rarely fail because engineers are careless. They fail because a source API times out, a file lands late, a schema changes without warning or two runs collide. Resilient pipelines assume all of that: retries with backoff on anything that touches a network, restart-safe processing so re-running a failed step gives the same result rather than duplicate rows, checkpoints so a job resumes where it stopped, and isolation so one failed feed does not stop the other nine. Keeping raw data unchanged in the bronze layer is what makes a pipeline replayable: if a bug is found downstream, history can be reprocessed without re-extracting from the source.
Alerts people act on
Most teams switch off pipeline alerts within a month, because the first version wakes someone at 2am for a warning that fixes itself, and after the third false alarm nobody trusts the channel. Alerts that last have thresholds set to what a person can actually do about them, a severity that matches the impact, and a route to a named owner rather than a shared inbox.
Refresh the report when the data is ready
A pipeline that lands clean gold data and then waits hours for a scheduled dataset refresh has not finished the job. Trigger the Power BI or Fabric semantic model refresh from the pipeline itself, so reports update as soon as the data has passed its checks, and make that refresh one more monitored step rather than a separate schedule. Our post on automating ETL for Power BI reporting covers the full pattern.
Related reading
The Data Engineering Layer That Decides Everything
Data model or ontology? What the meaning layer actually looks like on a Microsoft estate
What good Azure data engineering looks like when it is running well
How do you automate ETL for Power BI reporting?
Testing a data pipeline: the checks that catch wrong numbers before the board does
James Morley
Analytics Solutions Consultant
Part of the Hopton Analytics team, delivering governed analytics programmes for UK mid-market organisations.



