Most Azure pipelines are monitored the way a smoke alarm monitors a house: they go off once something has already caught fire. Observability is how you build an Azure pipeline that tells you the data is wrong before the board pack does. Here is how, in practice.
We have written before that a pipeline does not have an uptime problem, it has a trust problem. This post is the engineering answer to that. If the last one was about why freshness and completeness matter to the business, this one is about how you actually build an Azure pipeline that tells you when it is wrong, before the board pack does.
Most Azure Data Factory and Fabric pipelines are monitored the way a smoke alarm monitors a house: they go off when something has already caught fire. The pipeline either completed or it failed. Green tick, move on. The problem is that the most expensive data failures do not fail the pipeline at all. A source system silently sends half the rows. A currency field arrives as text. A join quietly drops orphaned records. The pipeline runs green, the data is wrong, and nobody knows until a number looks off in a meeting three weeks later.
Observability is the discipline of instrumenting the pipeline so it can tell you the data is wrong even when the run succeeded. Here is what that looks like in practice on Azure.
The three questions observability has to answer
Good pipeline observability answers three questions at every run, not just did it finish.
Is the data fresh? When did each table last update, and is that within the window the business expects? A sales table that should refresh nightly and last updated 40 hours ago is a freshness failure, even if every run since was green, because the source stopped sending.
Is the data complete? Did roughly the expected volume of rows arrive? A feed that normally lands 50,000 rows and today landed 12,000 has not failed, it has been truncated upstream, and the pipeline has no idea unless you told it what normal looks like.
Is the data valid? Do the values obey the rules they must obey: no nulls in a key column, currencies in range, dates not in the future, referential integrity intact? These are the failures that pass every technical check and produce confidently wrong reports.
Uptime answers none of these. That is why uptime is a comfortable metric and a misleading one.
Building it on Azure: where the checks actually live
You do not need a separate observability product to do this well on Azure. You need to instrument the platform you already have.
Freshness and completeness at the silver boundary. As data is promoted from raw (bronze) to conformed (silver), write a small metadata record for every table: row count, load timestamp, source watermark. Compare each run against a rolling baseline, the average of the last N runs, and raise an alert when volume deviates beyond a threshold you set per table. In Fabric this is a lightweight notebook or dataflow step; in ADF it is a stored-procedure or Azure Function activity writing to a control table. The mechanism is boring. The discipline of doing it on every table is what separates a trustworthy platform from a hopeful one.
Validation as explicit tests, not assumptions. Every business rule that matters should be an assertion that runs as part of the pipeline and fails loudly when broken. Not-null on keys, uniqueness on identifiers, accepted ranges on measures, referential integrity between fact and dimension. Tools like dbt formalise these as data tests; you can achieve the same with SQL assertions in an ADF or Fabric pipeline. The point is that the rule is written down and checked automatically, not held in an engineer’s head.
Lineage so you can answer what else is affected. When a silver table is found to be wrong, the immediate question is which gold tables, which semantic models and which reports depend on it. Fabric surfaces lineage across the workspace; if you are on ADF and Synapse, Microsoft Purview gives you the same map. Without lineage, every data incident becomes a manual hunt, and the business loses trust faster from the response than from the original error.
Alerting to a human, with context. An alert that says pipeline failed is nearly useless. An alert that says customer_orders landed 76% below its 7-day average at 02:14, downstream: Sales gold mart, Exec semantic model tells someone what broke, how badly, and what is now at risk. Route these to where the team already looks, Teams, email, or an incident channel, not to a monitoring console nobody opens.
Reliability is a design choice, not a hope
Observability tells you when something is wrong. Reliability engineering reduces how often it happens and how badly it hurts when it does.
Idempotent, replayable loads. Every load should be safe to run twice. If a pipeline fails halfway and you re-run it, it must not double-count or half-populate. This is why bronze exists as an immutable landing zone: you replay from bronze rather than re-pulling from a production source that has moved on. A pipeline you cannot safely re-run is a pipeline you are afraid of, and fear is not an operating model.
High availability that matches the actual need. Azure gives you availability zones, geo-redundant storage and regional failover, but not every pipeline needs all of it, and paying for five-nines on a monthly finance load is waste. The right question is: what is the business cost of this data being late or unavailable for an hour, a day, a week? Match the redundancy to that answer. HA is a spectrum you buy deliberately, not a box you tick.
Security and access as part of the design. Who can read bronze versus gold, whose credentials the pipeline runs under, whether secrets sit in Key Vault or in plain text in a linked service: these are reliability concerns as much as security ones, because a pipeline that breaks when a rotated credential expires is an availability incident. Managed identities and Key Vault are not optional extras; they are how the pipeline keeps running when a password changes.
The SLA the business can actually read
All of this rolls up into a service level the business understands, and it is not 99.9% uptime. It is a promise in their language: the sales figures are complete and reconciled by 7am, and if they are not, you will be told before you present them. Freshness, completeness, and a commitment to warn, not a percentage about a server. That is the SLA worth writing, because it is the one the business can hold you to and the one that actually protects them.
Frequently asked questions
What is data pipeline observability?
It is instrumenting a pipeline so it can tell you when the data is wrong, not just when the pipeline failed. It answers three questions at every run: is the data fresh, is it complete, and is it valid. A pipeline can complete successfully and still deliver truncated or invalid data; observability is what catches that.
How is observability different from monitoring?
Monitoring tells you whether the pipeline ran and how long it took, the operational health of the process. Observability tells you whether the output is trustworthy: freshness, row-count completeness, and validation of business rules. Most Azure pipelines are monitored but not observed, which is why silent data failures survive to reach a report.
Do we need a separate tool for this on Azure?
Usually not. Freshness and completeness checks can be written as pipeline steps in Fabric or Azure Data Factory that log to a control table; validation can be done with SQL assertions or dbt tests; lineage comes from Fabric’s built-in view or Microsoft Purview. A dedicated observability product can help at scale, but the discipline matters far more than the tool.
What does a good data SLA look like?
Not 99.9% uptime. A good SLA is written in business terms: which datasets are guaranteed complete and reconciled, by what time each day, and a commitment to alert the business when that promise is at risk, before they rely on the data, not after.
Hopton Analytics builds Azure data pipelines that are observable by design, so your team finds out about a bad load before your board does. If your pipelines are green but you still do not fully trust the numbers, book a free audit.
James Morley
Lead Data Engineer
Part of the Hopton Analytics team, delivering governed analytics programmes for UK mid-market organisations.