Microsoft Fabric Data Pipelines & Dataflows - FAQs

Use Fabric Data Factory for new builds in most cases, rather than Azure Data Factory (ADF). The capabilities are similar (and growing closer) but the Fabric version integrates natively with OneLake and the rest of the Fabric platform without separate connection management. For existing ADF investments, the migration path to Fabric Data Factory is straightforward but not always urgent; existing ADF pipelines continue to work and can stay on standalone ADF for the foreseeable future. For new mid-market analytics builds, the default is Fabric Data Factory.

Yes — Microsoft Fabric and Databricks are increasingly competing for the same workloads. Both platforms can do most things: lakehouse, data warehouse, BI, ML, real-time. The competition is real. The differences are in optimisation and culture rather than capability. Fabric is built around BI-led mixed workloads. Databricks is built around code-first lakehouse and ML. Both can do the other thing, but each is designed for a different centre of gravity, and that is what shapes the decision.

Azure Data Factory (and Synapse) suits teams with an established Azure estate and heavy, cost-sensitive orchestration that want granular control. Microsoft Fabric suits teams who want ingestion, storage, transformation and Power BI unified in one governed SaaS platform and value simplicity. For most mid-market organisations, Fabric’s consolidation is the better fit; the mistake to avoid is buying a large multi-cloud platform you do not need.

We test data pipelines in Microsoft Fabric through three test types. Unit tests on the transformation logic itself (does this Python function produce the expected output for known inputs). Integration tests on pipeline behaviour (does this pipeline correctly read from the source and write the expected schema to the destination). Data quality tests on the output (does the produced data meet the quality expectations: row counts in expected ranges, no null values in critical columns, foreign keys resolve). The Microsoft Fabric ecosystem supports each of these through standard tools (pytest, Great Expectations, SQL assertions). Mature data teams run all three; ad-hoc teams run none.

A single view of pipeline health comes from a central monitoring view rather than checking each pipeline individually. Fabric's monitoring hub gives one place to see every pipeline run across a workspace: its status, how long it took compared with previous runs, and where in the pipeline it failed if it failed. That execution history matters as much as the current status, because a pipeline that succeeds but takes three times as long as usual is an early warning of a problem that has not caused a visible failure yet. On top of that, most pipelines we build include retry logic for transient failures, such as a source system timing out briefly, so a single blip does not require a person to intervene. That is the closest real equivalent of self-healing: automatic recovery from the failure types that are expected and understood, combined with an alert to a person for the failure types that are not.

Every pipeline is built with monitoring and alerting from day one, not added afterwards. That means logging at each stage of the pipeline, automated failure alerts rather than someone noticing a report looks wrong, retry logic for transient source-system issues, and data quality checks that run before a load is considered complete. Across the Azure and Fabric pipelines we run for clients, this approach holds an average 96% pipeline reliability rate. We treat a pipeline the same way we treat a report: it needs an owner, a defined expected behaviour, and a way to know quickly when it has drifted from that behaviour.

How important notebook-native development is depends on your team. For engineers who live in notebooks, it is core. For analysts who live in BI tools, it is rarely used. If notebooks are the daily working environment for more than a few people on your team, Databricks fits better. If notebooks are an occasional tool for a small group, Fabric's notebooks are sufficient.

A data pipeline has no fixed number of transformation steps, but most well-run Fabric estates settle on four: landing raw data untouched, conforming it into a consistent shape, applying business logic once, and serving it through a semantic model. Fewer steps usually means logic is being duplicated somewhere it shouldn't be, and many more often means the pipeline is solving problems that belong in governance rather than transformation.

Across our engagements, automated Azure and Fabric pipelines run at a 96% average reliability rate, meaning scheduled refreshes complete successfully without manual intervention in the large majority of runs. We get there through standard engineering discipline rather than luck: retry logic on transient failures, alerting the moment a pipeline fails rather than letting it fail silently, and reconciliation checks that catch data quality problems before they reach a report. Pipeline reliability is monitored from the first week of go-live, not assumed.

Fabric is ready for production workloads for most mid-market workloads. Fabric has been generally available for over a year and the components inside it are mature. The integrated experience is newer than the parts. There are still edge cases where the integration is incomplete, but for the workloads most mid-market organisations run, Fabric is production-ready. We have multiple clients on Fabric in production today.

For MBA in Fabric, use Python with mlxtend for most mid-market implementations. The data volumes are usually manageable on a single notebook instance. Spark with MLlib is the right choice for very large transaction volumes (hundreds of millions of transactions or more) where the parallelism matters. The output format and the rule interpretation are the same regardless of the implementation. Most retailers and wholesalers we work with run successful MBA on Python without needing Spark.

For forecasting in Fabric, use Python for most mid-market implementations. The data volumes per SKU are usually moderate, and Python on a Fabric Data Science notebook handles thousands of independent time-series comfortably. Spark with MLlib becomes worthwhile at very large scale (tens of thousands or hundreds of thousands of SKUs forecast simultaneously). The output format is the same; the implementation choice is mostly about scale. Most mid-market retailers and wholesalers run successful forecasting on Python without needing Spark.

Data should be transformed after it lands in Fabric, as a general rule. We extract and load first, raw and unaltered, into the bronze layer, rather than transforming data in flight before it arrives. This matters practically: if a transformation rule turns out to be wrong, the original data is still there to reprocess from, rather than already overwritten by a flawed transformation on the way in. Transformation happens afterwards, moving data from bronze to silver to gold inside Fabric itself, where it is easier to test, version and fix. For on-premises source systems, such as an on-site SQL Server or an ERP database that is not internet-facing, a self-hosted integration runtime acts as the bridge: a lightweight agent installed inside your network that lets Fabric pipelines reach that data securely without opening the source system directly to the internet. This is the standard pattern for hybrid estates where some systems are cloud-based and others are not.

There are six Fabric workloads. Data Factory for integration and pipelines (used in every implementation). Lakehouse for analytical storage with notebook-based transformation (used in most). Data Warehouse for SQL-based analytics (used where the team prefers SQL to Spark). Real-Time Intelligence for streaming and event data (used where genuinely needed, less common in mid-market). Data Science for ML (used in maybe a third of mid-market implementations). Power BI for visualisation (used in every implementation). Most mid-market Fabric estates run heavily on Data Factory, Lakehouse, and Power BI, with the others added as the use cases emerge.

A data engineering engagement with Hopton covers more than pipeline code. A typical engagement starts with an assessment of the current estate, moves through a migration plan for the workloads being moved onto Azure or Fabric, adds a governance layer built on Microsoft Purview for classification and lineage, and ends with a modernisation path that retires the parts of the old stack that no longer earn their cost. The architecture document produced during the Establish phase ties these together in writing: which Azure services are used and why, the cost model, the security posture, and the operational model for running it day to day. That operational model is where cost optimisation and workload tuning live in practice - right-sizing capacity, monitoring spend against actual usage, and re-tuning pipelines and Eventstream jobs as data volumes grow, rather than treating cost as a one-off sizing exercise at kickoff. Security governance is not bolted on afterwards either: sensitivity labels, access boundaries, and audit logging sit inside the same Purview-based governance layer used across Data Factory, Data Lake Storage, and the Fabric lakehouse, so the whole estate is governed consistently instead of pipeline by pipeline.

Data orchestration means coordinating the execution of data pipelines: when each pipeline runs, what it depends on, how failures are handled, how the orchestration recovers when something breaks. Fabric Data Factory provides the orchestration layer with pipeline definitions, scheduling, dependency management, and monitoring. The orchestration is what makes the data platform reliable: pipelines that run individually in isolation are fragile; orchestrated pipelines with proper dependencies and recovery are robust.

Replayability is the ability to run a pipeline again from the original raw data and land at the same, or a deliberately improved, result. It depends on Bronze staying untouched and on transformation logic living in version-controlled notebooks or Dataflows rather than being applied once by hand. When a quality rule is added or a bug is fixed, a replayable pipeline can reprocess the full history through the new logic in one run, rather than patching only the records someone happened to notice were wrong.

Azure Data Factory is used for moving and transforming data between systems. ADF orchestrates pipelines that extract data from source systems (ERPs, CRMs, databases, APIs, files), transform it as needed, and load it into target systems (data lakes, data warehouses, downstream applications). It is the data integration backbone for most mid-market Azure data implementations. The capability is comparable to enterprise data integration tools (Informatica, Fivetran, Airbyte) with the advantage of being native to Azure and integrated with the wider Microsoft ecosystem.

A data pipeline SLA is a documented commitment about the data a pipeline delivers, not just whether the servers were running. A good one covers freshness (how recent the data must be), availability (how often it is expected to be delivered on time), completeness (which sources and records must be present), the alerting that fires when a target is missed, and the incident-response expectation — who responds and how quickly. We tier SLAs by reporting criticality, so board and finance data carries a far tighter commitment than exploratory datasets.

Three orchestration patterns keep Azure and Fabric pipelines reliable as they scale: beyond the basic bronze, silver and gold layering, three patterns do most of the work. Parameterised pipelines: one pipeline definition with source, destination and load type passed in as parameters, rather than a hand-built pipeline per table or source system, which is what lets a platform scale past a handful of sources without becoming unmaintainable. Incremental loading: pipelines track a watermark, typically a last-modified timestamp or change-tracking column, so a scheduled run pulls only what changed rather than reprocessing an entire source every time, which is both faster and less likely to strain the source system it is reading from. Control flow with defined recovery: activities are chained with explicit success, failure and completion paths, so a failure partway through a load triggers a specific response, retry, alert, or skip-and-continue, instead of leaving bronze, silver and gold out of sync with each other silently. We also deploy pipeline changes through a proper release process, so a change tested in a development workspace is promoted to production deliberately rather than edited live, and every run is logged centrally so a failure is visible within minutes rather than discovered when someone notices a stale dashboard.

Three principles work best for pipeline design. Idempotency: a pipeline should produce the same result whether it runs once or many times, so reruns after failures are safe. Modularity: pipelines should be small and focused, composable into larger workflows rather than monolithic. Observability: pipelines should emit structured logs, metrics, and lineage information so failures can be diagnosed quickly. These principles produce data platforms that operate reliably with minimal intervention; absence of any one of them produces platforms that consume engineering time disproportionate to the value they deliver.