Microsoft Fabric Data Pipelines & Dataflows - FAQs

Use Fabric Data Factory for new builds in most cases, rather than Azure Data Factory (ADF). The capabilities are similar (and growing closer) but the Fabric version integrates natively with OneLake and the rest of the Fabric platform without separate connection management. For existing ADF investments, the migration path to Fabric Data Factory is straightforward but not always urgent; existing ADF pipelines continue to work and can stay on standalone ADF for the foreseeable future. For new mid-market analytics builds, the default is Fabric Data Factory.

Dataflows Gen2 can be more expensive to run in Fabric than code-based alternatives for large volumes or complex transformations, because of how they consume capacity. For small and medium workloads they are usually fine. Monitoring capacity use per item shows whether a particular dataflow needs rethinking.

Yes. Microsoft Fabric and Databricks are increasingly competing for the same workloads. Both platforms can do most things: lakehouse, data warehouse, BI, ML, real-time. The competition is real. The differences are in optimisation and culture rather than capability. Fabric is built around BI-led mixed workloads. Databricks is built around code-first lakehouse and ML. Both can do the other thing, but each is designed for a different centre of gravity, and that is what shapes the decision.

Choosing between Azure Data Factory and Microsoft Fabric for data pipelines comes down to your existing estate and priorities. Azure Data Factory (and Synapse) suits teams with an established Azure estate and heavy, cost-sensitive orchestration that want granular control. Microsoft Fabric suits teams who want ingestion, storage, transformation and Power BI unified in one governed SaaS platform and value simplicity. For most mid-market organisations, Fabric’s consolidation is the better fit; the mistake to avoid is buying a large multi-cloud platform you do not need.

Business Central data can be loaded incrementally, because its tables carry system fields such as SystemModifiedAt and SystemRowVersion that identify changed rows. Deletes still need separate handling. Our Business Central analytics page explains the extraction options.

Power BI developers can build Fabric transformations using Dataflows Gen2, because they use the same Power Query editor as Power BI Desktop. For larger or more complex work, notebooks and engineering practices take over, which is where many teams bring in support from a partner.

Fabric notebooks do not always need Spark. Alongside Spark notebooks for large-scale processing, Fabric offers Python notebooks that run without a Spark cluster, which are quicker to start and cheaper for small jobs such as calling an API or processing modest files.

You do not need to migrate from Azure Data Factory now. Microsoft's new upgrade path assesses each pipeline and upgrades them one at a time, which makes migration easier when you have a reason to move. A stable estate is not, on its own, a reason to move.

You do not need to migrate from Synapse or Azure Data Factory to Fabric immediately, but new work should be built on Fabric and existing estates moved opportunistically: Fabric Data Factory covers around 90 per cent of ADF activities, mapping data flows become Dataflow Gen2, and self-hosted integration runtimes become the on-premises data gateway. See Azure Data Factory vs Fabric Data Factory pipelines.

For most reporting needs, Business Central mirroring into Fabric, announced at FabCon Europe 2026 with public preview expected in a Business Central 29.1 update, will replace a custom API extraction or nightly export. Selected tables and companies will replicate continuously into OneLake with no pipeline to build. A consultancy still proposing a bespoke extract for a standard Business Central estate without a plan to move to mirroring is working from last year's architecture. Our Business Central analytics page and Business Central to Fabric in zero ETL steps explain the approach.

No, Fabric Data Factory does not have Azure Data Factory's mapping data flows as such. Their role is taken by Dataflow Gen2, which is Power Query based, with Microsoft's upgrade tooling converting supported mapping data flows into Dataflow Gen2 transforms and a Spark-based transformation activity in preview. Heavy Spark transformations belong in a Fabric notebook. Notebooks, Dataflows Gen2 or pipelines explains which to use when.

Mirroring puts some load on the source database, because it reads changes using the source's own change mechanisms, but it is designed to be light compared with running reporting queries directly against the live system. The impact should still be reviewed with whoever owns the source database.

No, zero-ETL mirroring removes the extraction and loading work, not the modelling. Mirrored tables keep Business Central's operational schema, so turning G/L Entry, Value Entry and Item Ledger Entry into a trustworthy finance model still requires dimensional modelling, agreed definitions and a semantic model inside Fabric.

Fabric pipelines are priced against the capacity you already own rather than per run: a copy activity consumes 1.5 capacity unit hours per throughput resource hour and any other activity run consumes 0.0056 capacity unit hours, which on a UK South capacity at £0.1585 per capacity unit hour is under a tenth of a penny per activity run. Azure Data Factory bills per activity run, per data integration unit hour and per data flow vCore hour as separate pay-as-you-go items. Fabric capacity explained covers how capacity is sized.

Fabric pipelines connect to on-premises data through the on-premises data gateway, the same gateway Power BI uses, in place of Azure Data Factory's self-hosted integration runtime. The gateway supports up to ten nodes, can change region and throttles CPU and memory; private Azure sources connect through a virtual network data gateway deployed in your own VNet, replacing Data Factory's Microsoft-managed virtual network. Microsoft Fabric governance in 2026 covers the security settings around it.

Incremental loads based on a watermark do not detect deleted records, because a deleted row no longer exists to be found. Deletes need a separate approach, such as change data capture, a periodic comparison of keys between source and target, or a soft-delete flag maintained by the source system. Our Microsoft Fabric page covers how we deliver it.

We test data pipelines in Microsoft Fabric through three test types. Unit tests on the transformation logic itself (does this Python function produce the expected output for known inputs). Integration tests on pipeline behaviour (does this pipeline correctly read from the source and write the expected schema to the destination). Data quality tests on the output (does the produced data meet the quality expectations: row counts in expected ranges, no null values in critical columns, foreign keys resolve). The Microsoft Fabric ecosystem supports each of these through standard tools (pytest, Great Expectations, SQL assertions). Mature data teams run all three; ad-hoc teams run none.

A single view of pipeline health comes from a central monitoring view rather than checking each pipeline individually. Fabric's monitoring hub gives one place to see every pipeline run across a workspace: its status, how long it took compared with previous runs, and where in the pipeline it failed if it failed. That execution history matters as much as the current status, because a pipeline that succeeds but takes three times as long as usual is an early warning of a problem that has not caused a visible failure yet. On top of that, most pipelines we build include retry logic for transient failures, such as a source system timing out briefly, so a single blip does not require a person to intervene. That is the closest real equivalent of self-healing: automatic recovery from the failure types that are expected and understood, combined with an alert to a person for the failure types that are not.

You make data pipelines reliable by checking freshness, completeness and validity on every run rather than only whether the run finished, alerting a named person with context when a check fails, making every load safe to replay from the bronze layer, and agreeing a service level the business can read, such as the sales table updated by 7am and reconciled to the ledger. See observability for Azure data engineering.

We monitor and ensure the reliability of Azure data pipelines by building every pipeline with monitoring and alerting from day one, not adding it afterwards. That means logging at each stage of the pipeline, automated failure alerts rather than someone noticing a report looks wrong, retry logic for transient source-system issues, and data quality checks that run before a load is considered complete. Across the Azure and Fabric pipelines we run for clients, this approach holds an average 96% pipeline reliability rate. We treat a pipeline the same way we treat a report: it needs an owner, a defined expected behaviour, and a way to know quickly when it has drifted from that behaviour.

Microsoft Fabric supports AI and machine learning workloads in four layers over one copy of the data in OneLake: Copilot writing DAX, report pages and answers over a semantic model from an F2 capacity; Fabric data agents and Fabric IQ reasoning across models through an ontology and acting through Copilot Studio; a data science workload with notebooks, MLflow, AutoML and SynapseML for custom models scored back into the lakehouse; and Real-Time Intelligence with Activator raising a decision when a value crosses a threshold. Our AI and analytics page describes how we sequence them.

How important notebook-native development is depends on your team. For engineers who live in notebooks, it is core. For analysts who live in BI tools, it is rarely used. If notebooks are the daily working environment for more than a few people on your team, Databricks fits better. If notebooks are an occasional tool for a small group, Fabric's notebooks are sufficient.

CI/CD in Azure Data Factory uses ARM templates released through Azure DevOps or GitHub, whereas Fabric uses built-in deployment pipelines across development, test and production workspaces with Git integration behind them, and has no separate publish step. The Fabric model is simpler and less flexible, so a team with a mature ARM-based release process should plan the change rather than assume it. Deploy only what changed covers deployment pipelines in practice.

Change data capture is different from a watermark-based incremental load because it reads the database's own record of changes rather than querying rows by a modified date. That means it captures deletes and any change the watermark might miss, and usually places less load on the source. Our Microsoft Fabric page covers how we deliver it.

A data pipeline has no fixed number of transformation steps, but most well-run Fabric estates settle on four: landing raw data untouched, conforming it into a consistent shape, applying business logic once, and serving it through a semantic model. Fewer steps usually means logic is being duplicated somewhere it shouldn't be, and many more often means the pipeline is solving problems that belong in governance rather than transformation.

The Azure and Fabric data pipelines Hopton builds are automated and run at a 96% average reliability rate across our engagements, meaning scheduled refreshes complete successfully without manual intervention in the large majority of runs. We get there through standard engineering discipline rather than luck: retry logic on transient failures, alerting the moment a pipeline fails rather than letting it fail silently, and reconciliation checks that catch data quality problems before they reach a report. Pipeline reliability is monitored from the first week of go-live, not assumed.

No, Azure Data Factory is not being retired and existing pipelines keep running, but Microsoft has said new features are developed for Fabric Data Factory and are not backported to Azure Data Factory or Synapse pipelines, so the two have separate roadmaps and Data Factory is standing still. New pipelines are better built in Fabric. What is Microsoft Fabric? explains the wider platform.

Fabric is ready for production workloads for most mid-market workloads. Fabric has been generally available for over a year and the components inside it are mature. The integrated experience is newer than the parts. There are still edge cases where the integration is incomplete, but for the workloads most mid-market organisations run, Fabric is production-ready. We have multiple clients on Fabric in production today.

No, using several ingestion methods in one Fabric estate is normal and usually correct: mirroring for the ERP, shortcuts to a group data lake, a metadata-driven pipeline for bespoke sources and dataflows where the business owns the logic. What matters is that each choice was deliberate and documented rather than accidental.

For MBA in Fabric, use Python with mlxtend for most mid-market implementations. The data volumes are usually manageable on a single notebook instance. Spark with MLlib is the right choice for very large transaction volumes (hundreds of millions of transactions or more) where the parallelism matters. The output format and the rule interpretation are the same regardless of the implementation. Most retailers and wholesalers we work with run successful MBA on Python without needing Spark.

For forecasting in Fabric, use Python for most mid-market implementations. The data volumes per SKU are usually moderate, and Python on a Fabric Data Science notebook handles thousands of independent time-series comfortably. Spark with MLlib becomes worthwhile at very large scale (tens of thousands or hundreds of thousands of SKUs forecast simultaneously). The output format is the same; the implementation choice is mostly about scale. Most mid-market retailers and wholesalers run successful forecasting on Python without needing Spark.

You should migrate Azure Data Factory pipelines to Fabric when the data they land is moving into OneLake, when their mapping data flows need changing and Microsoft's upgrade tooling can convert them, or when you are paying for Data Factory and a Fabric capacity side by side. Pipelines that depend on SSIS packages or the Microsoft-managed virtual network should stay for now. Our free audit assesses an estate and recommends the order.

You should consider replacing an existing Business Central pipeline with mirroring when the pipeline next needs significant work, rather than immediately. A stable, tested pipeline feeding a trusted model has value, and the strongest case for switching is removing the failure-prone extraction layer and the scheduling and watermark maintenance that come with it.

Data should be transformed after it lands in Fabric, as a general rule. We extract and load first, raw and unaltered, into the bronze layer, rather than transforming data in flight before it arrives. This matters practically: if a transformation rule turns out to be wrong, the original data is still there to reprocess from, rather than already overwritten by a flawed transformation on the way in. Transformation happens afterwards, moving data from bronze to silver to gold inside Fabric itself, where it is easier to test, version and fix. For on-premises source systems, such as an on-site SQL Server or an ERP database that is not internet-facing, a self-hosted integration runtime acts as the bridge: a lightweight agent installed inside your network that lets Fabric pipelines reach that data securely without opening the source system directly to the internet. This is the standard pattern for hybrid estates where some systems are cloud-based and others are not.

Incremental tables should be fully reloaded or reconciled periodically, for example weekly or monthly, to catch any changes or deletes the increments missed. It is a cheap insurance policy against small gaps quietly building up into numbers that no longer match the source. Our Microsoft Fabric page covers how we deliver it.

You should standardise on a clear rule for which Fabric tool does which job, rather than necessarily one tool for everything. The rule matters more than the choice, because it lets anyone find where a transformation lives and support it. Our data platform team helps agree these standards.

There are six Fabric workloads, and how often mid-market businesses actually use each one varies. Data Factory for integration and pipelines (used in every implementation). Lakehouse for analytical storage with notebook-based transformation (used in most). Data Warehouse for SQL-based analytics (used where the team prefers SQL to Spark). Real-Time Intelligence for streaming and event data (used where genuinely needed, less common in mid-market). Data Science for ML (used in maybe a third of mid-market implementations). Power BI for visualisation (used in every implementation). Most mid-market Fabric estates run heavily on Data Factory, Lakehouse, and Power BI, with the others added as the use cases emerge.

Parent-child pipeline decomposition gives testability, because a child can be run alone against a test configuration; reusability, because one child implementation serves every source matching the pattern; and debuggability, because a failed overnight load can be re-run for the single failing branch rather than the entire graph.

EtLT means extract, a light transformation, load, then a full transformation. The small t covers controls that must be applied before data is stored anywhere, such as masking personal data, enforcing data types and removing duplicates. The capital T covers the business rules that change over time, such as customer hierarchies, KPI definitions and margin calculations, which are applied after the load. In Microsoft Fabric this usually means a pipeline or notebook doing the light pass into the bronze or silver layer, with the Warehouse or a gold lakehouse holding the business rules. Our post on ETL, ELT and EtLT in Fabric walks through the pattern.

A data engineering engagement with Hopton covers more than pipeline code. A typical engagement starts with an assessment of the current estate, moves through a migration plan for the workloads being moved onto Azure or Fabric, adds a governance layer built on Microsoft Purview for classification and lineage, and ends with a modernisation path that retires the parts of the old stack that no longer earn their cost. The architecture document produced during the Establish phase ties these together in writing: which Azure services are used and why, the cost model, the security posture, and the operational model for running it day to day. That operational model is where cost optimisation and workload tuning live in practice, right-sizing capacity, monitoring spend against actual usage, and re-tuning pipelines and Eventstream jobs as data volumes grow, rather than treating cost as a one-off sizing exercise at kickoff. Security governance is not bolted on afterwards either: sensitivity labels, access boundaries, and audit logging sit inside the same Purview-based governance layer used across Data Factory, Data Lake Storage, and the Fabric lakehouse, so the whole estate is governed consistently instead of pipeline by pipeline.

Data orchestration means coordinating the execution of data pipelines: when each pipeline runs, what it depends on, how failures are handled, how the orchestration recovers when something breaks. Fabric Data Factory provides the orchestration layer with pipeline definitions, scheduling, dependency management, and monitoring. The orchestration is what makes the data platform reliable: pipelines that run individually in isolation are fragile; orchestrated pipelines with proper dependencies and recovery are robust.

Replayability is the ability to run a pipeline again from the original raw data and land at the same, or a deliberately improved, result. It depends on Bronze staying untouched and on transformation logic living in version-controlled notebooks or Dataflows rather than being applied once by hand. When a quality rule is added or a bug is fixed, a replayable pipeline can reprocess the full history through the new logic in one run, rather than patching only the records someone happened to notice were wrong.

The Copy activity in a Fabric pipeline moves data from a source, such as a database, file store or SaaS application, into a destination such as a lakehouse or warehouse. It is built for moving data at scale with little transformation, and is usually the first step in loading the Bronze layer.

Azure Data Factory is used for moving and transforming data between systems. ADF orchestrates pipelines that extract data from source systems (ERPs, CRMs, databases, APIs, files), transform it as needed, and load it into target systems (data lakes, data warehouses, downstream applications). It is the data integration backbone for most mid-market Azure data implementations. The capability is comparable to enterprise data integration tools (Informatica, Fivetran, Airbyte) with the advantage of being native to Azure and integrated with the wider Microsoft ecosystem.

Dataflow Gen2 is best suited to low-code, business-accessible transformations built in Power Query by analysts rather than data engineers, and to sources where Power Query connectivity is the practical route in. It is less suitable for high-volume core pipelines where cost and performance dominate, or for logic that must be version controlled and reviewed.

A data pipeline SLA is a documented commitment about the data a pipeline delivers, not just whether the servers were running. A good one covers freshness (how recent the data must be), availability (how often it is expected to be delivered on time), completeness (which sources and records must be present), the alerting that fires when a target is missed, and the incident-response expectation, who responds and how quickly. We tier SLAs by reporting criticality, so board and finance data carries a far tighter commitment than exploratory datasets.

A metadata-driven pipeline is a single parameterised pipeline driven by a control table that holds one row per source object, covering the connection, object name, load type, watermark column and target table. It replaces multiple hand-crafted pipelines, so onboarding a new source becomes a one-row configuration change and improvements to logging or error handling are inherited by every source.

A watermark in data loading is a column whose value increases whenever a row changes, such as a last modified date or row version number. Each incremental load records the highest value it has loaded, and the next load requests only rows above it. Our Microsoft Fabric page covers how we deliver it.

The difference between Azure Data Factory and Fabric Data Factory is in the platform around the pipeline rather than the pipeline itself: Fabric uses the on-premises data gateway instead of the self-hosted integration runtime, Dataflow Gen2 instead of mapping data flows, connections instead of datasets and linked services, deployment pipelines with Git instead of ARM templates, and capacity units instead of pay-per-run billing. The canvas, activities and expression language are almost identical. Our Microsoft Fabric page describes how we build on the Fabric version.

The difference between ETL and ELT in Microsoft Fabric is when the data is transformed. ETL transforms data before it is loaded, typically with Dataflow Gen2 or a pipeline that calls a notebook before writing to a lakehouse or warehouse. ELT lands raw data in OneLake first, through Mirroring, shortcuts, copy jobs or pipelines, then transforms it inside Fabric with notebooks or Warehouse T-SQL. Fabric’s architecture leans towards ELT, but both are supported and most real estates use a mix of the two.

Related reading: Microsoft Fabric IQ: Sort Your Definitions First.

The difference between a Fabric pipeline and a Dataflow Gen2 is their job. A pipeline orchestrates and moves data, running activities in order with dependencies, retries and schedules. A Dataflow Gen2 transforms data using Power Query's low-code editor and writes the result to a lakehouse or warehouse. Pipelines often run dataflows as one of their steps.

The difference between a full load and an incremental load is how much data is copied each time. A full load copies the entire source table and replaces the previous copy. An incremental load copies only the rows that are new or changed since the last successful load, which is faster and cheaper for large tables. Our Microsoft Fabric page covers how we deliver it.

Three orchestration patterns keep Azure and Fabric pipelines reliable as they scale: beyond the basic bronze, silver and gold layering, three patterns do most of the work. Parameterised pipelines: one pipeline definition with source, destination and load type passed in as parameters, rather than a hand-built pipeline per table or source system, which is what lets a platform scale past a handful of sources without becoming unmaintainable. Incremental loading: pipelines track a watermark, typically a last-modified timestamp or change-tracking column, so a scheduled run pulls only what changed rather than reprocessing an entire source every time, which is both faster and less likely to strain the source system it is reading from. Control flow with defined recovery: activities are chained with explicit success, failure and completion paths, so a failure partway through a load triggers a specific response, retry, alert, or skip-and-continue, instead of leaving bronze, silver and gold out of sync with each other silently. We also deploy pipeline changes through a proper release process, so a change tested in a development workspace is promoted to production deliberately rather than edited live, and every run is logged centrally so a failure is visible within minutes rather than discovered when someone notices a stale dashboard.

Three principles work best for pipeline design. Idempotency: a pipeline should produce the same result whether it runs once or many times, so reruns after failures are safe. Modularity: pipelines should be small and focused, composable into larger workflows rather than monolithic. Observability: pipelines should emit structured logs, metrics, and lineage information so failures can be diagnosed quickly. These principles produce data platforms that operate reliably with minimal intervention; absence of any one of them produces platforms that consume engineering time disproportionate to the value they deliver.

A data pipeline is the right choice in Fabric when transformation, orchestration with dependencies, error handling and retry logic, or metadata-driven incremental loading are genuinely required. It is the wrong choice when its only job is copying a table on a timer, because that creates a schedule, logic and maintenance obligation a mirror or copy job would avoid.

A pipeline is still better than mirroring when heavy transformation is needed on the way in, when BC data must follow the same ingestion pattern as sources that cannot be mirrored, or when you need point-in-time history the source system does not keep. Many estates run mirroring for BC alongside pipelines for other sources.

You should use a full load for small and medium tables, such as customers, products or reference lists, and for any table without a reliable way to identify changes. Full loads are simple, handle deletes automatically and correct themselves on the next run. Our Microsoft Fabric page covers how we deliver it.

You should use a notebook instead of a Dataflow Gen2 in Fabric when data volumes are large, the logic is complex, such as merges and history handling, or the transformation needs to be tested and version controlled like software. Dataflows Gen2 suit smaller, business-owned transformations built by people skilled in Power Query.

The Microsoft Fabric tool to use for ETL, where data is transformed before it is loaded, is usually Dataflow Gen2, because it is Power Query and the analysts who own the business rules can read and maintain it. For higher volumes or engineering-led work, a data pipeline that runs a notebook before writing to a curated lakehouse does the same job with code under version control. Load-first tools such as Mirroring, OneLake shortcuts and copy jobs are not ETL tools: they land data raw, and the transformation happens afterwards in notebooks or the Warehouse. Our decision guide to shortcuts, mirroring, pipelines and dataflows compares the Microsoft Fabric tools, and ETL, ELT or EtLT in Fabric covers when to transform.

An incremental load usually misses changes because the watermark column is not updated by every process that changes a row, or because the watermark was moved forward after a failed or partial load. Testing the watermark and only advancing it after a fully successful run prevents most gaps. Our Microsoft Fabric page covers how we deliver it.

Watermarks should only be updated after all child pipelines succeed because advancing the watermark when one table has failed permanently excludes that table's missing rows from any future window. The rows do not produce an error, they simply never arrive, which is the hardest kind of data quality problem to detect because nothing alerts.