Everyone can define bronze, silver and gold. Far less is written about the engineering decisions that keep a Lakehouse maintainable at scale. These are the patterns we build on, and the one we changed after FabCon.
The medallion architecture is easy to describe and surprisingly hard to run. Bronze for raw, silver for cleaned, gold for business-ready: everybody can draw that diagram.
What the diagram does not show is why, eighteen months in, one Lakehouse is a calm estate where adding a source takes an afternoon, and another is forty hand-built pipelines that one person understands.
The difference is not the layer names. It is a handful of engineering decisions taken early. These are the ones that matter most in our experience building Fabric platforms for mid-market organisations, with reference to Nikola Ilic's session at FabCon Europe 2026, including the place where his recommendation sharpened our own approach.
Bronze is a tape recorder, not an editor
The most common bronze layer mistake is doing work there. A little light cleaning, a type cast, dropping a column nobody needs. It feels efficient and it quietly destroys the thing bronze exists for.
Bronze should be append-only and faithful to the source: raw values, source types, nothing dropped, with the load timestamp and source metadata added alongside. It captures what arrived and when, and it never edits the recording. What that buys you is worth more than the storage it costs:
- Provenance. When finance disputes a number you can show exactly what the source sent on the day.
- Replayability. When a transformation bug is found in silver you rebuild from bronze, rather than asking the source system for history it may no longer hold.
- Source independence. When the upstream system is replaced, your history survives the migration.
The discipline required is small: resist the urge to fix anything in bronze. If a field arrives malformed it lands malformed, and gets handled in silver where the fix is visible, tested and reversible.
One metadata-driven pipeline beats five hand-crafted ones
This is the decision that most changes how a platform ages. The intuitive approach is one pipeline per source: one for the ERP, one for the CRM, one for the warehouse system. Each is clear on its own. Five are five places to apply any change, and twenty is a maintenance problem nobody signed up for.
The alternative is a single parameterised pipeline driven by a control table.
The control table holds a row per source object: the source system and connection, the object name, the load type, the watermark column, the target table, and whether the object is currently active. The pipeline reads the table, iterates the active rows and processes each one. Onboarding a new source table becomes a configuration change rather than a development task, and an improvement to error handling, logging or retry logic is written once and inherited by every source.
The cost is a slightly harder first build and genuine discipline about keeping logic out of the per-source branches. It pays back the first time somebody asks you to add eight tables by Friday.
The watermark window, with a ceiling
Incremental loading is where correctness quietly goes wrong. The naive pattern reads everything with a modified timestamp greater than the last watermark, then sets the watermark to "now". It works until it does not.
The pattern that holds up uses a window with an explicit ceiling. Capture a ceiling timestamp at the start of the run. Select rows modified after the stored watermark and up to that ceiling. Process them. Only then write the ceiling as the new watermark. Two properties fall out, and both matter in production:
- Idempotent. If the run fails halfway and is re-run, it processes the same bounded window rather than a moving target, so nothing is skipped and nothing is double-counted in a way the merge cannot resolve.
- Resumable. Because the watermark only advances after the work completes, a failure leaves it where it was and the next run picks up exactly where the last one stopped.
The rule that goes with it is the one people break under pressure: update the watermark only after all child processes have succeeded. If a parent pipeline loads twelve tables and one fails, advancing the watermark for all twelve means the failed table's missing rows are gone from the window forever. They do not error. They simply never arrive, which is the worst kind of data quality problem because nothing alerts.
Parent-child decomposition
A single pipeline that does everything is hard to test, impossible to partially re-run and painful to debug. Decomposing into a parent orchestrator and child pipelines, each responsible for one well-defined job, buys three things that matter in operations:
- Testability. A child that ingests one object can be run on its own against a test configuration, without triggering the whole graph.
- Reusability. The same child is called for every source that fits the pattern, so there is one implementation to improve.
- Debuggability. When a load fails at 3am you re-run the single failing branch rather than a four-hour full graph, which is the difference between fixing it before the business arrives and not.
The parent's job is orchestration and state: read the control table, call children with parameters, collect results, decide whether the run succeeded, and update watermarks only on full success.
File layout, and where we have tightened up
This is where the FabCon session matched our approach in most respects and sharpened it in one.
Delta tables accumulate small files. Every incremental load writes new Parquet files, and left alone a table becomes thousands of them, which slows every read and makes Direct Lake transcoding significantly more expensive. The guidance is to consolidate toward roughly 1 GB blocks through optimisation, and to apply V-Order encoding so the engine loads tightly packed data faster and can in some cases compute directly over the compressed data.
Where we have changed our own practice is vacuuming. We had treated it as periodic housekeeping, run when somebody noticed storage growth. It needs to be a scheduled step in the pipeline with a known retention window, for a reason that is not about storage at all: a Direct Lake semantic model is framed against a specific Delta commit version, and if a vacuum removes the files that version depends on, queries fail. Scheduling vacuum with a retention period comfortably longer than your framing interval, and running it in a known order relative to refreshes, turns an occasional mysterious outage into a non-event. That is now standard in how we build, and we have gone back and applied it to estates we maintain.
What this looks like in practice
A production Lakehouse built this way has:
- An append-only bronze layer nobody argues with.
- One metadata-driven ingestion pipeline with a control table that doubles as documentation.
- Bounded, idempotent incremental loads.
- Child pipelines you can run and debug individually.
- A scheduled optimise and vacuum cycle that keeps file layout healthy for Direct Lake.
None of it is exotic. All of it is the difference between a platform that absorbs a new source in a day and one that needs a project every time the business asks a question.
Hopton builds and runs Fabric data platforms for organisations across the UK and Ireland as a Microsoft Solutions Partner. Our Microsoft Fabric and data platform and warehouse pages set out how we approach this work, the free audit is a straightforward way to get a view on an estate you have already built, and you can always just get in touch.
Related reading
Lakehouse or warehouse? The Azure architecture patterns that actually hold up
Bronze, Silver, Gold: What Each Layer of a Medallion Architecture Is Actually For
Kiran Sahiba
Analytics Solutions Consultant
Part of the Hopton Analytics team, delivering governed analytics programmes for UK mid-market organisations.