Subscribe
Share
Share
Embed
Medallion is the recommended design approach for Fabric. It's also one building block out of five, and the one that sends you a maintenance invoice nobody put on the architecture diagram. What the file-count guardrail means for Direct Lake on OneLake, why schema-enabled lakehouses can't use the automated maintenance activity, and when the whole pattern is overkill.
Episode 42 • 2026-10-23 Duration: 10:17
Medallion is the recommended design approach for Fabric. It's also one building block out of five, and the one that sends you a maintenance invoice nobody put on the architecture diagram. What the file-count guardrail means for Direct Lake on OneLake, why schema-enabled lakehouses can't use the automated maintenance activity, and when the whole pattern is overkill.
AI-generated voices. Matthias — cloned voice. Fabia — designed AI co-host. See Matthias live on YouTube (Fabric Friday), at his meetups, and at conferences like FabCon.
Hosted by Matthias Falland — Microsoft Data Platform MVP and community architect behind the Fabric Periodic Table. New episodes every Friday.
Have an architecture decision you are wrestling with? DM Matthias on LinkedIn — find him as Matthias Falland. Three to five sentences about the decision, your team size, and your current stack. We anonymize before airing.
This podcast was generated by AI. Both voices are synthetic: Matthias is a cloned voice, Fabia is a designed AI co-host.
Brand design based on fabricperiodictable.com.
Architecture decisions for Microsoft Fabric. Anonymized real customer scenarios, cost realism, counter-arguments included. Weekly episodes aligned with Fabric Friday recordings.
Five Thousand Files From a Blank Report
Episode 42 | 2026-10-23
Fabric Architecture Podcast | Hosted by Matthias Falland with AI co-host Fabia
---
[00:08] Fabia: Every write to a Delta table adds new Parquet files. The old ones stay on disk until someone runs VACUUM.
[00:15] Matthias: And on a lakehouse, that someone is you.
[00:17] Fabia: On an F64, the guardrail is five thousand files per table.
[00:21] Matthias: Who's counting? That's what this episode is about — medallion architecture in Fabric, and what it actually costs to keep running once the diagram's done.
[00:32] Fabia: Medallion is the recommended design approach for Fabric. Microsoft's words, from the OneLake medallion article.
[00:39] Matthias: That's the headline. What's underneath it?
[00:42] Fabia: The newer architecture-patterns article frames it as one building block out of five. Unified data access, medallion, data mesh, platform consolidation, external sharing. And most environments combine more than one.
[00:56] Matthias: So medallion answers...
[00:58] Fabia: One question. What is each stage of data trusted for.
[01:01] Matthias: That's... actually the gap I keep seeing. Teams treat medallion as the whole architecture. Three folders, call it done.
[01:09] Fabia: And then they've got no answer for how data arrives or who outside may read it. Those are different bricks entirely.
[01:17] Matthias: What does layer actually mean here?
[01:19] Fabia: Microsoft's sentence: each layer is a contract with downstream consumers, and data only advances after it meets that layer's quality standards.
[01:29] Matthias: Contract. That word does a lot of work.
[01:31] Fabia: Bronze: raw, immutable, data engineers only with limited access. Silver: cleansed and conformed, engineers and trained analysts get in. Gold: business-ready tables and semantic models, open to every BI and AI consumer.
[01:46] Matthias: And bronze — what I keep running into is teams copying everything into that first layer as if copying IS the job.
[01:53] Fabia: Learn's rule is strict. Prefer a shortcut when the source supports no-copy access. Use mirroring when the source needs a synchronized Delta copy. Build a pipeline only when those two don't work.
[02:06] Matthias: And whenever there's a copy—
[02:08] Fabia: —record the reason. That's Learn's own requirement. Every mirror, every pipeline, even silver and gold — they're transformed Delta output.
[02:17] Matthias: In the video, the bronze lakehouse has shortcuts to a mirrored SQL database. Fifty thousand orders, and bronze hasn't stored one of them.
[02:27] Fabia: Silver is where materialized lake views earn their keep. GA since September 2026. You write a SQL definition with named constraints, and every rejected row gets counted in the lineage view.
[02:39] Matthias: Counted how?
[02:40] Fabia: DROP removes the row and logs it. FAIL stops the entire refresh. FAIL's the default — which matters, because a refresh that silently drops half your data isn't a contract.
[02:51] Matthias: The video showed sixty-three rows dropped across three named rules. Twenty-six wrong-sign amounts, twenty test orders, seventeen missing customers.
[03:01] Fabia: And that refresh ran as a full rebuild. The source — a shortcut to a mirror — had no change data feed. Fabric told you honestly: FullRefresh, CDFDisabled.
[03:11] Matthias: So incremental refresh is the headline feature and it didn't even kick in?
[03:16] Fabia: It can't without change data feed on every source table. Without it, the engine skips or rebuilds. That's the catch people miss when they wire up MLVs over shortcuts.
[03:27] Fabia: Someone on the Fabric community forum writes: "I am thinking of having four workspaces per environment. Code. Data, landing and bronze lakehouses. Data 2, the silver lakehouse. Data 3, the gold lakehouse. Is this good practice, or should code be mixed in with the data workspaces?"
[03:47] Matthias: The layer part... yeah, Learn recommends a separate workspace per layer. More control, better governance. But the real reason it matters is underneath that.
[03:57] Fabia: A workspace is the unit Fabric hands out roles on. Three workspaces give you three short role lists that match each layer's contract.
[04:06] Matthias: Engineers on bronze, analysts on silver, consumers on gold. One workspace means one role list and a pile of OneLake security exceptions inside it.
[04:16] Fabia: And those exceptions have teeth. Every lakehouse ships with a DefaultReader role that grants read access to everyone who has ReadAll on the item.
[04:26] Matthias: So if I add a narrow security role—
[04:28] Fabia: —it changes nothing while DefaultReader still covers the same user. You have to edit or remove DefaultReader first.
[04:36] Matthias: That's... really easy to miss.
[04:38] Fabia: And there's more. New SQL analytics endpoints start in delegated mode. In that mode, OneLake security roles aren't applied at all. A workspace Viewer queries through plain SQL permissions.
[04:50] Matthias: So a team builds three workspaces for governance, sets up the roles... and the endpoint ignores all of it because nobody checked which mode it's in.
[05:00] Fabia: Switching to user's identity mode fixes it. But the switch cancels every running query on every endpoint in that workspace. And your existing SQL roles are deleted.
[05:11] Matthias: Off-hours only.
[05:12] Fabia: Now. The files.
[05:13] Matthias: The number from the top.
[05:15] Fabia: A Fabric Warehouse auto-compacts data files and auto-updates statistics. A Fabric Lakehouse is user-managed. Auto compaction exists for new Spark writes if you enable it, but OPTIMIZE for backlogs and VACUUM for stale files — that's on you.
[05:31] Matthias: There's the Lakehouse Maintenance activity though. That orchestrates both.
[05:35] Fabia: It does. And it doesn't support schema-enabled lakehouses yet. Known issue, on the activity's own Learn page.
[05:43] Matthias: The same schema-enabled lakehouses that materialized lake views require?
[05:48] Fabia: The same ones.
[05:48] Matthias: So the team that follows the recommended path — MLVs, schemas, the whole thing — is the exact team that can't use the automated maintenance. I'd still build it that way, for the record. Run OPTIMIZE from a scheduled notebook until the activity catches up.
[06:05] Fabia: That's a judgment call. The activity is documented as a known issue, which isn't the same as a committed fix with a date. You're betting on a gap closing.
[06:16] Matthias: Fair. But the alternative is skipping MLVs, and I think that costs more.
[06:21] Fabia: And here's where the file count comes back. Direct Lake on OneLake — the recommended option for new semantic models — runs in DirectLakeOnly mode. There's no DirectQuery fallback.
[06:33] Matthias: Meaning when a table crosses five thousand files on an F64—
[06:37] Fabia: —the refresh fails. The model can't be queried. Report visuals stop rendering.
[06:42] Matthias: The reports go blank.
[06:44] Fabia: Until you OPTIMIZE those tables back within the guardrails. And nothing in the product alerts you that you're approaching the limit.
[06:52] Matthias: There's got to be a way to check.
[06:55] Fabia: EVALUATE TABLETRAITS() in DAX. The DirectLakeFallbackInfo column gives you the fallback reason per table. None means it's genuinely in Direct Lake mode.
[07:05] Matthias: One DAX query that's been there all along.
[07:08] Fabia: Nothing tells you to look.
[07:09] Matthias: What does the maintenance itself burn in capacity?
[07:13] Fabia: OPTIMIZE and VACUUM run as Spark jobs. Each capacity unit maps to two Spark vCores. Your maintenance competes with your transformations for the same capacity pool.
[07:24] Matthias: So if I'm running OPTIMIZE on silver while gold's MLVs refresh... that's the same F64 sweating.
[07:30] Fabia: Unless you've turned on Autoscale Billing for Spark. Then Spark jobs use serverless resources instead of the fixed capacity.
[07:38] Matthias: A lever you'd better know exists before the capacity saturates on a Tuesday afternoon.
[07:44] Matthias: I've got to push back on the whole framing though. You've spent this episode making medallion sound like a maintenance invoice. When is it the wrong pattern entirely?
[07:55] Fabia: Learn names the condition. Multiple teams build on the same source data and need consistent quality. That's the first line of 'use this pattern when.'
[08:05] Matthias: One team, one clean source, one consumer?
[08:08] Fabia: A shortcut and a semantic model. Three layers there would be three tables of the same truth.
[08:14] Matthias: The video said it plainly — the demo itself, by its own flowchart, should've been one workspace with three schemas. The four workspaces were built to show the access model, not because the project needed them.
[08:28] Fabia: And a SQL-first team with structured data has a documented warehouse-only medallion. T-SQL or dbt end to end.
[08:35] Matthias: Here's where I'd land for anyone starting this today. You need to budget for three things that aren't on the architecture slide. A maintenance schedule — who runs OPTIMIZE, how often, and what monitors file counts before they hit the guardrail. Change data feed on every source you own, because without it your MLVs rebuild fully every time. And the endpoint mode. Delegated or user's identity. The security model you drew on the whiteboard only works in one of them.
[09:07] Fabia: The OPTIMIZE target: 128 MB to 1 GB per file. Direct Lake performs best with row groups between one and sixteen million rows.
[09:15] Matthias: Architecture is not religion. Medallion answers one question — what is each stage trusted for — and it leaves the other four to the other four bricks.
[09:25] Fabia: The price of that answer is maintenance. A schedule, a monitor, and someone who knows where TABLETRAITS lives.
[09:32] Fabia: Somewhere right now, there's a table sitting at four thousand files on an F64. Eighty percent of the cliff. And nothing in the product is going to mention it.
[09:43] Matthias: Until the Monday report goes blank.
[09:46] Matthias: DM me on LinkedIn — find me as Matthias Falland.
[09:49] Fabia: Go check your file counts.
---
End of episode. ~09:51 estimated.