Subscribe
Share
Share
Embed
Fabric's Schema Registry promises type-safe streaming pipelines. But non-conforming events vanish without a dead-letter queue, compatibility checks don't exist, and you can't retrofit schema mode on a running Eventstream. We dig into what the contract actually enforces — and what it silently doesn't.
Episode 21 • 2026-05-22
Event Schema Set is Fabric's contract layer for streaming data — but it ships in Preview with real gaps. Matthias and Fabia unpack the retrofit trap, the dead-letter gap everyone worries about, and when Confluent Schema Registry is honestly the better call.
The Fabric Friday video on this topic: https://youtu.be/dY7fGdiP9dE
Built on ElevenLabs voice synthesis. Matthias — cloned voice. Fabia — designed AI co-host. See Matthias live on YouTube (Fabric Friday), at his meetups, and at conferences like FabCon.
Hosted by Matthias Falland — Microsoft Data Platform MVP and community architect behind the Fabric Periodic Table. New episodes every Friday.
Have an architecture decision you are wrestling with? DM Matthias on LinkedIn — find him as Matthias Falland. Three to five sentences about the decision, your team size, and your current stack. We anonymize before airing.
Built on ElevenLabs voice synthesis. Brand design based on fabricperiodictable.com.
Architecture decisions for Microsoft Fabric. Anonymized real customer scenarios, cost realism, counter-arguments included. Weekly episodes aligned with Fabric Friday recordings.
The Contract That Drops Your Data
Episode 21 | 2026-05-22
Fabric Architecture Podcast | Hosted by Matthias Falland with AI co-host Fabia
---
[00:08] Fabia: Schema mode on a Fabric Eventstream is set at creation. Can't retrofit it. Four streams running in production, you decide you need schema validation — you're rebuilding all four from scratch.
[00:20] Matthias: There's no migration path? No config flag?
[00:23] Fabia: None. The working explanation is that enabling validation changes the internal processor mid-stream, and Microsoft chose the hard cut over risking silent data loss during transition.
[00:35] Matthias: So today we're pulling apart Fabric's Schema Registry — what it actually enforces, and where the contract has fine print that'll cost you.
[00:45] Fabia: Schema Registry shipped in Preview in September 2025, as part of Real-Time Intelligence. The core concept — it's a tenant-wide catalog of event contracts. Your Schema Set lives in a workspace, holds event types and versioned schemas, and every producer and consumer references the same registered definition. Instead of Team A saying 'the field's called Temperature, capital T, it's a string' while Team B reads it as lowercase temperature expecting a number — both reference the registered schema, and the system enforces the shape.
[01:20] Matthias: Every schema registry promises that. What's different about this one living inside Fabric?
[01:26] Fabia: Integration depth. The Schema Set never moves data — it sits above Eventstream, Activator, Eventhouse, and Real-Time Hub as a pure catalog. When you endorse a schema as Promoted or Certified, that badge surfaces across the Real-Time Hub for the entire tenant. Sensitivity labels cascade from the schema set down to validated streams. There's no separate cluster to operate, no client library to wire in.
[01:53] Matthias: So governance comes baked in. That's genuinely useful — if the enforcement underneath holds up.
[01:59] Fabia: Fabric Schema Registry doesn't enforce compatibility checks on schema updates. Confluent gives you backward, forward, full compatibility modes — the registry rejects an update that'd break consumers. Fabric lets any update through. Microsoft's own documentation is explicit about it: schema compatibility isn't enforced, so any change you make might break your pipelines, and that's on your team.
[02:26] Matthias: So the contract's enforced at ingest but not at authoring. You can publish a schema change that silently corrupts every downstream consumer.
[02:35] Fabia: And silently is precise. Rename a field — KQL queries referencing the old name return null, no error. Change a field type — data still flows, values come out garbage. The schema validated the shape of the event. Nobody validated the evolution of the schema.
[02:52] Matthias: I keep running into this pattern. A team ships a schema update through a pull request, someone approves because the JSON looks reasonable, and nobody catches that it broke the Eventhouse query feeding the operations dashboard. The registry gave them confidence. The confidence was misplaced.
[03:12] Fabia: We got a question from the community that cuts right to this gap. 'When a non-conforming event hits a schema-validated Eventstream, where does it go? Is there a dead-letter queue I can inspect? Or do I just lose the data?'
[03:28] Matthias: That's the question every architect asks within ten minutes.
[03:32] Fabia: The answer: you lose it. In Preview, non-conforming events are dropped to preserve data quality. No dead-letter stream, no return-to-sender. The event is gone. The only signal is a line in Fabric Diagnostics.
[03:45] Matthias: A log line. For lost production data.
[03:48] Fabia: A log line you have to be actively watching. There's no push notification, no built-in alert. If you're not monitoring Fabric Diagnostics, you won't know events are vanishing until someone asks why the dashboard stopped updating.
[04:03] Matthias: So how do you design around this? 'Don't lose data' isn't architecture.
[04:08] Fabia: Two layers. First — validate at the producer. In your Notebook, your UDF, your upstream transformation, before the event ever hits the schema-enabled stream. Producer-side rejection is observable. You see it in your own logs and you can act on it.
[04:24] Matthias: And the second layer?
[04:26] Fabia: A shadow Eventstream. Same source, no schema validation, destination is a Lakehouse table. During migration windows or high-stakes periods, you ingest the same events into both paths. The Lakehouse becomes your dead-letter equivalent — every event lands regardless of conformance. If something vanishes from the validated stream, you go find it in the shadow.
[04:50] Matthias: That's the pattern I'd ship. The shadow costs you an extra Eventstream's worth of CU, and on a high-throughput pipeline that's real money. But compared to losing production events you can't recover — I'll take the CU bill. What I'd add on top: wire an Activator alert to Fabric Diagnostics validation errors. That way you know in minutes when events start dropping, not days later when someone finally looks at the numbers.
[05:18] Fabia: On cost — Schema Registry doesn't have its own meter in the Capacity Metrics App. Validation happens inside Eventstream, so it rolls into Eventstream's existing CU consumption. No per-schema fee, no per-version fee. Register and version aggressively — there's no billing reason not to.
[05:37] Matthias: Free as in 'already paying for the capacity it runs on.'
[05:41] Fabia: And if your capacity's already saturated, schema validation pushes more processing onto it. The increment per event is small, but on a stream doing millions of events per hour, small compounds. Benchmark before you turn it on at scale.
[05:57] Matthias: The Avro question trips people up too. I've talked to teams with years of JSON Schema definitions in source control, and they hear 'Avro only' and assume they have to —
[06:08] Fabia: They don't have to rewrite everything, just their assumptions about scope. Schema Sets only accept Avro — that's a hard limitation on the current Preview, no JSON Schema path anywhere in the UI. But you don't have to register every contract. For the one or two high-traffic schemas that need true ingest-time rejection, convert those to Avro. Leave the rest as JSON Schema outside the registry, and validate downstream in KQL or your consumer code.
[06:38] Fabia: The steel-man for Confluent Schema Registry is this: if your team's had production incidents from schema drift, Confluent's compatibility enforcement is a guardrail you can't replicate manually at scale. It rejects the breaking change before it's published. Fabric makes that your team's responsibility, and a code review that misses a field rename costs you a midnight page.
[07:03] Matthias: I'd go further. If you're already running Kafka with Confluent, the migration isn't justified. Eventstream can deserialize Confluent-encoded payloads — that integration exists today in Preview. Keep Confluent as your registry, use Fabric for the downstream analytics. Don't move what's working.
[07:22] Fabia: The decision simplifies when you frame it that way. Existing Kafka shop — Confluent. Greenfield Fabric project, team with strong review discipline — Fabric Schema Registry, and the native integration with endorsement and Real-Time Hub discovery is a genuine win. Greenfield but you've been burned by drift — wait for GA, or enforce compatibility through CI checks on schema pull requests.
[07:48] Matthias: Which is what you should've been doing anyway. The risk I want to name, though — this is Preview. Three sources in, three destinations out. Custom Endpoint, Azure SQL CDC, Azure Event Hubs on the input side. Eventhouse Push Mode, Custom App, Derived Stream on the output. That's a narrow corridor to bet an architecture on. If those lists don't expand at GA, you've painted into a corner.
[08:15] Fabia: And Preview means the behaviour itself can change. The silent-drop pattern might become dead-letter routing at GA. The Avro constraint might open up. You're building on a surface that's still moving under you.
[08:29] Fabia: Simplicity on the slide is not simplicity at runtime. If your event architecture needs a contract layer and you're building on Fabric, Schema Registry is the right shape. Just know that the contract it enforces today has gaps it doesn't advertise — and the discipline the system doesn't provide lands on your team.
[08:49] Matthias: Schema validation is a filter. And a filter that destroys what it catches without telling you — that's something you monitor like production infrastructure. Because it is production infrastructure. The schema just doesn't know it yet.
[09:05] Fabia: Somewhere right now, a schema-validated Eventstream is dropping events that don't match. The team that owns it hasn't configured an alert. The dashboard still looks fine. It'll look fine for a while.
[09:20] Matthias: If you're building streaming on Fabric and want to talk through the schema decision — DM me on LinkedIn, find me as Matthias Falland.
[09:28] Fabia: See you next Friday.
---
End of episode. ~09:30 estimated.