Subscribe
Share
Share
Embed
The grounding prompt decides everything inside a Fabric data agent, and the single highest-impact factor is the one nobody wants to hear. This episode follows the pipeline from question to query to answer, and finds where accuracy actually lives.
Episode 26 • 2026-06-26 Duration: 9:21
A Fabric Data Agent generates queries from your column names, not your intentions. This episode pulls apart the grounding pipeline to show where accuracy lives and dies — and why the schema investment you skipped three years ago just became urgent.
AI-generated voices. Matthias — cloned voice. Fabia — designed AI co-host. See Matthias live on YouTube (Fabric Friday), at his meetups, and at conferences like FabCon.
Hosted by Matthias Falland — Microsoft Data Platform MVP and community architect behind the Fabric Periodic Table. New episodes every Friday.
Have an architecture decision you are wrestling with? DM Matthias on LinkedIn — find him as Matthias Falland. Three to five sentences about the decision, your team size, and your current stack. We anonymize before airing.
This podcast was generated by AI.
Brand design based on fabricperiodictable.com.
Architecture decisions for Microsoft Fabric. Anonymized real customer scenarios, cost realism, counter-arguments included. Weekly episodes aligned with Fabric Friday recordings.
Why Your Data Agent Gets the Wrong Answer
Episode 26 | 2026-06-26
Fabric Architecture Podcast | Hosted by Matthias Falland with AI co-host Fabia
---
[00:08] Fabia: Twenty-five rows. That's the hard cap on every answer a Fabric data agent returns.
[00:13] Matthias: And there's no setting to raise it.
[00:15] Fabia: Someone asks 'show all customers in Germany,' gets twenty-five, and opens a support ticket.
[00:21] Matthias: So the real question's what kind of tool this actually is... because that cap says something about its purpose.
[00:29] Fabia: That's the episode. What the grounding pipeline does under the hood, where accuracy lives and dies, and the one investment that outranks every prompt you'll ever write.
[00:40] Fabia: A data agent is the Azure OpenAI Assistant API orchestrating read-only tools. You can't swap the model. What you shape is the grounding prompt, and that prompt decides query quality.
[00:52] Matthias: What goes into that prompt?
[00:54] Fabia: The user's question, the schema of your sources — fetched under the caller's Entra ID — your instructions up to fifteen thousand characters, the top few-shot examples ranked by vector similarity, and the conversation history.
[01:09] Matthias: Schema meaning table and column names.
[01:12] Fabia: And if the caller can't see a table, the agent never gets its schema. No schema in the prompt means no query against that table.
[01:20] Matthias: So security sits upstream of generation.
[01:23] Fabia: Then a tool picks the language. SQL for a lakehouse or warehouse, DAX for a semantic model, KQL for an eventhouse. It validates, runs read-only, and caps the result at twenty-five by twenty-five.
[01:36] Matthias: And the generated query's visible to the user... so you can actually read what it wrote.
[01:42] Fabia: That's your debugging surface. When an answer looks wrong, expand the steps.
[01:47] Matthias: What about sharing? I give this agent to my whole org — does everyone see what the builder could see?
[01:53] Fabia: No. Every query runs as the person asking. The agent doesn't hold its own credentials.
[01:59] Matthias: RLS on semantic models?
[02:01] Fabia: Still applies. Same agent, same question — different results for different people.
[02:06] Matthias: Someone tries to prompt their way past the instructions?
[02:10] Fabia: Four-layer precedence. Organization at the top, workspace governance below it, your developer config next, user's question at the bottom. Each layer's bounded by the one above. A prompt can never override a Purview policy.
[02:25] Matthias: That's structural. I'll give it that.
[02:27] Fabia: The caveat: it's still a new query surface. Purview Audit can log every interaction — factor that into your compliance posture.
[02:36] Matthias: Treat it like any other query endpoint in the audit plan.
[02:40] Matthias: So the governance holds up. What actually goes wrong?
[02:43] Fabia: We got a question. 'We built a data agent over our sales warehouse. In the demo it was great. Then a director asked a slightly different question and it joined the wrong table and gave a confidently wrong number. How do we know when to trust it?'
[03:00] Matthias: That's the story I keep running into. Works in the demo, falls apart in the meeting that matters.
[03:07] Fabia: The governance can't save you here. The query was allowed. Read-only, right identity, every policy respected. It was just wrong.
[03:15] Matthias: What's the answer?
[03:17] Fabia: You measure it. Microsoft ships a Preview evaluation SDK. Build a benchmark — the thirty to fifty questions you actually expect, with answers you know are correct. The SDK counts correct, incorrect, and unclear.
[03:31] Matthias: So you publish only after you have a number... the evaluation's the gate, full stop.
[03:36] Fabia: And re-run it after every schema change. An unmaintained agent silently loses accuracy.
[03:42] Fabia: The evaluation tells you where it's wrong. The deeper question is what drives accuracy. And the answer's unglamorous.
[03:50] Matthias: Column names.
[03:51] Fabia: The single highest-impact factor. IsCustomerActive gives the model a clear signal. C1 gives it nothing. The agent reasons over your column names — it can't read your mind about what flg_3 means.
[04:04] Matthias: So the biggest accuracy lever for an AI feature is renaming columns in a warehouse.
[04:09] Fabia: In the video, the generation film walks through this. The agent rephrases the question using only the vocabulary you provided. Margin — gross or net? Fiscal year — whose calendar?
[04:21] Matthias: The demo had that too, right? Same question before and after adding instructions?
[04:26] Fabia: Same question — margin by region in fiscal year 2013. About a forty percent difference in total sales, depending on whether the agent picked calendar year or fiscal year.
[04:38] Matthias: Forty percent. Same question. That's the difference between—
[04:42] Fabia: —a useful tool and a career-ending number. And the fix was one paragraph defining three terms.
[04:48] Matthias: I'd lose sleep over that one. A team ships the demo, leadership's impressed, they publish to the org. And the numbers are quietly wrong by forty percent because nobody defined three business terms. The model did exactly what it was told — the instructions just weren't there.
[05:07] Fabia: Then examples. Up to a hundred per SQL or KQL source, but the agent pulls only the top few by vector similarity per question. Cover different shapes — a join, a date filter, an aggregation. And semantic models don't take examples; invest in Prep for AI on the Power BI side.
[05:25] Matthias: What does running one of these cost?
[05:27] Fabia: Runs on your Fabric capacity. F2 or higher, or Power BI Premium P1 or higher. The Azure OpenAI compute behind it is Microsoft-managed.
[05:36] Matthias: No separate AI bill.
[05:38] Fabia: The cost that sneaks up is maintenance. Instructions and examples rot as schemas drift. Budget a quarterly review.
[05:45] Matthias: That's the part nobody budgets for. You build it in a sprint, demo it to applause, ship it. Six months later the schema's changed, nobody's touched the instructions, and accuracy's quietly degrading. By the time someone important gets a wrong number, it's been wrong for weeks.
[06:04] Fabia: Cross-region's a wall too. Agent capacity and source capacity have to share the same region, or the query won't run.
[06:12] Matthias: Give me the steel-man for skipping this and calling Azure OpenAI directly.
[06:16] Fabia: If you need your own model, your own RAG, or your own orchestration logic — the agent locks you into one LLM, five sources, twenty-five rows, and English only.
[06:27] Matthias: For a team already deep in Foundry with multiple—
[06:30] Fabia: —tools, the five-source cap and the fixed model might shape your architecture more than you'd like. Go direct, and you own the design.
[06:39] Matthias: The tradeoff being you rebuild governance yourself.
[06:43] Fabia: Identity passthrough, read-only enforcement, Purview integration. That's what the agent hands you. Governance is the piece teams underestimate when they roll their own.
[06:54] Matthias: So my call: if you're building your first conversational interface over Fabric data and governance isn't solved in your stack, take the agent. It gives you identity, read-only, and Purview. If you're already orchestrating in Foundry and the five-source limit's too tight, go direct — knowing what you're rebuilding.
[07:15] Fabia: One risk to flag. The Foundry FabricTool uses On-Behalf-Of authorization. It doesn't support service principals.
[07:22] Matthias: SPN doesn't work through Foundry at all?
[07:25] Fabia: Service principal auth is Preview, direct REST only, and doesn't cover KQL sources. Plan for user identity in Foundry.
[07:33] Matthias: Here's what I'd tell a team starting this week. Before you open the agent builder, go look at your column names. If you can't explain what flg_3 means, the model can't either. That's your real architecture decision.
[07:47] Fabia: Simplicity on the slide is not simplicity at runtime. The agent looks easy. The schema work underneath is the hard part.
[07:55] Matthias: And for the team that hasn't done that work?
[07:59] Fabia: Fix the schema first. The agent will wait.
[08:01] Matthias: Somewhere right now, a data agent's joining the wrong table because a column's still called C1. And nobody's measured it.
[08:10] Fabia: That column name is the architecture decision.
[08:13] Matthias: If you're building one of these and want to talk through the tradeoffs — DM me on LinkedIn, find me as Matthias Falland.
[08:21] Fabia: See you next week.
---
End of episode. ~08:22 estimated.