Subscribe
Share
Share
Embed
AI Functions let you call GPT from a SELECT statement with no API key. Zero infrastructure — until you check the capacity meter. Two billing streams, no result cache, and a default model eleven days from retirement.
Episode 25 • 2026-06-19 Duration: 9:01
AI Functions let you call GPT from a SELECT statement with no API key. Zero infrastructure — until you check the capacity meter. Two billing streams, no result cache, and a default model eleven days from retirement.
AI-generated voices. Matthias — cloned voice. Fabia — designed AI co-host. See Matthias live on YouTube (Fabric Friday), at his meetups, and at conferences like FabCon.
Hosted by Matthias Falland — Microsoft Data Platform MVP and community architect behind the Fabric Periodic Table. New episodes every Friday.
Have an architecture decision you are wrestling with? DM Matthias on LinkedIn — find him as Matthias Falland. Three to five sentences about the decision, your team size, and your current stack. We anonymize before airing.
This podcast was generated by AI.
Brand design based on fabricperiodictable.com.
Architecture decisions for Microsoft Fabric. Anonymized real customer scenarios, cost realism, counter-arguments included. Weekly episodes aligned with Fabric Friday recordings.
Speaker 1
One line of T SQL, no API key, no Azure OpenAI resource. You write AI classify in a select and GPT just answers.
Speaker 2
And nobody's checking what that costs per row. Architecture decisions for Microsoft Fabric. This is the Fabric Architecture Podcast.
Speaker 1
Today we're pulling apart how Fabrics AI functions actually work underneath and where the gap opens between a one-line enrichment call. and a capacity bill nobody planned for. The reason it's so frictionless is that Fabric ships a built-in LLM endpoint Your capacity token handles authentication. There's no API key anywhere in the flow. An admin flips one tenant switch, the one labeled Copilot and Azure OpenAI integration. And every notebook and warehouse in that tenant can call GPT through column operators. No Azure OpenAI resource to provision. The default model is GPT 4.
Speaker 2
1 Mini. So the trade-off for zero infrastructure is you're on whatever model Microsoft chose as the default.
Speaker 1
And that default retires June 30th, 2026. 11 days after this recording. Two models people might expect as upgrades, GPT-4. 1 and GPT-5, already retired on June 11. If you're hearing this later, the default in your tenant may already be something else entirely. The step-up options with no retirement date listed are GPT 5. 1 and GPT-5 Mini.
Speaker 2
That's a production concern hiding behind a convenience feature. If you don't pin your model in config, your pipeline's writing a moving target.
Speaker 1
Three execution surfaces, and they're not identical. Notebooks give you pandas, single-node, 200 concurrent async calls by default, or PySpark, distributed across the cluster at 50 calls per worker. The Warehouse and SQL Analytics endpoint give you T SQL functions. All preview.
Speaker 2
Every surface in 2026.
Speaker 1
Every single one, as of this recording. And the function lists don't match. Python gets nine functions. tSQL gets seven. The two missing from SQL are embed and similarity. Both depend on the embedding model. Anyone designing a warehouse native RAG pipeline will need to route the embedding step through a notebook Write vectors to a delta table, and join back from SQL.
Speaker 2
That's two execution surfaces for one logical pipeline. Both preview. You're coupling two things Microsoft hasn't committed to yet.
Speaker 1
Someone from the community asked exactly this. I ran AI classify over a column in my warehouse Some rows come back null and the whole query crawls. Is it broken?
Speaker 2
I've seen that exact reaction play out. Someone demos AI classify on 10 rows and it's magic. They pointed at a real table. Half the results come back null. The query takes 20 minutes.
Speaker 1
The preview is doing what it documented. Two causes for the nulls. Either the responsible AI filter blocked the input, or the text exceeded the 15 kilobyte limit per call. And the throughput, 20 to 100 rows per second. Each row is a separate round trip to an external AI endpoint. That's enrichment speed.
Speaker 2
20 rows a second on a warehouse built to scan millions.
Speaker 1
And here's the part that turns a slow query into a capacity problem. There's no result cache. If you put AI classify inside a view, every select against that view recalls the LLM. Every row. Every time.
Speaker 2
Wait, there's no caching at all? Every read is a new call?
Speaker 1
None. The docs say it directly. Materialize results into a staging table, then query the stored column. And the billing hits two separate meters simultaneously. Spark compute goes under the spark meter. That's your cluster time. Token usage, input, and output goes to a completely separate copilot and AI meter, shown as an AI functions operation in the Capacity Metrics app since March 2026. Both meters run at the same time.
Speaker 2
How bad does that get at scale?
Speaker 1
1 million rows, roughly 100 input tokens each, on GPT-4. 1 Mini Input tokens alone come to about 373 CU hours before output tokens, before Spark Compute. On an F2, that's a dominant share of your daily smoothed capacity from a single classify run.
Speaker 2
On the cheapest model, and output tokens run three to four times the input rate. I'd push back on the instinct to walk away from this entirely though. The zero infrastructure story is genuinely valuable. I keep running into teams that spend weeks provisioning an Azure OpenAI resource, wiring up managed identity, handling rate limits, parsing JSON. and they haven't enriched a single row yet. AI functions skip all of that. For 10,000 customer reviews you classify once and store, the cost is trivial and you're done in minutes.
Speaker 1
The Conditions That Make It Right Batch Enrichment Thousands to A few hundred thousand rows Mostly right and reviewable is the acceptable bar, and you materialize the result instead of recomputing it.
Speaker 2
And it's the wrong tool the moment someone needs stable auditable classification. Regulatory labels, compliance tags. I tell any team in that situation to build rules or a lookup table. Temperature zero doesn't make an LLM deterministic across model versions.
Speaker 1
The docs carry a disclaimer on every single example. This code uses AI. Always review output for mistakes A seed improves consistency across runs. It doesn't make the function pure.
Speaker 2
The architecture I'd actually recommend is almost disappointingly simple. Run the AI function once, in a notebook or a create table as select. Write the enriched column to your gold layer. Query the column, never the function. Schedule the notebook to re-enrich new rows on whatever cadence matches your data freshness.
Speaker 1
And before any full run, call ai. staining on a sample. 100 to 1000 rows. It returns prompt tokens, completion tokens, exception count, even how many rows the responsible AI filter flagged. Multiply out, check against your capacity, then decide.
Speaker 2
The part that keeps me up isn't the cost. It's the null. If AI fix grammar returns null because the filter tripped. and you wrote that straight back into the source column without a guard, you've just blanked production data.
Speaker 1
Is null of the AI function result, comma, the original value? The docs recommended explicitly. It's the kind of thing everyone skips in a demo and discovers in production.
Speaker 2
The lesson here carries beyond AI functions. The easiest thing to call is often the hardest thing to cost. And every select that touches one of these functions is an invoice from something that looked free because it shipped with the platform.
Speaker 1
Pattern dictates platform. If your pattern is enrich once, read many, AI functions are the shortest path in fabric right now. If that pattern drifts into enrich on every read, you're billing the LLM like a column default
Speaker 2
That quiet meter in your capacity metrics app, ticking once for Spark, once for tokens. On every row nobody thought to cash.
Speaker 1
Ticking right now in every tenant where someone put AI classify in a view and moved on.
Speaker 2
If you're watching that meter spike and can't figure out why. DM me on LinkedIn. Find me as Matthias Falland.
Speaker 3
We'll sort out which meter to blend. This was the Fabric Architecture Podcast. This podcast was generated by AI. New episode every Friday