When Autolog Stops Logging — Fabric Experiments

Show Notes

When Autolog Stops Logging

Episode 23 • 2026-06-05 Duration: 8:47

Fabric's MLflow autologging promises zero-effort experiment tracking — until you train with LightGBM or XGBoost and discover your metrics column is blank. We pull apart the framework gap, the compounding exclusive flag, and the Git backup that isn't one.

What we discuss

• How it actually works underneath the abstraction
• Where the obvious answer breaks
• A real Reddit/Microsoft Q&A question unpacked
• The pattern we keep seeing in the field
• Risks of the recommended path
• F-SKU realism — what this actually costs
• When the rejected approach is actually right
• The concrete recommended architecture
• The architectural principle to take home

Key takeaways

• Somewhere right now, a training run just finished.
• And name your runs. Pass run_name to start_run. Forty runs all called Run followed by a number, and the comparison pane becomes useless regardless of how clean the rest of your tracking is.
• The takeaway I'd leave anyone starting with Fabric Experiments — the abstraction is real, but it has seams you need to know on day one.

Resources

• mlflow-upgrade
• Machine learning experiments in Microsoft Fabric
• Autologging in Microsoft Fabric
• What is Data Science in Microsoft Fabric?
• Analyze and train data in Microsoft Fabric
• Hyperparameter tuning in Fabric
• Perform hyperparameter tuning with FLAML
• Training visualizations for AutoML
• MLflow 3 in Fabric Data Science
• Tutorial: Create, evaluate, and score a churn prediction model
• Tutorial Part 3: Train and register a machine learning model
• Train models with scikit-learn in Microsoft Fabric
• Machine learning experiments and models Git integration and deployment pipelines (Preview)
• Data science roles and permissions
• Lineage for models and experiments

About the show

AI-generated voices. Matthias — cloned voice. Fabia — designed AI co-host. See Matthias live on YouTube (Fabric Friday), at his meetups, and at conferences like FabCon.

Hosted by Matthias Falland — Microsoft Data Platform MVP and community architect behind the Fabric Periodic Table. New episodes every Friday.

Submit your case

Have an architecture decision you are wrestling with? DM Matthias on LinkedIn — find him as Matthias Falland. Three to five sentences about the decision, your team size, and your current stack. We anonymize before airing.

---

This podcast was generated by AI.

Brand design based on fabricperiodictable.com.

What is Microsoft Fabric Architecture Podcast?

Architecture decisions for Microsoft Fabric. Anonymized real customer scenarios, cost realism, counter-arguments included. Weekly episodes aligned with Fabric Friday recordings.

Speaker 1
Four frameworks, zero metrics locked, and the run status says finished.

Speaker 2
Model artifacts right there, parameters all captured, the metrics column blank.

Speaker 1
And autolog was on the whole time. Architecture decisions for Microsoft Fabric. This is the Fabric Architecture Podcast.

Speaker 2
That's what we're pulling apart today. Fabric Experiments, the ML flow tracking layer that promises zero boilerplate observability for your training runs. The promise is real. The seams are where it gets dangerous. When you open a fabric notebook and write import ml flow, autologging is already active. Fabric calls mlflow. autolog behind the scenes. You don't write it. Parameters, model artifacts, datasets, the model signature you'll need for predict downstream, all captured. For Scikit Learn, PyTorch, TensorFlow, it's genuinely zero effort. You train a model, and the experiment item in your workspace has a complete run record waiting for you

Speaker 1
I've watched people demo exactly this and walk away thinking they're covered. Trained the model, saw the run appear, never opened the metrics tab.

Speaker 2
And if they used Scikit Learn, they are covered. That's what makes the next part so disorienting. ScikitLearn autologs everything. Params, metrics, the model artifact. Same for PyTorch and TensorFlow.

Speaker 1
But Lite GBM and XGBoost, the ones everyone actually picks for tabular data.

Speaker 2
Those plus Spark and Synapse ML Autolog captures the parameters and the model, but metrics? Empty. No error, no warning. The column just stays blank.

Speaker 1
Why though? What's different about tree boosters?

Speaker 2
Callback architecture. XGBoost and LiteGBM compute evaluation metrics inside their own training callback systems. MLflow's autolog hooks can read the estimator object. That's where the parameters and the saved model come from. But those custom callbacks are an API boundary autolog can't cross. It's documented in the hyperparameter tuning article, which nobody reads until they're already staring at an empty column. We got a question that lands right here. I enabled mlflow. autolog and trained a light GBM classifier in a fabric notebook. The run shows up, the hyperparameters are all there, the model artifact is there. But the metrics section is completely empty. I'm calling. fit exactly like the sklearn example. What am I doing wrong?

Speaker 1
Nothing. That's what makes this one sting. It's a documented framework limitation. Lite GBM Autolog captures params and the model but skips metrics entirely The right instinct is to add manual mlflow. log. metro calls for your F1, your AUC, whatever you're evaluating on.

Speaker 2
And that instinct is going to fail the first time you try it.

Speaker 1
Why?

Speaker 2
Fabrics Autolog ships with exclusive equals true as the default. That flag means Autolog is the only logging channel open. When you add your manual log micro call, which is the right fix, it gets silently ignored. No error, no exception. Your code runs, the metric still doesn't appear, and you're staring at the same blank column, wondering what you broke.

Speaker 1
So the failure compounds. Framework doesn't auto-capture your metrics. You write the code to capture them manually. Platform silently swallows your fix. Three things going wrong and zero error messages. Every time I see a team start with fabric data science, the first notebook they write is a light GBM classifier on some churn dataset. That's the hello world. And it hits both of these traps simultaneously.

Speaker 2
The fix is one flag. mlflow. autolog with exclusive equals false Set it before the run starts. Every official tutorial uses it, but nobody reads that parameter name and thinks it's the thing holding their metrics hostage. Fabrics Git integration for experiments is in preview. and it's metadata only. Display name, version, lineage, those sync to git. Your actual run data, everything from parameters to model artifacts, stays in workspace storage. Nothing in the UI flags that distinction when you set up the connection.

Speaker 1
So if the workspace gets deleted.

Speaker 2
Runs are gone. Git won't restore them. The docs are explicit. Data is not stored in Git. Only artifact metadata is tracked.

Speaker 1
I keep running into teams that connect the workspace to Git and think they've got a safety net. What they've got is lineage and display names. The actual run history sits in workspace storage with no second copy. Treat notebooks and training configs as your source of truth. Regenerate runs per environment and stop expecting Git to carry your experiment data across.

Speaker 2
On the economics, experiment tracking doesn't add a separate line item. It runs on your existing fabric capacity, same pool as your spark jobs and everything else in the workspace F SKU or trial required, no dedicated ML compute tier, and no native GPU in fabric notebooks.

Speaker 1
So if you need GPU accelerated training, you've already outgrown this.

Speaker 2
Which raises the honest question. When should you walk away from fabric experiments entirely? If you need GPU clusters for vision or language models, Azure Machine Learning has them. If you need a full MLOps audit trail where run data is version controlled and downloadable, Azure ML covers that. And if you're already invested in Databricks with Unity Catalog governing your model lifecycle, stay there. Fabric experiments don't interop with Unity Catalog.

Speaker 1
The case for Azure ML is genuine. If training and deploying models is the center of gravity for your team, the primary workload, Azure ML is the more complete surface. Fabric experiments make sense when ML is one workflow among many and you don't want to stand up a separate tracking server. Architecture is not religion. It's a fit question

Speaker 2
Pattern dictates platform. If your training runs feed a lake house, the lakehouse feeds a semantic model, and everything lives in one fabric workspace with shared lineage Then experiment is the tracking layer that belongs. The moment the ML workload outgrows the capacity or needs hardware fabric doesn't offer, That's your signal to split it out.

Speaker 1
To take away I'd leave anyone starting with fabric experiments. The abstraction is real, but it has seems you need to know on day one. Mfflow. autolog with exclusive equals false. Manual metrics for every tree booster. And never assume Git is backing up your runs.

Speaker 2
And name your runs. Pass run name to start run. 40 runs all called run followed by a number, and the comparison pane becomes useless regardless of how clean the rest of your tracking is.

Speaker 1
Somewhere right now a training run just finished. Status has finished, model artifacts there, and the metrics column is empty. The person who ran it won't notice until they try to compare it against tomorrow's run, and there's nothing to compare.

Speaker 2
That run says finished. The metrics are still waiting.

Speaker 1
If you've been there or you want to argue about when Azure Mail is the better call, DM me on LinkedIn. Find me as Matthias Falland.

Speaker 2
See you Friday.

Speaker 1
This was the Fabric Architecture Podcast. This podcast was generated by AI. New episode every Friday.