LLM.co

Your production traces are already labelled training data — you just need the right pipeline to use them. This episode walks through how self-hosted Langfuse turns live LLM output into fine-tuning datasets for open-weight models, inside a regulated boundary.

Show Notes

Production LLM traffic is one of the most underused assets in AI engineering. Every inference your model runs in the real world captures exactly the inputs, outputs, and edge cases you'd want in a training set — yet most teams only look at that data when something breaks. This episode of LLM.co unpacks how to close that loop: from instrumenting traces in self-hosted Langfuse, through structured scoring and curation, all the way to a QLoRA or DPO fine-tuning run on an open-weight model, entirely within a private deployment. The ideas are drawn from the Langfuse production-to-fine-tuning training loop article on the LLM.co blog.

The episode covers the full workflow from raw trace to trained adapter, including:

  • Why Langfuse is a safe long-term bet for private deployments — its MIT-licensed core, self-hosted architecture (Docker, Compose, or Kubernetes), and open-sourcing of all product features in mid-2025 mean the self-hosted path will not disappear behind a paywall.
  • What makes a trace training-ready — three fields matter above all else: the complete prompt the model actually saw, the raw completion before post-processing, and rich metadata tags covering route, model version, retriever version, and PII redaction status. For HIPAA workloads, data classification fields are also required.
  • Turning observability scores into dataset labels — Langfuse score configs (numeric, categorical, or boolean) can be attached via user feedback, LLM-as-a-Judge evaluators, or human annotation queues. A four-score rubric — Correctness, Groundedness, Policy, and Preference — covers both supervised fine-tuning and DPO pair construction. Teams running AI agent traffic monitoring can feed the same scoring infrastructure into their trace pipelines.
  • Curation as a filter-and-promote operation — filtering traces by score thresholds and promoting matching rows into a Langfuse dataset is a deterministic step, not a judgment call. Prompt experiments produce DPO pairs by running variants against production inputs and scoring completions head-to-head.
  • Export formats and nightly automation — Langfuse exports JSONL in ChatML schema directly from the UI; the public API supports filtered Parquet shards written to a private training bucket on a scheduled basis.
  • Training on a constrained GPU budget — QLoRA fits Llama 3.1 8B into roughly 10–14 GB of VRAM with four-bit base weights, making workstation-class hardware sufficient. DPO is the preferred alignment step over full RLHF, as it trains directly on preference pairs and avoids PPO instability. Organizations evaluating on-premise hardware for this kind of workload may find the LLM Appliance a relevant starting point.

For more on hardware selection for the models discussed in this episode, listen to GPU Server Configs for Open-Weight LLMs: Match the Build to the Model.

LLM.co

hardware.co

What is LLM.co?

Private and custom large language models — the build, the boundaries and the bill. Fine-tuning versus retrieval, running models in your own environment, evaluation you can actually trust, data governance, and the questions to ask before a vendor answers them for you.

Each episode takes one decision a team is facing — whether your problem needs a custom model at all, how to evaluate output without fooling yourself, what "private" has to mean contractually — and works it through concretely. Written for engineering and data leaders putting a model into production. Five or six minutes, one idea, no demos.

Topics include fine-tuning versus retrieval, self-hosted and private deployment, evaluation you can trust, prompt and context design, data governance and retention, cost and latency tradeoffs, and what "private" has to mean contractually.

Produced by LLM.co, private and custom large language models. Full details, services and further reading at https://llm.co