DEV

AI agents that nail the demo but fail in production almost always share one root cause: poor context design. This episode breaks down what context actually means for agents, why it's so easy to get wrong, and how to fix it with discipline.

Show Notes

Demos lie. An AI agent can look flawless in testing and then consistently produce confident, plausible, wrong answers the moment it touches real work. This episode of Development digs into the single most common — and most expensive — failure mode teams encounter when deploying AI agents inside their businesses: the context problem. It's not about the model. It's about what the agent actually knows when it has to make a decision.

The episode covers what context really means for an AI agent in production, why the gap between what you assume the agent knows and what it actually has is where failures live, and what a disciplined approach to fixing that looks like. Key points include:

  • Context defined practically: everything an agent has access to at decision time — instructions, data, history, and tool results — not what you assume it knows because it seems obvious.
  • The confidence trap: unlike a confused new hire, an AI agent won't ask a clarifying question — it will make its best guess and deliver it with complete certainty, which is precisely what makes context failures so damaging.
  • A worked example: a customer support agent that handles retail tickets perfectly but goes off the rails when it encounters a wholesale partner — not because the model is bad, but because the agent was never told that wholesale partners exist.
  • Structured context design: the discipline of deciding what an agent always needs to know, what it should look up dynamically, and — critically — what it should be explicitly told is out of scope.
  • Four concrete practices: writing an agent brief that precedes the system prompt; treating context retrieval as a first-class engineering concern; auditing failures through a context-first lens; and versioning context alongside the agent whenever business rules change.
  • The two-job reality: most teams do the engineering work well and chronically underinvest in context architecture — the ongoing work that determines whether a custom internal tool actually holds up in production.

The broader argument is that the model is increasingly a commodity. What separates a reliable AI employee from one that generates cleanup work is the quality of the context surrounding it — and that quality is a discipline, not a one-time setup task. Teams building workflow automation around AI agents will find this framing directly applicable to almost any deployment scenario. For more on building software that works the way your business actually works, visit VB. Also worth your time: the related episode The 30-Day Response, Budgeted Backwards, which tackles how operators plan and prioritize the build decisions that follow from getting the fundamentals right.

VB

RFP

What is DEV?

Software and web development from the side that has to ship it and then live with it. Architecture decisions with a cost attached, scoping, technical debt, hiring and vendor selection, and the AI tooling question every engineering team is now answering whether they planned to or not.

Each episode takes one decision — rewrite or refactor, framework choice, build versus buy, how to scope a fixed-bid project honestly — and works through the tradeoffs, including the ones that only show up in year two. Written for engineering leads, technical founders and the people who fund them. Five or six minutes, no hand-waving.

Topics include rewrite versus refactor, build versus buy, scoping fixed-bid work honestly, technical debt you should keep, framework and platform choices, hiring and vendor selection, code review culture, and where AI tooling actually helps.

Produced by DEV.co, web and software development. Full details, services and further reading at https://dev.co