DEV

Your AI tool works great in testing — then real users show up and it starts misbehaving. This episode makes the case that a system prompt isn't a creative exercise: it's a policy document, and writing it like one is the difference between a reliable internal tool and an unpredictable one.

Show Notes

System prompts are the invisible policy layer inside every AI tool your business runs — and most of them are written like rough drafts rather than operational documents. This episode of Development tackles a failure pattern that founders and ops teams hit repeatedly: an internal tool that performs well in testing collapses into inconsistency the moment real users, real inputs, and real edge cases arrive. The fix isn't technical. It's in how the prompt is written.

The episode walks through four concrete principles for writing system prompts that hold their shape under pressure, covering:

  • Job, scope, and edge — all three. Most prompts define what an agent should do but skip what it shouldn't, and what it should do when it hits a situation outside its lane. All three elements are required to write an actual policy rather than a job description.
  • Worked examples inside the prompt. Abstract rules leave room for interpretation; concrete examples of both a clean output and a correctly handled ambiguous case communicate in ways that instructions alone cannot. This is one of the most underused techniques in custom internal tools.
  • Explicit failure-mode handling. Every agent has predictable failure modes — the complaint arriving through an intake form, the brief missing a budget, the transcript with no decisions. Listing the five most likely off-path inputs before launch and writing handling instructions for each is a stress test you run before your users do it for you.
  • Version-controlling prompts like code. A system prompt is a policy document and should have a change history. When an agent's behavior shifts unexpectedly, the first question is always what changed — and without a version log, that question is unanswerable.
  • A full worked example. The episode contrasts a vague vendor-inquiry prompt with a policy-grade version, showing how roughly twenty additional minutes of writing translates into meaningfully more reliable behavior at volume.

The broader argument is that when you build on top of a language model — rather than renting off-the-shelf software — you own the consistency problem. Prompt-as-policy thinking is how teams building workflow automation or standing up AI employees inside their operations keep that consistency from eroding over time. For more on this episode's themes, the show previously explored related decision-making in How to Read an Evaluation Criteria Section Before You Write a Word.

VB

RFP

What is DEV?

Software and web development from the side that has to ship it and then live with it. Architecture decisions with a cost attached, scoping, technical debt, hiring and vendor selection, and the AI tooling question every engineering team is now answering whether they planned to or not.

Each episode takes one decision — rewrite or refactor, framework choice, build versus buy, how to scope a fixed-bid project honestly — and works through the tradeoffs, including the ones that only show up in year two. Written for engineering leads, technical founders and the people who fund them. Five or six minutes, no hand-waving.

Topics include rewrite versus refactor, build versus buy, scoping fixed-bid work honestly, technical debt you should keep, framework and platform choices, hiring and vendor selection, code review culture, and where AI tooling actually helps.

Produced by DEV.co, web and software development. Full details, services and further reading at https://dev.co