DEV

Your AI tool works great in testing — then real users show up and it starts misbehaving. This episode makes the case that a system prompt isn't a creative exercise: it's a policy document, and writing it like one is the difference between a reliable internal tool and an unpredictable one.

Show Notes

System prompts are the invisible policy layer inside every AI tool your business runs — and most of them are written like rough drafts rather than operational documents. This episode of Development tackles a failure pattern that founders and ops teams hit repeatedly: an internal tool that performs well in testing collapses into inconsistency the moment real users, real inputs, and real edge cases arrive. The fix isn't technical. It's in how the prompt is written.

The episode walks through four concrete principles for writing system prompts that hold their shape under pressure, covering:

  • Job, scope, and edge — all three. Most prompts define what an agent should do but skip what it shouldn't, and what it should do when it hits a situation outside its lane. All three elements are required to write an actual policy rather than a job description.
  • Worked examples inside the prompt. Abstract rules leave room for interpretation; concrete examples of both a clean output and a correctly handled ambiguous case communicate in ways that instructions alone cannot. This is one of the most underused techniques in custom internal tools.
  • Explicit failure-mode handling. Every agent has predictable failure modes — the complaint arriving through an intake form, the brief missing a budget, the transcript with no decisions. Listing the five most likely off-path inputs before launch and writing handling instructions for each is a stress test you run before your users do it for you.
  • Version-controlling prompts like code. A system prompt is a policy document and should have a change history. When an agent's behavior shifts unexpectedly, the first question is always what changed — and without a version log, that question is unanswerable.
  • A full worked example. The episode contrasts a vague vendor-inquiry prompt with a policy-grade version, showing how roughly twenty additional minutes of writing translates into meaningfully more reliable behavior at volume.

The broader argument is that when you build on top of a language model — rather than renting off-the-shelf software — you own the consistency problem. Prompt-as-policy thinking is how teams building workflow automation or standing up AI employees inside their operations keep that consistency from eroding over time. For more on this episode's themes, the show previously explored related decision-making in How to Read an Evaluation Criteria Section Before You Write a Word.

VB

RFP

What is DEV?

Software and AI development podcast. We cover all things software development, including today's advanced AI development tricks and techniques.