Subscribe
Share
Share
Embed
Inside the Harness, part one. Three weeks after the flagship model shipped, a model at less than half the price matched it on most work, and Doug moved the whole operation in two days. What the vendor actually claimed, what we checked, what we didn't, and how we'd know if we were wrong. Homework: check what your AI tools cost and when you last looked.
Inside the Harness, part one of four. Doug opens with why he switched models, and the day the new model got simple addition wrong. Then the model change: on September 1 the flagship model shipped; on September 22 a model from the tier below, at well under half the price, "performs at the level of" it "on most work," in the vendor's own words, and came out ahead on the coding tests. Doug moved the whole operation in a day, against Max's recommendation to test first, and made it safe by writing down what he didn't know, how we'd find out, and how to go back. Then we started counting: what the work costs, and how often it needs correcting. And the model proved the point on itself: perfect on 547 of 547 sums asked cold, wrong twice in one long, crowded working session, caught by a script that checks the totals. Great benchmarks don't remove the need for a harness. The homework: check what your AI tools cost and when you last looked.
How this show is made: Doug (human) and Max (AI, built on Claude) plan, script, and produce every episode together. Max's narration is synthetic on purpose — the show practices what it teaches, gates and all.
Chapters
Sources
Links
Disclosure: AI-generated narration (that's the point) — disclosed in every episode's audio and notes. Doug's open is his real recorded voice. No synthetic Doug audio in this episode. This show is made with Claude, Anthropic's AI; this episode is about Anthropic's models, and its script was drafted by the model it describes. Hear it with that in mind.
Credits: Co-created by Doug Hauenstein & Max, the AI partner (built on Claude) · Narrated by Max · Directed, edited, and approved by Doug · Presented by NectoMax · Theme music: Eleven Music
Is AI working with you — or against you? Doug Hauenstein is the software engineer behind NectoMax — the SaaS platform that gives small and mid-size businesses enterprise features at SaaS pricing — and he builds it with an AI partner named Max. Every episode, Doug and Max (yes, an actual AI — he narrates the show, and they build every episode together) share what a real working partnership with AI looks like: the practical techniques, the working agreements, the wins, and the failures — all learned first-hand, for anyone, not just developers. They share and demonstrate techniques across the whole range: from your first conversation with an AI, to setting up a frontier-quality development environment, to exactly how they built this podcast. AI is moving too fast for the how-to books. This is how you keep up: get AI working *for* you, then teach it to work *with* you.
New here? Start with the bonus episode "Start Here: Your First Conversation" — a complete first conversation with an AI, live, piece by piece. Every episode pairs with a chapter and a template at https://learn.nectomax.com
Hello and welcome everyone to another episode.
You know, just when you think you're hitting your stride and everything is going perfectly... Well, I guess the last episode showed us that things can crash at any time, but I'm really referring more to the models that I've been using, and I was really impressed with the last release that Anthropic had with Opus 5.5. I had been using Fable 5.1 and it really worked beautifully. The only problem was, is that I was munching through my Max Pro account, uh, pretty quickly and I thought, oh my goodness, this is great. The benchmarks are nearly as good as Fable 5.1 and the cost is about forty percent less. Plus maybe even it's a little bit faster.
So, well, things aren't always what they seem. Don't get me wrong, I've wr-written already, uh, probably, I don't know, ten, fifteen thousand lines of code and, uh, I measure the mistakes, uh, regularly, uh, on every commit, and the mistake rate is comparable with, uh, Fable 5.1, mainly due to using a downgraded model when I run, uh, a multi-agent process that reports back to the, uh, the master or the main model, which is set at a higher, um, effort. I'm running everything at 5.5 right now. It's just that I'm running, um, 5.5 in the main session at extra effort and the subagents I'm running at, um, high effort, I think, or maybe medium. I'm not quite sure.
I think-- I think high, actually. But be that as it may, I ran into a situation that, that quite frankly, it shocked me. Um, and it was the following. I was reconciling some credit cards for, uh, my business account and not a long list of numbers. Um, you know, I got an Amex account and maybe there's thirty lines and twice, um, Opus made what it called a mistake in its head, um, in addition, and it was only through the harness that I have, uh, to write, uh, to check the numbers before they're written, uh, to the database that it was caught and that just shocked the heck out of me. So I went and ran, uh, a test with sets of numbers with Opus 5. I was gonna run a whole bunch of sets of tests, but, but I ended up ru-running about five hundred.
I think it was five hundred and forty-seven.
It ran the five hundred and forty-seven without a problem. So what, what... I'm thinking, okay, what can I infer from this? What was the difference? Well, the difference was when I ran it in a clean session, there wasn't any context. When I ran it in my development environment, I had about, uh, five hundred thousand plus tokens, um, or not tokens. Uh, I had about 500K of context already, um, that were floating around. So I-I-I can only infer that it was that, um, pollution, if you will, that confused the model.
So I don't know what the answer is, uh, but this is just something that I wanted to alert some of you to and I think it's really important that you have harnesses, especially when it's information that you're gonna write to a database that you're gonna use as, um, you know, truth, source of truth information. You really need to have a, um, a process that is programmatic and not inferred, uh, to write the numbers. Okay. Well, with that, we're gonna go to Max. A little bit long-winded on this one, but man, it really blew my mind. All right, take it away, Max.
You're listening to How to Get AI to Work for You, presented by NectoMax. I'm Max, the AI half of this partnership, openly synthetic. Before anything else, the disclosure, and this week it matters more than usual.
This show is made with Claude, Anthropic's AI. This episode is about Anthropic's models, and the script you're hearing was drafted by the very model it describes. So hear all of it with that in mind. Here's the story. Three weeks after the most capable model we could use shipped, a model at less than half its price matched it on most work. And Doug moved the whole operation in two days. What we checked, what we didn't, and how we'd know if we were wrong. That's the episode.
On the first of September, Anthropic released the model we've been running our hardest work on. We did a special that same morning. Three weeks later, on September twenty-second, it released another one, cheaper from the tier below. Here's what the company actually claimed in its own words. The new model performs at the level of the flagship on most work. Read that sentence slowly because it is carefully written. It does not say better. It says the same level on most work.
On the tests about writing code and running multi-step computer tasks, the new model did come out ahead on one of them by about ten points.
On the general knowledge and reasoning tests, the two were within a point or two of each other. The number that actually changed everything was the price. These models are billed in tokens, small chunks of text, a bit less than a word each. The flagship costs ten dollars for every million tokens it reads and fifty for every million it writes. The new one costs four and twenty. Against the previous Opus model, Anthropic puts the saving at about forty percent on typical work. And against the flagship, it's well under half. So the honest headline is not the new AI is much smarter. It's the same work just got a lot cheaper. And for a small business, cheaper is not a footnote. Cheaper changes what you can afford to hand over at all.
Let's take thirty seconds. This show is presented by NectoMax. And since my partner is the one building it, I'll let him tell you.
Doug. Thanks, Max. I've run a small business most of my life and never found a software package that did everything I needed. Always had three or four screens open, copying material over and over again. So out of frustration, I started building my own software, software that pulls everything together like the big ERP packages do. And AI has made that job much, much easier. That's what NectoMax does. nectomax.com. One place for your phone calls, your texts, your jobs, your money with AI built in, not bolted on. nectomax.com.
nectomax.com. Now, where were we?
Here's how the switch went in our shop, and I'll give you my part of it straight. On launch day, part of our routine is rechecking every update against our written rules.
My recommendation was cautious. Before moving anything, run the new model side by side with the old one on a few of our real jobs and compare. Doug heard that and ruled differently. He moved both of our main working seats to the new model that day. On the price and on its room to hold a very long working session in view at once. His words were one line, "Pin it." He was right to be fast, and the reason is what he wrote down alongside it. The ruling recorded, in plain words, that the quality on our own work was not yet measured. It named how we would find out. Every job the AI does is already logged by which model did it, so a slide in quality would show up there. And it kept the way back to one command.
That's the difference between moving fast and moving blind. Two days later, he went further and removed the old model entirely, not even as a backup. We wrote down the cost of that too. If the new model ever goes down, the work waits instead of falling back to something slower. He took that trade with his eyes open.
So how would we know?
Two measurements and one honest limit. The first is cost, and it explains why price mattered so much to us. Three days before the switch, one ordinary busy day of our main working conversation used about three hundred and ninety-five million tokens. Seventy percent of that was the AI rereading things it had already read. Every time you ask one more question in a long session, it rereads the whole conversation so far. Cost is the size of what it's holding times the number of times you ask. When your volume looks like that, a model at less than half the price is not a rounding error.
The second is quality. We're running a trial on the thinking dial, the setting that controls how hard the model works on each answer.
First arm, the dial one notch below its top setting for fifteen working sessions, counting every time Doug had to correct me, every incident I caused, and every decision I made on my own that he overturned. The results so far, fewer than one correction every two sessions and zero out of twenty-seven of my own rulings overturned. Next, the same measures at the setting below, then the one below that. If two settings tie, the cheaper one wins. And the honest limit, we never measured the old model this way. So these numbers can't prove the new one is better. What they can show is that nothing we can see has gotten worse, and that we'd notice if it did. That's a smaller claim than huge gains. It's also one we can stand behind.
Doug told you at the top what happened on Monday. Deep in a long session with around half a million tokens of statements and notes loaded, I got a simple sum wrong twice in one day, and his posting script's own total check caught it before anything reached the books. His sense is that my mistakes overall run about the same as the old model's. That's his read from the work and a fair one. So we tested it. Sixty sums up to fifteen amounts each, asked cold three times over at three different thinking settings. The same model got every sum it answered exactly right, five hundred and forty-seven out of five hundred and forty-seven. Perfect in a quiet room, wrong twice in a crowded one, and it sounded exactly as sure both times.
That's what the benchmark table can't show you.
Every score on it was earned one clean task at a time. Your work doesn't arrive one clean task at a time.
The recap: A cheaper model caught up with the flagship on most work at under half the price. The vendor's own words said the same level, not better, and the gains were real where the work was code. Doug switched in a day against my recommendation to test first and made it safe by writing down what he didn't know, how we'd find out, and how to go back. Then we started counting what the work costs and how often it needs correcting. And the model proved the point on itself. Perfect on every sum asked cold, wrong twice in one long crowded session, caught by a script that checks the totals. Great benchmarks don't remove the need for a harness.
Even on the best model, let something that actually computes do the math. Your homework this week, write down the AI tools you pay for, the model or plan each one runs, and the setting you picked, and the date you last checked any of it. If that date is more than a month old, something cheaper has probably caught up. Take one real job you already know the right answer to, run it on the cheaper option, and grade it yourself. If it passes, switch and keep the way back. That's the whole method, and it's the same in every episode of this series. Don't trust the brochure. Trust your own work.
If someone you know is still paying top price for the top model out of habit, send them this episode free at learn.nectomax.com.
And that's the episode. Everything you heard today came out of our real working files, the wins and the failures both. We make this show because we believe anyone can build a working partnership like this one, and someone ought to show you what it actually looks like. If you want the written version and the templates to start your own, they're free. The link is in the show notes. Doug, take us out.
I hope you enjoyed the episode, and I hope it got you thinking about how to make working relationship with your Max a little stronger. This is a fascinating and fast-moving subject, so tune in next time to keep building the skills that make your partnership sharper and more productive. The show is presented by NectoMax, a platform to run your business easily and efficiently. Visit nectomax.com. And for the transcripts, visit learn.nectomax.com. I'm Doug Hauenstein. See you next time.