Welcome back to AI Systems. This is episode two, and the question is: how much does the interface shape what the model can do?

Quick recap. Last time we pulled apart four places an agent's behaviour can come from. Training changes the model's weights. The context changes what the model knows about this particular task. Inference settings change how much computation it gets. And the harness, the software around the model, turns the actions the model proposes into things that actually happen, then hands back the results.

That gave us a more careful way into the argument the series is built on. One engineer wants a stable platform that works with any model, so the team can switch providers. The other thinks newer models have learned enough useful behaviour that a lot of the old scaffolding is becoming unnecessary.

We're not picking a side yet. Today's question is narrower, and both engineers need an answer to it. Say two tools can make exactly the same edit to a file. Should a model do equally well with either one?

It's tempting to say yes. If the model understands the task, and the tool can do the job, why would the interface matter? To see where that reasoning breaks, you have to follow the actual decisions the model makes. You also have to tell apart two different situations: the model genuinely can't do something, or the model can do it but the interface makes that hard to express.

We'll keep the parser from episode one. An empty input crashes it. The fix is small: before the code reads the first character, check whether the input is empty, and if it is, return the empty result the code already uses. The fix must leave the function next to it unchanged.

The file, the tools and the failures in this example are all made up. We're using them to show how things work, not pretending we've measured a real product.

So first: what does the model actually see when it's given a tool?

Not just a function sitting on a server. It sees a tool name, a description of when to use it, a schema for the arguments, and later, the result that comes back. The schema is the shape of a valid request: which fields exist, what kind of value each one takes, which are required, and what limits apply.

Behind that description sits the code that does the work. That code decides what a request really does, what permissions it needs, and what happens when something goes wrong. The model sees a description of this deal. It doesn't automatically know every assumption buried in the code.

Picture a tool called edit. Its description says: update a resource. Its arguments are called target and payload. Those names might make perfect sense inside your own software. The model still has to guess. Is target a file path or some internal ID? Is payload the new text, or a list of changes? Does the edit replace the whole file or just part of it?

Now picture a tool called replace exact text. Its arguments are file path, old text and new text. Its description says it makes one replacement, the old text has to match exactly one place in the file, and if there's no match, or more than one, it changes nothing.

The second tool leaves less unsaid. That's a good reason to expect fewer misunderstandings. It isn't proof that it works better. A model that's used to the first style might handle it fine, and the code behind the second one could still be broken.

There are several separate steps here, and an agent can fail at any of them. The model has to realise it needs to act, pick the right tool, fill in the arguments, and then make sense of what comes back. Separately, the software that carries out the request, which we'll call the executor, has to check the request and actually do it.

So a valid request to the wrong tool is a selection failure. The right tool with the wrong file path is an argument failure. A correct request that hits a permission error is an execution outcome, not a model mistake. A helpful error message that the model then ignores is a failure to use feedback. If you lump all of those together as "the model is bad with tools", you throw away exactly the information you need to fix the system.

Let's walk through one run step by step. In the jargon, the full record of a run is called a trajectory: everything the agent saw and did, in order.

The agent gets the bug report. It asks to read the parser file. The executor reads it and returns the relevant function. The model proposes an edit. The executor applies it and reports what changed. The model asks to run the test. The test fails, showing that the function next door now behaves differently. The model reads that part of the file again, revises its edit, and runs the test again.

Notice that the model doesn't get to make up the test result. The result comes from the outside world. A model can write text that looks like a test result, but only actually running the test tells you what happened. That's why we keep three things apart: the action the model proposes, the operation that really runs, and the result that comes back.

The same split matters in training. A training example can show a good tool request and the result that followed. Training teaches the model to write good requests and to use the results it gets back. Usually the model is only scored on the parts it wrote itself, not on the tool results in the record. So seeing tool results in training doesn't teach the model to invent them later.

A trajectory also tells you far more than the final fix does. It shows how the agent found the file, how it coped when its edit no longer matched, and what it did with a failing test. A dataset with only the bug report and the finished fix doesn't show any of those decisions in between.

There's more than one way for a model to pick up tool use. Toolformer, a research paper from twenty twenty-three, had a model write candidate tool calls into ordinary text, actually ran them, and kept only the calls whose results helped the model predict the text that came next. Then it trained the model on that enriched text. That's one documented approach. It isn't a description of how every coding model today was built.

ReAct is a useful contrast. Its headline results came from prompting: showing the model a few examples where reasoning steps and actions take turns, then letting it carry on in the same pattern. That shows a way of working you can bring out through the input alone. The paper also tried training smaller models on the same kind of examples, which is a different intervention, and it doesn't show that every useful agent needs to spell out its reasoning.

So when someone says a model has "learned tools", ask what they mean. Was it trained on trajectories? Is it following examples it's been given right now? Or is the program around it forcing the sequence? More than one can be true at once, but they're not the same thing, and they don't call for the same fix.

Now let's look more closely at learning from rewards, which we met last time.

Imagine two attempted fixes during training. In the first, the model reads the function, adds the empty-input check, and passes the hidden tests. In the second, it reads the same function, edits a similar-looking line in the function next door, and fails.

The first attempt gets the better score. That can be used to make some of its choices more likely next time. Both attempts started by reading the function, though. The score on its own doesn't tell you whether reading was the step that mattered, whether the particular edit was what made the difference, or whether some other approach would have worked just as well.

That's the credit assignment problem from episode one. Training turns scores into changes to the weights, following particular rules. A score at the end isn't a precise note saying which earlier decisions caused the success.

Now change the environment. Say the edit tool rejects an ambiguous replacement and explains why clearly. The agent can recover by giving more of the surrounding text. In a different environment, the same request quietly changes every match in the file. That can cause a failure before the model ever sees a useful warning.

So the tools on offer, and what they report back, affect which attempts succeed. If those attempts become training data, that way of interacting is part of what the model has learned from. The environment shapes the learning problem, and later the trained model has to work through an environment of its own.

That's the basic idea behind what people call model-environment co-adaptation: the model and its tools shaping each other over time. It happens through training, or through deliberate changes people make to the system. It doesn't mean the model's weights change every time it uses a tool in your live product.

Next, a phrase that gets used a lot without much explanation: distribution matching.

A distribution just describes how often different kinds of case come up. Here, think about the kinds of tool names, argument styles, results and recovery situations the model runs into. The training distribution is the mix it saw during training. The deployment distribution is the mix it meets in your product.

They'll never match exactly. The practical question is whether the differences affect the decisions that matter.

Here's an example. Say the model has seen, over and over, a tool where a request contains the old text and the new text. In your product, though, you ask for a numbered list of operations, with special rules about character positions. The edit you want might be identical. The model now has an extra job: translate that edit into an unfamiliar action language and follow all its rules.

Some models might do that reliably. Others might make more mistakes or need more attempts. Those are ideas to test, not conclusions. Familiarity is one plausible reason for a difference in performance. You can't read it off a score.

It helps to separate two complaints that sound alike: an interface is unfamiliar, or an interface is badly documented. An unfamiliar interface can still be perfectly clear. A familiar-looking one can be dangerous if it behaves a little differently from what its name suggests. You have to check both what the model is shown and what the tool really does.

Back to our two editing tools. The first is exact text replacement: give it the file path, the old text and the new text. The second takes a patch: a description of which lines to remove and which to add, plus a few lines around them so the change can be found.

For our parser fix, either tool can describe the same change. That's the limited sense in which they're equivalent, and it's all we need. We're not claiming every version of each tool supports exactly the same operations, or fails in exactly the same ways.

The replacement tool asks the model to copy out the original piece of code and write the revised piece. The patch tool asks it to identify the changed lines and include enough surrounding lines to pin down where they are. Each format has its own rules about matching, layout and ambiguity.

Think of it like giving directions. When a person calls the two tools equivalent, they're describing the destination: the file ends up with the same check at the top. The model still has to plan the route, and writing a valid patch is a different job from writing a valid replacement request.

There are also two kinds of correct. A request can be correctly formed, meaning it follows the required format, and still change the wrong thing. Or it can describe exactly the right change in a badly formed request the executor can't accept. In the jargon, the first is syntactically valid but semantically wrong.

Forcing the model's output to fit a schema can help with the format. It doesn't make the chosen file, old text or new text correct. A perfectly shaped request can still delete the wrong line.

Here's where the parser example shows this. The file has two functions that start with the same opening line. The model needs to change the parser, not the helper function below it. If it gives the replacement tool only that shared line, it matches in two places, so under the rules we set, the executor should refuse it as ambiguous.

A patch can be ambiguous too, if it includes too little surrounding context. Using a patch format doesn't guarantee the right change. Both tools need enough information to find the intended spot, and code behind them that handles ambiguity in a predictable way.

Now say the patch tool keeps the file's line endings as they were, but the replacement tool quietly rewrites every line ending in the file. You compare them and see different test results. You haven't measured the model's preference for one format. You've also changed what the executor does.

Or say the patch tool shows a useful snippet of the file when it fails, while the replacement tool just says "error". Again, you've changed more than the format. That's not a reason to give up on the comparison. It's a reason to be clear about what you're actually comparing.

For a product decision, comparing two complete tools exactly as they ship might be exactly what you want. To explain why the model behaves differently, you need a narrower test. Knowing which option works better and knowing why are related goals, but they're not the same goal.

Now we can separate two ideas: capability and elicitation.

Capability is what tasks a model can do successfully, under stated conditions. Elicitation is how the prompt, the tools, the information and the settings bring that performance out. There's no single true ability score hiding behind the interface that you could read off directly.

If the model writes the correct parser fix as plain text, but fails to get it through your edit tool, that's evidence the problem is in turning the fix into an action, or in carrying it out, at least for this case. If you hand it a correct tool call and the rest of the run then succeeds, you've narrowed it down further.

Those are diagnostic tests. They're not fair scores for a fully independent agent, because you helped. Keep the helped result separate from the unhelped one. The point is to find where it breaks, not to make the scoreboard look better.

If the model still writes the wrong check after you've made the interface clearer, you may be looking at a reasoning problem, missing information, or some other limit. A better tool description can't guarantee the right fix.

The tempting but incomplete view is that a powerful model should just figure out any equivalent interface. It might. Figuring it out still costs steps and attempts, and it can go wrong. You still have a budget, a deadline and a reliability bar to hit.

The opposite mistake is to blame every failure on elicitation. You could keep rewriting prompts forever and never admit the setup you're testing isn't good enough. Set a limit on how much effort you'll spend improving it. Test on held-out tasks, meaning tasks you didn't look at while tuning. And be willing to reject the setup if it doesn't meet your bar.

So what do the people building these systems actually report?

Cursor's harness write-up from April twenty twenty-six says it tunes prompts and tool formats for each model, inside shared abstractions that work across models. It describes giving models editing formats they're familiar with. Treat that as Cursor's own engineering account. It doesn't give anyone a view into every provider's training, and it isn't a permanent rule for future models.

Anthropic's article on writing tools for agents, from September twenty twenty-five, describes evaluating tools, making them return useful information, improving their descriptions, and checking changes against held-out tasks. Those are practices it reports from its own systems. They give us concrete things to try, not guaranteed results.

So the claim we can honestly make is narrower than "every model needs a custom harness". Particular tool designs and integrations can change how well a model performs, in the setups people have reported. How big the effect is, why it happens, and whether it carries over to your product all still need evidence.

Let's finish the failed edit, because what comes back from a tool is half the interface.

The model asked for a replacement. It gets back: edit failed. What should it do now? It doesn't know whether the file was missing, the old text wasn't there, it matched in two places, permission was denied, or the write actually went through just before the connection dropped.

Each of those needs a different next step. Missing context might mean reading the file again. An ambiguous match calls for a longer, more specific piece of text. Permission denied means the action isn't allowed. A write that might have happened needs checking before anything is retried. One vague error has squashed several very different situations into the same message. It's like a delivery update that just says "problem with your parcel": you can't tell whether to wait, go to the depot, or call the sender.

Now suppose instead, under the made-up rules for our tool, the executor says: two matches found, nothing written. And it names the function each match sits in. Now the agent can tell the parser from the helper, and write a more specific replacement.

That doesn't give the tool any new editing power. It just makes the situation visible enough for the model to choose its next move. That's one reason tool design can change how well an agent does over several steps, even when both tools could in theory produce the same final file.

The most useful message isn't always the shortest. If the result leaves out the function name the model needs to tell the matches apart, saving a few tokens can cause more mistakes and retries later. If it dumps the whole repository, the useful clue gets buried. Design the message around the next decision the model has to make, not around a blanket rule to always return more or always return less.

The same goes for tests. A result that just says "failed" tells the model less than one that names the failing check and points to where it broke. Even a detailed test result only tells you about that one run, though. It doesn't prove the rest of the system is correct.

Recovering from errors also depends on whether an action has side effects.

In the parser example, reading the file is usually safe to repeat, as long as the file hasn't changed. Writing may not be. If the connection drops after a write, repeating it could make a second change, or fail because the file has already changed. The model shouldn't have to guess whether the first attempt happened.

A good design can give each operation an ID that the agent can look up later to check its status, or check which version of the file it's working on before retrying. That's the system's rules supporting recovery. It doesn't rely on trusting the model's guess about what probably happened.

Standard protocols can help organise all this. The Model Context Protocol, a shared standard for connecting tools to AI models, sets out how tools are named and described, how they're called, and how their results come back. That gives software a common way to find and use tools. It doesn't make sure the model picks the right one, that the result is useful, or that your business rules about who's allowed to do what are right.

Working with the protocol, getting the task done, and doing it safely are three different things. Passing a check on one doesn't tell you about the other two.

Now let's give both engineers the strongest version of their case.

The engineer who favours the provider's own runtime can say: if the provider trained and tested its model around a particular way of working, its runtime may keep that way of working with less integration effort from us. Forcing the model through a generic interface could add extra translation, extra errors and extra maintenance. We'd rather buy something that works than rebuild it badly.

That's a coherent position. It gets stronger if a realistic comparison shows a meaningful advantage, with matched budgets and real product requirements, and if maintaining your own version costs more than the flexibility is worth.

The engineer who wants to keep options open can answer: working well with a model doesn't mean your whole product has to adopt one provider's way of doing things. We can own a stable contract for the tasks, the permissions, the outputs and the logging, and put a thin adapter in front of each model. An adapter is a translation layer: it presents our tools to each model in whatever form suits that model. We change how the job is expressed without changing what the job is.

For the parser, the fixed job is something like: change only the file you're allowed to change, at the version you expect, keep everything else working, and hand back the changed file plus the evidence that it was checked. One adapter offers a patch tool. Another offers exact replacement. Both have to stay inside the same boundary, even though what each model sees is different.

That isn't identical behaviour across models. It's the same product meaning, delivered in different ways. It isn't free, either. Each adapter needs its own tests for matching, errors, results and recovery. A supposedly clean abstraction can leak hidden assumptions, and keeping up with every provider's changes can turn into a real job.

So the key question becomes: which differences should the platform smooth over, and which should it keep? If squashing a provider's useful features into generic text loses something important, the abstraction may be too narrow. If the provider's own runtime can't give you the evidence or the permission controls your product needs, adopting it has a different cost.

Today we've strengthened both positions and narrowed the gap between them. Familiarity with tools can justify tailoring the integration to each model. That on its own doesn't decide who should own the whole harness. And a broadly capable model may be less sensitive to the interface, without that meaning every task and every tool has stopped mattering.

Here's a small experiment that lets you see the interface's contribution without building a full agent.

Prepare one short file containing the parser and its helper, both starting with the same opening line. Write down the exact check you want added, and a separate way to confirm the helper hasn't changed. Start from the same file every time.

In one version, ask a single model to express the change as an exact replacement, using a piece of text that only matches in one place. In the other, ask it to express the same change as a patch with surrounding lines. Give it complete, accurate rules for the tool each time. Use a fresh session for each run, the same task, and similar settings.

Do two runs of each format if you have time. Apply the results with a tool whose behaviour you've written down, or check them by hand and note that you did. For each run, record three things: whether the request is valid, whether it points at the right function, and whether the final file passes your separate check.

Four runs won't tell you which format is better in general. They can reveal one specific failure worth looking into. If one format keeps losing its place, look at how the action is expressed. If a clearer error lets the model recover, the results the tool sends back matter. If both formats succeed, you've learned that this easy task didn't show a difference, not that interfaces never matter.

Be clear about what you kept the same and what you didn't. A person checking the patch by hand is a form of help. A different error message changes the feedback as well as the format. A different limit on the number of attempts changes how much room there is to recover. Those aren't small footnotes if you're trying to explain the result.

For a harder version, put both tools on top of the same underlying code, and let the agent recover from one deliberate ambiguity. Then compare the full trajectories. Keep that separate from the simple experiment, so you know which question each one answers.

Now try the mastery check without your notes. The same model fixes the parser inside a provider's own agent, but fails through your generic editing tool. Give at least four explanations that fit what you saw. Propose tests that would tell apart a problem with how the action is expressed, a mismatch in what the executor does, missing feedback, and a genuine limit on solving the task. Then say what more evidence you'd need before choosing the provider's runtime as your architecture.

A strong answer won't jump straight to "the model was trained on that tool". It will look at the actual request and what came back, compare the budgets and the starting context, check exactly what each tool does to the file, and test whether giving some help moves the failure. It will also ask whether the gap matters on realistic tasks, not just one convenient example.

Last, here's where we stand on the evidence. What we know: two tools can describe the same change while needing different valid requests, and an error message can hide or reveal what the model needs to recover. Both follow from the rules in our made-up example. What's been reported: research shows particular ways of training or prompting models to use tools, and system builders describe tailoring integrations and improving tool design in their own systems. What we're inferring: familiarity with a way of working could be one reason performance depends on the interface. What we don't know from a score alone: what a private model was trained on, whether familiarity caused the gap, or whether the gap is worth the cost of owning another adapter. What would change our minds: careful comparisons showing matched interfaces perform the same on the tasks and limits we care about would weaken the case for tailoring. Gaps that repeat, matter to the decision, and survive checks on the code, the context and the budget would strengthen it. Neither would be a final verdict on all future models.

Here's the picture to keep. The interface decides what actions the model can express and what it hears back. The model chooses what to do within that. The executor decides what actually happens. Performance depends on how well those pieces fit together, and on how much room there is to recover when they don't.

Next episode, we widen the lens. Editing is only one part of the harness. We'll look at context, memory, state, permissions, retries and knowing when to stop, and ask which of those belong in a stable platform and which should stay specific to each model. That's where a model-agnostic design becomes something concrete, rather than a promise that every model will behave the same.