Subscribe
Share
Share
Embed
Train custom decision models on stored decisions, verified corrections, and outcomes. Luis and Carina explore AnyJev, Jev features, and the tests a memory-learning system would need to pass.
Contextually Aware · Luis and Carina
An assistant remembers a temporary demo instruction and accidentally replaces the production decision. What would help its next version handle a different exception correctly?
Luis and Carina examine a proposed learning loop built around stored decisions, verified corrections, and later outcomes. The episode starts with Luis's documented rule that project facts belong in memory and reusable lessons belong in versioned skills. It then explores a small learned decision layer for deciding whether instructions apply permanently, temporarily, or need clarification.
The database scenario is hypothetical. The combined system is a proposed design: no model training, benchmark reproduction, or deployment result is claimed.
Listener question: Which mistake have you corrected twice, and what evidence would distinguish a retrieval failure from a judgment failure?
These sources were checked in the accompanying research through September 26, 2026. This adaptation does not add a new research pass or claim independent reproduction.
Scripted with AI assistance and performed using Luis’s approved synthetic voice and the Carina synthetic co-host voice. This is not a recording of a live conversation.
What product managers can actually build with AI today—and where it still breaks.
LUIS: You tell your assistant, “For this demo only, use an in-memory database.” Next week, you ask about production deployment. It recommends the demo database. You correct it. Again. The assistant remembered your words and lost the part that mattered: “for this demo only.”
CARINA: That is a hypothetical example, but the failure is easy to picture. The assistant has more history, and somehow the next answer gets worse. What would it take for that correction to improve the next decision, instead of becoming another note?
LUIS: I'm Luis Calderon. Welcome to Contextually Aware. I have an operating rule for my agents: project facts belong in memory; reusable lessons belong in versioned skills. Today we're exploring a learned decision layer around that rule: memory that learns from memory.
CARINA: And I'm Carina. We'll follow that database example through the whole loop. We'll look at what gets saved, what could be trained, and how to tell whether anything improved. This is a proposed build, with research behind it, rather than a system we're claiming to have benchmarked.
LUIS: Start with the existing record: “We chose PostgreSQL for production.” The demo instruction should sit alongside it with a limited scope. Replacing the production choice damages useful memory. Saving both without scope leaves the next assistant to guess which one applies.
CARINA: Before we train a model to recognize that distinction, I'd want to see what it actually received. Did retrieval supply the production decision? Did extraction preserve “for this demo only”? We could be blaming judgment for evidence that disappeared earlier.
LUIS: Exactly. Keep the source message, retrieved records, proposed action, and verified correction together. If retrieval missed the record, investigate retrieval. If the decision component saw complete evidence and still treated the exception as permanent, then we have a different problem to work on.
CARINA: And if the deadline simply changed, update the deadline. That doesn't require the assistant to become a better learner. Your distinction gives each failure somewhere to go: a record update, a procedural repair, or an experiment with the decision model.
LUIS: For the first model experiment, I'd ask one narrow question: does this instruction apply to ongoing work, the current task, or is its scope unclear? That gives us specific labels and a mistake we can recognize. “Make memory smarter” doesn't give us either.
CARINA: Where does Jev fit? People hear “custom Jev” and could reasonably assume we're talking about fine-tuning the provider's model. The documentation makes a distinction there, and it changes what the builder would actually own and maintain.
LUIS: TypeSafe says Jev uses shared weights across customers. You customize the state, question, and criteria; you aren't getting a customer-specific fine-tune. I'd use that as a baseline. A clearer scope question may solve enough of the problem before we take on a training pipeline.
CARINA: There's also a way to learn downstream. TypeSafe's feature-discovery example uses Jev's answers as numeric inputs to CatBoost, a separate model. That example predicts wine scores. Applying the approach to memory would be our experiment, and we'd need our own evidence.
LUIS: I would try questions about source support, duration, and conflict. Their answers become features for a small model choosing save, update, skip, or review. We train that decision layer on checked examples. Jev stays fixed, while our questions and downstream model can change.
CARINA: AnyJev offers another route. Its documented levels separate changing the readout, calibrating with labels, and fitting a small decision head. At the fitted-head level, the base model's weights remain unchanged. The new head is tied to a particular question and model.
LUIS: Its observe interface can collect labeled feedback and refit a question-specific head. That supplies a useful mechanism. We still have to establish that a correction is trustworthy, separate development from evaluation, and decide whether the new version deserves to control memory writes.
CARINA: So the assistant's own confident explanation cannot automatically become the correct label. Neither can every thumbs-down. A bad answer might come from missing context, a changed preference, or an ambiguous request. Training on the wrong diagnosis would teach the wrong lesson.
LUIS: I'd review those cases and preserve uncertainty. Training examples should include quoted instructions, hypothetical preferences, and temporary exceptions. “My colleague likes short answers” should not become my preference. Those awkward cases help define what the question actually means.
CARINA: Then comes the test everyone wants to skip: fresh conversations. Keep entire conversations out of development, so nearby messages don't leak into both sides. Repeating the corrected database example can show a repair. It gives weak evidence that the judgment transfers.
LUIS: Try a temporary price, a holiday schedule, or a one-off reporting format. Can the candidate preserve the lasting rule while honoring the exception? Also test ordinary lasting changes. A system that calls everything temporary has avoided one failure by creating another.
CARINA: Measure wrong replacements, missed lasting instructions, and how often the system sends work to a person. Review is useful, but referring every difficult case can make a model look cautious while the operator does all the thinking.
LUIS: The recent papers help separate possible jobs. Jev-Mem puts a fast controller in charge of memory operations while a larger model answers. That supports the division of work. It doesn't establish that the controller continually retrains from a user's corrections.
CARINA: JitMem trains a curator to select useful context when the next task is known, with the executor frozen. Interactive Memory Learning studies storage and retrieval policies using delayed feedback. That feedback comes from model judges, so its quality needs scrutiny too.
LUIS: My design takeaway is to keep original evidence alongside compact memories. Later tasks may reveal what the first summary omitted. But those papers evaluate their own systems. Combining ideas from them doesn't give our proposed assistant their reported results.
CARINA: What about faster, cheaper, and more powerful than frontier models? We'd need a named comparison on the same workload. Include extraction, retrieval, retries, human review, hosting, and training. A cheap decision call inside a costly workflow may barely change the bill.
LUIS: I'd run the candidate beside the current system with writes disabled. Review disagreements, evaluate fresh cases, then release narrowly if the chosen outcomes improve without unacceptable regressions. Keep the previous version available. Learning needs a way back when an update makes things worse.
CARINA: The listener question is practical: which mistake have you corrected twice, and what evidence would distinguish a retrieval failure from a judgment failure? I'm Carina. Thanks for joining us on Contextually Aware.
LUIS: And I'm Luis Calderon. My first milestone is one verified correction helping the next version handle unfamiliar exceptions while preserving permanent decisions. That's the experiment. The sources and proposed steps are in the show notes. Thanks for listening, and we'll see you next time.