This is the worst AI tutor we'll ever have | DataCamp

DataCamp co-founder and CEO Jonathan Cornelissen and Chief AI Officer Yusuf Saber explain how their AI tutor tests teaching behaviors, handles latency, and balances learning value against cost per hour.

Show Notes

DataCamp tested an early version of its AI tutor against its classic courses, and the tutor won: willingness to pay, engagement and satisfaction were all higher. It was also easily ten to twenty times more expensive per hour of learning than DataCamp's previous setup. Financially, Jonathan says, shipping the better product would have made the business worse.

He expects that cost to fall until it looks like the cost of serving DataCamp's interactive courses. The only question, he says, is whether that takes four to eight weeks or another year.

He's building the tutor on that bet.

Jonathan Cornelissen is co-founder and CEO of DataCamp and is joined by Yusuf Saber, DataCamp's Chief AI Officer and the founder of Optima, the learning company DataCamp acquired. They walk Conor through how the tutor works: teaching behaviors it has to show on cue, evals that test each one repeatedly, a mix of models picked for speed and cost, and a latency budget measured in milliseconds.

We cover:
- How DataCamp turns what great human teachers do into about 30 core behaviors, and how the tutor gives feedback: one main issue at a time, hints phrased as questions, and asking before it reveals a solution
- The A/B test against DataCamp Classic, and why Jonathan calls today's tutor the worst one DataCamp will ever have
- The models behind the tutor: Claude Sonnet handles about 70% of the work, Gemini Flash used to handle much of the fast work, and a cheaper tier is starting to take on more
- Why open models keep failing on speed, hosting and caching, even after one outperformed DataCamp's best model at the time
- Going from almost nothing to a run rate of over a million hours of learning a year, and why the same three-hour course can take one learner half an hour and another six hours
- Pre-generating responses while the learner thinks, and why a stronger model lets them aim for cache hit rates in the mid-90s
- What Bloom's tutoring research says, why doing better on a test isn't proof of learning, and what happens to junior roles when AI takes the grunt work

Connect with Jonathan Cornelissen:
LinkedIn: https://www.linkedin.com/in/jonathan-cornelissen/
Twitter/X: https://x.com/CornelissenJo
DataCamp: https://www.datacamp.com
DataCamp on X: https://x.com/datacamp

Connect with Yusuf Saber:
LinkedIn: https://www.linkedin.com/in/yusuf-saber-ba380b15/

Connect with Chain of Thought host Conor Bronsdon:
Newsletter: https://newsletter.chainofthought.show/
Twitter/X: https://x.com/ConorBronsdon
LinkedIn: https://www.linkedin.com/in/conorbronsdon/
YouTube: https://www.youtube.com/@ConorBronsdon

More episodes: https://chainofthought.show

Chapters:
(0:00) Better product, worse economics
(0:59) How the tutor helps a struggling learner
(7:09) From static courses to AI tutoring
(9:25) Multi-agent architecture behind the tutor
(13:10) Scaling costs and testing open models
(18:30) Economics of cost per learning hour
(22:13) What great looks like for DataCamp
(25:33) Latency budgets and precomputed lesson plans
(32:53) Measuring learning outcomes with Bloom's research
(39:20) AI skills gap for early-career workers
(43:22) Evals built on tutor behavior expectations
(50:12) Closing thoughts on how skills are changing

Thanks to CoralBricks for sponsoring this episode. CoralBricks serves open models through one API, fast and at low cost: DeepSeek V4.1 Flash streams at over 300 tokens a second. Get $5 in credits on signup, plus another $5 on your first purchase: https://www.coralbricks.ai/r/Z6V2RE

Thanks to Walrus, presenting sponsor of season four of Chain of Thought. Walrus Memory gives AI agents portable, verifiable memory that carries context across apps, sessions, and other agents. Get started at https://walrus.xyz/cot

Thanks to Inngest, presenting sponsor of season four of Chain of Thought. Agents in production run long - they call models and wait on APIs and people. But the longer agents run, the more they break. Inngest handles that with durable execution. You build your agent as steps in TypeScript, Python, or Go. When a step fails, Inngest retries it with exponential backoff, and completed steps are saved and skipped. Try it out: https://inngest.link/cot-pod

Thanks to WisdomAI for sponsoring this episode. In WisdomAI's latest benchmark, Claude Code alone topped out at 73% accuracy; WisdomAI closed the gap, delivering accurate answers 3x faster at about a third of the cost. Get "The Blueprint for Trusted AI Analytics": https://wisdom.ai/benchmark

Creators and Guests

Host

Conor Bronsdon

Creator and Host of the Chain of Thought Podcast

Guest

Jonathan Cornelissen

Co-founder and CEO of DataCamp

Guest

Yusuf Saber

Chief AI Officer at DataCamp; founder of Optima, the learning company DataCamp acquired

What is Chain of Thought | AI Agents, Infrastructure & Engineering?

AI is reshaping infrastructure, strategy, and entire industries. Chain of Thought is the podcast where builders reason through what's changing. Host Conor Bronsdon sits down with the engineers and founders shipping AI in production to get past the hype into what's working and what isn't. Episodes cover model infrastructure, inference, agent frameworks, evaluation, and developer tools.

Guests have come from NVIDIA, Google DeepMind, AMD, Databricks, Vercel, and more. Every episode carries a full transcript and show notes at chainofthought.show. New episodes weekly.

Conor Bronsdon is an independent consultant and angel investor in AI infrastructure and developer tools. He led technical ecosystem at Modular, acquired by Qualcomm in 2026; led developer awareness at Galileo, acquired by Cisco; and ran developer marketing at LinearB, where he was GM of the Dev Interrupted podcast and community.

Views expressed by the host and guests are their own.

[0:00] Jonathan Cornelissen:
Easily ten to twenty times higher than our previous kind of cost. We would have made the business worse. The cost will go down significantly, and it's a huge area of focus as well.

[0:11] Yusuf Saber:
The tutor is essentially a system of several agents that collaborate. But for us, we have to respond in milliseconds.

[0:23] Conor Bronsdon:
Welcome to Chain of Thought. I am your host, Conor Bronsdon. For more conversations with the people building AI systems, such as Jonathan Cornelissen, DataCamp's co-founder and CEO, and Yusuf Saber, their chief AI officer and the founder of Optima, the learning company that DataCamp recently acquired. Um, they have been working through how to make personalized tutoring accessible to people around the world and exceedingly useful. And I wanted to get into the decisions behind how they're building DataCamp, what they're doing with tutoring through AI, and much more.

[0:59] Jonathan Cornelissen:
Oh.

[1:00] Conor Bronsdon:
Let's set some groundwork before we dive too deep into the conversation. Yusuf, when a learner is going through on DataCamp and something goes wrong, uh, let's say they fail a test and, uh, they're, they're having problems, um, what do they do when they pick up the DataCamp platform? And what's their experience as they start to interact with it in a way where they need to improve?

[1:28] Yusuf Saber:
Very, very good place to start. So if we, uh, begin, uh, a tiny step, uh, before that with, you think of the, what the tutor is trying to do versus, uh, what we call static content, so lessons or videos that have been pre-generated. Uh, we essentially try to solve two problems. The first one is, uh, we are very different. We have different interests, we have different goals. So even if we are learning the exact same material, that material will make a lot more sense if it's presented as a solution to our very different problems.

[2:02] Yusuf Saber:
Uh, that's one. And the second is we have different backgrounds, we have different gaps in knowledge. We are sometimes need different amounts of time to understand the same concepts. So the tutor can adapt the flow of the lesson, uh, what gets covered, what gets skipped, based on what the person knows. And to make that work, it can't just have a lesson or a video that was pre-created. It kind of creates that as it goes. from a well-defined set of inputs, so that it's not making up the information.

[2:35] Yusuf Saber:
Every fact in the lesson, every asset. Kind of like a good teacher would then let the students sit down as he tries or she tries to create slides. They come in with their figures pre-created. They've really thought through their material, but they adapt the delivery. That's exactly what, uh, what the tutor does. So at a high level, we, we think of a great teacher as someone who exhibits a set of behaviors. Uh, those are behaviors that come together to make someone an excellent teacher. And one of those behaviors is they're very good at giving feedback.

[3:10] Yusuf Saber:
So they're thoughtful about how they go about giving feedback to their learners on whatever it is they're working on. So this is, we, the way the tutor works is we take those behaviors and we try to recreate them in AI form. So this is an example, uh, uh, and it's the behavior that, uh, guides the AI tutor on how to deliver feedback to a learner on whatever they're working on. Yeah, very high level. First it would, uh, when the learner submits something, it would check, were the criteria for success met?

[3:39] Yusuf Saber:
Uh, if that's true, then it will celebrate their success. And that celebration is kind of proportional to the amount of struggle and effort that they put into the work, and it's focused on that as opposed to, like, their innate talent or, and then it will focus on recapping what was learned from that struggle. So sometimes we find learners who just try through trial and error, get to solve a problem, but they really don't pause and reflect, what did I learn out of this? So it ensures whatever learning was captured and, and clear.

[4:10] Yusuf Saber:
But what if the criteria are not met? Uh, so that we'll see, did they actually finish solving the problem? Or they are still halfway through it and they just wanna get some feedback. They're very different things, and in prior to AI, you'll submit and say it's wrong. It's not wrong. I'm still working through it. I just need some initial feedback on the direction. So it will figure out, okay, this is the current state, they're still starting, they want feedback on. So it will appreciate the progress and let them know that they're on track or off.

[4:37] Yusuf Saber:
But it will not criticize them for not having a complete solution yet. Then, um, if the solution is actually, like, there's actually an actual problem, which is the case you asked about, then first of all it will check, does it understand where that is coming from? Is it, like, a misconception, a typo mistake they made while they're just writing quickly? If it believes that the source of this error is a misunderstanding, yeah, it will clarify that and, uh, ask them a question, uh, instead of giving them the actual solution.

[5:09] Yusuf Saber:
But if it doesn't know what's the cause is, like, it's, it's not about, like, I know what the problem is, but where they're coming from, it'll actually ask them, walk me through how you got here, just to try and understand where the error is coming from. With one set most errors, uh, usually they're not one problem but multiple. It will focus on one. We found through our experiments, when we give the learner, hey, your solution is wrong for these six different things, they break. Like, most learners, they just don't continue.

[5:37] Yusuf Saber:
It's just too overwhelming. So we give them the main issue that will get them to move one step forward, and then, uh, the second set of issues are shared. And the process is if on one particular issue that they're kind of trying to work through, there will be a series of hints, uh, mm, phrased mostly as questions and increasingly more informative. So the first ones will try not to give away any detail, to let them figure out on their own. But eventually it's clear that they can't move forward.

[6:06] Yusuf Saber:
So in the past we used to share the solution, but then we stopped doing that. We actually ask them first. We give them that agency. They make the decision that they want to see the solution. We have found not, not many, but quite a f, uh, sizable, uh, population of learners who would actually want to try 20 times, 30 times until they figure it out. And alone has deprived them of that. So the teacher will ask, it seems we are stuck, uh, on this problem, or we're not, uh, there's something that we, we need to, uh, clarify together.

[6:35] Yusuf Saber:
Would you like to see the solution and then we discuss it? So if the learner says yes, then it will share the solution. If they say no, then it will respect that and let them, uh, keep trying. So that's high level how that behavior unfolds.

[6:48] Conor Bronsdon:
(Sponsor read: Sponsor thank-you) Before we dive in, I want to say a quick thank-you to our presenting sponsors for Chain of Thought, Inngest for long-running agents, and Walrus Memory, providing portable agent memory. Plus, this episode's other sponsors, CoralBricks.ai, providing lightning-fast open model inference, and WisdomAI, the enterprise agent analytics platform.

[7:08] Conor Bronsdon:
Jonathan, I'm curious from your perspective, how have you seen user behavior shift on the platform over time as you've implemented the AI tutor and expanded this program?

[7:18] Jonathan Cornelissen:
Very, very good question. So, so when we kind of embarked on this journey, um, obviously there was a lot of excitement, 'cause, 'cause our vision was always, yeah, we want to make sure people learn by doing, and we ultimately want to get the experience to be as close as possible to an actual tutor. Should we flip the switch? Um, so we actually started by doing, like, a clean A/B test where kind of we, we did a bake-off between the, the AI tutor experience and what we call the DataCamp Original or the DataCamp Classic experience.

[7:51] Jonathan Cornelissen:
And what was very exciting to see is that even this early version of the AI tutor, the version that exists at six to nine months ago, it already performed better on several important metrics. We have a freemium model on the consumer side, and, and so, uh, ultimately the, the, the willingness to pay is, is a key, uh, indicator of how well people are, are kind of perceiving the experience. And so willingness to pay was immediately higher, engagements was higher, um, satisfaction was higher, uh, compared to the DataCamp Classic experience.

[8:28] Jonathan Cornelissen:
So overall, it was, it was kind of a very clear win. Um, even though I feel like we're at the very beginning of building these amazing AI tutor experiences. And based on all the data we're constantly gathering, based on improvements in models, improvements in different parts of the technology, I think the, this is kind of the worst tutor that we'll, we'll ever have. Um, and, and so I'm, I'm very bullish on, like, the fact that we're at the start of this S-curve. Um, and, and, and that there's a lot more to come. Um.

[9:03] Conor Bronsdon:
I mean, I'll say it makes me personally excited for the next time I am actively seeking to take a course and learn something, because it's so clear that the technology around it is changing, and if you can access this type of tutoring, as you put it, Joe, there is such an opportunity to learn better. Yusuf, right now, is this all running off of a frontier model, or what's the approach actually underpinning, uh, these decisions for the tutor?

[9:34] Yusuf Saber:
The tutor is essentially a system of several agents that collaborate. So the, though, kind of the experience that's split out to the learner is this coherent single entity, but it's actually a set of entities that collaborate to deliver that experience. And there is at least four different models that underpin those different entities. Uh, so the core, uh, tier that does about seventy percent of the work, it's a frontier model. It's, uh, it's a Sonnet. Uh, and, uh, we just upgraded to five point five, uh, earlier today.

[10:09] Yusuf Saber:
Uh, and it was, yeah, like a good step up, not, not a, not a big step, but, but a, a decent step in, uh, outperformed over our evaluation. So, so far we have found Sonnet, uh, to be like the sweet spot for the tutor. So smaller models, they just, you saw one example of a behavior. There at least, uh, thirty core behaviors and another 50 secondary behaviors, like a massive, uh, amount of instruction that the model kind of needs to stay coherent about. And with a smaller model, we found that it just loses that coherence.

[10:46] Yusuf Saber:
This is a very long time horizon task. Like, the tutor will run a lesson that can go anywhere from 15 minutes to an hour, and it has to stay coherent all through that. We do a lot of work to keep the context within limit. But it also has to remember, like, if someone is submitting a solution, it has to say, oh, you're trying the same thing we tried earlier, why don't we think of something different? So if it doesn't remember what was tried earlier, it just can't show that coherence across the lesson, even across multiple lessons.

[11:13] Yusuf Saber:
So the world needs to be able to handle, like, a large volume. But at the same time, it needs to be fast, and, and, uh, the cost needs to be reasonable. So Sonnet has always been the tier that is just right for this application. But recently, we started to see the lower tier, I think the Haiku slash lunatier, becoming capable enough that we can actually use it more effectively for more substantial, uh, portion of the workload. Uh, before we used to use models like Gemini Flash.

[11:42] Yusuf Saber:
Uh, for, for quite a bit of the work. Uh, it's exceptionally fast, uh, mostly because we're hosted where we are hosted, so there, there's just almost no latency. So for applications that require very high speed, usually simpler cases, uh, we have relied on, on that. Maybe there's about five to seven percent of the work. But the exciting thing for us now is this, uh, what used to be the lowest tier of models is kind of stepping up that it can actually do some of the heavy lifting that was just out of reach for them in the past.

[12:14] Conor Bronsdon:
(Sponsor read: CoralBricks) This episode is also brought to you by CoralBricks. CoralBricks serves open models through one API at the lowest prices and the fastest speeds. DeepSeek V4.1 Flash streams at over 300 tokens a second. Plus, CoralBricks also just put out MotionBench, a benchmark that asks a simple question: can a model animate? I've personally contributed to the open-source MotionBench, and I was impressed that DeepSeek V4.1 Flash managed a close third place behind a leading Claude Opus 5.5 at about an eighth the cost per task.

[12:48] Conor Bronsdon:
(Sponsor read: CoralBricks) The animation for this ad came out of that benchmark. We used Claude Opus 5.5 and DeepSeek V4.1 Flash via CoralBricks to create and edit it. Get lightning-fast open model inference at chainofthought.show slash CoralBricks with a free $5 in credits on signup, plus an additional $5 credit on first purchase.

[13:09] Conor Bronsdon:
I assume scaling this has been a challenge given inference costs and the architecture involved. Jonathan, how have you been approaching the scaling challenges here?

[13:21] Jonathan Cornelissen:
Yeah, so. So, so maybe as additional context on the previous point, uh, like, cost is a huge consideration. Uh, but one of the limiting factors is speed as well. Like, people, like, really good teachers, mm, if th, if they take five to ten seconds to answer people, people don't care. They will be gone. So, like, the speed really matters and, and puts a whole constraint. But then from a business perspective, um, cost really matters as well. For context, we, we had over 10 million hours of learning on the platform last year.

[13:52] Yusuf Saber:
Hippie. B?

[13:54] Jonathan Cornelissen:
Um, we've scaled to tutor from kind of almost nothing nine months ago to, now we have a run rate of over a million hours, uh, of learning per, per year. Um, so it's, it's grown quite, quite rapidly. Um, and the cost is, is, initially was measured in many dollars per hour of learning. Um, and so it's a real business challenge. Uh, 'cause that's easily ten to twenty times higher than our previous kind of cost setup. Um, and so even though a lot of our customers, a lot of our clients really like the experience.

[14:32] Jonathan Cornelissen:
It was essentially too expensive to scale. Um, and so that's been a huge consideration, um, in terms of how we approach this. It's also one of the reasons we, yeah, believe in open source and, and open source's ability to, to get costs down. Um, but it's good to see even at the frontier labs, like, there's more competition now. Um, so there is, there is a more price pressure, uh, that we can take advantage of as well.

[15:01] Conor Bronsdon:
Yeah, I'm curious if you've tried, like, DeepSeek V1 Flash or any of the newer open source frontier models and how that has impacted decision making, uh, here.

[15:12] Jonathan Cornelissen:
Yusuf, you w, you wanted to speak to that.

[15:14] Yusuf Saber:
Yeah, happy to. So we, we, uh, have this, uh, ongoing experimentation stream where whenever an open model kind of comes out, we immediately run it through the evaluations. Uh, we have them running, uh, on, like, an internal system where we can choose any of the models, run, uh, lessons with it for, for, like, mm-hmm, like, to get a kind of quick feel for how it's performing, where it's failing. So both test, like, uh, evaluations, which are, uh, quite accurate, but that feel for, is, like, there's this spirit of the model, which varies quite a bit and can't be captured easily in evaluations.

[15:55] Yusuf Saber:
So we run them through both. And what we found is, Lee, usually many of the open models are disqualified, like, uh, ethics. I just did an arrival. Like, right off the bat, it's at first, it's usually speed. So, quite a few of the models, even running on proper hardware, uh, they're, they're very slow. So they take quite a bit of time to respond. And what we found is learners, if, 'kay. For a lot of agentic applications, you can take minutes to work on a problem. But for us, we have to respond in.

[16:26] Yusuf Saber:
Milliseconds. So speed is critical, otherwise learners just lose interest and their attention drifts. Uh, so that millisecond kind of time window makes it, uh, quite a challenge for many of the open models. The second, uh, feature that we've, we've found is getting proper hosting, uh, is, is not easy. So to, you could, you know, for experiments or, or for some applications you could run quite a few of them. But getting hosting that meets all our criteria, uh, is, is often, uh, a bit tricky. Even with Google's own models, getting them hosted properly on GCP, sometimes there isn't enough.

[17:06] Yusuf Saber:
Uh, supply of, uh, GPUs that are available. Sometimes it ends up being more expensive, like once we get the, the hosting set up right. Like, because, uh, to, to get the same throughput, uh, it ends up being, uh, a bit more expensive. And you need a supplier who has set it up, has built, like, a really robust, uh, caching architecture. Like, what you get from the major labs, uh, you get this incredible caching architecture, which allows you to run, like, frontier models at, like, a ninety eight percent discount.

[17:38] Yusuf Saber:
Uh, which is not as developed, I'd say, for, for some of the open model options out there. But I think, like, every month we, we are closer. Like, very recently, just a few weeks back, uh, actually a bit lo, a bit more than a few weeks back, when the gemfor, uh, Flash of model was generated. Actually one, one of those models did really, really well. Actually, uh, outperformed, uh, our best performer at the time. Uh, but most of the challenges to getting it run at scale.

[18:10] Yusuf Saber:
Were just getting reliable hosting. Most of the options we found just didn't have enough capacity for what we were asking for, or had enough capacity, but latency was a serious issue, or didn't have the right caching infrastructure, so we end up not saving, uh, significantly over, uh, what we are, oh, paying at present.

[18:30] Conor Bronsdon:
Interesting. Um, just, uh, talk to me a bit more about the economics here. If you are seeing great savings from some of the open models, but you're also having challenges reaching the goals you need, is that more about optimizing for, you know, l per learning hour? Is it about how to get learners to mastery? What's kind of the unit here you're looking at?

[18:57] Jonathan Cornelissen:
I think from a business perspective, it's actually quite simple. From a business perspective, what I care about is kind of the cost per hour of learning. Um, an hour of learning is generally a proxy for value delivered, uh, from, from a customer's point of view. Um, and, and that costs, I think, eventually will be similar to kind of our kind of server costs to run our interactive learning experiences. That trend seems clear. The only question is, do we get there in four to eight weeks, or, uh, does it take us another year?

[19:36] Jonathan Cornelissen:
Um, but the trend is, is very clear, and that's also the bet we're making. Um, at the current cost, our, we don't have negative margins, but our margins are not super interesting, I would say, on kind of the AI tutor experience. Even though we deliver a better product to our customers, from a kind of financial point of view, we would have made the business worse. Um, but I, but I'm very confident based on everything I'm seeing, mm, that the cost will go down significantly, and it's a huge area of focus as well.

[20:09] Jonathan Cornelissen:
Um, and I think eventually will be and become kind of a competitive advantage as well. If you can have a really excellent tutor, um, but you can deliver that at a lower cost per hour of learning, th, that will be kind of an important competitive dynamic in, in my minds. Um, okay. But Conor, y, y, you implicitly refer to another angle to look at this, is, like, well, is it true that one hour of learning is the right metric, um, to kind of think about value?

[20:42] Jonathan Cornelissen:
And this is actually an area where switching even from interactive learning experiences to the AI tutor gives you a very different result. Um, so, so for context, if you were to take a course in DataCamp, DataCamp Classic course, um, you would have a range. Maybe a three-hour course, some people would finish it in two hours, some people would finish it in four hours, but most people would be around three hours. Right. And DataCamp already had a range, where if you, if you think about a video course, like the classic Coursera type, type experience.

[21:15] Yusuf Saber:
Stern.

[21:16] Jonathan Cornelissen:
Three hour course is a three hour course. Now fast forward to working with a tutor, you actually get a much, much wider distribution. Because guess what? Some people actually can get through this material and can achieve the learning objectives in, uh, maybe in half an hour. And some people will take six hours. Uh, and it depends on many things, like how much background they have, maybe how much they know about similar topics, how fast they're kind of able to learn. Um, and so, this is kind of a new way of thinking about value delivery.

[21:51] Jonathan Cornelissen:
Because if I'm an organization, I want my best people to spend, to learn the skills they need to learn but to spend as little time as possible. Um, but at the same time, if I have a group of people that needs more time, I actually want them to have the skills. I don't want them just to take the three-hour course and then half of them still don't really know. Um, and they still don't have the skills.

[21:55] Yusuf Saber:
As.

[22:13] Conor Bronsdon:
Jonathan, as you think about the goals for DataCamp going forward and for building the best AI tutor that you can, what will great look like for you internally, and how is that going to align with the goals of businesses that are sending their team members through DataCamp?

[22:34] Jonathan Cornelissen:
Great question. So, so there's really two parts to the system we're, we're building and the system we're envisioning. There's the AI tutor itself, which we feel is already in, in a very good place and, and, and is kind of leading in the markets. Um, so there's internal improvements that make it a better teacher, a more effective teacher, a more efficient teacher. That will be hard to immediately see, but you'll kind of know when you go through it. Um, but then there's things like modalities. I'm, I'm very excited, there, there will be a time where there will be an audio-only version of the tutor.

[23:13] Jonathan Cornelissen:
Um, where you can go back and forth, and if you're, uh, on your commute on a mobile device, you can kind of have this private tutoring session, just kind of going back and forth with your tutor on the phone, which I think for a lot of people who are busy or who are on the road in commercial functions will be a great experience. But that's all the AI tutor system itself. Um, where I think this is going to be a real game changer, and this is a system that's much, much earlier stage, is where we're also developing technology to prepare the material for the AI tutor.

[23:53] Jonathan Cornelissen:
Um, we internally call it the AI Creator. Um, and that will eventually enable us to not just teach data and AI skills, but to teach any professional skill, and to have a system that ingests existing knowledge, could be their own internal company knowledge, could be, uh, a really good book, a really good YouTube video. Um, and then kind of transform that into something that the tutor can really teach. Um, and, and that's where I'm very excited, because once you have the whole system, um, you kind of have an omnipotent, a tutor that can kind of teach anything.

[24:31] Jonathan Cornelissen:
So that's what we're kind of working towards. Obviously it's gonna take many steps to get there. Um, but we're very excited about that, that kind of, uh, future that we're, or, building towards.

[24:43] Conor Bronsdon:
(Sponsor read: Walrus Memory) Season 4 of Chain of Thought is presented by Walrus Memory, the portable memory layer for AI agents. Your agent has learned your codebase and how you work. Now you want to use that context in another tool. Exporting a file gives you a snapshot, but what happens when the context changes? Walrus Memory is a portable memory layer that lets your agent store context for later use across tools. Python and TypeScript SDKs plus the native MCP support let you connect it to your agents, no matter where they are.

[25:12] Conor Bronsdon:
(Sponsor read: Walrus Memory) You said who can read and write, so sharing context doesn't mean opening up your whole memory store. If you're building across tools or model families, take a look at Walrus Memory. Learn more at walrus.xyz slash cot. That's walrus.xyz slash cot.

[25:32] Conor Bronsdon:
And I assume one of the challenges you face around this is latency. Yusuf, you took us through a bit of the architecture earlier. How do you think about latency challenges with the tutor, as you obviously have to work with learners around the world?

[25:48] Yusuf Saber:
So we found that latency is extremely important to success of the tutor and of the learners. Uh, so even if it delivers the correct response, uh, the fact that it needed several seconds to get to that response can often, like, really subtract from its value. So wh, what we try to do is we have this, mm-hmm, kind of set of limits on how long we can allow it to take in terms of time at every, no, kind of interaction. Some interactions, you know, it needs to be really quick.

[26:25] Yusuf Saber:
Some interactions, it's generally difficult problem that it has to solve. And a wrong solution to the problem can derail the lesson, so we can allow it to have slightly more time. Like, there's about six percent roughly of the responses during a typical lesson where we have to turn on thinking, so it needs to make a critical decision. Like, one of those critical decisions is, it's about to start teaching something, and then the learner says, says something, they ask a question or they share something that implies that they probably know this material.

[27:00] Yusuf Saber:
So it's a very important judgment call. Is this, uh, statement? Good human teacher would just quickly catch on this. You said that, then you probably know this. And then they wouldn't just walk them through it as if they're hearing it for the first time. That's a kind of a critical judgment call. If it gets it wrong, it can really take the lesson in a direction that would take several minutes to recover from. And at those moments, we kind of allow it more time to think before responding.

[27:28] Yusuf Saber:
Whereas in some other cases, it's just a continuation of an idea that's being delivered. There is no new question. There's, so those moments, there is a lot less time. The other thing is we have this, uh, massive asymmetry in how the system works. So once the learner responds or asks a question, they want an immediate reply, but then they take a very long time to think about that reply, to do work on an exercise, and then respond. So we try and use that downtime to do a lot of the preparation.

[27:55] Yusuf Saber:
If you look at a lot of the common, like, uh, engineering architectures, they rely on, like, tool calling. So you set up a bunch of tools and you allow the agent to kind of call tools. For tool calling, you immediately add a round trip. So it has to decide this is the right tool, it has to generate a response, wait for the tool to return, and then generate a second response. So for us, this is almost always off the table, but we carry something that is essentially the benefits of tool calling.

[28:22] Yusuf Saber:
But where most of the work happens during the downtime, where the learner is processing the previous response. But we have to kind of make several guesses. Is the learner likely to, uh, answer in the positive or the negative? Are they likely to get the response right or get it wrong? And because the, where the lesson goes from that point can be different. And we have to prepare, like, some material that doesn't get used because the lesson went a different way. A lot of the work we do is kind of optimize that prediction.

[28:52] Yusuf Saber:
So the chances that we throw away tokens we generated is, is very low. And through that, we can also make the response, like, really fast. Even audio, you see it almost starts speaking immediately because part of that audio has also been pre-generated during that downtime. So it runs. But we could end up not using that because the learner said something that took the lesson in a different direction. That's why we try to optimize that, to get latency really, really low, by kind of taking a bit of risk on where the lesson might go next.

[29:23] Conor Bronsdon:
Super interesting to think about this risk idea. Are you able to use caching across different learners to kind of reuse specific responses that maybe didn't get picked up by one learner? You can apply it at scale across other learners, or how do you think about avoiding some of this potential, like, attacks if, uh, alert goes a different dire, direction?

[29:45] Yusuf Saber:
Yeah. So think about, there, there is a sequence, or there's a plan that's pre-prepared for the flow of the lesson. And that plan, we keep optimizing it based on real lessons and how they actually unfold. Uh, so the plan, if, you know, that plan has been, you know, well thought, and it's been optimized through actual, uh, trajectories of how lessons were, were taken, then, uh, it's less likely that the learner would need to go off plan. And also usually when going offline is certain, is, it's more often we're skipping part of the plan as opposed to coming up with an entirely new plan, because the objectives of the lesson are, are fairly, uh, kind of preset, uh, in advance.

[30:23] Yusuf Saber:
Other part is, um, a lot of the time, what can be shared between lessons is a lot of, like, the infrastructure. So a lot of the learning plan itself across the different lessons. Uh, we can, like, you can do the same lesson. Mm-hmm. We're using, let's say, a lesson on, like, SQL, for instance, a data set about, like, you know, a podcasting platform, uh, with data that's like podcasters and shows and so on. And doing that in a learning platform, and all of the data I'm seeing is like courses and students.

[30:55] Yusuf Saber:
So that the, the example scenarios that are used to run the two different lessons are very different. The material is the same. Uh, so we try to have, like, a good, uh, of, uh, subpopulation of learners that, uh, is in each of those groups, so there is a variant of the lesson that's cached through some of the other learners. A lot of the, uh, power comes from, like, deciding where to put different parts of the prompt, so as much of it as possible is common across the largest possible base.

[31:28] Yusuf Saber:
So we spend a lot of time on that. How can we get to 90%, 94%, 95% cache hit rate, as opposed to, you know, something lower? We found that a better model allows us to take liberties with getting the caching to the high, mid to high nineties that the weaker model just can't handle. I got a lot of the tricks that we kind of do to help a weaker model involve, like, dynamically altering the session. Because the way it would unfold it is just too much for it to handle.

[32:00] Yusuf Saber:
So, like, we remove previous responses, we hide certain things, we summarize certain parts. But that means those parts are no longer cacheable, so the, the price could, could grow higher. Uh, for, for those reasons.

[32:12] Conor Bronsdon:
(Sponsor read: Inngest) Season 4 of Chain of Thought is presented by Inngest. Long-running agents need endurance. They must endure waiting for users and weather tool failures. They must take each new jump in model intelligence in stride. Inngest makes long-running agents durable, observable, and improvable over time. Check out their generous free tier and a sleek local dev server for testing your workflows, flows. Grab the Inngest link in the description, or visit inngest.link slash cot dash pod to get started.

[32:48] Conor Bronsdon:
(Sponsor read: Inngest) Build for the long run with Inngest.

[32:52] Conor Bronsdon:
And as you make these architectural decisions around latency, caching, reliability, obviously, you still have to make sure that this is actually serving your end user. Jonathan, what are you doing to test that a tutor is actually helping someone learn rather than simply producing an answer the learner may like?

[33:14] Jonathan Cornelissen:
I think the high level answer is this is actually quite well established, uh, in, in kind of scientific lit, literature. Um, even forty, fifty years ago you had this researcher called Bloom, um, who essentially kind of show that conventional classroom education versus one-on-one tutoring, um, yields very different results, uh, in terms of educational outcomes. Um, so you can move someone from kind of an average performance as a student, you can move them into the 95th percentile by kind of having one-on-one tutoring. And if you look at the real worlds.

[33:55] Jonathan Cornelissen:
This is a common thing, right? Like, if people can afford it, they will often get p, private tutors, uh, for their children. Um, and that's for, for very good reasons. Because if you think about what a, what a good tutor does is they adjust the pace. They kind of adjust all the examples to, uh, the, the kind of world that learner lives in. They will catch any misconceptions really fast. They will challenge them, but not too much. So there's so many things a really great tutor does.

[34:27] Jonathan Cornelissen:
That we essentially have encoded into different behaviors, and that we have a whole eval system to kind of check, are we, or is the tutor actually demonstrating these behaviors in a reliable way? Um, and that's a, a continuous improvement effort where, um, we monitor obviously all the interactions students have with the tutor, uh, and, and some of the friction points between a learner and the tutor. may then give us ideas around how can we further improve things. Um, now, Conor, all of that being said, like, this is not yet scientific proof that our tutor is better, which is obviously what we, what we want to get to. I was actually reviewing the scientific literature a little bit last night.

[35:14] Jonathan Cornelissen:
Um, and, um, so there is research that, uh, I think one in Nature, uh, uh, from last year, that already kind of researched AI tutors that have this kind of skeleton around that, uh, are demonstrated to be better, uh, than conventional kind of teaching approaches. Um, just the LLMs without the scaffolding? Uh, I think, I think even, there's, there's research that indicates that's, that's not necessarily, uh, helping, uh, learners progress their kind of understanding and their skill level. Uh, it may make them do better at the test, but that doesn't mean there's more learning.

[35:54] Jonathan Cornelissen:
Uh, that happens. Um, but, like, long story short, like, I can't wait for the day where we have a scientific collaboration and we can actually show you some hard data. For now, it's a collection of the existing scientific literature and then actually looking at the data, that I, g, I have a very high degree of confidence that it already is better. Um, and, and, and especially more effective. Um, where you only need 30 minutes to get people there. In some cases, you may need an extra hour.

[36:31] Jonathan Cornelissen:
But the end point that the, those two learners reach is much more similar versus if you put someone behind a laptop and say, watch this video for half an hour. You have no idea, like, were they even paying attention? Um, uh, do they ha, have they absorbed the skills? Have they not? Like, you have no idea. Uh, and unfortunately a lot of corporate learning, that, that's still what it is. Yeah. Yeah, but you have, mm-hmm, you say, oh, people watch these videos, great, we, we've got a check mark and, and we're done.

[37:02] Jonathan Cornelissen:
Um, and so, so I feel very confident that we're already beating that. Um, but there, there will be a lot of research on this topic in, in the next few years, and, and we definitely wanna, if, if anyone listening is, is on the, on the research or on the academic side, we're very open to collaborate and, and find ways to, um, to share data.

[37:26] Conor Bronsdon:
That's a great call out. I will say anyone who is listening to the show that wants to reach out, drop a comment or reach out to me, and I'm happy to put you in touch with Jonathan, Yusuf, 'cause they are doing some very interesting things here on the front of learning. And I think, especially as we shift our day to day for work to be one where so much of execution is happening via the AI model, where a chat window is our interface with a lot of work that is, is to be done.

[37:57] Conor Bronsdon:
We are going to have to rethink how learning happens and be very structured about it. Uh, because right now it is so reliant on individuals to keep their skills from atrophying as we move away from execution. Yeah, really interesting philosophical thing that we have talked a bit about in the show already, particularly in our recent episode with Laurie Voss, is the gap that has opened up for early career entrants, where it used to be that if I was joining a field, I would be given a lot of the grunt work in order to learn.

[38:32] Conor Bronsdon:
Because look, yeah, my, my leadership, my mentors within the industry didn't necessarily want to be taking on that grunt work themselves. They had done their time. But now they can hand it off to an AI model that is often doing it more efficiently, uh, that they can direct more easily and that can do things in a repeatable loop. And so these tasks that were previously set aside and often sent to interns or, you know, other junior professionals are being, mm-hmm, taken care of by agents in many cases.

[39:06] Conor Bronsdon:
Uh, it's some of the first work that gets picked up by good AI loops. And I think as an industry, and frankly, as a world, we have to be thinking about how do we stop there being this gulf between the people who are far enough along in their career and can leverage AI models to 10x themselves, as we put it, uh, versus those who are, are picking things up and having to build expertise in their skills and in their industries in order to have the leverage to execute these models.

[39:41] Conor Bronsdon:
And so, Jonathan, I'm curious from your perspective, as you think about where DataCamp plays a role there, and more broadly how we should be thinking about enabling juniors to enter industries like software engineering, et cetera.

[39:56] Jonathan Cornelissen:
Yeah, see, this is a really important, uh, uh, observation and really pertinent question. The, at, at the highest level, I would say it's the reason we exist, like, we're going through this really fast transition in, in skills. And the reason for existence for the whole company is we want to help people who are long humans. We want to help the humans kind of, okay, take advantage of, of all the new technologies that, that come about. Um, and it's, things have never changed as fast as, as they have today.

[40:30] Jonathan Cornelissen:
Um, if you look at the, uh, it's kind of market as a whole. Um, in the last six to nine months, mm, two things are true in parallel. So, on the one hand, you have this gap for juniors, which is very real. On the other hand, if you look at AI engineering roles, data engineering roles, they are growing faster than anything we've seen before. So there's actually an acceleration of certain types of roles where people have AI skills. Um, but, but, but that, that is true even though it's, it's harder for junior people to, to get a new job.

[41:09] Jonathan Cornelissen:
Um, and my personal opinion is I think it's a big mistake, and it's a kind of a change management challenge for a lot of larger companies. You're absolutely right. Historically, you would hire juniors. You would give them some of the grunt work, and, and that's how they would kind of understand the business and learn the business. And some of those tasks are also the tasks that are the easiest to kind of automate with AI. Uh, so, so you have a temporary gap there. But, but I think the right solution for organizations is to recognize that it's the young people that will adapt AI.

[41:46] Jonathan Cornelissen:
The fastest, that will understand how to leverage AI, um, better, who are, like, truly AI native. Um, and hiring those people, but just having them skip a few steps, will ultimately yield way better results, even though you have a short-term effect where some organizations might be tempted to say, hey, we don't need, uh, a lot of juniors. I, I, I believe and I hope that's a temporary kind of, uh, imbalance in the markets, uh, that will, will hopefully go away, uh, very soon.

[42:19] Conor Bronsdon:
(Sponsor read: WisdomAI) Any frontier model can query your data. But can you trust its answers? This episode's sponsors, WisdomAI, put that to the test in their latest benchmark study. Alone, Claude Code topped out at 73% accuracy. WisdomAI closed the gap, delivering accurate answers 3x faster at a third of the cost. I sat down with Kapil Chhabra, WisdomAI's co-founder and CPO, for episode 75, and he explained the analytics architecture that makes these kind of results possible. Check it out for a full deep dive, plus get the blueprint for trusted AI analytics at wisdom.ai slash benchmark so you can deliver accurate answers on Claude, ChatGPT, or wherever your business gets their insights. Download your copy at wisdom.ai slash benchmark.

[43:12] Conor Bronsdon:
So, as we hopefully solve this problem, and DataCamp I know is eager to be part of that solution, this still creates major challenges for the industry more broadly. And as you work through what the next stage of the tutor needs to look like to help solve this challenge, Yusuf, what are the changes you might make around architecture or model decisions? What are you bringing in-house on the eval side versus saying, oh, no, we're surely letting an external team handle that? I'd love to understand your thinking around architecture and what the future of the platform looks like.

[43:56] Yusuf Saber:
Do you want to get that? Let me start on, on the evals point, because this is absolutely critical. Uh, a lot of, uh, like, teaching or tutoring, even in, uh, with human teachers, it's, it's a very hard, yeah, situation to be in, uh, as an evaluator. Like, how can I say that this is actually working? The tutor is performing, uh, their duties in the ways that are expected. Uh, so we took this, uh, approach, which I think is, is essential for anything we build to work.

[44:30] Yusuf Saber:
And we've broken it into, like, two problems. The first question is, what should a great tutor do? And this is what guides us into how to design these behaviors. And then the second question is, is our tutor living up to those behaviors? And that's where the eval comes in. So, everything we discussed earlier, whether learners are, uh, actually benefiting from that learning, and proxies for that when it comes to, like, engagement, a willingness to pay, uh, a completion of learning material, or, or at all, is on the first question, which is, what are the right behaviors that a good teacher should exhibit?

[45:07] Yusuf Saber:
And then the second question is, now that we know them, is our tutor exhibiting them? So this is another example of a behavior. So we looked at one earlier that was, uh, about, our learner, you know, makes, submits their work for feedback. This is when the learner asks a question. So, this is one of the most common behaviors. So, a good teacher handles this moment really, really well. It's a moment where learners show curiosity, they're speaking up. And if that's not handled well, that could really hurt.

[45:37] Yusuf Saber:
This learners, not just with this lesson, but their, you know, their whole, mm-hmm, attitude towards learning could be impacted. Uh, so once the learner asks a question, the tutor would try to put it in one of four classes. So, the first one, this is a question about what you're teaching right now. This is, they are trying to understand it better, they're confused about something, they want clarification. If that is the case, then it will answer it in full. And it will check that it landed, and it will not move on to the next point until it makes sure.

[46:07] Yusuf Saber:
That this is clear. The learner is essentially helping the teacher do his job by telling it, I don't get this part. So it has to kind of cater to that fully. And the second is the learner is curious, and they're asking a question about something that is coming soon. That's one of the most common situations you have with a bright learner. They're slightly ahead and they're asking a question about something that's coming soon. So here the, we, we give them a short, concise answer. We appreciate the curiosity, give a short, concise answer, and then reassure them that this is coming at this moment.

[46:37] Yusuf Saber:
Usually that's, that's sufficient. Uh, we don't shut them down, but we also don't derail the flow of the lesson, because there are usually dependencies that need to be addressed. And third, it's beyond the course. It's, like, a very good question, but it's beyond. Again, a lot of focus on appreciating the curiosity and then pointing them to where, whether it's in our course catalog or elsewhere, where they can go and kind of fulfill that curiosity. And finally, something that's, like, totally off topic, like people, you know, oftentimes they just do it for fun, or they just.

[47:06] Yusuf Saber:
They asked something that has nothing to do with learning. And here the creator would, the tutor would not engage. So it would say here, uh, but just, uh, fake a laugh and then say, let's stay on track and continue learning, or it will say, hey, if you'd like to take a break, and then then you know when you're back. Kind of a behavior. Now, how do we ensure that the tutor is actually doing this in those situations? This is where the evaluation system comes in. Correct.

[47:33] Yusuf Saber:
So the, every behavior is spinned by a set of expectations. So the behavior we just saw, it kind of gives us a set of four expectations. So, answers fully when on topic. Checks that it landed. So these are, we expect it to do those things, so we can say that it actually exhibits this behavior. It stays brief when out of scope, or doesn't engage in an off-topic. So, for every behavior, we connect with a set of expectations. They can be anywhere from, like, you know, three to.

[48:01] Yusuf Saber:
You know, tens. And then each of those expectations we kind of evaluated through a set of tests. Now a test is an actual lesson that has unfolded up to this point. So a real lesson where you have the lesson started, the teaching is going on. And then we get to a point where the learner asks a question. And then we see how the tutor responds, uh, that's being evaluated. So for each expectation, there is a set anywhere from three to low tens of what we call tests.

[48:29] Yusuf Saber:
A test is this whole thing. Maybe it's a lesson from start to that very point. I just check that it produces the next response correctly. And we try to assess, uh, mm, we do is slightly different from this simplified version. So there are two questions. Can it do this, and can it do this reliably? So if we give it five situations. So, each test we run it multiple times. So, if we have 10 tests for one expectation, they can be run each five times. So, and then if it gets them consistently right, then you know this is.

[49:02] Yusuf Saber:
Uh, this, the, the tutor can reliably exhibit this behavior. If it gets them sometimes yes, sometimes no, then that means it's not as reliable, and we have to iterate on it. So that's at a high level how we answer the question, if we assume that these are indeed the right behaviors, is it actually exhibiting those behaviors? And in a way that we can run, this can take, you know, like, several minutes to go through the whole evaluation run, but then we get something that is, you know, close to a rigorous answer to that question.

[49:34] Yusuf Saber:
Uh, in the past we'd have to, you know, have, like, a bunch of, uh, you know, lessons, and then we'd have, you know, the team go and take them, so just to get, like, a maybe answered. It spent, know, four to six hours, and then the end, maybe I just got them right this time, they have no idea. If it would get them right every other time, and we haven't tested every aspect of every behavior. So I think with this setup, we can actually see that's the, we're not sure if we are covering all the right behaviors within our tutor, but the ones we do cover, uh, are actually working in the way we expect.

[50:12] Conor Bronsdon:
Yusuf, thank you for walking our audience through this approach. I think it is instructive. As we close out here, Jonathan, Yusuf, I'd love to get some thoughts from both of you, but, but Jonathan, why don't you start on, where do you see DataCamp going and the role that you look to play in the future? And perhaps just maybe give our listeners, hey, we want to check out DataCamp, where should they go?

[50:34] Jonathan Cornelissen:
If anyone wants to learn about anything related to data and AI, go to datacamp.com. Um, you'll see kind of the, the courses that have the AI tutor. Um, and Conor, to link back to what you said earlier, I think it's a scary time for a lot of people where things are changing extremely fast. Expectations for jobs are changing extremely fast, faster than ever before, at least in my lifetime. Um, and so we want to be a force that helps people kind of through these transitions. Focusing on the essential skills, and, and our vision is that we'll do that by building the best AI tutor and then building all the systems that enable that tutor to teach, uh, every relevant professional topic. That's kind of what we're working towards, and, and that's the role we want to play.

[51:28] Conor Bronsdon:
Love it. Any closing thoughts from you, Yusuf?

[51:31] Yusuf Saber:
If I were to add something, something to this same point, we spend a lot of time thinking about that. And we see one in that the nature of the substance itself, like what we teach, is actually changing. And every new course that we're creating now, we ask this question, what would someone doing this in 2027 need to know? And that's usually quite different from what we learned when we did data science or, uh, machine learning, or back what we, we learned, like, t, 10, 15 years ago.

[52:04] Yusuf Saber:
So it's not enough to, to create a course for a tool or a skill. It's also the, the very nature of using those tools and skills has changed. I try to see, uh, what's the pat, balance point between giving learners enough of the underlying, like, really how it works, like how kids learn to do long division by hand to kind of get a feel for that. So how much of that should still remain in the course, even though it's not done at all in real life? You would not do it, you would have the model do it for you.

[52:31] Yusuf Saber:
And how much of the course should be about the meta skills, about how I work with models and, and kind of manage the development process or the execution? And I think it's still an open question. And I think it's, it's a responsibility for educators everywhere to, uh, it's easy to teach what we learned, right? What we've been teaching for a very long time. I think it's hard to kind of try to reimagine what should people learn now that actually sets them up for success in this world. And we try to do that as much as we can with the courses we are creating. Oh. Yeah.

[53:06] Conor Bronsdon:
Amazing. Jonathan, Yusuf, thank you so much for joining me today. And for everyone who enjoyed this conversation, make sure you have hit that subscribe button on your platform of choice. And maybe drop a comment, just let us know what you thought, and let us know what you'd want to learn on DataCamp or what you learned from this conversation. But yeah, Jonathan and Yusuf, great to see you guys, and thanks so much.

[53:27] Jonathan Cornelissen:
Yeah, thanks so much, Conor.

[53:28] Yusuf Saber:
Thank you for having us.