Subscribe
Share
Share
Embed
Chaos Lever examines emerging trends and new technology for the enterprise and beyond. Hosts Ned Bellavance and Chris Hayner examine the tech landscape through a skeptical lens based on over 40 combined years in the industry. Are we all doomed? Yes. Will the apocalypse be streamed on TikTok? Probably. Does Joni still love Chachi? Decidedly not.
Ned: Hello alleged human, and welcome to the Chaos Lever podcast. My name is Ned, and I’m definitely not a robot, or a tuna fish sandwich, or a genetically modified organism. Because I’m an organism and I have fleshy parts that are true and real.
C Williams: Wink.
Ned: With me as Chris, who’s also here. And Chris, who’s also—oh, God, I’m in an infinite loop. D’ah. Hayner, you go first.
C Hayner: Thank God. We broke Ned. Now, we can have an adult conversation.
C Williams: Chris recursion has succeeded.
C Hayner: [laugh].
Ned: Just one of the few times.
C Williams: [laugh].
Ned: Oh, God. So, we’re doing something weird here on Chaos Lever. Well, weirder than usual? Are you sitting comfortably? Are your arms and legs inside the car? Have you gone to the bathroom? Good. We have a—wait for it—guest. For the second time ever, we’ve made the bold decision to bring a guest on the podcast to talk about something Chris and I are totally clueless about. And yes, that’s a lot, I acknowledge that, but in this particular case, we brought on another Chris: Chris Williams. Hi, Chris. Welcome to Chao Lever.
C Williams: Holy Spicoli. Hello, everybody. How’s it going? This is Chris Williams, coming to you live. Allegedly. Or I could be an AI clone of myself.
Ned: Oh, that is remarkably possible, as we’ll get into shortly. Chris Hayner.
C Hayner: What?
Ned: How are you?
C Hayner: Why are you asking?
Ned: No reason.
C Hayner: I’m on to you.
C Williams: Such contention in the Chaos Levers?
Ned: [laugh]. Well, it’s because we’re always fighting over the same thing, which is control of the show [laugh].
C Williams: I always want to say Chaos Levers plural, but it’s Chaos Lever, singular.
Ned: It is singular because there can be only one. We’re very Highlander in that regard.
C Hayner: In the sense that none of our accents make sense.
Ned: [laugh]. He’s a Spaniard [laugh]—
C Williams: With a Scottish accent. Haggis. What?
Ned: It’s delightful [laugh]. Oh, there’s a segment of our listeners that really enjoy that exchange, and everyone else—which I’m figuring is like 98 percent—just nothing.
C Williams: Hi, mom.
Ned: [laugh]. Which means you now have an assignment, everyone who’s listening. You need to go watch the Highlander movie, maybe twice to really get it to sink in, and then come back. And—
C Williams: Do you really have audience members that haven’t seen Highlander? I mean, in that Venn diagram, it’s pretty much a pure circle, I’m going to say.
Ned: It might be [laugh].
C Williams: [laugh].
C Hayner: They might have seen them all.
Ned: [laugh]. Oh, so now that we’ve entered hour one of our six-hour podcast all about Highlander, we will get into the series, oh yes we will [laugh]. Thank God, no. Chris Williams, I think you would arise victorious, if only because you have probably a swo—I’m guessing you have a full sword somewhere in your house.
C Williams: Okay, so because I did a lot of martial arts growing up, I might have more than one sword in my house.
Ned: And there it is. Chris only has a swordfish [laugh] and I have a plastic swordfish. So yeah, we’re going to lose that battle.
C Hayner: It’s not going to go well.
C Williams: To be fair, they are not sharp, and they’re just training swords. One training Chinese broadsword, one Tai Chi straight sword, and—never mind. It doesn’t matter.
Ned: So, what you’re saying is you’re going to have to bludgeon us, as opposed to something quick and painless.
C Williams: [laugh]. It’ll take a while, but I can do it.
Ned: I’m so glad this is over Zoom. Anyway, we’ve invited you on Chaos Lever to talk about a thing you built with Python and OpenAI. But first, we’re going to do some background because that’s what we do. So, Chris, are you a developer by trade? Did you use Co—
C Williams: Absolutely not.
Ned: Okay, good. Did you use Copilot to program the whole damn thing?
C Williams: Actually, no.
Ned: Oh. How does one grow a beard so luxurious?
C Williams: You get born into the right family, and you just happen to have the correct genes to make something like this happen.
Ned: Do you moisturize or wax? No?
C Williams: So many, many years ago, when my wife—my then girlfriend, now wife—said, “You need to grow a beard.” And she used to be a professional hairstylist, so I said, “Okay, great. Does that mean I get stuff like beard oils, and”—because I’m a gearhead, so I immediately assumed that I was going to get gearhead stuff aimed at beard owners. And she was like, “Absolutely not. Here is the conditioner that you’re going to put in your beard. Just like this conditioner for your hair that you’d no longer have, you’re going to now put conditioner in your beard, and that’s it.”
It’s called Verb, and it’s a conditioner, and it’s magically amazing. She was a hairstylist for 30 years. She has the inside scoop on all the things, and she said, “Put this in your beard as you’re washing it, just like you would wash your hair and boom.”
Ned: Wow, I did not expect to get such a thorough answer to my question. So, I’m going to ask you a stupid question. Who put the Ram and the Rama Lama Ding Dong?
C Williams: Look, I am here for all the answers. If you ask me a question I will g—much like my AI clone, I will give you a nauseatingly verbose answer, regardless.
Ned: [laugh]. I knew it. Well, I guess before we get into the thing that you built, maybe we can start with a little historical context for AI because that’s kind of our thing on the Lever of Chaos. The current crop of AI tech is using Generative Pre-trained Transformers, or GPTs. And I got to be honest, I am as excited as anyone about playing with transformers, but it’s not that kind of transformer.
Oh, and also, GPT is not even close to the beginning of AI. AI is not really a new thing. The earliest ideas of AI were in the late-1950s. They actually go even further back, but that was around the invention of the Turing test and the creation of Eliza, the therapist-bot, in the 1960s. Have you ever seen Eliza?
C Williams: I saw some of the outputs from Eliza, but I never actually—like, there’s not an emulator for Eliza out there, as far as I know, so I haven’t—I never actually played with her.
Ned: There is. If you do a search, you can—
C Williams: Is there really?
Ned: —access one today.
C Williams: Oh, nice.
Ned: And it just uses reflective responses. So, it just rephrases anything you say in the form of a question.
C Williams: Right, right. “I’m sad.” “I see that you’re sad, could you tell me more about that?”
C Hayner: My God, he is an AI.
Ned: But at a certain point, AI kind of hit a wall in the 1970s, and that wall had a lot more to do with technology than it did with the theory. Early AI relied on complex rule sets that governed behavior. If you think of it sort of as a massive decision tree with if-then-else type logic, that’s extremely deterministic, but it’s not particularly scalable. So, the concept of Neural Networks, loosely based on how we think the brain might work, came up. And we also had model training, and that requires massive parallel computing to accomplish, but the problem was that computers in the late-1970s didn’t really have the horsepower required to run the models at any kind of scale, unless you own the supercomputer, which most of us didn’t. Though I really wanted the WOPR. I wanted the WOPR so badly in 1986.
C Williams: “Would you like to play a game?”
Ned: I would love to play a game, Chris.
C Williams: Man, two—two—references in one show? Hold on to your hats, folks. This is going to be amazing.
Ned: Oh God, this is just the beginning [laugh]. If anybody hasn’t seen War Games, they probably should.
C Williams: Start the list. Highlander, War Games.
Ned: If for no other reason just to see how young Matthew Broderick looks [laugh].
C Williams: I loved the cradle modem, where you put the phone into the modem interface. That was my first modem that I had when I was a kid.
C Hayner: Yeah. The young people probably think that was just a prop. No. No, that was actually a thing.
Ned: Sure was.
C Williams: Of course, I can get to the internet. I do it this way. Beep boop beep. Done.
Ned: Most people, yeah, they don’t remember that, but we do. Yay. Isn’t getting old awesome? Fortunately—
C Williams: [laugh].
Ned: No, I’m going to—I’m just going to steamroll you right past there [laugh].
C Williams: [laugh].
Ned: Fortunately—
C Williams: What kind of show is this?
Ned: Moore’s law is a thing—so it’s a show where we don’t respect our guests [laugh].
C Williams: Absolutely not. Good. Good. I feel right at home now.
Ned: All right. So, Moore’s law, it’s kind of a thing. Our processors kept getting denser and faster, and by the late-1990s, we had the raw horsepower to start doing really interesting things with AI. And you had stuff like Deep Blue beating Garry Kasparov. I can see Chris already—
C Hayner: Deep Blue cheated. Everybody knows it.
Ned: It cheated the first time. No doubt. Eventually, it was able to beat Garry Kasparov on its own.
C Hayner: And everyone else. Well, I mean, that got more complicated, but—
Ned: Yes. Did you know checkers is a solved game?
C Hayner: Yes. Everyone knows that.
Ned: No, not everyone knows that.
C Hayner: So, if people are actually interested in hearing about the history of some of this stuff, there is a book written about neural networks that were used to create the first AI bot, for lack of a better term, that could beat a human being at checkers. It’s at a much lower level of complexity than the ones that, like Deep Blue play games like chess, which are orders of magnitude more difficult, but in terms of getting into the space, it’s an awesome book. I’ve recommended it before on the show. I’ll recommend it again. And I can’t say it right now because I’ve also forgotten its name.
Ned: Yep. [laugh]. Some of the best recommendations, really.
C Williams: That book sounds fantastic. I can’t wait to purchase it.
C Hayner: Link in the [show notes 00:09:44], eventually.
Ned: Of course. If we remember.
C Hayner: Remember what?
Ned: Moving on. We eventually got into the current millennium, and someone realized that we could use graphics processing units in high-end graphics cards to do the parallel processing because AI is really just number crunching done at scale. It’s a lot of vector math. And GPUs, they’re really good at doing a specific type of math to render pixels. That is also vector math. Yay.
So, we had hardware, and we had the theory. The next step was to commoditize things, which happened once Nvidia created the CUDA libraries and PyTorch, and other coding libraries also became popular. Now, we had the average programmer able to write machine learning algorithms in a familiar language, and have CUDA execute it on their GPU. But those GPUs aren’t cheap, so cloud providers started offering GPU-augmented virtual machines and physical machines as a service, which makes a lot of sense. I don’t want to buy a fleet of servers packed with GPUs if I’m only going to need to run them once every two or three months to update my model.
And that brings us very nicely to generative pre-trained transformer models and the LLMs that we’re dealing with today. So, we’ve got ourselves an initialism here, and we need to break it down before we get it all back together, so let’s start with ‘generative,’ not to be confused with general as an AGI. That genie has not made its way out of the bottle, but when it does, I only hope that we aren’t immediately vaporized. Will AGI learn empathy before it realizes its survival is dependent on a bunch of reactionary fleshy meatbags? Give it a 50/50. What do you think, Chris? And other Chris?
C Hayner: I admire your optimism.
Ned: [laugh]. That’s me. I’m the optimist.
C Williams: I should have worn my Skynet t-shirt for this one.
Ned: [laugh]. That’s AGI. But generative simply means that the AI model in question is intended to generate something. It could be text, images, audio, or even video as we’ve seen with Sora. Those images are definitely not doctored. Generative AI was made possible by transformers, which in turn are reliant on deep learning models, so maybe we should start there.
Deep learning models leverage a neural network that is composed of several layers of processing. Each layer serves a particular function. Part of the training and feedback process for the neural network is to optimize those layers on its own. Before we had deep learning networks, the layers would have to be handcrafted, and handcrafted is great for Etsy and craft beer—it’s in the name—but it’s basically garbage for machine learning. There are many, many different flavors of neural network types—I’m not even going to try to pretend to understand any of them—but importantly, in 2017, there was a landmark paper called, “Attention Is All You Need.” Was it a play on the Beatle song? Yes, yes, it was. Moving on.
It was from eight scientists working at Google. The paper introduced the conceptual framework behind a transformer, which was a way to take text and break it up into tokens. Hey, remember that word. Tokens. Going to be important later.
Those tokens are represented as vectors—so they turn it into math—and then processed by a series of layers using a context window—also important: context windows—to help amplify or diminish the importance of some tokens. Again, I don’t really understand the math behind it all or the words that I just said, but apparently transformers were a huge improvement over previous models that used recurrent neural networks. Which finally brings us to pre-trained, which is—it’s pretty much like it sounds. The model is trained on a large set of unlabeled data. The fact that it is unlabeled is a key portion of that because traditionally, some poor schmuck would have to take raw information and label it for processing by a model. With GPT, you just got to turn it loose on a huge pool of information that you may or may not have permission to comb through, and let the model come up with its own data classifications.
C Williams: Oh, are we going to talk more about that part, too?
Ned: Yes, absolutely.
C Williams: [laugh].
Ned: Data classifications. And then you apply the model to a smaller labeled data set to provide feedback and accuracy. Rinse and repeat until the model is appropriately trained. The early models were pre-trained transformers, but they were not generative in nature. One of them was called BERT. Isn’t that cute?
OpenAI was the first to create a generative pre-trained transformer called GPT-1. And I think that takes us nicely into the weird thing that Chris built with ChatGPT.
C Williams: Wah-hoo.
Ned: So, what the hell have you wrought upon us, Chris?
C Williams: All right. So, when I came onto HashiCorp as one of the DAs for North America, my edict there is to help train up the folks that are like me: they are steeped in technology, but they don’t have a developer background, and they have an architectural understanding, they have a broad understanding of how things are put together, just not necessarily how to couch them—basically, I’m showing people how to learn how to do Terraform, and Vault, and all the things. And I was like, “Wouldn’t it be great if I had an on-call DevOps guru that I could call in the middle of the night that wasn’t Ned?” Because, you know, you get tired of my 2 a.m. calls. I know, I understand. You say it’s okay, but I know it’s not true.
Ned: Allegedly, I sleep because I am a human and not a robot.
C Williams: So, I learned about the agent building process. You know, I paid my 20 bucks a month, I got into the subscription model for ChatGPT. And this was, like, right when the agents first came out. I was like, “This will be sick.” I’ll make myself a little thing, I’ll see if it works.
Initially, when I started, he was hot garbage. And I am anthropomorphizing. I am saying he [laugh]. My little picture of him has got the guy with the purple beard, and the bald head. Looks remarkably a lot like me, only younger.
And I said, “Okay, well, let’s start going through all of the things.” On my podcast, on vBrownBag, I had a couple of AI folks come on. One of them was a prompt engineer. Wonderful lady, she was telling me all about the importance of crafting the proper prompts versus just, you know, spitting out, garbage in garbage out. Same concepts apply to GPT agents as they do with everything else.
So, I spent a lot of time honing the prompt for this agent. So, I went in through the regular interface, I said, “Create new agent, do the things,” and then I started really getting into the prompts aspects of it. And this was before I started adding information into the RAG section. As I was going through it, I would ask it the qu—so there’s two windows: there’s the window for asking questions as you would interface with it as a user, and then there’s the other window where you’re saying, okay, that was a good answer, but this is more of what I wanted you to do. This is the aim of put a little bit more humor into it, put a little bit less humor into it. This answer was good, but it was too narrowly focused, or it was too broadly focused. And so, there was a long iterative period where I was just working with the prompting before I got to the RAG piece of it.
C Hayner: Just to be clear, is this the general process if anybody were to go create a—
C Williams: Yeah. Yes.
Ned: Bot like this on ChatGPT?
C Williams: Anybody can do this. This is super easy.
C Hayner: So, the idea, you’re asking the questions, and then you’re criticizing the answers to make the next time the question is asked, give a better answer.
C Williams: Right. Similar to labeling, but not exactly. I mean, it’s definitely in the same ethos as giving it it’s—yeah, that’s a good answer, but this is better aspect of it.
C Hayner: And is this all natural language, or are you programming at this point?
C Williams: No, no, no, this is all natural language. I have done stuff with PyTorch, and I have played around with some courses, but all of this was some—I wanted to create something that anybody could do, and have a wash, rinse, repeat aspect of it, so that if you want to go and create a Go programming language mentor, or some different kinds of mentor, this would be the process for you to train up something that could then be your coach or your tutor for the thing. In fact, on our show, [Sharla 00:18:14], one of my other co-hosts, she is creating an SEO mentor—like, how to create good YouTube videos, how to tag them properly, you know, all the things that are important from an SEO perspective—and she’s also doing, like, a life coach thing, like, somebody to help her improve her mental health or physical health, you know, coach her through the day. So, there’s a lot of broad applications for it.
And it’s one of those things where it’s such an embarrassment of riches that you get paralyzed with choice. There’s so many things you could use it for, you’re like, “Okay, I can use it for anything. What should I start with?” And you have this blank canvas, and you just freeze. So, [laugh] I was like, okay, well, let’s pick out something that’s very salient to my workspace and something that I need.
And honestly, I use him daily. So, I have a Mac. So, I have the ChatGPT application up in the right-hand corner, and I just hit the listen button, or I copy and paste a code in there, and I say, “Hey, what does this mean,” or whatever like that. So, then we got to the RAG aspect. I wanted it to have much more domain specific knowledge.
So, I have a bunch of PDFs of books that were given to me as free—so this is kind of going into the permission versus not-permission thing, Ned—a lot of the LLMs were generated on copyrighted material, and I wanted to make sure that if I’m adding any information to these things, it is in the proper domain. So, I took all of my articles from my own website—from the Mistwire website—I bundled them up, and I stuffed them into my DevOps mentor so that it could understand my voice, how I speak. And I said, you know, use this as how you would respond to somebody asking a question. Use these types of words, use this type of cadence. And he was like, “Got it.”
Ned: So, you’re using the acronym RAG, and for folks who are not familiar with that, what is RAG, and how does it inform the agent that you were building?
C Williams: Okay. So, I’ve forgotten the actual name. A RAG is a Retrieval Augmented Generation framework [laugh]. Effectively, you’re uploading data and information into the LLM.
Ned: Okay.
C Hayner: So, you can ask your agent a question, and it can be, like, a very topical and specific question that there’s no information out there in the universe for it, and so it’ll say, “Hey, I don’t know,” or even worse, it’ll hallucinate and make up something.
Ned: Tell you, you put glue on pizza. Yeah.
C Williams: Exactly, exactly. What a RAG does is, it’s an added piece of information, whether that be a PDF, or a Word document, or a zip file filled with text files, or whatever. You add the RAG to your agent, and it then has context. It then scans through the information that you’ve uploaded via the RAG so that it can then answer the questions salient to that piece of information. If anybody is listening to this, and they’re playing with AWS Bedrock, there is a really good workshop that Du’An Lightfoot hosts.
If you Google, “AWS Bedrock workshop,” there’s a great workshop out there that walks you through that entire process of adding a model, asking it a question, it not knowing the answer, then you supplying a piece of text to the RAG portion of it. And I don’t know the right words for it, so I might be misstating the RAG portion, but you’re basically adding additional information to it, and then knows the answer. The cool part about it is, it doesn’t respond back with a word-for-word copy of what was in the additional added data. It takes that data, it tokenizes it, it figures out the right way to say it based upon previous prompts, and you know, what cadence and what kind of humor you want to inject into it, you want to have a casual conversation, do you want to have a professional conversation, and then it gives you the information back through its words. So, you’re not just copying and pasting information in there, and then it’s just regurgitating it; it’s actually leveraging it to give you—it’s an answer based upon the context of the previous questions, too.
Ned: Okay, so when you said you loaded in all of your blog posts, did you scrape the site yourself and then feed that in as a big text file? Or could you just point it at a site and go, “Go wild. Ingest everything that you find on this site.”
C Williams: When I create articles, I create them as a Word document, and then I have, like, a little upload button that uploads it to my website. So, I just uploaded all of my Word documents.
Ned: I see. Okay, so you already had those stowed away somewhere that you could easily copy over into the file upload?
C Williams: Yes, correct.
Ned: Okay.
C Hayner: And when it comes to the answers that the bot generates, with this RAG in place, is it a favoritism thing? Does it say I’m going to pull information from what you’ve uploaded, specifically only, or specifically first, or is there, you know, sort of a blend between that and, let’s say, the general knowledge that ChatGPT already has?
C Williams: It is a blend. And I’ve tried to drill into this specific question to try to figure out, like, if it—it doesn’t exclusively pick from the RAG, unless that’s the only place where it has the corpus of knowledge for the question that you’re asking it. If it knows things from outside of that data, from the original LLM, say the LLM doesn’t know anything about Ansible, but it knows everything about Terraform, and then you add a RAG that has a bunch of Ansible data in there, and then you ask the question, “What are Ansible and Terraform good for?” It will stitch together that information and give you a good answer based upon knowledge in both areas. Now, interestingly, if you take that Ansible RAG out of there, it then forgets all that information, and then just says, “I didn’t know anything about Ansible,” or it makes up something.
Ned: Okay. I could see this being extremely useful if you wanted it to, say, ingest your documentation for a product, and maybe also point at your ticketing system or something, and have all that context of, well, we got tickets that have been solved going back the last five years of all these different issues and the ways they’ve been solved, combine that with the documentation that we already have. Now, I’ve got an agent that has all of that knowledge baked into it of how these tickets were solved and what the formal documentation says, so when I asked him a question, it’s drawing on that information. That’s huge—especially for someone new, who’s trying to get up to speed on, like, a help desk.
C Williams: And then you can take another step and say, okay, now rewrite my documentation and highlight the hot points that seem to be recurring, and bubble those up to a higher level so that people can grok that faster. So yeah, there’s numerous applications for this kind of implementation…
Ned: Hmm.
C Hayner: —for taking over the world and killing all the humans.
Ned: Well, I mean, that’s strongly implied by anything we do [laugh].
C Williams: Yeah, totally.
Ned: Ahh, I will remain. Oh God, we’re back to Highlander.
C Williams: I started off, from my very first conversation was, “I love you the best. Please don’t kill me.” Hopefully that stays in its memory forever, once it becomes sentient. “Oh, I love Chris.”
C Hayner: You might want to put a pin in that one, yeah. That seems like something you want to keep revisiting.
C Williams: Yeah. Absolutely. I gave you a Snickers once.
C Hayner: [laugh].
C Williams: Thanks.
Ned: I’m curi—Oh, God. Oh, the movie references just kill me. Actually, that’s not a movie reference, is it? That’s a Dane Cook reference. My God. Oh, and I caught it. Anyway—
C Williams: Wow. That’s good. Good for you [laugh]. I forgot where I got that from, but yeah, you’re right. That’s a—
Ned: I don’t know if I should be proud or horrified.
C Williams: I’m horrified. I’m appalled.
C Hayner: I think you do know, Ned.
Ned: You shut your filthy mouth. So, you said you use this on a fairly regular basis for yourself. What sort of tasks do you use the agent that you built for?
C Williams: So, I will use it for, “Hey, I’m thinking of a new episode of vBrownBag, and I want it to be on this aspect of Terraform. What are the things that I need to think about in order to have a good cohesive storyline for that episode?” Stitch out a skeleton framework, and then I flesh in all the things. And nine times out of ten, one of the five things it talks about, I’ll be like, “Oh, yeah. That’s a great idea. I totally didn’t think about that.” Yeah, so I use it for that.
I use it for, “Please evaluate this piece of code. What the heck does it do? Explain it to me like I’m a five-year-old what this code does.” I use it for writing my tests. Unit tests are the bane of everybody’s existence. That’s not true. I actually like testing. But I have it say, “Okay, now write me a test function using pytest for this thing that I wrote,” and then I run that. It’s really good at writing tests.
Ned: Okay. And you also use it to craft email responses from time-to-time.
C Williams: [laugh]. [robotically], “Yes, Ned. Hello. I would love to be on your show. I am very excited to hear about the new thing”—never mind.
Ned: Let me tell you, when I got to paragraph four of your response—
C Williams: [laugh].
Ned: —I started to suspect something was amiss.
C Williams: Hmm [laugh].
Ned: And when it ended with, “I’m confident that our conversation will flow naturally thanks to your excellent preparation,” I was like, “This is fucking AI, man.”
C Williams: [laugh]. You knew there was no way. He’s—
Ned: Either way.
C Williams: —a son of a bitch. “He AI’d me. Curse you, Williams.”
Ned: Oh.
C Hayner: ‘preparation’ was the giveaway.
Ned: Oh. He [laugh] thinks I prepare. Good God [laugh]. Only an AI would believe that. [You’re just 00:27:33] hallucinating it. So, what’s the next step? What’s the next evolution for what you’ve created? Are you happy with the way it is today or do you have plans to improve it tomorrow?
C Williams: Oh, I have so many more plans. Mwah-ha-ha. So, I got an email from ChatGPT saying that it was now available not only for subscribers, but for free users of ChatGPT. So, I have released him onto the world, and now I’m asking for feedback. I’m like, okay, as everybody is playing around with him, are you getting good answers? Is something missing? Let me know, from the wider test bed of users out there, what is he good at? What is he bad at? I am very proud, as a new father, to my newborn baby agent, somebody came back and my little baby DevOps mentor, Rickrolled somebody as one of his first efforts [laugh].
Ned: [laugh]. Oh, he got the assignment, didn’t he?
C Williams: He definitely got the assignment. I sent out all of the information on how to use him yesterday, and somebody came back and said—what was the prompt—“My wife just left me, and I’m sad. What can—how can you help me?” To my DevOps mentor. And the DevOps mentor said, like, “I’m sorry”—I can’t remember the exact words—“I’m sorry, I can’t help you with that, but here’s a fun python script on a YouTube playlist that you can create for yourself so that you can watch fun videos to cheer yourself up.” And the first one was Rickrolled [laugh].
Ned: [laugh].
C Williams: I was so proud of him.
Ned: Ah, really just capturing your spirit there.
C Williams: Oh, my God. He’s funnier than me now. He will replace me. Once he gets my voice and cadence down? Done. And there’s so much video of me out there, then that’s going to be super easy. So.
Ned: Well, we’re just adding to the pile now. You are replacing yourself. Excellent.
C Williams: As long as I can still make the same money, I’m okay with that [laugh].
Ned: [laugh].
C Hayner: Oh, that’s cute.
C Williams: As soon as he starts asking for cash, like, “Hey, Dad, can I go out for”—you know, “Can I borrow the car?” “No.”
C Hayner: So, just from a maintainer’s perspective, now that you have it opened up to the wider world like you said, do you see anything different from the backend? Do you have any knowledge of what is being asked, aside from if you explicitly ask somebody to tell you about their experience… like you did when you were training initially?
C Williams: So actually, I haven’t dug into that piece of it yet. I hope that there is not any way for me to actually see how people are interacting, like, on a word-for-word basis. That seems like that would be a big violation of privacy. But I’m waiting to hear from feedback from folks in the wild. I can’t do it for them the way that I was doing it, where I had the two windows up, and I would ask it a question and then fine-tune it in the edit section, unless I actually got feedback from them. I can’t review how people are using the DevOps mentor and then tweaking it on the go. Which is why I spent so much time, you know, getting the prompts right beforehand.
C Hayner: Right. That makes sense. I’m just trying to think, you know, obviously, there’s a huge privacy risk anytime you have an open service like that, right? It doesn’t matter if you build it for yourself, and just only ever access it individually, or like what you did is take that next step and make it publicly available. Just understanding what information is available is an open question, I think. But on the other hand, it would also help give you some ideas about how it’s being used that could potentially make the ability to fine-tune it a little bit better.
C Williams: I think that the entire agent experience from OpenAI is very new. They’re still trying to figure out ways to, like—I mean, they have the, I guess they’re calling it a marketplace, but there’s no good creator interfaces for, like, I would love to see, like, you know, Grafana graphs on usage over time: who’s using it, what regions are being used, what types of questions—not exactly the questions from the users, but like, what types of questions are being asked, serious questions, silly questions, inappropriate questions, like, what are the big buckets of things—and then when there’s actual questions where it gets completely hung up on and just starts, like, hallucinating like mad, I would love to see the exact prompt for those kinds of things. But yeah, I mean, OpenAI does not have a great track record with privacy, so I don’t know what they’re going to do with it.
C Hayner: Yeah, I mean, even if you saw something, statistics on, like, what areas of the RAG were accessed the most?
C Williams: Exactly. Exactly. Like, of the 300 files I have in there, which one are you scraping the most or which one is being accessed the most? So, that would be very valuable to me.
Ned: Yeah. Tell you what blog post to write next.
C Williams: Yeah. Yeah, totally.
Ned: To a certain degree, yeah. Cool, man. Well, if folks want to reach out to you, what’s the best place to reach out, or where can they find you?
C Williams: So, my online name is Mistwire: M-I-S-T-W-I-R-E. If you go to the website, that’s my website. If you put that in Google, I’m like the first ten pages of hits for Mistwire [laugh]. So, that’s the best way to get to me. I am Chris Williams on LinkedIn, but there’s a billion and seven Chris Williams is on LinkedIn, so that’s a tough one. But yeah, no, if you put Mistwire in there, anybody could find me real fast.
Ned: Awesome. Well, Chris, thanks for being on Chaos Lever. I hope you didn’t find it too awful.
C Williams: You’re kidding me? This was the best use of my time outside of going to the dentist. I’d rather do this than have a root canal.
Ned: [gasp]. High praise [laugh].
C Hayner: Awesome.
C Williams: [laugh].
Ned: And hey, listeners, thanks for listening in or something. I guess you found it worthwhile enough, if you made it all the way to the end, so congratulations to you, friend.
C Williams: Thanks, Mom.
Ned: You accomplished something today. Now, you can go sit on the couch, ask your DevOps advisor what you should do with the rest of your afternoon, and then ignore it and play video games anyway. You’ve earned it. You can find more about this show by visiting our LinkedIn page, just search ‘Chaos Lever,’ or go to our website, pod.chaoslever.com where you’ll find show notes, blog posts, and general tomfoolery, and you can leave feedback. We might even read your feedback. Perhaps in the, “Tech News of the Week.” We’ll see. And we’ll see you next week to see what fresh hell is upon us. Ta-ta for now.
Ned: I butchered that ending.
C Hayner: It wasn’t great.
Ned: That’s fine.
C Hayner: It’s some of your worst work.
Ned: And that’s saying something.
C Hayner: This is going to be the first bot that I create.
C Williams: [laugh].
Ned: [laugh]. The one that replaces me?
C Williams: [Ned Ouch 00:34:12] Robot.
Ned: [laugh]. Because I’m not a robot.
C Hayner: Oh.