Subscribe
Share
Share
Embed
The Open Source AI Radar
Hello, everybody, and welcome to the first ever episode of Slop Overflow. Mariah and I are so excited to be talking to you about AI Slop.
Miriah Peterson:Pretty much. How disgusting AI is and... But we're all addicted to it, we gotta we gotta deal with the Slop.
Jonathan Hall:Yeah. So the point of this show is it's not really to talk about Slop, although that's definitely gonna be a a topic. We're here to talk about open source AI and dev tools, and what's happening in that world, sort of a weekly update on what's happened in the last week or so. That's what we're hoping to do, we hope it's valuable to you, we hope we can both learn a lot and have fun doing this at the same time.
Miriah Peterson:Exactly. It's our first episode. So I I was planning going a little bit further back than a week, but Mhmm. We will have a good cadence and precedent going forward. And to kick us off, Microsoft is trying really hard to get people using their, PCs.
Miriah Peterson:They partnered with NVIDIA, and now you can run models natively on Windows. So you can... If you've got a gaming PC that you only use once a week, you can have some AI models on your, you know, your 30 sixties or or or 40 sixties just running natively. You can use Unslow Studio that works on Windows, you can do agent sandboxing on Windows, so you can have a lot of fun with local AI on the machines you already have.
Jonathan Hall:That's cool. This CUDA ported to Windows, is it more complicated than that?
Miriah Peterson:So, previously, you would have to do WSL. How I've done Windows in the past. Most people I know, but now it is Windows native.
Jonathan Hall:Very cool. Very cool.
Miriah Peterson:Another hardware from NVIDIA is they have announced a 64 gigabyte version of their unified memory DGX Spark. Unified memory, I think, is my favorite dedicated hardware for AI right now, if you want some dedicated hardware, because you can run larger models on them. You can... You use the entire memory because it's unified memory. So a 64 gigabyte, you could get a thirty four eighty gigabyte model on there, have enough room for inference and KV cache.
Miriah Peterson:So that's a really good Quen model. I think that's too big for GLM flash... Too small for GLM flash, but you can always couple them together. So you can have four or five. Well, not five.
Miriah Peterson:You can have two or four, and then that gets you two fifty six, which is a a GLM. GLM flash is... Or GLM three five, I think, would fit on two fifty six Okay. Which is a really good open source model.
Jonathan Hall:Yeah, definitely. Definitely.
Miriah Peterson:If you wanna just drop, you know, 20 k and get four Yeah. Of
Jonathan Hall:And who who doesn't? I mean, that's... I'll just go dig that out of my couch cushions. Meanwhile, in the world of Nvidia and hardware, rumor has it, I think this is a rumor, don't know if it's confirmed, but it seems to be widely circulating, so it seems pretty solid that the GeForce RTX fifteen ninety is halting production. This is to shift supply to AI data center and professional GPUs where NVIDIA can make more money because they don't have enough yet.
Jonathan Hall:So if you were thinking of buying a fifty ninety, then you'll have to get it on the used market very soon. Although, that's possible probably, but you're paying a high premium. Or their new flagship model is the fifty eighty with 24 gigs.
Miriah Peterson:So what you do is you go on eBay, you find a forty ninety, and then you can get it altered in Taiwan to actually get 32 gigabytes of VRAM. You won't get any warranty guarantees, but it will get you all of the same capacity. I haven't done this. I have friends that do it. I have a fifty ninety.
Miriah Peterson:If you see the video, behind me is my fifty ninety that is actively coding right now. So I I like the fifty ninety. It's expensive. Yeah. But it's got 32 gigabytes of VRAM, and that's an entire model in VRAM, as opposed to a mixture of expert models, which only puts part of the model in VRAM.
Miriah Peterson:So,
Jonathan Hall:it's
Miriah Peterson:a great hobby. I think you should consider it, but it's expensive.
Jonathan Hall:Yeah. Alright. A lot of this stuff is expensive.
Miriah Peterson:But you know what's not expensive? Tell me. JEV is It's not actually so much cheaper than all the other models that I know people that are switching to JEV to save money, because it's like parts of a penny per token. Mhmm. If you haven't been on the Internet and seen JEV take it over, JEV is a specialized decision model.
Miriah Peterson:So instead of doing full language, it takes, certain categories and makes decisions based off of them, and you can call it over API. It's... It was created by a couple of former NVIDIA... Not NVIDIA, and former OpenAI employees. Yeah.
Miriah Peterson:And it's mostly just really good reinforcement learning. I haven't actually played with it, but everybody I've played, I want to, I'm just too busy with my startup.
Jonathan Hall:Yeah. Yeah. I've been reading about it. One of the things I do three or so times a week when I'm putting my kids to sleep, they're lying next to me and I'm looking at my phone and I ask Gemini, what is the latest in the open source alternatives to JEV world? And it changes every couple of days, but we're gonna talk a little bit about that here.
Jonathan Hall:Probably the most interesting advancement in that area, open source alternatives to JEV, is that Llama C plus plus now supports these decision models natively, which means that all of the sort of children of Llama C plus plus like Olama Pronk, which is written in Go, and any of these other things that build on top of it also have that support now.
Miriah Peterson:So it's it's that they added the same API spec. Mhmm. So you can host the decision model yourself, and you've been able to. I don't think you could have hosted it with Lama C plus plus but... So gLiner, which is an NER, which is named entity recognition, have been open sourced for a long time.
Miriah Peterson:And they do very similar categorization like decision models do based on text, which is essentially the intent. But the fact that it keeps it API compatible means it's really easy to switch models without having to change the way you change the way you build your software. Right? You have a really stable interface to be able to change these models as you get new inter... Interactions.
Miriah Peterson:It is... It did seem like it was a beta release now, but I love Llama C plus plus and so this is a really great, I think, iteration on making decision models very accessible.
Jonathan Hall:Absolutely. I'm looking forward to trying this. I've been using Cronk mostly for my own local LLM, which supports this now too, and it'll make it really easy to use, like you said, the same interface for standard style LLM inference, plus this at the same time on the same hardware without having to bother with Python dependencies or whatever else you might need if you're trying to run lay... Is it Laya? Is that the right name?
Miriah Peterson:Laya's one of the options. That was the one that that people picked up really quickly. But, like, I said, there's lots of decision models. Classification models are not new. No.
Miriah Peterson:And like I said, this is mostly just reinforcement learning on top of a traditional classification model, is a normal NLP thing. So... But like, it's really, really... If you have hardware, I think... And they're small.
Miriah Peterson:These models are are really small, couple of gigabytes max. So great thing to mess around on any hardware you have.
Jonathan Hall:Right. Right. We've made it we've made it through the first episode's main list of of news. We have some bunch of models we're gonna talk about in a second, but before we do that, a word from our sponsors.
Miriah Peterson:This episode is brought to you by Goodwill and our viewers. It's our first episode, so it's really important that we get good feedback. So please give us comments, likes, and share this with your friends, anybody who is really interested in building with AI and open source AI options. We would love to hear from you, and we really need your support.
Jonathan Hall:That's it. Love our sponsors. They're so great.
Miriah Peterson:They're so great. They... They're really interactive, and they give us lots of lots of feedback and love.
Jonathan Hall:We hope. Alright. Let's do a laundry of a bunch of models that have come out. Our intention... We'll see how this goes.
Jonathan Hall:Our intention is just sort of tell you what the models are, what came out, not go into any depth on these because that would take forever. But just to let you know, if you're watching for, you know, what's the latest version of a certain model that you're curious about, you can just listen to the show and we'll we'll tell you, and then you can go look up the the release notes. What's the first one?
Miriah Peterson:Muse. Meta Muse released Muse Wonder and Muse Wonderful. If you're looking for deep dives, I'd say check Theo's channel. It's their competitor to Claude's. Next, we've got...
Miriah Peterson:You know, Claude released Haiku, their cheap one
Jonathan Hall:Mhmm.
Miriah Peterson:Which is Haiku 5.5. Mistral. Mistral's one of my favorites. They have large four slash LeChonk. Cloudflare released Clef, which is Cloudflare's version of JEV.
Miriah Peterson:Yes. OpenAI has the decisions API, which is their version of JEV. Gronk has a new voice model, three one. JetBrains released Melium two one under an Apache license, I think. It's something crazy.
Miriah Peterson:I hope that that's coding focused. And then, we've got glm three five flash.
Jonathan Hall:Five three.
Miriah Peterson:Oh, that one. Yes. Five three flash.
Jonathan Hall:Five three flash. Fast. Five three fast. Don't know what Fast. Fast and flash mean that are different, but...
Miriah Peterson:Well, it might have been flash and I spelt it wrong.
Jonathan Hall:Yeah. Could be. But I've been using five three flash for a while, so I think this is something new. But anyway, we'll have we'll have links in the show notes and in the in the description if you want to go check out any of these, you can go find all the details there.
Miriah Peterson:Open source models, I highly recommend Hugging Face and Unsloth, and then everything else is an API. So, get your token purses ready.
Jonathan Hall:That's right. Well, we made it through our first episode faster than expected. It's... But I guess that's fine. We'll try to be back here next week with some more AI news related to development, software development.
Miriah Peterson:Yeah. Thanks everybody for listening. Yeah. Please comment and share and like, and let us know if we missed anything, and how you're gonna use your open source AI tools.
Jonathan Hall:Yeah. Absolutely. Looking forward to hearing from you. See you next week.
Miriah Peterson:See you next week. Bye.