Slop Overflow

What is Slop Overflow?

The Open Source AI Radar

Jonathan:

Welcome to Slop Overflow. I'm Mariah.

Mariah:

I'm Jonathan.

Jonathan:

We're here to help you navigate all the crud that's coming out. We are the open source AI radar. We've got news and information, and breaches, and everything you can imagine, just fifteen minutes or less.

Mariah:

And we've got some Slop too.

Jonathan:

Gonna be

Mariah:

talking about Slop, yes.

Jonathan:

All Slop. We'll start with OpenCode Security Breach. Yeah. OpenCode Interesting. Yeah, it's gross.

Jonathan:

I use OpenCode.

Mariah:

I I use it too. Yeah. I use it all the time.

Jonathan:

But it does not affect anyone who's doing model hosting. Like, I do all my local model hosting. If you keep your keys on your computer, you're fine. It's just if you were using OpenCode as a key store

Mariah:

Right.

Jonathan:

For model credentials and stuff that they had a... An attack that said it impacted 1.4% of open code users, and they were mostly on model keys and providers' information.

Mariah:

Yeah. So, basically, someone stole access to credentials or they got access to credentials and stole a bunch of API to to Google and to Anthropic and OpenAI and whatever, so. You should have already received an email if that affected you, so you probably don't need to go scrambling unless you got that email and haven't read it yet. Yeah. Good to know that this is apparently resolved fairly quickly.

Mariah:

It's kinda scary though.

Jonathan:

I have VLM. VLM was has a new release. The new release of... Has better KB caching, which will make it way faster, better decoding improvements, and then a generic tool call parsing, which is actually really, really nice. Before, you had to have specify when you started up VLLM, specify the tool call flavor.

Jonathan:

So, for example, when I tried to run MiniMax three, they didn't have that built into the binary yet. They only had MiniMax two tool call flavors, so I couldn't run mini max three. So with generic tool call parsing, you do not have to be favor specific. Generic tool calls are typically JSON, so it might affect some of those Chinese models that tend to use XML. But it should allow you to just essentially run any model without having to wait for BLM to support that tool call.

Mariah:

Does it detect the tool call format, like, on the fly, like, as it parsing each request, or or how does how does that work? Do you know?

Jonathan:

Like I said, I do not know how it works. Right Mhmm. Now it's probably just saying, hey, this is the generic tool call format. So if you're going to be using a JSON generic format, very similar to how the OpenAI API works, there is an OpenAI tool call spec that's just generic JSON Mhmm. That a lot of models also support.

Jonathan:

Right. It's just those are going to not always have the... The models themselves do reinforcement learning on the tool calls during... With their tool call flavors. Mhmm.

Jonathan:

They also do generic tool call for what we call structured output. And so if you have regular structured output with predictable methods, you would get a generic tool call parsing. They're just saying now you can provide that as a flag, and you don't have to be flavor specific. So it's gonna be very model dependent, but it's going to allow... Open up a lot of availability for other models.

Mariah:

Very cool. I was excited to see this week something I wasn't aware of, but I saw this from the Llama CPP and Olama releases that they support a new model called Flint, which is cool, they're always supporting new models, but I didn't know what Flint was, so I went to go look up Flint. And Flint is a 3,980,000,000 parameter MOE model, I love how specific they are, we're usually just like 40 b or 67 b or whatever, this is 39.8 b, it's very specific. But it's it's intentionally designed for local coding. So they've done some tweaks that I don't fully understand or some, I shouldn't say tweaks, they sort of built this to be very specifically tailored for use on local machines, Apple laptops and so on.

Mariah:

I can't run it yet, don't have enough, a big enough VRAM or integrated ram pool to to run it yet. But when when I do, I'm looking forward to trying it out.

Jonathan:

That's interesting. So typically, where... How they do most of those is they will do model comp... Like, essentially, they have accelerators. And accelerators are typically at the inference engine level, and MLX is the accelerator that these use.

Jonathan:

So I'm interested to see how the model itself... How the model architecture affects unified RAM. Yeah. Because usually that's just been handled at the inference engine being accelerated for the GPU.

Mariah:

Mhmm.

Jonathan:

So interesting. I also don't use MOEs that much, so you have to tell

Mariah:

me Mhmm. Mhmm.

Jonathan:

How you like it. I used to use MOEs. I don't use Mixtripe Expert anymore.

Mariah:

So Okay.

Jonathan:

Goose. Goose is an open source. It's an open source agent that lets you just call generic MCPs and create your own workflows via MCP. They released a new thing that's... I've seen it.

Jonathan:

It's really cool. You can put it on any computer. It has a nice little UI. Super open source. Great for doing your workflow agent orchestrations.

Jonathan:

So it's super...

Mariah:

Is this like an MCP wrapper around arbitrary tools?

Jonathan:

No. It's more like n eight n. So n eight n is a workflow orchestrator. So it's more like you're creating agents that do specific workflows and tasks, but they connect to public MCPs. Oh, okay.

Jonathan:

Or private MCPs.

Mariah:

Sure. Okay. Cool. The next one on our list here comes from Apple. This is one that you pointed me to.

Jonathan:

It did. It's really cool.

Mariah:

Yeah. Apple has this new foundation models framework. Explain that to me.

Jonathan:

So Apple, with their... They... Their new OS 27 is going to allow anybody with an Apple computer to run foundation models, while their version of a foundation model. So a small LLM on your Mac hardware natively. You can only run it from a terminal, and it just serves an API that you can inference with.

Jonathan:

So you'll able to do your local coding or your local inference or whatever you want. They are shipping with the models customized per your chip. So they will have m one optimized flavors up to m five optimized flavors. And the size of the model is also going to depend on the chip and the RAM specs that you have. So Apple did already know all of this because it's registered with the serial number of your computer.

Jonathan:

So it's not like they're they're taking any information they didn't have. But, essentially, what that's... Like I said, they're saying, we're gonna own MLX GPU acceleration, and we're gonna build models specifically for our machines that are best in class. Apple's been doubling down on MLX machines and local inference for a while. That's the whole reason they're doing their big Mac Studio push and why they pulled back on the five twelves to then rerelease five twelves with the m five.

Jonathan:

And so... But I I do think... I think there is two problems. The first problem is a lot of people are still scared of the terminal. And so I think there's a user experience problem that people who are not comfortable with the terminal will never use this.

Jonathan:

Mhmm. And then the other problem is we have no benchmarks for these Apple models. We have no clue if they're even good. We just know that they're Apple native. So

Mariah:

Okay. Yeah.

Jonathan:

We could... And we don't know how... What their release cadence is gonna be. Right? We see a very proactive release cadence in a lot of models.

Jonathan:

And so if these guys don't match those aggressive release cadences with these models, it's gonna be very interesting. It... It's... Right now, it's shipping with the operating system, so you're getting that big download up front of several gigabytes. But models are also several gigabytes.

Jonathan:

So how are they going to manage those updates? It's like, great. Great. Your computer's gonna be off for four minutes while it downloads a 20 gigabyte model. Like, it's going to be a very interesting maintenance problem for them, but it also, I think, opens up a lot of opportunity for self hosting.

Mariah:

How how much of this is just now you can run an app or or or a model locally easily and how much of this is that it enables developers to build apps that use a local model? Do you see what I'm saying? Because it it it says that it includes a native Swift API which which tells me that if I'm a Swift developer, I can now build an app that talks to a local LLM and I could ship it on

Jonathan:

Yes.

Mariah:

To other people.

Jonathan:

Anybody... So Macs come with the launch CTL, and launch CTL is the way essentially for any... Anything with terminal access to interact with other Mac applications. Mhmm. So as long as you have a Mac OS developer license and, obviously, they're building in Swift bindings, you should be able to natively integrate anything via terminal access.

Jonathan:

Like, that's part of the built in Mac OS API. So it... Yes. It should allow for native integration to local AI. But I don't know...

Jonathan:

Yes. And that means that any Mac app would be able to do that, which is cool. Mhmm. But again, like, there's also... You could already do that with web GPU, I guess, is what I'm saying.

Jonathan:

Like, you can already host GPUs in a web app. You've written to the same problem, which is the models are huge. Like, that's always the bottleneck is model size

Mariah:

Mhmm.

Jonathan:

Not ability to connect apps to AI. I think I think it's cool. I think it's awesome. I think it's... But I I don't I I don't know what the user experience is gonna end up being like.

Mariah:

That would be kind of my concern too, is you know, if if I build some amazing new app that a million Mac users now download, do they have to download a 50 gigabyte model and how does that experience differ based on how much memory they happen to have on their own laptop? If it's a really small model for edge computing or whatever, know that's probably fine, but if it's something serious it it seems like a a big hit and miss. Back back to the old olden days of when we were building websites for different size of monitors.

Jonathan:

That's kind of the thing that Apple's saying they're handling for you. They're saying, we pick the model, you just interact with the interface. That's essentially their whole point. But that is... Picking the model influences user experience so much.

Jonathan:

I think it's a little naive. Great for them. I'm interested to see how it shakes out.

Mariah:

Yeah. I'm very curious to see how it goes too. Next up, last week I reported that there were rumors that NVIDIA was discontinuing the fifty ninety. NVIDIA has finally weighed in on this and they say, no they're not. However, they do admit that there are some supply change problems and they don't have them necessarily available right now, but they aren't discontinued, they're just having a glitch in the distribution I guess, the manufacturing distribution.

Mariah:

I don't know if that really means a whole lot in practical terms. If you're trying to buy one, they're still not available, at least not from the manufacturer right now, but they probably will be at some point in the future.

Jonathan:

Just don't buy a fifty ninety. Get a forty ninety d on eBay from Taiwan. They're just modified. There are no guarantees they work, but they have the same vram. That's what you gotta do.

Mariah:

Who needs a guarantee anyway?

Jonathan:

I'm not recommending it either. I have a fifty ninety. I bought it. I just am saying that's the only thing.

Mariah:

You probably got it for half the price that they're they're demanding these days.

Jonathan:

I got the whole machine for 4,700, so yeah.

Mariah:

Yeah. Yeah. That's about what it cost for the the GPU alone now, think.

Jonathan:

Seriously. Last thing I've... We found is that the Linux Foundation released a report that they did an audit of pull requests being sent to Linux Foundation progress... Projects, and they're finding that about 25% of them are just slot pull requests. No value.

Jonathan:

They did say, yes. They're having a productivity boost. They... They're they're having acceleration. There's some good things coming from AI enabling these open source projects, but there's also garbage.

Jonathan:

25% is a lot of garbage for maintainers to sift through. Like, that's trivially... This is... It's... 25% sounds low, but for a maintainer who has to review everything sounds...

Jonathan:

That sounds Yeah.

Mariah:

Yeah.

Jonathan:

So, don't I'm do

Mariah:

really curious how representative this is. Like, as you said, it's for the Linux Foundation and they're not all of open source. But they're but they're big. There's a... They have...

Mariah:

There are a lot of Linux Foundation projects out there. I'm I'm guessing I'm a little surprised that it's actually so low. I would have expected, I don't know, 50 or 70% for popular open source projects to to be getting crap PRs.

Jonathan:

I I think I think part of it is what projects are part of the Linux Foundation. Mhmm. I I think those are the the the... One of the things article says is that a lot of OSS maintainers think that the Gen AI has reduced a barrier to entry. I agree with that.

Jonathan:

But I think there is... Like, the Linux foundation projects are more daunting than, like, NPM projects or PyPI projects. Right? So I think, like, if you were to just do a survey of NPM projects, I think it would be much higher than It's very likely for Linux foundation projects. And I'd I'd be interested to see Apache.

Jonathan:

I think Apache would be lower because those are also, like, kind of more daunting. CNCF are not as scary. So it's like like, I think there's a there's a scale of complexity. And I think the more complex the project is, the lower we'll see those slot pull requests just because people don't care to maintain... Like, to do things to them, but I I definitely...

Jonathan:

I would have expected more. I would have expected more. Alright. Now we do our break. We're...

Jonathan:

Now we do our our our... This is sponsored by you or whatever we were doing.

Mariah:

Yeah. I don't know what we're doing. But that's... This is the point for that part that that I don't know

Jonathan:

you're This podcast is sponsored by our agents finding articles and you giving us likes.

Mariah:

That's perfect. I don't know. Let's... I'll I'll I'll edit that in somehow.

Jonathan:

Okay. I would add something like, if you have news or topics or anything else you find interesting that we missed, you want us to cover, put it in comments Yes. We will address it in the next episode.

Mariah:

We do have a laundry list of model releases. I I was I was thinking about this last week as I was writing the show notes. Like, what do we call this? Like, we could just do a model lineup. But I thought, you know, a model lineup is called a catwalk.

Jonathan:

That's amazing. The model lineup is a catwalk. So, we should... This is our catwalk. I This love

Mariah:

is catwalk. Yeah. Alright. First up, Quan 3.627 b had an update October 21. I don't know much about it.

Jonathan:

I'm good.

Mariah:

It's updated.

Jonathan:

I use 28 b, it's fine. DeepSeek v one flash ten seventeen, which means it was released on the October 27. Big meat MOE update. It's supposed to be way faster and way more accurate.

Mariah:

Beam five zero one b. I've never heard of Beam, but it was released on October 16. This is an Apache licensed model, has an update out.

Jonathan:

I've heard really good things about Beam. Really good things about Beam. Muse Spark Open. This is the Facebook Muse model. You can run this on your local machines.

Jonathan:

It is a 100 and... No. 816 parameters MOE. There are quantized versions if you wanna run this on consumer hardware.

Mariah:

Nemotron 3.5 super v l one twenty b from Nvidia, October 15. This is a vision model.

Jonathan:

I... Jayce three. I've never heard of Jayce. Oh, they're a state of the art Arabic native foundation. Nice.

Jonathan:

I've never heard of them.

Mariah:

Is it Jayson or is it Jai? I have no idea how you pronounce...

Jonathan:

It's probably Jai. I don't I'm going with Jace.

Mariah:

Mini CPM V 4.7. That sounds like something from the eight bit computer days, CPM.

Jonathan:

Yes.

Mariah:

From OpenBNB, October 15, multimodal model family featuring a whole bunch of stuff. I don't know.

Jonathan:

So that that that 35 b means it's a 35,000,000,000 parameter total and your active parameters are 3,000,000,000. So each expert's a 3,000,000,000 parameter model.

Mariah:

Also have a lightweight one b develop... Deployment option. So it's That's currently not just a single model, it's a

Jonathan:

Yeah. I think that's a new a new lab out in the Bay Area. We've got Amos one three suite by Circa. It's a suite of four lightweight to mid tier open range models from point five b to 27 b. Point five b runs anywhere.

Jonathan:

They're super lightweight.

Mariah:

Cool. Drama three from fish audio. Open weight text to speech model. Might have to try that. I have some blog posts I wanna turn into some audio books to listen to in the car.

Mariah:

Maybe I'll give this one a shot.

Jonathan:

Good luck. Good luck. Quinn's great. And then open source framework. If you haven't heard of LLMD, LLMD is great if you're trying to do orchestration on the cloud of models and specifically switching models.

Mariah:

Okay.

Jonathan:

The... We have a whole episode on it on my other podcast. It reached one point o. That's fabulous. For multi GPU run times and serving of LLM deployments.

Jonathan:

It's right now the... Really the only way to do cloud native GPU deploys.

Mariah:

Awesome. Cool. We'll put a link to that episode from your other podcast in the show notes if you wanna listen to all the details on that. I think that's it for our catwalk, and I think that's it for our episode this week. Thanks for listening.

Jonathan:

Yeah. We'll see you see you around. Don't forget to like and share and comment. And we'll have to figure out a sign off to keep your slop. Just clean up your Slop.

Mariah:

Clean up your Slop?

Jonathan:

Find a Swiffer.

Mariah:

That works for today. We might think of something better for the next time, but for now, clean up your Slop. Find a Swiffer.