- Go 1.22.4 & 1.21.11 released
- Proposals
- Blog: Flaky Tests Overhaul at Uber
- Blog: Redpanda acquires Benthos to expand its end-to-end streaming data platform
Creators & Guests
What is Cup o' Go?
Stay up to date with the Go community in about 15 minutes per week
This show is sponsored by you, the listener. Stick around till after the news to hear more about that. This is Cup of Go for our June 7, 2024. Keep up to date with the important happenings in the Go community at about 15 minutes per week. I'm Jonathan Hall.
Shay Nehmad:And I'm Shay Nehmad, and I'm really hoping nothing important in the Go community is gonna happen between the 6th 7th Yeah. Because we're actually recording to this, on 6th.
Jonathan Hall:Recording a date early because I'm traveling. I'm in a different time zone right now.
Shay Nehmad:And a different mic.
Jonathan Hall:And a different mic.
Shay Nehmad:I I catch that all the time in other podcasts where people switch up their setup, and I'm, like,
Ashley Jeffs:that's not your voice. Do you have
Shay Nehmad:a podcast that you listen to for long enough that you recognize, you know, the sounds in the room and what
Jonathan Hall:Yeah. And then a few of the especially if they do a live recording, you know, on location somewhere, it's very different sound.
Shay Nehmad:What's your number one podcast? This is not a show. We're gonna get into there's super interesting news. But what what's your number 1 podcast? Just hours.
Jonathan Hall:I don't know. I've I've actually mixed up a lot lately. The one I'm listening to right now is called Your First Million. And it's about a cup it's a couple of, entrepreneurs talking about and interviewing other entrepreneurs to make money.
Shay Nehmad:Cool. Your first million. That's such a American sounding It
Jonathan Hall:sounds a lot more egotistical than I think it really is. It's actually kind of a cool shot. It sounds really like like, startup bro type thing. I don't I don't think it really is. Anyway, let's talk about Go.
Shay Nehmad:Yeah. Let's get into Go.
Jonathan Hall:So last week, we mentioned that Go 122.4 and 121.11 were about to be released. They have been. So, this included 2, security fixes. The first one was mishandling of corrupt central directory record in zip files. So if you're using zip files, be sure to upgrade and the other is related to net IP unexpected behavior from the is method.
Jonathan Hall:Is that right? Is that is or LS? My thought doesn't tell me the difference, for IPV 4 and IPV 6 addresses. So if you use IP addresses Or zip files. Then you should be upgrading.
Shay Nehmad:Sounds like you do. I know of a of a tool that uses a zip as a Go dependency, the GitHub, CLI. Because the bug I fixed was in zip files. So I don't know. Maybe we should really I really was on top of upgrading the CLI when my version came out.
Shay Nehmad:And ever since then, every time we're on GH and the they command that it's like, oh, upgrade you with brewing. Like, no, I don't think I will. So I might.
Jonathan Hall:But, yeah,
Shay Nehmad:you know, new security releases always good to upgrade. You should stay on the latest patch always. Right? The there's no reason to not upgrade patches in, Go Miners as you need to worry about a little bit, right, when you upgrade from
Jonathan Hall:Yeah. Right.
Shay Nehmad:Like, the latest upgrade, broke the in it, imports and whatever. But also edge cases. Right? It's pretty stable. It's a pretty stable language.
Jonathan Hall:It is.
Shay Nehmad:But you don't need us to remind like, convince you that, oh, zip file. I need to check if I have zip just upgrade. Nothing bad's gonna happen. It's like arguments with my kid. Like, give it up.
Shay Nehmad:But I don't I'm not using it right now. So what does it matter? Just give it up.
Jonathan Hall:Let's talk let's talk about a couple of proposals.
Shay Nehmad:Yeah. So for a future, you know, patch release that includes a security fix, although this might be, you know, something that may wait for a minor version, There's a proposal here from Neil, Damien Neil, to add some safer file open functions to OS. You know, there's a big class of vulnerabilities around files. I think the most common one is, you wanna serve files from your HP server, and then someone asks for the dotdot backslash.backslash.backslash.backslash passwords cat file and your etc password, I mean. And, you know, you travel outside the intended location that you wanted to serve the files from.
Shay Nehmad:This is called directory traversal. And if you're, into security or want to get started, just find any place that lets you download files and start typing these, paths. One of them is gonna work. I promise that. To try and, you know, circumvent these cases, there's a proposal here for new functions in the OS, package, which just on the surface, I'm I'm starting with a minus 100 points.
Shay Nehmad:Like, I don't like expanding even more the the OS library. It seems to me like it's pretty complete, and any other functionality you wanna add above it or, you know, alongside it or inside it should be in a in a different pack because you wanna keep it, like, small. And the reason to to opt for these functions is because operating systems offer them. Right? So the always package basically wants to provide an interface that allows you to interact with, you know, win API or, you know, modern Linux functions, and there is.
Shay Nehmad:You don't only have open to open a file descriptor in Linux, you have open at. And if you're if you're familiar with win API, ntcreatefile has root directory just to prevent these specific cases. Right? So the operating systems offer these functionalities, and this proposal, you know, is just for, hey, let's expose it in the Go library as well. This is really, really well documented and people seem to like, the proposal is really well written and people seem to like it.
Shay Nehmad:Even though I started a minus a 100 points, I ended up deciding to upvote it as well. It is important to note that you don't have to wait for this proposal. If this sounds like something super relevant to code you're writing right now, there is safe open. It's like a package, from Google, just called, safe open that has like open beneath. And you basically say, only open files that are beneath this path and then it's, like, immune to directory traversal, you know, like, exploitations, I should say.
Shay Nehmad:I'm wondering what do you what do you make of this? Like, should we do stuff like this if it's that justified? Like, OS supports it. It's for security.
Jonathan Hall:Yeah. I do think it's justified. I also agree with your sort of knee jerk reaction that you why do we need to add more variant to those in our library? But I think this is justified.
Shay Nehmad:The thing that's killing me is, alright, someone's gonna do open, and then someone's gonna oh, it's open. They've written Go, like, 5 years, and they try to open a file, then someone's like, oh, no. It's, open in now. Like Mhmm. Why would you change it?
Shay Nehmad:Even if it's justified, it feels like a frustrating transitionary period. Right?
Jonathan Hall:Definitely. And and I don't I don't think we're gonna be in a place where we can, like, deprecate with GoVet, for example, the old ones because they're still definitely relevant for certain certain cases. So it's not nearly as clear cut as just stop using the old one and use the new ones now, which is gonna kinda be unfortunate, but that that's reality.
Shay Nehmad:Yeah. And and by the way, it's not like operating systems offer everything out of the box. Like, Windows for, I think, for delete file does not have delete file at or delete file beneath it. So it is going to have to be a pure go implementation using the, you know, other capabilities that the operating system, or, you know, Windows file system API, exposes today. And, you know, I ended up being on, sure, let's go ahead with this.
Shay Nehmad:But I really hope that the documentation is gonna be great and the errors are gonna be really, really good, you know, and the error text is gonna be very specific and there are gonna be different error types for, you know, traversal attempted and root directory access attempted and what happens if you pass in a a symbolic link that actually points to the, open end. Should it work or shouldn't it work? Like, every time you touch file systems, it looks simple on the surface. Oh, if it's in the directory, no problem. But when you actually get into it, you can either go with whatever the person who wrote the c function in the Linux, you know, kernel decided, that's the truth, or tried to make sense.
Shay Nehmad:Sometimes these line up, sometimes they don't, but it's another, case of someone taking a really, really good package. In this case, it's Google Safe Open and offering it in the standard library. And I think there's a big difference between what I did, a few months ago where I said, let's take a really good package called yaml and suggest it in the standard library, which is a huge dependency, really big and doesn't really have justification other than, you know, maybe if we offer JSON, why why not the ammo? Where, you know, there's a ton of real world examples of people accidentally using open file, and exposing themselves to direct reach reversal. But it's gonna be a tricky, proposal and it's gonna be a tricky implementation.
Shay Nehmad:I think if you're into, like, operating systems, and I don't know, files and security, this is something you should really jump on. Both the discussion and the implementation. And it's still, like, open right now. It's still active. So, you know, it's it's all open.
Shay Nehmad:I'm I'm not sure, how the team even feels about this one other than adding it to the active column, which is a good sign.
Ashley Jeffs:Mhmm.
Shay Nehmad:So link in the show notes if you wanna dive into it.
Jonathan Hall:So if that one wasn't full of enough, nuance, let's talk about another proposal.
Shay Nehmad:Yeah. This one's technical, but I I think it's interesting as well.
Jonathan Hall:Yeah. So the, you may recall that, presumably in Go 123, unless something drastic happens, they have to revert it. We're going to have the ability to range over function. So you can create a function with a certain signature that that can be fed into a for loop, a for range loop to iterate over the return values of this function. And so there's a new proposal.
Jonathan Hall:How shall we say?
Shay Nehmad:I think it like categorizing it as a new proposal is not as relevant as just as imagine someone in your, work Slack. There's there's a channel called big refactor whatever, and someone or it'd be huge ass new feature, and someone's like, wait, new thread. We need to discuss this specific sub feature.
Jonathan Hall:Because That's really weird.
Shay Nehmad:The whole Range Rover fun thing is a really big issue. I'm not sure even most people understand it or like it or or they're not sure where they use it. And then people get into it, get, you know, I've been writing with it for a while, and it's working really well. The discussion here is about a specific implementation detail of, let's imagine you're, ranging over the seek, right, the sequence Yeah. Yeah.
Shay Nehmad:Which is, let's, range over a map, Right? And get they have keys and values.
Jonathan Hall:Right. So your your function returns
Shay Nehmad:2 keys.
Jonathan Hall:The value. And then you call it multiple times, and you get multiple copies of your multiple instances of key value pairs.
Shay Nehmad:And I remember discussing this on the show and being like, damn. They say other than the fact that the type is called seek 2 with the with the number 2 in it, which is just fugly, there's I had no issues with it. Like, I couldn't think of a problem. But, as stuff happens in software development, the team has been using this for a while. Right?
Shay Nehmad:And they've come up with a case where it doesn't work so well, where it's not key value, which you just imagine, oh, for kv equals whatever. That's what I've been doing in all the other languages. What happens when it's for value and error? And it's been very unergonomic to write it.
Jonathan Hall:And the reason this is a problem is because by convention, and this is already true with the slices of maps, you can emit the second variable, right? But if you only want the keys in your map, you just say for key, both equals range map, and you just get the keys and you don't get the values.
Shay Nehmad:But for functions, obviously, that doesn't work because you have to do let's say you wanna ignore the error, which, by the way, in a in a function that you iterate over kinda makes sense, you know, because you expect the iterator to finish if it's, EOF.
Jonathan Hall:It's very context specific, I think. In some cases, you might wanna ignore the error. In some cases, you don't.
Shay Nehmad:And it's it's mostly it's it's mostly frustrating having to write, you know, I explicitly ignore in some places. The way you ignore the return values in go is you assign them to underscore. Right? So in your range expression, you assign the arrow to an underscore. Now that doesn't sound so bad and honestly, it it really isn't.
Shay Nehmad:I think they're really workshopping all the little tiny frictions that they are going into and they're not afraid to rework it, which in my opinion is good overall, like, for the success of this feature. But it it does end up with, I don't know, a 100 comments discussion on whether we should or shouldn't omit, allow to omit this, this, you know, variable, which you can ignore anyway. Right? You can assign it to underscore. This entire discussion is about whether you can drop the underscore and just do like for value in whatever.
Shay Nehmad:And let's imagine this whatever, this function is you iterating over, lines from a buffer. Right? You iterate over lines from your WebSocket. So you write a function that reads a line from the WebSocket or, you know, the next message or whatever. I imagine like a Kafka iterator or something like that.
Shay Nehmad:And you don't care about errors at all in this case. Because you, I don't know, handle it in some other level. That makes a lot of sense. Let's say if we're talking about Kafka, you have one layer that reads from Kafka, deals with all the parsing error, deals whatever, and then you just wrap this whole thing with a message iterator that you just wanted to work. You don't wanna worry about anything.
Shay Nehmad:You just just wanna worry about the mess you just wanna iterate over the messages. But you still need to be able to return error in case something horrible happened, But then you wanna ignore it, let's say, in one in the test. Because in the test, you don't care about the error. Or maybe the other way around, by the way. Maybe you need the error in test, but in production, you're like, ah, whatever.
Shay Nehmad:It's good enough. This proposal is all about allowing you to just iterate over a function that returns 2 values and ignoring the second one. The main case being ignoring er. I don't know if I'm for this one, by the way, overall. Even though I I've been praising the whole, process, like, the workshopping it of it is beautiful.
Shay Nehmad:Yeah. I don't love the fact that you can omit, like, the the value and key value Yeah.
Jonathan Hall:I think I I would be in favor of requiring the explicit underscore. Oh, honestly, I'd be in favor of that in all cases, even for maps and slices that we have today. Of course, they're not gonna make that change. But
Shay Nehmad:Yeah. Because that's gonna piss off
Jonathan Hall:a lot of people. Yeah. I I I think more explicit is better. Less
Shay Nehmad:I've never I've never written, like, you know, slightly more explicit code, and have been like, oh, I really regret that time invested. It just never happened to me.
Jonathan Hall:Exactly. And there have been times when I've done 4 k comma blank equals whatever, and then my linter or whatever tells me I have to remove that. I'm like, okay. I guess I will since you told me to, but I wish I could leave it there because it it's more it's more explicit. Yeah.
Jonathan Hall:I'm in favor of this
Shay Nehmad:I'm gonna do it, but I don't like it.
Jonathan Hall:Right.
Shay Nehmad:This this whole episode, I don't know if this is just because we're both kinda, tired out of our our minds, but this whole episode just sounds like quotes from my daughter. I'm gonna do it, but not because I'm happy about it. Yeah. So this proposal is new. Again, it's part of a whole big thing.
Shay Nehmad:So it's not, new in the same sense that we just discussed with the, you know, safer OS stuff. Right? It's already doing implementation of other thing. We changed we wanted to change one thing to another thing. Most people here, by the way, agree with dropping error, which is surprising to me because I'm not in favor.
Shay Nehmad:There is the I don't wanna say unfortunate because it's I don't think it is unfortunate. But there is the thing to know that Russ, who I think is leading this feature for sure, is the one who opened this discussion and clearly has an opinion. So I think unless something drastic will happen, you know, he's the person, in the trenches. Right? So I trust his opinion a lot more than my $2 analysis here on the show.
Shay Nehmad:So I got a cool link on our channel.
Jonathan Hall:I like cool links.
Shay Nehmad:Our channel, the ad break is really soon, so I'm gonna plug it, quickly. Capago, kebab case on the go for Slack, has been home for really interesting links and and discussions and and jokes and japes. And, Peter and Links. And Peter, has been listening to the show and actually been listening to me, which is interesting. Funny decision.
Shay Nehmad:And I've been working at work, on fixing flaky tests. Actually, since we've, discussed this, we already implemented a flaky test, like, detection pipeline that works automatically and sends messages, and then we got some responses from some teams. Cool, so you're gonna fix them? So, you know, some teams are really overloaded with the features right now, so we're actually fixing some tests. It's a really interesting, like, cultural discussion as well.
Shay Nehmad:And just in time, 2 days ago, Uber, released a flaky test overall at Uber. They have a hue if you don't know, Uber has a really big engineering culture thing. Like, they have a lot of open source. They have a lot of, blog posts. I think they're really big into engineering branding.
Shay Nehmad:You know what I mean?
Jonathan Hall:Yeah. Yeah.
Shay Nehmad:I assume they even have an Uber, like, developer podcast, just assuming. But maybe we need to find it. So they, started tackling, flaky tests. Just, you know, for our listeners who maybe aren't familiar with the term, what are flaky tests, John?
Jonathan Hall:Well, they're tests that that don't they aren't consistent. Yeah. Is that is that a fair way to put it? They aren't consistent?
Shay Nehmad:I I would categorize them as the devil, but, you know, to each their own. So it's basically tests that fail randomly, and then let's break down randomly into okay. So they break down if you run them in the incorrect order. They break down statistically because they rely on on the some DB residue or something that has to do with time or something that has to do with undefined behavior. Right?
Shay Nehmad:Let's say, in it, order in Go up until a recent version was undefined. So if your test relied on that, it is flake. Even if it is passing deterministically, it can be flake. But the the fact that a test can be flaky is not the part that matters. The part that does matter is you're working on a super critical feature at 5 AM before a demo.
Shay Nehmad:Right? You've been working all night and you're trying to push your branch with a critical fix and the test fails and you're debugging it, and then you rerun and it passes and you wanna die. This is a really bad developer experience. Yeah. And they went all out, man.
Shay Nehmad:This blog post is is amazing. It really shows the difference between small scale and big scale enterprise y, software development, where, like, they have a goal. They developed a whole subsystem in Uber. They're calling it Testopedia that shares you know, they share the structure where, basically, they monitor, all the tests, and every test has a a life cycle, Right? Where it can be stable, which is the good state.
Shay Nehmad:It can be unsound, which is the service marks it as, oh, this is, flaky basically, and then a human can either delete it or disable it, and the system can decide to automatically delete it as well if it's too unsound for too long. So if a flaky test is flaky and the team doesn't get to it, the system automatically deletes it and, you know, the moment something gets into unsound, a Jira ticket is fired off. It's a really interesting system. They even had to worry about, like, the scale of it, right, analyzing the test data. This is just something that, you know, we have a lot of tests at Orca where I work, but this is not the scale we're looking at.
Shay Nehmad:Like, this is pretty amazing. There's some really interesting Go specific stuff here as well. I think the way they share their data model is really good. If you're DevOps, you should read this. If you're engineering enablement, you should read this.
Shay Nehmad:I got this link for from, like, 7, different people, but, thanks a lot, Peter, for, sending it to me. You you were only 7th because, I checked the the Kapago Slack slightly less than the work Slack. That's the only reason. But it's a really, really good you know, there's something about a really good blog post that you read. It has a lot of charts, has a lot of code.
Shay Nehmad:Everything's written super well that you feel like the person who wrote it was like, I just want to give this to you. I wanna gift this to you. It's not like AI generated slop top five list whatever. And I'm really interested in your thoughts about their future plans. So they built this whole system and now they're working on developing LLMs to, like, generative AI to both resolve flaky tests automatically, like suggest a fix, and find the, categorization, like explain why this test is flaky, not just the fact that it is flaky.
Shay Nehmad:In your experience, you know, dealing with these tests, do you think that's even feasible, or is every flaky test a unicorn?
Jonathan Hall:I think in some cases, it's feasible, for some I mean, I actually have a few flaky tests I deal with on a regular basis. I I have a pretty strong idea why each one happens, that they've been happening long enough, but I've I've built that knowledge. Mhmm. When I'm faced with a brand new flaky test, often it does require an investigation to to determine that. And I'm I imagine an LLM could could help in many cases to help, you know, pinpoint that.
Jonathan Hall:You know, this is a data race or this is a, a timing bug or this is, you know, test execution order, whatever.
Shay Nehmad:I'm not sure. I just have a feeling, like, it's slightly too complicated for just shoving it to do chat gpt and asking it why it's flaky. But maybe if they have their own, like if they're at the point of developing and training their own models, then maybe. But I'm not sure.
Jonathan Hall:I don't know. We'll see. I mean, it wouldn't have to it wouldn't have to solve even a large percentage of them to be helpful. If it if it could point you to the right answer for 10%, that's that's the big value.
Shay Nehmad:The the problem I have is if the other 90% is, like, pointing you towards a solution that ends up wasting your time because it looks correct, but it's not really. That would be helpful. Anyway, their numbers at the end are also staggering. They're like, quote, in the Go monorepo, we are steadily detecting around a 1,000 flaky tests. And I'm like,
Ashley Jeffs:what? That's horrible.
Shay Nehmad:And then you finish reading it, and it says, quote, out of 600,000 in total. That's so many tests.
Jonathan Hall:A lot of tests.
Shay Nehmad:This is a texti company. Right? Anyway, so thanks a lot for sharing this link. It was super interesting to me. I also it was, you know, how you know a link is really good?
Shay Nehmad:The time between you took you to read it and the time you sent it in work Slack. For me it was almost instant. So thanks a lot, Peter. Keep keep them coming, man. We really like your contributions in the channel.
Jonathan Hall:One last story that was, brought to us by the channel, the same channel.
Shay Nehmad:Sounds like a cult, the channel. The greater good.
Jonathan Hall:So I I found a different article talking about the same story, though. But the the headline is Red Panda Acquires Bentos expand its end to end streaming data platform. Bentos, as you may recall, is a data streaming platform. We've interviewed the creator of it on the show before. But, you know, the the biggest surprise to me here was that Red Panda is an actual thing and not just a made up, you know, logo.
Jonathan Hall:You know what a red panda looks like?
Shay Nehmad:The animal? Yeah. I've seen one up close. It almost bit my, my nose off
Jonathan Hall:Wow.
Shay Nehmad:In a zoo so one time. They're really cute.
Jonathan Hall:Like, I would have thought it was a Firefox mascot if, somebody hadn't told me otherwise. But, yeah, they look really cute.
Shay Nehmad:They're also, like low maintenance zoo animals. My mother-in-law told me because she used to work in a zoo.
Jonathan Hall:I'll keep that in mind next time I open a low maintenance zoo.
Shay Nehmad:Yeah. I can't imagine you're running high maintenance one. So what's up with this article, man? This sounds like huge news. Do you know any way anything about it?
Shay Nehmad:Can you can you grok it, figure it out?
Jonathan Hall:Yeah. So, like, I don't know. One of the most interesting things here, other than the fact that Red Pandas are real and they're cute, is that as a result of this announcement of this acquisition that Red Panda is acquiring Bentos, a new team has come out to fork Bentos. It's it's like Terraform all over again.
Shay Nehmad:Terraform and OpenDoFu. Right? Yeah. I think I I tried to read about it. There's a lot there's a lot of nuance here.
Shay Nehmad:It's not like Yeah. All of Penthouse got bought, and then everything got forked. We need to dive deeper here.
Jonathan Hall:I'm kinda con well, I honestly, I find it kinda confusing. I I I wonder if if so we can get an expert. Oh, you know what? I think I have actually Jeff's on my speed dialer here. Maybe we could get him on the show.
Shay Nehmad:Do the speed dial mobile. Welcome to our ad break. We have, Ashley on hold here, so I'm gonna rush really fast. If you wanna help sponsor the show like the rest of our beautiful, beautiful, members on Patreon, we would love for you to do it. This show is a hobby, but it's pretty expensive.
Shay Nehmad:We have to pay for editing and hosting fees and stuff like that. So if you like the show, if it's useful for you at your work or university, supporting us, $8 a month is, really nice. To find the link to our Patreon or maybe to our swag store where you can hook yourself up with some hoodies, cups, whatever, to find our Slack channel where, we just talked about the, link Peter shared. You can find everything at cupago.dev. That is cupago.dev.
Shay Nehmad:If all these fancy communication, platforms are not your thing and you're not into Slack, you can also email us at news@capago.dev. Another way to support the show that we really appreciate, we don't pay to advertise anything. Well, I don't know. I pay for Internet to share people the link to the show. I don't wanna say anything anything.
Shay Nehmad:But we don't pay for ads as you would think about them. This show is only spread by word-of-mouth. So share the show with a friend, a colleague, co student, whatever, and leave us a review on Spotify, Apple Podcasts, or wherever you listen to the show. It helps us, you know, get through the rankings and, get into some random listeners' hand. We've gotten some really good feedback about the show recently on LinkedIn and on Slack, and I even got some face to face at work.
Shay Nehmad:And I'm really hoping to, you know, run into some, listeners in Amsterdam, where we're gonna have our, I'm gonna go to DevOps Days, and we're gonna have a meet up on, June 19th. So we're really excited about the direction the show is taking, and your support is a big part of that. Thanks a lot. And I think we're just gonna jump into the interview. See you next week.
Ashley Jeffs:Listen, I told you not to call me at this number again. Next time, there's gonna be consequences.
Shay Nehmad:This is an emergency, Ash. We know this is the never call you on this number line, but the people need to know about red pandas. Wait, that's not the oh, red panda. Oh, oh, wait.
Ashley Jeffs:There's only 1.
Shay Nehmad:Yeah. Hello, Ash.
Ashley Jeffs:Hello.
Jonathan Hall:Thanks for taking my call Hello, Willow. Even though this is this is the wrong number, The red phone. When when we're done
Shay Nehmad:with the call, can you, return this briefcase with this phone to the president? No. So welcome, Ashley Jeffs, also known as Jafail, online. For the people who don't know you, can you briefly present who you are?
Ashley Jeffs:So I'm that guy what did the, blobfish, Bentos open source project thing. You may if you're in the ghost space, you might also know me from my project called Gabs, and there's another even less popular one called Tunny. I've got a few few few fingers in various open source pies in the Go space, all with rather silly logos. But obviously, the more successful one is is Bentos, which is an open source stream processor, and that's all written in Go. Yummy.
Ashley Jeffs:Yum. Yum.
Shay Nehmad:Yum. Mhmm.
Ashley Jeffs:And that's pretty much I do lots of YouTube stuff. People might know me from that because I've got I actually have content that's not related to my various open source projects, but it's related to code sometimes. So people might know me from that, but probably fewer. Cool. But you can find me on there and Twitter and all the other places.
Jonathan Hall:We'll have links
Ashley Jeffs:here to all
Jonathan Hall:that in the show notes.
Shay Nehmad:And if you're a long time listener, you may remember we had Ash and, Mihai on the show, like, I don't know what it was a year ago.
Jonathan Hall:Probably. Let me look it up.
Shay Nehmad:That I think about it. Damn, that was that that was long ago, man.
Ashley Jeffs:Are you gonna sound bite all the stupid stuff I said?
Shay Nehmad:That's a really good idea. We might ask, Filippo to edit all that stuff in. So a lot has changed since, that last recording we did, since that last interview, and we need to talk about it. What do you think about my new tattoo?
Ashley Jeffs:It's not a blobfish, so I'm not interested.
Shay Nehmad:So the big news this week, obviously, is everybody on both sides of the aisle, agree that your code is great, and, Ventas got acquired by Red Panda, now it's called Red Panda Connect. Like, there's a thing called Red Panda Connect, and there's still the Bentos open source library. And we wanna talk about that and try to understand what happened. We wanna congratulate you, by the way, 1st and foremost.
Ashley Jeffs:Thank you.
Shay Nehmad:Dude, that's great. That's a really, really good move. Red Panda, by the way, in my opinion, this is just my opinion, really good stewards of open source stuff, really cool guys. So, you know, this would have been a different tone if this was, like, IBM. But, not if we have listeners than IBM.
Shay Nehmad:Not not, like
Ashley Jeffs:They're they're actually I would say they're not doing a bad job with Ceramic. I actually think the changes
Shay Nehmad:I'm trying to think about a company that I really wanna, like, name and shame in in terms of so then I thought about it a bit some more, then let's steer away from that specific part of the conversation. I don't know. If Saudi Aramco Nukes Tech.
Jonathan Hall:Yeah. Exactly. Seal Club Inc, if they brush them I wouldn't have been No.
Ashley Jeffs:They're doing really well with their open source libraries.
Shay Nehmad:They they actually built the pprof. I'm sure. So anyway, congrats about that. We
Jonathan Hall:We really
Shay Nehmad:wanna get into it and what happens in the future. But just in case someone, you know, wants to understand this news but doesn't know what Ventas really is. And they we don't wanna send it back to old episodes. Can you give, like, a one minute summary of what this open source project is and why is it so big?
Ashley Jeffs:I can't promise 1 minute. Basically, what we had so for about 10 years, I've been working on this open source project. It's a stream processor. If you don't really know what that is, it's basically something that reads from message queues and things like Kafka and your NATs and RabbitMQ, that kind of stuff. And what I built is a sort of engine for doing all that stuff with the decorative config.
Ashley Jeffs:So rather than having to write code and all that stuff, you can just write some structured YAML. And just this emphasis on operational simplicity. So making sure it's really easy for people to build those configs and maintain them, but also when you deploy this streaming processor, it doesn't do any nasty stuff. And that's basically, ingrained in me from my time working on services like that and seeing how badly the operations people suffered when they had to deploy services like that. So things like data loss, back pressure, recovery, metrics, observability, all that stuff.
Ashley Jeffs:And it's very, very hard in the stream processing space. As the project grew, the number of connectors we had, got bigger and bigger and bigger, and the community got bigger and bigger and bigger. It was all organic. It was all just me living on my own off various. So I've I've gone through the rounds of monetization strategies.
Ashley Jeffs:I've tried about it's been, like, 5 different phases, which we can go into in detail if you want, of me trying to monetize it essentially without VC funding. I ruled that out a few years ago that, basically, I didn't wanna do that for a number of reasons. And the project was getting to the point where it was too much for me to be the only person working on it full time. And we have contributors, but they're also doing their own thing, trying to live off, you know, their own means and figure stuff out on their own. So we didn't really have the ability to expand from that.
Ashley Jeffs:So I was trying to find lots of ways of bootstrapping some sort of company without sacrificing family life and the ability to watch TV and video games. And I've been pretty open over the years of how that's been going, to very varying levels of success. And, eventually, it got to the point where, okay, I needed to make a decision, and the timing was right. It wasn't exactly when I expected to do this, but I was basically This was on my list of options at the end of the journey, and it was it was coming to that point anyway. So what's happened now is I've sold the project.
Ashley Jeffs:So there's been an acquisition of the company behind Benthos. And what's happened is so it used to be fully MIT licensed, all the connections and the engine. We already had the plan in the open source community just in general to separate the core stream processing engine from the connectors themselves. And the idea is that you would decentralize so people can have any suite of connectors that they want, because they all have different dependency trees and different runtime, weirdnesses. And it was just becoming really cumbersome having all of them baked into this one open source project.
Ashley Jeffs:So we wanted to split those out anyway. So what the acquisition has done is it fast tracked the splitting of we've got an MIT licensed core engine, which is the plug in engines, the stream processing engine, all the important stuff, all the stuff that makes Bentos, Bentos. That's still called Pentos. Red pounder Bentos, if if you wanna, fully adhere to trademarks, but you're not gonna get sued just for saying Bentos. But the other bit is called Red Panda Connect, and that's essentially the distribution of, essentially, using Bentos as a library to build all of these plug ins.
Ashley Jeffs:We've relicensed most of them to Apache 2, and then there's 2 that we've enterprise licensed as part of the monetization strategy. So, basically, the acquisition purpose is to try and find a way of sustaining a team of engineers to work on this project and not sacrifice their lives doing it. And, hopefully, that's gonna be healthier for the project, but who knows? It's only been a week. Who knows what's gonna happen?
Shay Nehmad:Pretty fresh.
Jonathan Hall:Yeah. So are you gonna stay on the team? Will you be still working there, or are you going to the Bahamas now and you're done?
Ashley Jeffs:No. I'm staying I'm staying on it. I don't have to, but I'm staying I'm sticking around. I mean, my my intention now is to build out enough people who so there there's, like, lots of different groups of people that I can I can create, I can spawn? So there's obviously people who just are able to support Red Panda Connect users.
Ashley Jeffs:Red Panda Connect is a lot, so I'm I'm starting to shorten that to, rednecked. Just the the red at the beginning and the necked at the end. Redneck. Alright.
Jonathan Hall:We need we need Jeff Foxworthy on here now.
Shay Nehmad:Yeah. I can hear the the radio, ads already. Right? Like, banjo twang. Welcome, y'all.
Ashley Jeffs:Shout out to all my, southern US friends.
Shay Nehmad:Do you know I I have to do do this. Do you know that, in old Unix I'm not sure if it's Unix or Linux, but really old distributions. Redneck was a language. Like, you could install it in English, like Spanish, French, Hebrew, like whatever, Russian, and redneck. Not as a joke.
Shay Nehmad:1 of the engineers was a redneck and it'll be like, yep. Nope. Because in the I'll look that up.
Jonathan Hall:There's a screenshot here I found. In 1998, Red Hat had redneck as a language option.
Shay Nehmad:Yeah. Yeah. Red.
Ashley Jeffs:Oh, okay.
Shay Nehmad:It could happen.
Ashley Jeffs:Damn. That means I can't I can't trademark it. But, yeah, anyway, so I'm I'm trying to I'm I'm building out people who can help support. Because right now, it's the developers who work on the Bentos ecosystem, and they they are also in the questions channels, the Discords, the Slacks, answering all the questions. And most of the time, you're you're you're basically consulting for free.
Ashley Jeffs:Right? Because people are coming in and saying like, how do I do x to y and do this blah blah blah and use case blah. And it's like, that's that's it's great that we can help those people, but we're not gonna be able to do that forever because the community is getting bigger. So we're not we're not gonna be able to just answer those questions.
Shay Nehmad:We're not
Ashley Jeffs:gonna spend half an hour every morning answering some random consulting questions. So the idea is we're gonna expand the number of people who can help, just generally help, and understand the the service. Then we're expanding the people who can work on it. Because right now, there's lots of people who've contributed features on top of the engine. So people who are adding, like, fields to connectors and documentation and a new connector here and there.
Ashley Jeffs:There's, like, a small pool of people who contribute quite a few bits consistently, but the the number of people who really get down and dirty in the core stream processing engine, It's basically just me and maybe Mihai to a to a slight degree, but if you look at the the real nasty, like, gnarly bits, that really takes a lot of brainpower to get all the factors of stream processing right. Imagine, like, delivering guarantees, vertical scaling, back pressure, and, you know, all the other bits, you you know, usability, error handling, all those things. To get all of that right for some of these bits is really difficult. So I obviously can't be the only person on the planet who knows how to do that stuff because it's just gonna degrade over time. Like, the the project will degrade if I disappear from it.
Ashley Jeffs:So the idea is I want to onboard more people with enough time to get stuck into that stuff, which is already happening. So it's not just me who has to work on that stuff in theory. And then it's just better for everybody. Right? And then, you know, obviously, more people working on the connectors, more people more people consulting.
Ashley Jeffs:So it's it's just a general, endeavor to expand the community. Because, I mean, as well, I'm doing all the graphics, all the documentation, all the videos, all the training materials, all of that stuff was just me, which is great fun, and it it means that all the content can be equally ironic and just generally silly because, you know, I'm not a professional, so I'm just doing all this stuff fun, which has been great, but that can't that can't last forever because you can't you can't just have one person doing all those things forever.
Shay Nehmad:I really love the the the, like, student listening to the show being, like, the guy who just sold his 10 year project to a reputable streaming company. Like, no. I'm not a professional.
Ashley Jeffs:My family are like, did you did Ash, did you sell the blobfish? Is that what's happened?
Jonathan Hall:So
Ashley Jeffs:They they didn't get
Shay Nehmad:any Yeah.
Jonathan Hall:So I'm curious. Before this acquisition, how much of your time was dedicated to this? Was this full time for you?
Ashley Jeffs:Yeah. Well okay. So full time is so the the public face of what's happening, right, is a fraction of my time spent on the project as a whole because what I'm having to do to survive is spin all these plates, not just for my current income, but also, like, trying to figure out what am I gonna do in, like, a year? And what what Mhmm. What am I gonna live off in a year's time?
Ashley Jeffs:And trying to get to that point where I can hire more people is the all the the main goal. So I'm you know, I was dedicating a lot of my time to just that. You know, that it was getting to the point where I would almost spend all my time some weeks working on that stuff. Because I don't, like, I don't know how much of your audience are solo entrepreneurs, but just doing stuff like procurement, like getting getting a company. You have a company that is desperate to throw money out.
Ashley Jeffs:Mhmm. And they want to they wanna have a support contract, and then it goes to procurement. And it's like, oh, no. We can't. We aren't giving you money.
Ashley Jeffs:You're just some guy in the UK. Mhmm. No. We can't do that. And, you know, I'm I'm spending my time reading through, like, 60 page contracts and talking to random people through email, trying to figure all this stuff out.
Ashley Jeffs:And it it means that the amount of time I was spending on the project was getting smaller and smaller.
Shay Nehmad:Just because of
Ashley Jeffs:that? Working on Benfoss Studio, which is like a complete closed source, spin off thing, which is basically a visual UI for it, which I like, that was bay that was my big plan for the neck. For for essentially this year and last year was to try and monetize that. But, again, it's it's the procurement that ends up killing me.
Jonathan Hall:So now that the acquisition is done, are you employed by Red Panda? Or okay.
Ashley Jeffs:Yeah.
Jonathan Hall:And
Ashley Jeffs:Yeah. So I'm
Shay Nehmad:badge and everything? Company badge, company uniform?
Ashley Jeffs:Company badge yet. I got a laptop.
Jonathan Hall:Oh, there you go.
Shay Nehmad:Damn. I'll
Ashley Jeffs:just fight my way into a physical office. It's everything. But, yeah, I've I've got a a official title. I'm Connect technical
Jonathan Hall:lead. Alright.
Ashley Jeffs:So I'm I'm running the the project to to continue, you know, working on this thing, expanding the team, expanding the resources. And then, like, I'm still gonna be doing the pull request features, all that stuff.
Jonathan Hall:So from one day to the next, the main thing that changed was who pays your salary. Yeah. But but over the coming months, you're building a bigger team to help offload a lot of those tasks.
Ashley Jeffs:But but also all that time I was spending on trying to, like, essentially solve next year's income Mhmm. I no longer have to spend any time on any
Shay Nehmad:of that.
Ashley Jeffs:I've now got a there's a sales team that I can hand stuff over to. There's, you know, there's sales engineers that I can liaise with, and, you know, I don't have to worry about any of that stuff now. That's great. So, I mean, that's that's freed up a huge amount of my time. Obviously, I've just joined this company, and I'm getting all of them up to speed.
Ashley Jeffs:So that's taking up, you know, some of my time. But, already, I'm I'm spending more time on pull requests. And I'm not like, there was there was this horrible sensation. I've generally always enjoyed working on the project and especially being free to just do whatever the hell I want. Right?
Ashley Jeffs:Make whatever decisions I want, has been a wild ride, but it was getting to the point where because I was stressing about, it doesn't look like this is gonna work, and I was worrying about, oh, no. What if I have to because I'm not I I decided against VC funding just because I don't think I'm gonna enjoy that job, if I'm, like, the CEO of, like, a VC funded start up. And it was gonna eventually lead to tough decisions that I wasn't gonna wanna make around licensing and all those things. And the the worry that I had over the last, like, year or 2 is I'm merging pull requests, and I know that I have to support what I'm merging. That's always the contract.
Ashley Jeffs:Right? Because I'm I'm the face of the project. People are coming to me when they have questions, and ultimately ends with me. If there's nobody else volunteering their time in the in the chat channels, then it will land on me eventually. And anything that I build to try and live off, I'm also supporting this whole bag of this massive burden of connectors, all these different technologies.
Ashley Jeffs:And what I was finding is I I used to be able to fit all of it in my head. So somebody would come with a feature. I'd merge it. I'd understand it, and it would just be there in my mind. And it was getting to the point where that wasn't the case anymore.
Ashley Jeffs:So I was merging, you know, like, MongoDB connectors or, Google Cloud connectors, and people come along and say, oh, how how does this thing work? And I wouldn't even be able to tell them that that component exists. Like, I wouldn't even know what's in my own thing that I'm running. And it's it's stressful, you know, like, because I don't want to stop people's poor request from being merged. Right?
Ashley Jeffs:I don't want to, like, prevent because people have already done that work. They've they've put the effort in. So I don't wanna block them, but I I don't have enough time to do the merging. I mean, even when I do, I'm, like, freaking out about it. Whereas now, I have a team of people who can help me with all that stuff.
Ashley Jeffs:So me merging pull requests is just about is it good code quality? Does it fit the, you know, the overall purpose of the project? Is it gonna help people? If that's all yeses, then merge, and that's it. Mhmm.
Ashley Jeffs:As simple as it gets.
Jonathan Hall:Awesome. That's nice. We're laughing because, the the fire was just scared. Yeah. The thumbs up and
Shay Nehmad:Yeah. And do you see a difference or I guess, what differences do you see between stewarding, you know, a herd of open source cats developers, in trying to direct this sort of open source chaotic energy versus, you know, a team, in a company that, like, has a purpose. I'm trying to abbreviate the the title as well. So your rednecktied, your Red Petna Connect technical leadership is probably a lot more I get that you've only been there a week. Maybe that's an unfair question.
Shay Nehmad:But you plan for it to be slightly more structured, I guess, than
Ashley Jeffs:there's gonna be aspects of it. Yeah. So there there's obviously gonna be, like, enterprise features that we're gonna wanna have, to unblock customers and, you know, the the usual business y stuff. Those are obviously gonna be things that we're gonna try and fit in on a deadline. There's also gonna be open source features that we're I mean, same same applies.
Ashley Jeffs:You know, things that we're gonna wanna need is nice to have as we're gonna use. So we wanna get those in on a certain timeline. To be honest, that doesn't really change from my personal perspective because I was always doing that. I was always setting deadlines and then slapping the keyboard until it was done. But getting other people on board to be honest, it was at the point in the last couple years where I felt as though I would be able to say, hey, Mihai, can you do this thing?
Ashley Jeffs:And as long as it's something that's good like, obviously, it's gotta be good for the project. It's gonna be fun to work on. You go and do it. I mean, he's doing it for free. Right?
Ashley Jeffs:So, you know, there's obviously within reason, you you need
Shay Nehmad:to do
Ashley Jeffs:the stuff. But if it was something that was good for the project, there's been times where that's that's been the relation we've had. So it's it's kind of like having employees light.
Shay Nehmad:Mhmm.
Ashley Jeffs:And as long as you're respectful and everybody is kind of on the same page, then, you know, that's that's kind of what you're doing. So I guess you're kind of like it it's the training wheels version of running a team where you're using, you know, motivation and mission to get that sort of stuff done. Whereas now I can just say, money next week or now.
Jonathan Hall:So I I wanna talk a little bit about the acquisition. So Red Panda bought Bentos. Did you approach them? Did they approach you?
Ashley Jeffs:I've known Alex for, like, 5 years. There there was other suitors. It was kinda funny because any anybody following my movements for, like, 2 2 months ago, I think it was, I was at I was at Kafka Summit hanging out with the WarpStream guys at their booth. And then, you know, I'm, like, spending a week in San Francisco. And everybody anybody that I'm interacting with online can just track my movements to see what's happening.
Ashley Jeffs:So it's like you can keep it you can keep it a secret to an extent, but, I mean, if people want that info, it is it is actually just there. It's it's not really particularly easy for me to I've I've had a good relationship with Senadia for a long time as well. Mhmm. They've been they've been OGs. I mean, so my main monetization has has just been nothing for a couple years, and I've been living off sponsors, and they're by far my biggest sponsor.
Ashley Jeffs:They they basically kept me alive when I had my kid, because I didn't have time to to really chase up support contracts, and they just happened to appear at the right time. I was like, what do you need? So, yeah, I mean, I've obviously had conversations with lots of companies kind of the same the same ilk where they're all partners in that sphere. And, I mean, I didn't I didn't really have, like, a a a strict agenda or a timeline thing. Basically, I can't really go into details specifically of what happened, but stuff suddenly moved very, very fast, and it became apparent of like, oh, okay.
Ashley Jeffs:This is gonna happen then. I guess this is gonna happen right now.
Jonathan Hall:You don't need to answer this. I mean, I'm gonna speculate. I'm pretty sure I'm right on this. The reason you chose Red Panda is because of their logo. It's an animal.
Jonathan Hall:And that is, with the Bentos eat ethos of animal logos.
Ashley Jeffs:They kind they kind of fit. I've already so there's obviously been loads of fans making, mashups of the 2 logos. In terms of in terms of mascot to, to blobfish ratio, it's actually pretty good, I would say. Definitely good. Better than better than any of the other candidates because, I mean, cute mascots is hard to get anyway.
Ashley Jeffs:Obviously, the red panda is very cute.
Shay Nehmad:Yeah. We wouldn't know because we knocked it out on the park, like, the first first try, but I assume for other people it's hard.
Ashley Jeffs:Same. I just slapped a plodfish together. It's been the same for, like, 10 years. But, yeah, it's it's a good brand. It fits fits very well with my brand vision, I think is what they say.
Ashley Jeffs:The brand ecosystem. The cine I keep saying the cinematic universe of blobfish. It kinda fits.
Shay Nehmad:The VCU.
Ashley Jeffs:They cut yeah. They they're compatible.
Shay Nehmad:And, you know, the the acquisition basically means, you know, some people have been worried about, wait, acquisition, does that mean all the open source work is going away? You mentioned that the core is saying MIT license, which basically, you know, if you don't wanna get into the legalese means open source and you don't have to worry about it too much because it's not the Red Hat license where it's copy left. And some of the plugins are enterprise, which makes sense, you know, Red Panda have to sell something. And the rest of it is Apache 2. Now Apache 2 is kind of a new kid on the block.
Shay Nehmad:It's, the Elastic, search, license. Right?
Ashley Jeffs:No. That is is Apache v 2 is just basic I mean, it's basically MIT with a bit more legalese around it. So, I mean, you can do whatever you want. AWS could take all of that if they wanted to.
Shay Nehmad:Oh, really?
Ashley Jeffs:Don't AWS. Yep. Apache 2 is,
Shay Nehmad:the people from, AWS on the on the on the podcast. You know
Ashley Jeffs:what I mean? At least Tell them no. Don't a actually, no. Go ahead. Go to town.
Ashley Jeffs:Take take all the connectors, I think. Yeah. So the the the move so, basically, the move from MIT to Apache 2, I don't know why. I was fine with it because it's basically the same license. MIT is obviously more is is less wording, so you you have even fewer protections, like, for what you're sharing.
Ashley Jeffs:But the the ethos is, I don't care. I'm just putting this out there. I don't really care what happens to it. But Apache 2 has a bit more legalese around copyright, that sort of stuff. I have no idea what the implications there.
Ashley Jeffs:I think it's to do with, you know, there are smarter people than me who understand the commercial side, figuring out that that's that's better for insurance or whatever. I don't know. And then the enterprise stuff, basically, it we just need, like, a bit of enterprise stuff. We're obviously gonna be adding cool features that are sort of, like, tacked onto Red Panda Connect, just to make it work cooler. So, I mean, an example is we've already got, a new config section specifically for Red Panda Connect where you can forward your logs directly to a topic, which is kinda cool.
Ashley Jeffs:People have been asking for that for a while. And there's obviously gonna be features that we're gonna put on on top just to make them all work. Cooler together, I think, is the official term. But, yeah, the the idea is just one connector or to the, enterprise. There's 2 it's the Splunk and the Snowflake ones.
Ashley Jeffs:And, you know, basically, the mentality there is you're already paying for Splunk. If you're using Splunk, you're paying for it. And if you're using Snowflake, you're paying for it.
Shay Nehmad:You're paying for it multiple times over because your data's getting leaked.
Ashley Jeffs:Yeah. Yeah. Exactly. Of the week. Or maybe they're making money from that.
Ashley Jeffs:I don't know. Maybe you get a kickback if you take it away. I don't know what joke. Don't you hear me?
Shay Nehmad:Insurance companies are the one pocketing all the the difference. Trust me. Cyber insurance.
Ashley Jeffs:Guess. Yeah. But, yeah, that that's the mentality. Obviously, we're we're already working on expanding them. I mean, peep people were, obviously hyper aware on the day of, well, you've just relicensed enterprise on something that was MIT, which is a 100% correct.
Ashley Jeffs:The goal was to show people what the separation was gonna be for the announcement, but, obviously, we're not gonna leave it as it is. Let's just leave the enterprise license. We're we're gonna expand it and and basically redo them and, you know, beef them up a little bit, stuff that we couldn't really do or have the resource. Because, I mean, I don't I can't maintain a Splunk connector on my own. It's like an open source thing.
Ashley Jeffs:It's currently a template, and it's just basically using the HTTP component under the hood. And it's like, I can't I can't like, if anybody asked me a question
Shay Nehmad:on that, I would have to go and fish out, the with the Splunk server.
Ashley Jeffs:Right? Yeah. Exactly. Yeah. Because I can't I can't run And there's no there's there's not really a lot of tooling for I think there is, like, a a VM you can run now.
Ashley Jeffs:But I I remember at the time somebody was asking me about it. I was like, I can't I can't do anything for you. Sorry. But, yeah, obviously, now we've got the resources. But, yeah, so there's there's 2 two components that have been enterprise licensed.
Ashley Jeffs:Then what we've done is because it used to be that you would import all of the connectors sometimes as, like, a full suite. If you just wanna have, like, a custom build that has everything in it, all the plug ins, they used to be like a macro. And, obviously, now that there's licensed components, that's a bit dubious for people. So what I've done is I've separated out packages that are all the Apache licensed connectors or all including the enterprise ones, and you can pick 1. And I've got explicit licenses now.
Ashley Jeffs:So, at least now you're not like, now you can be a 100% sure if you're making a build, whether it's fully open source or not. I mean, I've I've never run a multi license project before, so it was a bit of a learning experience for me. But, yeah, we're trying to make it as easy as possible to to have custom build with whatever you want in there.
Shay Nehmad:What's the use case for, like, I get why you would want all the processors because I don't see them as, like, super heavy and you oh, suddenly, I have protobuf. Suddenly, I just I wanna start playing on the blah blank, whatever. But why would you need all the inputs and outputs? Like, I can't imagine a system that gets I don't want to imagine a system that has Cassandra, Discord, the Twitter, SQL, Mongo, and HDFS as the inputs.
Jonathan Hall:I just, like, please tell me my website.
Ashley Jeffs:So the I think the the the main reason is, so I've got a bit of insight into this. But, basically, when when you're a company that's big enough, you've got teams all over the world that have all grown independently. So they all they all use different tools and services. Some of them came in through acquisition. So they, you know, they they made all these decisions before they were part of the company.
Ashley Jeffs:And you need to have the ability for all of these teams to kind of collaborate. So these sorts of tools that connect all of these different technologies together are really important, and they're so important that you end up having a team internally that is in charge of it. So they will have their own, like, custom plug ins that they'll write for it, and they'll have their own, like, framework for how you're supposed to deploy this thing and, best practices, all that kind of stuff, how to do it securely, all these things that you should you should be doing. And, you know, we use this deployment mechanism, and you do it with this and blah blah blah. Here's some templates.
Ashley Jeffs:They'll have their own internal documentation, all this stuff. And for them, they just wanna have this, like, mega binary that is gonna solve all the use cases that they maintain so they only have one build. Mhmm. And they they won't necessarily even know what connectors they're gonna need when they're when they're shipping this thing out. So it's almost like they are, they're they're almost like a a Benthos distribution that's just internal.
Ashley Jeffs:And I know a lot of organizations that do that. Like, when when they eventually get to that size where it becomes this massive problem to solve, it it becomes really important that if you're giving people this tool to solve it themselves, you're giving them, you know, everything, which is it does seem like a lot because the binary size is like a 100 meg. A 100 megabytes. In the Java world, that's not a big deal, but in in the Go world, that's massive.
Shay Nehmad:Yeah. Wow. With all the dependencies on, like, I assume the core engine is, like, you know, kinda tiny.
Ashley Jeffs:Yeah. Yeah. I think I think it is. I think it's pretty small.
Shay Nehmad:So you mentioned It
Ashley Jeffs:is all the client libraries. Specifically, AWS is pretty big.
Shay Nehmad:I saw you have, Azure. I remember I really hope you don't have to ever do as part of, you know, a client to deal whatever the, Microsoft Graph API as an input. Because let me tell you, man. They just took the entire c sharp codebase, and I don't know what they did do it. They, like, transpiled it into Go, and their Go package is, like, 2 and a half gigs if you add it to your build.
Shay Nehmad:Something crazy. I remember it literally crashed our GitHub, pipelines last time we tried to, integrate it. And we ended up, like, not using the the SDK. We just, like, copied the files out of it. And they were, like, yeah.
Shay Nehmad:We're not really gonna maintain it, so sorry on GitHub. Speaking of stuff that's not gonna get maintained no. I'm just kidding. What happened? So so you're talking about Snowflake and Splunk being, like, 2 connectors.
Shay Nehmad:It talks about growing a team. Are there any other exciting features on the horizon now that you have I assume
Ashley Jeffs:Oh, yeah.
Shay Nehmad:Internal Red Panda integrations. I don't know. Is the studio, like, getting a front, like, you know, it's gonna be front and center, the UI stuff? Like, what's going on? What's the what what are Bento's, fans, should look forward to now that you they don't have to worry about you, you know, ending up on the street?
Ashley Jeffs:There's, so console is another product we got. It's open source, Red Panda console. That is already able to give you a an editor with autocomplete for Pentos configs.
Shay Nehmad:Damn.
Ashley Jeffs:So they've been they've been working on that at the same time that, you know, we were working on the, repo split. There's also tooling around. We're probably gonna take Studio and just, like, break it up into bits because there's lots of problems that I solved with Studio that will probably fit in different places better than than just having one central thing. But, yeah, I mean, we're definitely gonna start moving, some of that good stuff out into better kind of, like, tools that we can give people. But, I mean, the the console, editing experience is is is like, people have been asking me for autocomplete for years, and I I basically just put my hands up and said, I'm I can't do it.
Ashley Jeffs:I'm just not capable. I I don't know what tools I would need to do. Like, I I built a visual editor before I could muster the front end magic to get autocomplete to work. And yet the, you know, the console team, they they had it pretty much immediately. Like, they they were like, yeah.
Ashley Jeffs:Give us the bits and tap tap tap. They did some stuff on a laptop, which looked sus, and then suddenly they've got autocomplete in console. So, yeah, I mean, there's there's obviously gonna be, like, integrations into Red Panda, but there's also gonna be stuff. Because, I mean, you don't need console with, you can just use that. But, obviously, most most of the functionality that we're gonna be talking about is is stuff that's gonna be cool with Red Panda.
Ashley Jeffs:But the stuff that the open source community is gonna be interested in specifically is just more support on the core engine. I mean, I wouldn't be surprised if at some point I'll be doing performance improvements, stuff like that that I just haven't had time for. Lots of more connectors. Pull requests are finally gonna be free to to get merged. The the pull requests will flow once more Mhmm.
Ashley Jeffs:And just more more hands on deck, more stuff, more bits. Is that specific enough for you? There's gonna be stuff and bits.
Shay Nehmad:I'm really excited when you said performance because the the the moment you say, okay. My thing is stable enough. And when I say my thing, I mean, your business and both, like, both your business and your project are stable enough that I can, like, I just wanna come to work one day and profile it and make it better. Yeah. That that means that it's actually providing so much value for so many people that humanity has, like, okay, little funny man, sit in your box and make it better.
Shay Nehmad:Although, I don't know if that analogy is apt because I don't know if you're actually little. Like, all your I
Ashley Jeffs:think blog posts. I'm pretty small. I'm pretty small. You need to meet me at a summit. I think you'll tower over me.
Shay Nehmad:I'm a pretty tall guy. Yeah.
Ashley Jeffs:I like it that way. I don't like being in
Shay Nehmad:the middle. Tall Jonathan is, but I'm gonna find out in 2 weeks. Oh, come
Ashley Jeffs:to Amsterdam. Okay.
Shay Nehmad:Let's hang out. Well, actually I'm
Ashley Jeffs:actually not far.
Jonathan Hall:No. Not too far. Oh. What?
Shay Nehmad:Off the record, we're gonna we're gonna course you to to come. What's wrong?
Ashley Jeffs:What do you
Shay Nehmad:One last question about the, you know, Red Panda stuff. You, said you knew these guys, like, before, and they actually sponsored a project and and things like that. Like, where did you meet them? Just through Bentos, or, you know, did you go to high school with these guys?
Ashley Jeffs:No. I didn't go to school. The, the I mean, you you end up just kind of, like, mingling through, like, blogs and stuff. So I I remember when Red Panda was announced, publicly. It was it was when it was, like, fresh, and my immediate reaction was, I can finally run Kafka integration tests.
Ashley Jeffs:Because up until that point, it was like zookeeper, and it was an absolute nightmare to get it running in a in a, like, test framework without having, like, a a dedicated server for it, which I was not gonna do. And then Red Panda comes along, and it's like you could just run it. Like, you just run it quick, and it's it's low memory and all that stuff. So I immediately jumped on that as, oh my god. It's amazing.
Ashley Jeffs:And then I think, basically, they had some some users who were already using Benthos, and they obviously talked about that. And then Alex reached out to me, and it's the whole, you know, we'll do a blog together and post on our website, post on your website, and, you know, mix. And it's it's a good way of getting people to kind of discover new tools because up until that point, I I had basically no connections. That was about 5 years ago. I had no connections with anybody else in I said, that was probably 2020, I think, actually.
Ashley Jeffs:Not that long ago.
Shay Nehmad:I said,
Ashley Jeffs:oh, no. That's almost 5 years ago. Yeah. But back when I was a little and back when I was a little kiddo, yeah, I I didn't really have any, like, interaction with the the the bigger, more established tools because, you know, I was kind of like this organic developer driven. I didn't really have any, like, corporate presence.
Ashley Jeffs:Like, no nobody at Kafka Summit was talking about this blobfish project by this kid. So it was all it was all kind of, like, developer driven. But I think we we kind of, like, matched in our general marketing pitch because, you know, the whole pitch with Benfoss is it's just way simpler than the other tools, both to use and to run. And the the term that I was kind of using and still use to an extent today is the the operational simplicity. Is you've got this thing, you're running it, and it's it's simple in how it behaves.
Ashley Jeffs:Fewer hours on pages in the middle of the night, that sort of stuff. And they're obviously very, very similar because it's, you know, it's Kafka without the hassle. It's basically Red Panda. It's better performance and better behavior and, you know, easier to deploy and manage. So, yeah, we we just kind of started interacting based on that.
Ashley Jeffs:And, you know, it's it's the same with all these other, like, companies that, you know, you end up you end up interacting online, have a call just to say, like, hey. What's up? And then, you know, sometimes it's through VCs as well. Venture capitalists like to connect you with people. So that's that's like a whole, avenue for for getting contacts.
Shay Nehmad:I heard someone say that's the only thing they're good for.
Ashley Jeffs:No. Not me.
Jonathan Hall:No. No. No.
Ashley Jeffs:I wouldn't say that.
Jonathan Hall:I wanna ask one question that might be a little bit sensitive. The day after the announcement, warp stream announced that they had forked Pentos. I don't wanna go into too many details there that their announcement was, I don't know, a little bit adversarial. What are your thoughts? And I'll just leave it open like that.
Ashley Jeffs:So the like I said, that month was a bit wild. Obviously, I was hanging out with WarpStream. I still I I like all the WarpStream people, personally. They're in a difficult spot because so I think, essentially, the big the big thing that happened is people discovered that Kafka Connect is bad. It's like a it's like a product to sell.
Ashley Jeffs:So, like, lots of companies now are dropping support for it, and that used to be their sort of connector ecosystem. That was what they were selling to connect to other, tools. And it's difficult to justify how costly it is to maintain and run, for these businesses. So the the sort of Kafka ecosystem, you I used to have no presence there. And I didn't really have the resources to kind of, like, reach out because it's very it's very established, very corporate, lots of consultants kind of dictating and thought leaders who, you know, they're they're obviously very much tied to the JVM space.
Ashley Jeffs:So I kind of carved out my own space, and I wasn't really privy to any of that stuff. But in the last year or so, I think because there seems to be this sort of collapse in trust on Kafka Connect, which I think is a bit it's a bit over the top. Kafka Connect is is obviously fine for a lot of things. It's just difficult to to justify, for some of these companies. They're obviously looking at other tools, and, obviously, Benfils fits that bill really well.
Ashley Jeffs:And there's there's a lot of crossover. Obviously, there's lots of stuff that Calf Connect Connect does that Benfos doesn't, but there's there's enough overlap there that you can kind of prioritize it. So a lot of these companies were seeing it at the same time, and they're all they're all kind of, like, dabbling with it. So, WarpStream can you know, they're they're they're playing around with the idea of running Benfos configs in their product, and I I think it's scary because they're obviously, like, serious competitors. So they they obviously knew that, some sort of deal was happening, just because of all the signs.
Ashley Jeffs:It's pretty obvious. So I think they they kind of had it queued up ready to go. Because, I mean, they they announced that, like, the day after. Yeah. So the day after the the acquisition was public, they're on Hacker News talking about their fork.
Ashley Jeffs:Mhmm. I'm not anti fork. We I've specifically structured the repos right now to enable people to have forks of the connectors.
Shay Nehmad:Mhmm.
Ashley Jeffs:And the beauty of it is there's an MIT licensed core engine. And if you fork all the connectors and put whatever license you want on it and support it however you want, it will compile with that engine, and other people can pull components from either. So they they could mix and match any vendor's connectors. That was always the plan from from the technical perspective, but what they've chosen to do is do, like, a monolithic fork of the whole thing. So they they have a monolithic repo right now, which is just everything in one thing, which isn't decentralized.
Ashley Jeffs:So, I mean, I'm not sure what their plan is. But right now, basically, my main gripe is that that's not good for the community. Because, obviously, if the whole point of this fork is is for the community anti Red Panda, then, you know, at least make it decentralized so people can share plug ins. Because right now, they can't share plug ins between the 2 forks. But that's there's no there's no technical reason why that couldn't be the case, and it would rely on MIT license codes.
Ashley Jeffs:So there's not really any reason for it other than just where.
Shay Nehmad:I think the we had a Ohad on the show from env0 and I think from sort of the other side of the of the fort. Right? They, Infzero and some other companies, were the people who forked Terraform into, OpenTofu. Now this is obviously not the same because, Terraform was monolithically open source and then monolithically source available, and this is a lot more open sourcey than that. Like, Red Panda Connect is a lot more open source than, Terraform is right now, for sure.
Shay Nehmad:But I think the the reaction, makes sense on both sides, and it it I think it does hurt the community in the short term, these sorts of decisions. Even Open TOEFL and and Terraform, it's like, we had all these people working on this beautiful thing together, and now they're separated.
Ashley Jeffs:Yeah.
Shay Nehmad:But I think, like, with Open Tofu, we're seeing, you know, in a few we still need to let it cook for, like, a year or 2 until we finally see see the results. But overall, I think it sort of stresses out, the core developers, like the actual people, like you said, in the gnarly bits to make sure that their side of the fork is really good. Basically, competition 101. Yeah.
Ashley Jeffs:Competition.
Shay Nehmad:It's and in the long term, it's not like these two projects are gonna live side by side for 15 years now, and, you know, we're we're gonna invest a ton of time into both of them. It's gonna get lost. It's not it's not the case. Right? So I think I sort of agree with you that it's kinda mad just because I saw the, announcement.
Shay Nehmad:I was happy for you. Then I read the blog post, like, oh, no. Is Ash a shill for has he been a shill for Red Panda for the last 10 years even though Red Panda's only been out for 6? Like, but I I overall, I really hope that this will end up being they do a fork when the Red Panda Connects project ends up being slightly better for it, you know, as an educational experience. And I hope, both companies end up succeeding in their own niche.
Ashley Jeffs:Yeah. Exactly. It's it's difficult because it's like you've got the competition side, The fragmenting of the sort of ecosystem around it was probably gonna happen anyway because I think that there was, like what I predicted was gonna happen was people who were just they they cannot tolerate, a company running a project, especially one that has to make money and is part of the capitalist system. So it's like they they have a a problem with that, and, you know, I I was expecting people to have kind of, like, community forks. And they're they're kind of there to keep the project honest.
Shay Nehmad:You know
Ashley Jeffs:what I mean? It's like it's not really that we're gonna diverge. We're not gonna do a bunch of changes that means we've completely diverged from your other project, but it's more just we're gonna keep this thing here, and it's gonna be known in the community as, like, you know, the the project x. It's just some extra thing that people are aware of. And the idea is that if Red Panda ever became bad stewards, then this is the thing that people could just flock to.
Ashley Jeffs:Whereas, because there's there's a big competition aspect between these two companies, I think it's it's it's a bit sad that they're kind of like they're the only dominating forces right now in the Forks space. I was kinda hoping it would be a community fork that made technical sense, rather than kind of like a panic fork. Like, it it just feel a little bit like it was like a panic maneuver, that's not necessarily well thought through. I mean, they've got load like, it it's obviously the first week. It's the 1st week of the acquisition, but it's also the 1st week of their fork.
Ashley Jeffs:So, I mean, they might they might change things so that there is more, shared code base, but I I don't know. I I don't really know what to to expect at this point.
Shay Nehmad:They might not knock you out of the park with features.
Ashley Jeffs:Well, yeah. Like, obviously, I can I can just keep slapping on features and pull requests and stuff, which is obviously the intention anyway? But the the thing I would personally be scared of, as the you know, this is kind of like my baby is is the idea of just willy nilly stripping out stuff from the engine in the name of, you know, benchmark numbers and just rolling out features from, like, a competitive standpoint and basically racing to the bottom of you you essentially eliminating guarantees. Because there's there's stuff in the Benthos engine that I know is critical for the 2 AM pages. So, like, the delivery guarantees that Bentos has is completely unnecessary for a product that you're putting out there and giving to people.
Ashley Jeffs:Because if you have weak delivery guarantees, well, you're just one of many products that people run that have weak delivery guarantees where the user won't realize. Because it's not an you know, it's a 1 in a 3 year event where you'll be hit by that stuff. And, obviously, the core engine of Bentos is specifically designed to be the one that doesn't have that. And, you know, this is a it's a technical decision. It's very, very difficult to get that balance right between, you know, back pressures, delivery guarantees, speed, scaling, all that kind of stuff.
Ashley Jeffs:So my worry is, you know, fork city happens. Everybody's got a fork. They're all stripping the engine out in weird, incompatible ways, and then we end up with this, like, race to the bottom because that that's what Bentos wasn't. Like, Bentos wasn't the tool that was kind of pressured to, sacrifice those things in the name of, you know, delivering all these things. I'm optimistic because I don't think the engine really needs to change that much.
Ashley Jeffs:It's quite stable now, but that does that does worry me. But, you know, I'll just leave and, you know, 10:10 years on, I'll just be doing something else when I say I don't know. Mhmm. I'm hoping to make myself, unnecessary in the project anyways.
Jonathan Hall:Yeah. Right.
Ashley Jeffs:Maybe it'll work out.
Shay Nehmad:So, Ash, you're actually, the first repeat interviewee that we have, that's gonna get both questions, stumper questions. Last year our stumper question was, what feature would you reap out from Go, and what, feature would you steal from another language? When we ended up with a full feature list on both columns, we were like, okay. We need to rephrase this. And I'm actually interested to know when you started learning Go, what was difficult or surprising for you?
Shay Nehmad:Go was your wasn't your first language. Right?
Ashley Jeffs:No. So I was I was a c plus plus kid. Did a bit of c sharp, and then went back to c plus plus, and then Go. And that was pretty much it from that point onwards. It was about I think it was about 2016.
Ashley Jeffs:No. It's actually must have been early. No. It was probably 2014, Maybe 2013 I started using Go. You can look at because it so that library I've got, Tunny, that's pretty much I made that immediately when I started learning Go.
Ashley Jeffs:Because the one of the first things I noticed was, hang on a minute. How is there no thread pool? Like, how is there no goroutine pool here? Because I was just expecting that in the standard library, and I was very surprised, so I made one. And it wasn't until I made it that I realized, oh, actually, this is very unnecessary because, you know, with with channels and coroutines, it's not it's it's not difficult to reimplement that when you need it.
Ashley Jeffs:And you normally, you know, you normally want to reimplement it.
Shay Nehmad:1st commit, March 23, 2015, but the copyright is 2014.
Ashley Jeffs:That's probably I probably,
Shay Nehmad:Copied it
Ashley Jeffs:over from somewhere else. Rebased it or something? Yeah. Yeah. 2014 is probably about maybe the end of 2014.
Ashley Jeffs:Let's find this out. We need this right now.
Jonathan Hall:This is not this
Shay Nehmad:is not that good of a code, I have to say, like, first, first commit from 2015. So, when you came in to go, what was, like, difficult for you to to grok and and wrap your head around? Or maybe what was surprising for you?
Ashley Jeffs:The thing I didn't like was, lack of some slash enums slash union types, whatever people want. Like, depending on your language is what you'll know it has. But it was it was that because, I wanted to do JSON parsing, and I found it really weird that it it has to pull things out into, you know, generic maps and generic arrays rather than, like, a some type where you can make it more explicit because that's just what I was used to. And then I found it really weird that, like, if you've got a generic map, why is there no syntax for just walking it? That blew my mind because it's like, this seems like everybody's gonna hit this.
Ashley Jeffs:Right? Everybody's gonna have a JSON API. Everyone's gonna have, like, a poorly defined structure because that that was my world. I I just assumed everybody had that problem. So I was I was shocked and devastated when I found out there's no, like, there's nothing there to kind of help you with that.
Ashley Jeffs:But, obviously, like I discovered, you could just, you know, you can just slap a library together. It's not it's not a big deal. You can just do that yourself. Just get get it. Pull yourself up by
Shay Nehmad:the bootstraps.
Ashley Jeffs:Yeah. Make yourself a library for it.
Shay Nehmad:Write another for
Ashley Jeffs:it. Generics.
Jonathan Hall:I think a lot of that's finally coming in 1.23.
Shay Nehmad:Yeah. If you're, the new ASH joining, you know, to go right now 10 years later, you'll get the Range Rover funk and all this, generics nonsense, and you can build
Ashley Jeffs:it up. Nonsense.
Shay Nehmad:I don't know. Russ Russ needs to keep himself busy somehow. You know? You need another connector on top of your 220 connectors, and Russ needs a seek 3 iteration.
Ashley Jeffs:I'm moving on to Zig.
Jonathan Hall:There we go. That'll solve all the problems. Yep.
Shay Nehmad:Ash, thanks all for coming, man. Again, Thanks
Jonathan Hall:for taking our call.
Shay Nehmad:Yeah. For taking our call.
Ashley Jeffs:Never again.
Shay Nehmad:Not on this number.
Jonathan Hall:I'll I'll burn this on after we hang up.
Ashley Jeffs:I've only had this SIM for a week.
Shay Nehmad:Again, I really wanna congratulate you on this, you know, we can dive into the technical details and and what happened, what's gonna happen, whatever. But I think a congratulation is in order. You mentioned somewhere doing this interview, an end of a journey, and I really, really feel, this marks a big, stone in your journey. I don't know if it's the end of it, but it's a really, really cool one. And I think it's a really big success story for open source and people who work on open source and, you know, have some sponsors and and are thinking about long term what they can do, can really look into your just tireless work on this well, I don't know if tireless, but very
Ashley Jeffs:I was very lazy at times.
Shay Nehmad:Yeah. Yeah. But I mean, like, relentless, dude. Working on the same project with the same, tone for many, many years, and having just this recognition. By the way, both the fork people and red red panda people and just random people on Hacker News, every nobody's like, yeah, but maybe the engine's not good.
Shay Nehmad:Everyone no. Asher's code is pretty great. Right?
Ashley Jeffs:And the logo's good. Everybody unanimously loves the logo.
Shay Nehmad:Yeah. For sure.
Ashley Jeffs:More so than the engine.
Shay Nehmad:Well, it's less gnarly. You know what I mean? It's cuter. So we just really wanna congratulate you, man.
Ashley Jeffs:Well, thank you for having me on. It's been great.
Shay Nehmad:Do it again next week
Jonathan Hall:to solve a business.
Shay Nehmad:Yeah. Next next question.
Ashley Jeffs:No. Thank you. I think I'm done.
Shay Nehmad:Alright. Thanks a lot, Ash.