Subscribe
Share
Share
Embed
Screaming in the Cloud with Corey Quinn features conversations with domain experts in the world of Cloud Computing. Topics discussed include AWS, GCP, Azure, Oracle Cloud, and the "why" behind how businesses are coming to think about the Cloud.
SITC-Hunter Leath-002
===
Hunter Leath: That was the goal of ours was to be in a position to offer this experience like EFS without having to charge the prices. That I know so many customers are frustrated with from a thing like EFS.
Corey Quinn: Welcome to Screaming in the Cloud. I'm Cory Quinn. I'm joined today by Hunter Leath, the CEO over at Archil. Uh, I get a. Bunch of questions periodically from folks who are building something new in the AWS ecosystem, and they want me to take a look at it. And I, I generally turn most of them down because I'm a skeptic.
But for whatever reason, hunter caught me on a good day and I looked at it and I really liked what it was he was building. Hunter, thank you for joining me and suffering my slings and arrows now public version.
Hunter Leath: Thanks, Corey. Excited to be here.
Corey Quinn: This episode is sponsored in part by my day job Duck. Bill, do you have a horrifying AWS bill?
That can mean a lot of things. Predicting what it's going to be, determining what it should be, negotiating your next long-term contract with AWS, or just figuring out why it increasingly resembles of. Phone number, but nobody seems to quite know why that is. To learn more, visit duck bill hq.com. Remember, you can't duck the duck bill.
Bill, which my CEO reliably informs me is absolutely not our slogan.
So I can do a piss week job of explaining and mispronouncing almost anything. But in the interest of time, why don't you describe what it is you're building.
Hunter Leath: Sure. So our company Archil, is transforming S3 buckets into Infinite Post six compatible file systems that provide instant access to massive data sets.
So you can run anything directly on S3.
Corey Quinn: Well jokes on you. If you improperly mount things and start using it as a file system, then people have been doing that since S3 launched and then they led to tears before bedtime when request charges were levied, uh, starting in late beta around a lot of that just because it caused a lot of problem for the front end stuff.
Uh, AWS has also come out with their Mount point open source nonsense. So, you know, after 15 years of, sorry, 20 years of telling people, oh, S3 is not a file system. Don't use it that way. They trot out and here's how to use it as a. File system because someone is trying to see if some customer somewhere is gonna snap and just go after them at reinvent.
Hunter Leath: Yeah. I mean, I don't know. You, you may have seen a year ago we posted on the AWS subreddit, uh, talking about the solution and almost all of the top comments were S3 is not a file system. This is a terrible idea. I would never recommend this to my customers, which I love. I love that kind of feedback because I think what we've done is we found a really unique approach to solve this problem.
That hasn't been done before by any of these fuse solutions in the past.
Corey Quinn: I, I, I wanna point out one other thing because this, this does sound to the uninitiated, given things I've said before, like I've somehow pivoted into becoming a credulous moron because a lot of people have tried to take a bite at this apple.
But one thing that makes you a little bit different is you spent eight years at Amazon working on the elastic file system. EFS product, which generally you can say a lot about Amazon, they don't tend to hire stupid and they certainly don't suffer it for the better part of a decade. You clearly understand the space.
You've worked in it, you know where the bodies are more or less buried here, and you think you see something here. That alone gets my interest and my attention more so than anyone starting it like, well, we're gonna build this thing in a product space we've never actually worked in. How hard could it really be?
Uh, and for some reason that pitch, if it uses the word AI and it somewhere raises a $30 million seed round.
Hunter Leath: Yeah, I, I mean, I think that's. Right, and I really enjoyed my time at AWS on EFS, and it taught me all kinds of things about running at scale storage solutions. I think I went home every day for eight years thinking about the problems associated with EFS and performance and cost and how to optimize storage placement.
And eventually came to this realization that what EFS does as this. Scale out file system that you can dump things in is probably the right shape for a lot of customers. If we could add that synchronization to S3 and the performance that they expect, which we think that we can bring to the table, and so rather than being just a fused library, like you might see S3 Fs or goofy Fs from the past,
Corey Quinn: you're going through user space in those cases and the latency hit is wildly unacceptable for an awful lot of things.
Hunter Leath: That's right. We are running this middle layer of NVME devices, which allows us to provide SSD, like EDS, like access to the data that's already there for our customers. So they get a file system like experience.
Corey Quinn: Uh, this can be done. I wanna be clear that back in 2008 at Reach Local, we ran the entire site on a MySQL database that lived on a NetApp volume.
This can be done over NFS. There's a lot of dialing in and tweaking and care you have to take. But if we were able to do it back then, we, this isn't knowledge of the ancients that has somehow been lost. You can run these highly critical transactional workloads on something over a wire that is, that is no longer in.
Doubt. I found that when EFS launched, it had some challenges. I was very vocal about them, talked to several people on the team. As I recall, they were very large at reinvent, had no neck, and I thought, well, this is the end of me. But the question was, tell us more about the use case. And over time, it's become one of my favorite services.
I I, for a while, I was running my home directory on my EC2 box out of EFS. I had to stop doing that because it took forever to instantiate a new uh, shell system. Turns out that every time it read my ZSH profile, it was making 1500 discreet read operations against disc. And maybe that's not the best use case over the wire, but that's also 20 years of legacy Croft in my case that I had to worry about.
So this is in AWS, this is living on prem, this is in other cloud environments. Where where is this thing based? You said S3, so I have some guesses.
Hunter Leath: We today have shared clusters that we sell to customers, similar to how S3 or EFS or any of these services work in a couple of regions in AWS and a single region In GCP, we're also working with customers for some on-premise deployments, but we want to be where the compute is that uses the data to reduce that time on the wire.
Corey Quinn: Right. The latency issue for anything that is being used by, as you said, presents as an NVME device, that means you've gotta be hooking the kernel somehow, uh, locally on the system, and there has to be a cache. You can't be going to S3 for every read and every rite. The, the, the request charges alone would make this a complete financial non-starter.
What's the overall architecture look like?
Hunter Leath: Yeah, so what we do is we are running a fleet of servers that have the SSD instances attached to them. Similar to, there's like a lot in the news about planet scale, planet scale metal, very similar thing that we're doing here, and those servers provide this distributed.
Durable cache for data that's going into or out of S3. And what that allows us to do is provide a very easy on-ramp for people who want to use S3 as a file system or use it as their backing store, but then also give customers this ability to tell us, well, maybe we actually only want this data to be stored on SSDs for performance or cost reasons.
And we give that flexibility to our users.
Corey Quinn: Now the right way a lot of these workloads have been done is you teach your workloads how to use object store and teach at those semantics and it grabs whatever it needs and life goes on. That is a wonderful end state, but there's a universe of applications out there that flat out are not built that way.
Where, oh, just teach your 30-year-old application how object store works. Oh, terrific. After you please, if it's not that hard, I like. Due to meet my mainframe. There are a bunch of legacy workloads. I can see where this is incredibly helpful. Do you see modern workloads taking advantage as well?
Hunter Leath: I agree. I think it's a combination and, and I've had conversations with people in the past where I kind of like correct the legacy term because in many of these fields it's, it's not even that these applications are necessarily old.
Like if you look at the HPC and academic fields, these are new things that are being written that are the best in class for what they do. But are built against file systems because most universities and labs are running on some kind of shared cluster mainframe thing with a file system sitting on the side.
And so I agree that we provide an opportunity to take this vast library of software that exists today and actually connect it to the cloud in a cloud native way so that the data could be used by. Lamb from someone on a different team. But I also think that we are going to see a resurgence in applications that are built.
In a modern way, four file systems, and my belief is that people have shifted development to using object storage natively because nothing like Archil has existed. And traditionally, the limits of the file system have been so painful for people. If you've run a NFS share and you know, run out of space or run out of iops or gotten paged at 4:00 AM it leaves this bad taste in your mouth about the entire technology.
Where it turns out, we can build file systems that scale in a very similar way to S3 and provide a very similar experience.
Corey Quinn: The S3 bucket that's backing this has to be, have the data laid down in it in a particular format that you folks, uh, uh, work with, or it can be hooked up to existing large data sets.
Hunter Leath: It's the latter, and that's what we are super excited about, is that if you are a user who has. An immense quantity of geospatial data or video data, or any of these data sets that you might need to process. We work with that data directly and we synchronize in both directions, so anything that you put into our file system then gets replicated into S3 and its original format too.
And you get to own that data.
Corey Quinn: You, you talk about, uh, shareable across multiple instances simultaneously for datasets. Um, I, I have to imagine that there are some race condition and concerning limits here. You, to use my, my insane example, you can't necessarily, I would think, have a MySQL database living in S3 that you then wind up pulling out, having multiple things, commit transactions to that thing, and then put it back and hope for the best.
I, but how does the, how does the fencing work?
Hunter Leath: That's also a very good question and something that stumped us early on around how we actually extend mutability to multiple instances. But we, we started from this premise that many customers who are running workloads in the cloud. We'll pick something like EBS as an easy starting point to store their data and then only move to shared storage when they find out that for high availability reasons, all the data can't be attached to a single instance, for example, and we want to build almost a version of EBS that doesn't come with any performance drawbacks, but allows you to have this sharing across instances in a safe way.
Has the synchronization to S3 and does the auto-scaling and the pay per usage that people like from services like S3 and EFS. But ultimately the application ends up being the one responsible for coordinating rights to that data. So we would not recommend that if you launch a MySQL database, you then attach multiple instances to that database and write to it simultaneously.
Because MySQL isn't designed for that.
Corey Quinn: Well, not with that attitude. Sorry. Please continue. I, I misuse databases because I have problems that I make everyone else's.
Hunter Leath: You can do it once, right? Like many things, you can do it. One time.
Corey Quinn: It worked in, it worked in my laptop, so off to production with it. Why not?
That's why I call my staging environment theory works. In theory, not in production,
Hunter Leath: but for many workloads that are maybe producing files or like transcoding videos and have a multi-stage pipeline where different instances are handing off that video file. Being able to share that data across instances is very helpful.
Corey Quinn: Oh, I can absolutely see where that, that makes sense. The, the pricing, at least as on your website, is also compelling. Uh, does the data live in the user's S3 bucket? Does it live in yours? Where is the, where does that. Live, for lack of a better term.
Hunter Leath: Yeah, so we attach to customers S3 buckets directly, so we use the data that's already in their bucket.
We synchronize any changes back to their bucket and they get to keep that data. We're working to simplify onboarding so customers can try us without maybe making an S3 bucket or attaching us to their Productionist three bucket. And so we're actually launching the ability to use a bucket that we manage.
Uh, so that you don't have to worry about any of that if you're not coming in with data.
Corey Quinn: Something that strikes me about this is your pricing. The first thing I always look for on a website is the, it's the pricing page, because that tells me who is this actually for? And you generally want two things.
One is the get started right now because it's two in the morning and I have a problem. And if it's call here to contact us, you're not going to space today. And the other is the enterprise end of it, which is contact us. We don't. Put our data with random websites because we are a Fortune 500 and strive to act like one and whatever's in between them almost doesn't matter.
And you have two options right now. One is the enterprise option and the other is the developer plan at 20 cents a month. And that is a very resonant figure for me because 10 cents a month gets you single availability zone EFS at. On demand prices and 30 cents a month gets you multi a z uh, EFS, you're smack dab in the middle of them with a better durability story than EFS itself offers.
And even with having to store the original data on, in S3 standard as well, which is another two and a half cents, you are, we're still talking that this is less expensive for active data than using EFS off the shelf.
Hunter Leath: Yeah. And that, that was the goal of ours was to. Be in a position to offer this experience like EFS without having to charge the prices that I know so many customers are frustrated with from a thing like EFS.
And I think too, if you, if you zoom out further. And you talk to users of AWS who are not familiar with EFS. What they compare it to is EBS pricing. And when they see 20 cents a gigabyte, it looks extremely large compared to the 8 cents a gigabyte that you get for provisioned GP3 volume. But we've actually run the math, and if you do an apples to apples comparison of taking data that was previously on EBS and moving it to our service, the pricing actually clocks in at something like 1.95 cents.
Per provision gigabyte that you would have had on EBS.
Corey Quinn: Right? Because you have to over, you have to over-provision an EBS volume because only a lunatic runs at a hundred percent, uh, utilization on it. You have to monitor that and care about it. And people from the EFS side will say, well, if you're not using the data, we have intelligent tiering, which makes it a lot less per gigabyte.
Yeah. You only charge per active gigabyte in the course of a month there at, I believe, a 30 day cycle. So yeah, you, you drop to zero. Where they drop down to, I forget the exact number, but it's, it is compelling. The economics alone are fascinating. All you have to worry about after that is okay. Does, does the thing actually work for a given use case?
Hunter Leath: Yeah, that's right. And it is rare, I think, to find these kinds of optimizations and infrastructure where you can both save people money. And make their software faster. And we think that this is one of those examples where that's possible, where if you're moving from something like an All SSD, an EBS or an EFS deployment, we can save you significant money on top of that existing deployment.
And if you're moving from something like using S3 directly or downloading a zip file and then zipping it, we can make that workload faster because we're actually keeping some of the data on SSDs. But yes, I think we. Plan to do a better job over the coming months of publishing more benchmarks and, and more customers who are using the service to highlight the applications that work well with Archil so that people know that it's an easy thing to adopt
Corey Quinn: if the pull through cash, more or less, with really great performance characteristics.
So a concern I have historically, I did a piece back in 20 19, 20 18, about how to compete with Amazon and. One of 'em obviously is good developer experience because that is something that has not gone super well for them, uh, in recent years compared to a number of other folks, but also to find things that aren't on the same rails that they're innovating on.
This feels like it could be a story of a feature that AWS puts out at some point, and at that if they wind up effectively. Fronting S3 with an EFS, like cash or something like that. How does that change your competitive positioning? I mean, I have to imagine this is come up as an idea.
This episode is sponsored by my own company, duck Bill.
Having trouble with your AWS bill, perhaps it's time to renegotiate a contract with them. Maybe you're just wondering how to predict what's going on in the wide world of AWS. Well, that's where Duck Bill comes to help. Remember, you can't duck the duck bill. Bill, which I am reliably informed by my business partner is absolutely not our motto.
To learn more, visit doc bill hq.com.
Hunter Leath: Honestly, Corey, I, I hope they do. I have many friends still who are on the EFS service, and I would be ecstatic if they were able to build something kind of as exciting as synchronizing to S3 because I think it's important for customers. But I think that what we are building goes beyond that, which is that AWS, at least from my experience, is very focused on how to build building blocks and give them to customers.
And I think even. Peter DeSantis said, uh, reinvent, uh, several years ago, said something to the effect of we will not build frameworks. We want customers to build frameworks. And so our ability to help our customers is going to be based around how we tie things tightly together so that they don't have to cobble a bunch of AWS services in order to get the same solution.
And the way that we do that is through a combination of performance work. Our goal is actually not to be as fast as EFS. It's to be as fast as EBS so that customers can replace EBS volumes with us, which we think is an entirely new market segment that Amazon won't be able to capture.
Corey Quinn: When you say Eeb, can I compete with EBS?
Does that. Compete with the bring money IO2 volumes as well, performance wise. 'cause every time someone talks to me about those, it's like, oh great. Uh, and everything they say after that turns into, and here's how you build your own sand from Popsicle sticks in the cloud. Which great. Not really what I wanna do.
Hunter Leath: Yeah. I think in time we will. Uh, for now we see us as like a very good alternative to GP3, which are where we expect the majority of workloads, which are not transactional database workloads running, where if you're doing video transcoding, you're probably using a GP3 volume to store that data and not like a finely tuned IO2 volume.
Corey Quinn: Oh yeah. Every time I see an IO2 volume, I have many questions, uh, most of which are the answers become, uh, maybe you shouldn't be using this volume for this use case. I also, normally, I tend to be relatively down on the idea of AWS trying to compete with someone else who's making money somewhere because their developer experience is crappy.
But the, the lockin halo effect of the ecosystem does become challenging. Um, especially if we take a look at something on the storage side as an easy example where. Staying first party, even if it's crappier from an experience, is a lot more justifiable for folks than bringing something into critical path with something as fundamental as disc access.
Hunter Leath: You had mentioned to me at some point previously that uh, the file system is not the place where you want to be betting on new technology, and I think that's absolutely true. You know, these, these things are not developed frequently because there is so much. Thought and care and safety that needs to go into these critical components.
And so I think that as a business, we are going to have to build our way up to some of the enterprises that would otherwise be happy to pick an AWS solution. But if you look at the vast, at just the L. Of storage workloads that are out there. There are lots of startups who are interested or need to capture these marginal cost savings or marginal performance savings in order to win in their product category where we can start.
And there are also lots of enterprises who have enormous shared caches for things like. Docker images or CICD artifacts where if things did go wrong, it's not a huge loss for them. And so being able to build our customer base from these easier to win people who are more open to tech, to new technology and this place workloads and work our way up to an established member of the storage community like A ZFS or an XFS, is the path that I think we're going to take.
Corey Quinn: One question I have because you have, uh, you announced at the top of your website that you've raised a $6.7 million seed round. Congratulations. Uh, but what I don't see on this, in, at least above the fold, and as I scroll down, I don't see it here either. I see no mention of ai. So first is that legal? It toward the end of 2025 to raise money without an AI story front and center sucking all the oxygen out of the room.
So what is your AI story?
Hunter Leath: Yeah, Corey, I can't believe you've brought this up. And my investors are gonna find out that we, we've not placed AI above the fold. I, I think that. What is exciting about our technology is that it is so horizontal and broad that it's applicable to all of these workloads that exist today.
Like we talked about video transcoding, CICD, geospatial, these things that were in the world prior to 2025. But I also think that there's a lot of interesting things happening in the AI space that's going to intersect with us. And what I hear most frequently from customers is that these models are being.
Trained on these data sets that include an immense amount of like Unix tool usage and file system usage. And so there are many companies out there that are trying to connect LLMs to, you know, slack or Salesforce through these purpose built tools. But the models actually perform better if you're able to expose data in a file system that the model can just cre.
Corey Quinn: Oh yeah, the, the command line tools are the best MCP out there. It can do basically anything you need it to do, and that knowledge of how to use these things is baked into the foundation model itself.
Hunter Leath: And so I think it's my hope that we can become. As the file system was originally intended to be this universal access layer for data, no matter where it lives, and then allowing AI models, applications, agents, what have you, to use our system to access that data is going to allow them to be more efficient and more productive than they otherwise would be.
Corey Quinn: When you tell the story of what you're building to folks, what are some of those common misunderstandings you see?
Hunter Leath: Our product is complex as, as you can probably tell from the conversation that we've had.
Corey Quinn: It's one of those things that feels extraordinarily simple and never is.
Hunter Leath: So we have folks who come in and aren't sure if we are here to save them money on their S3 bill, which would be great, but of course we are not, and they're not sure necessarily.
Corey Quinn: Technically, if you're treating it like a file system and just have her in the crap out of request, yes, yes, we will, but you're doing something wrong that's right up there with help cut your AWS, uh, by bill by 98% by no longer checking your credentials into GitHub.
Hunter Leath: Yeah, yeah, exactly. And so I, I think that we have this, this narrow focus today.
On how we can help people who are either working in the AI space like we talked about, or have these applications which are built around file system semantics and connecting those applications to a data lake or downstream object storage native systems where we will win ultimately.
Corey Quinn: I, I think that that is probably a, it's a user education problem.
There's, which is kind of a, a testament to how reliable storage has gotten across the board. This is something that it, it takes a relatively narrow subset of the industry to actually even think or care about these things, but those who do really care a lot. Bout it, it, it's also kind of wild that we've watched EC2 alone get to a point where we can trust it now for things of this level of latency requirement as well as durability, the world has changed.
Hunter Leath: Yeah. I, and there's like a tremendous amount of engineering effort and, uh, work that goes into making EC2 a durable service for sure. But it is funny that you say that about people caring. Uh, because a week or two ago, I think I reached out to the CTO of a company doing 200, $300 million in annual revenue.
And he picked up the phone immediately and told me that this is a problem that he's been thinking about for six years. Because for, for some segment of storage workloads, they're just woefully underserved by the offerings today. Whether it's an Amazon offering or a pure WCA vast, uh, NetApp, none of these things have the right latency and throughput characteristics for all workloads, and that's why we see such a vast array of storage options that you can buy.
Corey Quinn: It's, I, I can't find it at the moment, but there, the EFS site used to say with their intelligent tiering offering, uh, what percentage of data was, uh, there but not accessed in the course of a calendar month. And it was something north of 80% in the common case, which is effectively some of the best marketing you could do for this.
The challenge is, is great, but. I do need to access some of it. I never really know which one, and it has to be capable of being reached via a file system. Semantic. Well, this sounds like a relatively great way to get started. It does not sound from the try here to get started. Uh, documentation you have that it's a particularly heavy lift to integrate into a test environment.
This is, it's, I wouldn't, I didn't expect to see a lot of innovation in the block storage world in the year of our Lord 2025. And yet here we are.
Hunter Leath: Yeah. And, and that's what I, I go around telling people is funny is that we're in San Francisco doing a startup in 2025 and we're building a file system.
Corey Quinn: Sorry.
It's a file level semantics and not, uh, block level semantics. I, I could, it feels like it's right on the cusp.
Hunter Leath: Yes. I, I think that you are correct. In that one of these learnings that I had being focused on EFS and talking to customers is that. Many people don't view the file system as a place where they want their data to live long term or, or they don't like the idea of having to pick, if I'm going to store this data on a NetApp file system forever and lock it away from the rest of the ecosystem that's evolving around object storage in the cloud.
And really what, what we believe people want is this ability to use the file system to access that data. And whenever they're using the data, it makes sense to use it through a file system, but when they're not using it, they want it to just live in S3. So it can be like any other piece of data that they're keeping track of.
Corey Quinn: I I do have to ask, what is the response? If some, if a piece of data lives in S3 via intelligent tiering, for example, and it's been aged out into some of the, uh, archive tier where it will take some time, often measured with the calendar in order to come back. Uh, does it just blow itself the chunks? Does it just read a file, read error in some way that gets surfaced to the file system?
That this is not a catastrophic event, but try later?
Hunter Leath: Yeah. I, today what we do is we just. Basically wait, which obviously isn't ideal for a lot of applications, but I think this is one of those give and takes with file systems that we mentioned earlier. There's not an easy way for us to signal via Linux that some piece of data is maybe there but not immediately accessible.
And obviously I think we're going to spend time with our customers to understand as we hit people who are using these Glacier Deep Archive kind of storage classes, what kind of experience they would prefer. Then surface that to them in a usable way.
Corey Quinn: Do you present as a custom file system? Are you, uh, presenting as EXG something else?
Hunter Leath: We are a custom file system. We are based on, we run infuse right now, but we're working our way into the kernel over time.
Corey Quinn: Excellent. Uh, via module or are you dreams of one day being mainlined?
Hunter Leath: I absolutely, I think that we should be mainlined. I think that we see development as this iterative step.
Towards stability and what Fuse offers us is an ability to develop very rapidly and try different things because we're not messing with the kernel that's going to eventually stabilize and we'll move to a module and then, uh, hopefully. Like, uh, as that stabilizes further and the Linux community becomes used to us, then we would love to be mainlined.
Even.
Corey Quinn: I imagine you would have to open source a fair bit of that. It's like, well, except for this file system thing, that's a giant binary blo. Yeah. I, I see people having apoplectic fits already just at the notion.
Hunter Leath: Yeah. And we'll, absolutely, like when we get to the point where it makes sense for us to be in the mainline kernel, we will absolutely open source the client.
Corey Quinn: I look forward to it. I think this is going to be an interesting company to watch a d interesting space to watch, which again, I did not see myself saying at this late date.
Hunter Leath: It is very interesting, I think, how much innovation is going to happen in the storage layer in the next 10 years, and I hope that we are an important part of that.
But you'll see with time a lot of these AI workloads and the massive data centers of inference and training that open AI and Core Reef and Lambda Labs are building out. That people desperately need a faster, better way to get data into these GPUs, and so I expect that we will not be the only people in this space in the years to come.
Corey Quinn: I really wanna thank you for taking the time to speak with me. If people wanna learn more, where's the best place for them to go?
Hunter Leath: So the website is Archil.com, A-R-C-H-I-L, and then you can also find us on Twitter at Archil data or myself at JH Le L-E-H-T-H on Twitter,
Corey Quinn: and we will of course put links to that in the show notes.
Thank you so much for your time today, I appreciate it.
Hunter Leath: Thank you, Corey.
Corey Quinn: Hunter Leath, CEO at Archil. I'm Cloud economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please, we have a five star review on your podcast platform of choice, whereas if you hated this podcast, please, we have a five star review on your podcast platform of choice, along with escaping comment, tearing apart the idea of this new file system without bothering to mention that your favorite file system was ripped out of the kernel after its creator killed his wife.