EAS-198
===
Overview of Cultured Code and Things App
---
[00:00:00]
Leo Dion (host): Welcome to another episode of Empower Apps. I'm your host, Leo Dion. Today I'll be talking with Vojtěch and Werner from Cultured Code, the makers of things. Gentlemen, thank you so much for joining me for today's episode.
Werner Jainek (guest): Thanks for having us.
Leo Dion (host): Yeah. Before we begin, I'll let you gentlemen introduce yourself.
Werner Jainek (guest): Okay, so I'm Werner. I'm, I'm the one of the co-founders and the CEO of cultured code.
Vojtěch Rylko (guest): I am Vojtěch Rylko, but you can call me Vojtěch and I'm software engineer responsible for things cloud.
Leo Dion (host): Awesome. So today we're gonna talk about your story moving over to Server Site Swift. Before we do that, you introduced yourself. Maybe you wanna introduce things and cultured code and the story of your company.
Werner Jainek (guest): Sounds great. Yeah, so currently we're a team of 11 people and we're spread all over the world. A fully remote company wasn't always [00:01:00] like that. We have headquarters here in Stuttgart, in Germany, that's in the southern part. But my, by now there's only three of us going there. The rest is all over the place.
So we're fully doing that. Yeah. And for the past 18 years now.
Leo Dion (host): Whew.
Werner Jainek (guest): We've been working on exactly one product and it's called things and it's a delightful to-do app that you can use on all the Apple platforms. We're on all of them. And I guess some of the audience members might have heard of it or maybe even tried it already.
We've been there from the beginning. The iOS app was there as one of the first 500 apps in the app store when the app store launched on iOS. And and then we've continued that, like when the iPad came out, we were there on day one, and then same for the Apple Watch, and most recently the vision.
So yeah, that's that's things. We we won two Apple design awards over, I. This course of time, which made us very proud. Yeah, very happy about that. And yeah, the most recent [00:02:00] version of the app is things three. It's been sometimes since we shipped that it was a complete redesign.
So if you go to our website, that's the one you see. And ever since we've been keeping it updated with all the latest features, apple ads to their platforms, new features we're building and so forth,
Leo Dion (host): That's
Werner Jainek (guest): and.
Migrating to Server-Side Swif
---
Werner Jainek (guest): So just related to this discussion I guess I should conclude what we've also done is we have our own sync service, which we dub things cloud and we've been running that for the past 13 years as well.
Leo Dion (host): Today we're gonna talk about your migration of things, cloud
Werner Jainek (guest): Mm-hmm.
Leo Dion (host): to server side, Swift what's kind of the backstory of things, cloud and how that worked. And well also, why have your own syncing service as opposed to iCloud.
Werner Jainek (guest): Yeah. So back when we started, we've been at it for some time, as I said there was no iCloud there was on, on iOS. There wasn't even core [00:03:00] data back then. Hmm. So when we built our, when we built our app we built our own layer on top of SQ lite and, uh. Yeah. And there we were, and our customers demanded that the to-do sync, of course, between the platforms.
So we had to sit down and develop it ourselves. And so we did. And the first version of the cloud shipped 2012.
Leo Dion (host): Okay.
Werner Jainek (guest): Been running it ever since. And we're quite happy how it worked out like later in the game. Apple, of course, introduced other technologies we could have. Right to use. But, but at that time we already had a, a great running system.
We actually invested quite some time making it basing it on a very clear theoretical foundation that we're quite proud of as well. And, it had proven itself already, like our customers loved it from the day we we, we shipped it and it's, you know, they praised it for its stability and everything.
And so why change that, right? Like today we're [00:04:00] very happy to own this piece of technology. It feels great. So here we
Leo Dion (host): Yeah, I mean, you'd be handcuffing yourself, like basically, or restraining yourself to, if you ever moved over to iCloud at this point. 'cause you have so many. We'll talk about some of the features you guys have with the syncing tool. But yeah, it's pretty, it's pretty sophisticated stuff. Yeah, so let's get into it.
What, what was it built with and then why did you migrate off of that to Swift?
Vojtěch Rylko (guest): Yeah, it was, yeah. The legacy system was built using Python two and run in Google App Engine. And how, while it was pretty reliable, there were various issues. We, we had like the unpredictable performance, pretty long latencies, like the Python was not that performant. That led to pretty high monthly bill and then the, it was pretty [00:05:00] old, and then some several deprecation loomed and we had to.
Decide what to do next, like,
Leo Dion (host): say deprecation? What do you mean
Vojtěch Rylko (guest): yeah. Yeah. So, obviously Python two became deprecated a few years ago. That was the, the biggest thing. But was there was, it was not only Python two, it were also some frameworks we were relying on and other technologies. So. The simple path forward to just rewrite to Python three was not possible.
We had to like rewrite through, truly rewrite entire, entire cloud.
Leo Dion (host): That makes sense. What was that decision process like as far as what you were gonna do next?
Vojtěch Rylko (guest): cool. Yeah. So, yeah, we, we sit down on top took blank paper. And started from scratch, like thinking about which language to use and all the options [00:06:00] were on table, like we're thinking J Java, obviously Coline go. We are also considered, again, the Python three and even c plus plus. Part 10 we realized that the Swift is also the option, like it was not a.
Some knowledge back at the time, like we are talking about like four years ago when we were making this decision and you didn't hear about people using Swift on server, like the, the kind of companies. Yeah. Not lot like the, the kind of companies like we are small indie developers. There were zero success stories back at the time.
We heard about Apple, for example, using that successfully internally, but it's a company with different kind of resources than we are. So, so we looked at the, at the Swift and its ability to run on server and we are nicely surprised that it [00:07:00] seems like it is possible and not only possible. It compiled on Linux for many years already and there was.
Server side work group, which was pretty established. And we started to get a little bit excited about this this proposition. And, but of course we also pretty worried because the ecosystem felt not major at the time, like in comparison with, let's say Java. It was a still young technology. So we approach that with. A caution and but there were clear benefits to that. Like the Swift, we've been already using that for our client side development for many years. We, we are used to, to that. It's, it's fast language. It performs well. It has, automatic reference counting for memory management, which makes memory managements [00:08:00] predictable. And it is a nice language, nice expressive type system. Strong type system, which is great for a maintainability of these larger systems. So all these benefits together, took our attention and we started to evaluate Swift on server. We, we started gradually we started to play with some experiments.
We're trying to run it, build some prototypes, proof of concepts. Always worried that we will hit some roadblock, some I missing library. But that was interesting. That didn't happen. Like all these libraries were available. Minuscule readies. AWS encryption, like everything was available and it worked well.
And so we started gradual building stuff and over the course of three [00:09:00] years, we were basically done with rewrite and so that's why we got to Swift.
Technical Challenges and Solutions
---
Werner Jainek (guest): And maybe one, one thing I, I should add just to paint a fuller picture of the past the technology on, on Google side was. Google App Engine and that is a very high level turnkey solution. And it served us very well in the beginning 'cause you can get something going fast, which is what we needed.
But over the time as we were adding more features to the cloud we kept hitting roadblocks where the. Black box nature really was in the way. There were performance issues that we couldn't explain. It was just lagging while accessing some data or a transaction sizes. There were various limits that we weren't fully aware of when we originally built it, and then we had to build workarounds for that in the old system and, and so forth.
It's like, it really dragged us down by had to implement one such workaround and. [00:10:00] It was nice. So we were in high demand internally for more control. I think that's, it's not just about the language, right? Like the entire stack. We wanted to be more in control of.
Leo Dion (host): Yeah, I wanted to ask about that. Like could you even like run Swift in Google App Engine? I mean, maybe now, but not at the time. I would assume.
Werner Jainek (guest): Good question. I don't know.
Vojtěch Rylko (guest): I don't.
Leo Dion (host): Okay.
Werner Jainek (guest): Probably.
Leo Dion (host): but that, I mean, that sounds like the basic motivation to get off a Google app engine. What, what s so you you did end up going with Amazon. What were, what were the other considerations you made as far as hosting was concerned?
You, you had made the decision you wanted to host, you wanna do Swift and that was like, okay, where are we gonna actually host this? Like, what was your decision process there?
Vojtěch Rylko (guest): Yeah, so I think the AWS is. Kind of obvious leader here or that, how we perceive that, and it felt stable. Although it feels like they take that seriously. [00:11:00] They Mm. And I think the question is more like, why not to use AWS? Like what would be the reason to not, to not to use that? We, and also we already had experience with AWS good experience.
We run some
Leo Dion (host): that's a big factor. Yes.
Vojtěch Rylko (guest): so, and. So we are familiar with that and also they're offering met our requirements like we had. We are looking for where to run Swift and where to store data and the database they of offered and the way how they provide Kubernetes. They made everything made sense for us.
So we went with AWS.
Leo Dion (host): Yeah, that makes complete sense. And so, what, what's, did you use EC2 or what did you end up doing as far as like actually hosting it?
Vojtěch Rylko (guest): yeah. Yeah. So, we are using Kubernetes, but the EKS, the managed one, so A AWS takes care for running Kubernetes cluster for us, and we are [00:12:00] basically just plugging in EC2 instances. Currently we have four EC2 instances for some extra processing. We can always scale up. It's pretty convenient. And yeah, so it is running on top of EC2 instances in the Docker image. We compile Swift and bundle into the Docker images, which run in the Kubernetes cluster. And yeah, it, it's com complex. The Kubernetes is complex, but it works well for us.
Leo Dion (host): I want to take a step back 'cause I know there's a lot of iOS developers who listen to this. Docker is kinda like a vm. Basically a virtual machine that runs the application, essentially. Command line app, that's a server. And then Kubernetes. Is that like a way to run multiple Docker images and manage 'em, or what, how does Kubernetes fit into that?
Vojtěch Rylko (guest): Yeah, so I would say. Is like operating system, which spans multiple [00:13:00] machines. So Kubernetes takes care of multiple, easy two instances. It manages them, it knows which are fine, which are crashing, for example. And if you tell Kubernetes to run this Swift executable five times, it'll distribute it on the machines and make sure the.
Networking is correctly set up so they see what they should see and the traffic is reaching them as you wish, so
Leo Dion (host): Okay.
Werner Jainek (guest): And maybe in addition what it also handles is the orchestration of when you're deploying a new update, right? There's a whole complex procedure where you have to stop traffic to the existing binary, bring up the new things already. Redirect traffic and so forth. Like, all of this management is handled and Kubernetes is usually like, if you want to run that yourself, it's a whole team in your company doing that.
So, and for us it's two people on, on the backend. Your VTECH is one, and then team is the [00:14:00] other guy. So that was not an option, but it being managed was certainly interesting and it, it works fine for us, I would say.
Leo Dion (host): Okay. What were some of the challenges you initially ran into in your rewrite to Swift? That you had to like wrap your head around that Python made easy,
Werner Jainek (guest): no, not really. Right? Like there wasn't I mean, you
Leo Dion (host): how was the, so how was the maturity of other Swift packages compared to the libraries that Python provides? Like I would assume that would've been
Vojtěch Rylko (guest): Yeah, like, yeah. So the challenge was our uncertainty about entire ecosystem. At the beginning we were approaching the, the Swift, we didn't know much about that. And we are very worried about, as you say about maturity of third party libraries.
Leo Dion (host): Party libraries. 'cause a lot of them are Apple or like, yeah, right.
Vojtěch Rylko (guest): Yeah, you are right
Leo Dion (host): Second, let's call 'em second party libraries. Yeah.
Werner Jainek (guest): Well, but let me quickly [00:15:00] interject. Like you're right, Swift, Neo, of course, is provided by Apple and I, I should say knowing that Apple provided that base layer of what we're building on top of was very important in our decision. Like that gave us a lot of confidence.
Leo Dion (host): They actually use it day to day in
Werner Jainek (guest): Exactly. Yeah, exactly. Yeah.
So that's great. And then, but there are other libraries from third parties that we're using and that is unclear, right? Like, how good will they work? Are they full of bugs, are they crashing all the time and so forth that we didn't know right?
Leo Dion (host): Right, right. So like, what which framework did you end up using to, to, for your server?
Vojtěch Rylko (guest): And so, there was almost obvious choice to use Vapor back then. And that got us pretty quick up and running. It was very con, very convenient. Now we are, I think, a little bit outgrowing it and maybe it's time to, to do it ourselves to not use framework anymore. And but it still serves well. [00:16:00] Like to, to this day.
We, we are happy with that. And it's great for, for quick start.
Leo Dion (host): Yes, I was gonna say that if you're looking to start,
Vojtěch Rylko (guest): Yeah, that's.
Leo Dion (host): highly recommend Vapor. It does a lot of handholding for you to get up and running. Speaking of frameworks that you use, what other services are you using on Amazon for your application and how do, how does Swift, Swift fit into that?
Vojtěch Rylko (guest): Mm-hmm. Yeah. So, the most important service is probably our database, and that is my school, Aurora, which is kind of. Different. My SQL managed and modified by a Amazon by AWS team, and it has different storage layer for data, which means that the data stored in SQ Aurora are more durable. They are stored in distributed storage, and that was very
Leo Dion (host): But it's, but [00:17:00] it's still relational, which I think is
Vojtěch Rylko (guest): yes.
Leo Dion (host): Big difference from, I don't know what the Amazon one is, not Dynamo, is it? Simple. Simple. Or what's the Amazon one that's just distributed, but simple key value stuff. I forgot the name of it, but this is an actual relational database.
Vojtěch Rylko (guest): exactly. It's for, for the developer's point of view, it is a regular, my SQL with transactions. Atomic commit installation. They're very nice features. Like we, we love them. That's why we wanted to use relational database. But we also wanted to have some guarantees about the durability of data.
And there's not many databases with such guarantees like the Aurora. This is very nice.
Leo Dion (host): durability, what do you mean by durability?
Vojtěch Rylko (guest): So if the, if you commit a transaction and main instance crashes, you, you still [00:18:00] have enough copies of your data. They are, so Aurora has a different storage layer where they write your data on to multiple locations,
Leo Dion (host): Mm-hmm.
Vojtěch Rylko (guest): and that's a great features.
Like the regular setup of my sql if you want to achieve something like that, is to have two instances running and you have a. Main writer instance and it you are, you have some follower reader instance, which is, which is building a copy of your data.
Leo Dion (host): Yeah.
Vojtěch Rylko (guest): But if your main instance crashes now, you have only one copy of data often, and yeah, so that's what I'm talking about, like the Aurora comes with guarantees, which are true all the time.
Leo Dion (host): And then my, so I wanna cover, had a question about any particular reason you went with MySQL instead of Postgres. Well. Is it because Aurora is a fork of MySQL, that that's why you ended up going with [00:19:00] MySQL and Aurora.
Vojtěch Rylko (guest): So that's the main point, like, there is our. For Postgres and for my SQL, but for my SQL that one has been around for longer, so we felt it is more mature back back at the time. So that's why we prefer my sq l And also we were thinking that my sq l should better handle frequent rights. When you have wider table and you modify just a few cells here and there. My should be more performant in this scenario, but I'm not sure if it's true even now, like this year.
Leo Dion (host): So you might like, if you were gonna do it today, you might consider switching over to Postgres.
Vojtěch Rylko (guest): Oh yeah. Yeah. Like I, I would
Leo Dion (host): Okay?
Vojtěch Rylko (guest): consider that I love Postgres, so.
Leo Dion (host): Okay.
Werner Jainek (guest): but also, I guess we should say that we didn't run into major issues. I guess, we did have an issue with the size of the database, but I'm not sure if Postgres would've been much better [00:20:00] there. Like that, that's one of the surprising things we found is like, you would assume that a few terabytes of data.
Can be handled by these databases these days. I dunno, that was my naive thinking going into this. But we found out that no, actually you can't store terabytes of data. I mean, you can, but then you can't upgrade the instance, for instance, to a newer version. 'cause it just crashes while the update is going.
Leo Dion (host): So how are you gonna do that when they deprecate what you're currently using?
Werner Jainek (guest): Yeah, exactly. So this is one of the surprises we encountered along the way. And then it's like, okay, I guess we have to solve this. And we went ahead and solved it by building a bottomless database thing. What I,
Vojtěch Rylko (guest): Yeah, we,
Werner Jainek (guest): He did, he did it.
Vojtěch Rylko (guest): yeah, we it was pretty late when we found this issue. Only once we had, in our data in our database, we found we are not able to upgrade, upgrade it anymore. So, we built mechanism for offloading of data from database to [00:21:00] S3, which is like, which we try to be opaque from the, from the code point of view.
And now we have a parameter which you are basically by this parameter you are changing how much of data we store in database and how much data we store in S3. Because storing it as three is more expensive. It's, there is more latency when you are getting the data. So we don't want to put all, all data there. So we have a parameter and if we need to upgrade the database, we we adjust this parameter. It brings the database off, uploads the data to S3, and then we can upgrade. But yeah, it was quite complex process. We had to build, which we didn't foreseen back then.
Werner Jainek (guest): And maybe I should say like when we're trying to solve these kinds of issues, I, I said in the beginning that we're very, quite proud of the way [00:22:00] the syn works. And the, our whole approach to how we built it is to. Try to get to a hundred percent correctness, not 99.9, because if you're a thousandth to do, doesn't sync, that's a problem, you
Leo Dion (host): Right, right.
Werner Jainek (guest): And so we, we care very much about correctness and this now extends all the way to the cloud. And when you're offloading data from a transactional database into a different system you have to be very careful to not accidentally break the transactionality here and the consistency of the data.
And so we did spend quite some time engineering that, and it works great. Like another challenge is that in the database, each row is very small, right? And you can't just put each row individually into, into S3. That would be insane. So we compacted them together and so forth. Yeah, so it's funny, I, as I said, I didn't think we needed to solve that problem, but we had to.
So.
Leo Dion (host): It's, it's, yeah. So [00:23:00] the database is in the server, but what, what, sorry, what part is an S3 and what part is in the actual Docker instance? Or Aurora I should say, or do you use Aurora and then you have the files in S3, or how does that work exactly?
Vojtěch Rylko (guest): Yeah. Yeah. So, we basically store all data in, in database, and there are like two parts, like two main categories of data. One part is the regular relational stuff, like user accounts, usernames, and like this obvious stuff. And then there's actual user data there to do.
Leo Dion (host): Oh, okay.
Vojtěch Rylko (guest): these guys, this to produce this content of for users.
We don't care about the content on the server. It's, it's a, for us, for the search server, this is like binary block. As soon as possible, we compress and encrypt that and store in database and we don't [00:24:00] ever change, we don't ever mutate this data. So when user changes their to do. We just add another record about send, about a change on top of previous changes.
So we are like building app and only log for every single user. And that's, that is a nice property because if you don't mutate this thesis, you can put them somewhere else eventually. So if, if, if you have, yeah, if you have enough of this. So they're worth putting to S3. You take them, put them together in our, some binary format we made and we put them to S3 storage where they are guaranteed to exist forever.
And you can basically put a pointer to database in database to to S3. Yeah, if you want to read the data now, you need to go through S3, which is slower, is more [00:25:00] expensive, and you need to think about transactionality of whatever you are doing.
Leo Dion (host): So you like all the to-do data is actually encrypted, is that what I'm hearing
Vojtěch Rylko (guest): Oh, yeah, yeah. Like, we the way how things thinks is that the server. Doesn't do much about with data. Like it's all up to clients how to, to understand data, to, to resolve conflicting situations. And the backend is like dump stor, dump storage. Like it doesn't look into to those. There is nothing what we need to do there.
So immediately when the change from user arrives. We just compress and encrypt that and put it into databases, binary block, and never need look inside anymore. Like only when another client wants that, that the client will read the data. Look inside.
Leo Dion (host): Okay. [00:26:00] That's amazing. Wow. So you mentioned that you use S3, you also use Lamb does Right? For some stuff.
Werner Jainek (guest): Oh, that's a good point. Yeah.
Leo Dion (host): Yeah. How does
Werner Jainek (guest): that we needed from, from Python.
Leo Dion (host): Okay. Okay.
Werner Jainek (guest): go.
Vojtěch Rylko (guest): Yeah. So, we have a feature called mail to Things. So every every user of things cloud gets a special email address
Leo Dion (host): Yeah.
Vojtěch Rylko (guest): where when they can, when they send the email there, it will appear as a, to-do in their things inbox. So, we are receiving these emails and. We need to process them somehow on server to create a, to-do for, for the user.
And processing emails is a ma that's a mess. Like, so we so we are using, we are processing incoming emails with Lambda, where we run a Python three script because Python has the best libraries for this kind of [00:27:00] like messy, messy work. Where there are internet standards, but nobody really follows them.
So you need very major libraries to be able to parse any email, any incoming email. So yeah, so we are, we still have some Python around.
Werner Jainek (guest): But it's very much a satellite, right? Like, it's just this little thing that runs in Zam. The majority of the code is in Swift inside Docker, running on Kubernetes, or managed by Kubernetes. Yeah,
Leo Dion (host): Is that the only lambda you have? Is the email? Okay. Okay. That makes sense. Yeah, we did the episode with Sebastian talking about, lamb done Swift stuff, so it's a pretty powerful tool. You have also like some background processes as well. Those are written in Swift. How does that, how does that work or, and what do you use it for?
Vojtěch Rylko (guest): Yeah.
Werner Jainek (guest): Mm-hmm.
Background Workers and Swift
---
Vojtěch Rylko (guest): So we have bunch of background workers and [00:28:00] they run in our Kubernetes cluster as every other, as every other staff. And what's maybe interesting is that we have one code base and one shirt coat, and we are building just one. Binary, but it can be configured to run as API or as the worker of type A or another worker.
And one of these workers is exactly that database of uploading. So, there is a worker, which is reading the database, looking for data which can, which it can offload and put to S3. So this is one of these workers. So is the Swift. So the guy another, another is the worker, which takes these processed emails by Python and actually creates the, to-do for particular user, like looks for the user and creates to, and so, and we have a bunch of [00:29:00] such workers.
Leo Dion (host): Do you use, like, what is it, service lifecycle stuff or how do you, how do you trigger that or how do you get it running?
Vojtěch Rylko (guest): Yeah, that would be great. Like, we would love to use this, but we, we still have some legacy stuff built on top of Evan Loops and it's a little, that part is a little bit messy.
Leo Dion (host): I will, I will ask later about Swift six, but before I get into that, I think one question I should ask that I think is important is, you've got an app in the app store, you've got a server on AWS, how are you sharing code between the two? And like, do you have, do you have, have you seen advantages with that?
I guess because that's the big question is like, okay, you've written in an iOS app, you have the backend and Swift too. Where's, is there any shared code in that, in that instance between the two? Or what advantages have you seen in that, in that way?
Werner Jainek (guest): Yeah, so that's very interesting. Like, even on the [00:30:00] client side, right? It's not just one app, it's multiple apps that we were shipping, one for the Mac, one for iOS, one for visional. And even there, the stories that the code bases have started out quite separate. Like we used core data on the Mac, but our own thing on iOS and hence entirely different models.
And entirely different view stacks and so forth. And so in the past ever since we've been around we had this effort of unifying more and more of the code base like today. The same model runs everywhere
Leo Dion (host): Right. That
Werner Jainek (guest): the way. Yeah. And even a lot of the controller code all the logic code the sim code of course is shared across the platforms.
And there's even more we want to do on the clients. And back when we decided to try out Swift on the server the possibility of co-chairing was certainly a, an interesting prospect. And so that was part of the reason why we were excited to use it on the server. Now so far, that has not come [00:31:00] to.
Realization, like we're not sharing code, maybe one or two files that I forgot. But the majority isn't shared. However, we think in the future there will be quite some opportunities for co-chairing and so I guess we'll have to chat again in few years and then we have a, a more interesting perspective on that.
I think how that
Leo Dion (host): Yeah, I think like for me, it's always ended up with like simple data models. Tho those are the ones that end up being shared or maybe like some utility functions and things like that. But yeah, I mean, your, your business logic is different on both and Yeah, I could see how, how that would not be as, as useful as you might think.
Is there, is everything in one, sorry? Is everything in like one Xcode project or multi repo mono repo.
Werner Jainek (guest): I mean, for the server it's all in one. And then for the clients it's all in one.
Leo Dion (host): Okay. So you've ke kept that totally separate. Okay. Interesting.
Werner Jainek (guest): Mm-hmm.
Leo Dion (host): So I'm gonna ask [00:32:00] a awkward question here. Swift six. Where, where are you with that? Are you excited? Are you scared? What, what's kind of the thought process with Swift six?
Swift 6 Adoption
---
Vojtěch Rylko (guest): Oh, actually we just recently, we switched to Swift six
Leo Dion (host): Oh wow.
Vojtěch Rylko (guest): and Yeah. And we like that
Leo Dion (host): Both client and server.
Vojtěch Rylko (guest): yeah.
Werner Jainek (guest): Yep.
Leo Dion (host): Okay.
Vojtěch Rylko (guest): for, for, for server, like we, we, the, we adopted structure concurrency everywhere. Now we are just waiting for that few remaining dependencies to, to catch up. We adopted, we, we completely switched to Swift testing and it went pretty well.
Like, I don't know, like we spent like one or two weeks adopting this structured concurrency. I. There were
Werner Jainek (guest): On the server
Vojtěch Rylko (guest): on the server speaking about server. And there were no major roadblock or something, just like some send ups here and there some [00:33:00] changes, but it was pretty nice experience on the server.
Leo Dion (host): almost easier on the server than it is on the
Werner Jainek (guest): It is certainly like, that's why I was correcting like only on the server. It took two weeks I think on the client. We invested more. And it was certainly more in the way on the client, I would say, on the server. It's very natural. 'cause already the structure is like you're getting the requests, you're isolated and
Leo Dion (host): Yeah. We've had that for a while, so it's like, yeah. Let's talk about, well, is there anything you're interested in when it comes to Swift 6.1? I may may as well ask that if you've even looked at it.
Vojtěch Rylko (guest): Yeah, not I'm not particularly like excited about any, any features. What I liked is that Swift language committee acknowledged that this concurrency is sometimes a little bit tricky, and they are trying to work on more approachable Swift concurrency. And that, that excites me. I think that's a good direction.
Leo Dion (host): Yeah. [00:34:00] Yeah. Agreed. Agreed. Let's talk about logging and metrics. How important is that when running a server application and where did that fit when it came to Swift?
Vojtěch Rylko (guest): Oh, that's, that's super important. Like without being able to observe what's happening. We are not able to, to detect that something's wrong. So, as Werner said we are team of just two on, on, on backend. But we, we have pretty high standard even for such small team. So for example, we have we have pager duties.
So we have, um, or all the call duties. So we have like, we have monitoring which triggers incidents, and that incidents wake up current developer who currently has on-call duty and for that to work towards, well, we need grade monitoring, logging metrics, everything. So we can set up incidents which [00:35:00] notify us only with something's really wrong.
Leo Dion (host): Hmm.
Vojtěch Rylko (guest): And we need to be able to detect all the potential issues. So, I think the logging is our, the, like, that's probably the, even the biggest part of our cloud bill currently, like the logging and metrics, that's the most expensive part currently. So we take that pretty seriously and we are using custom custom json logger we, we built and we are using the Swift Prometheus library or package, which is available and which ships the metrics. And we are watching. The efforts about s pricing and, and, and other stuff. I think currently there are some people working on this. So it's on our radar.
Leo Dion (host): What is Prometheus?
Vojtěch Rylko (guest): Yeah. Protive is a way how to, it's about [00:36:00] metrics. It's, it is about metrics. So currently we've switched from itus package. Our code is able to provide various metrics and these are then digested to CloudWatch, AWS CloudWatch, and then where we can see nice graphs and we can we can set up some, I dunno, incidents if something is over some threshold and so on.
Leo Dion (host): Okay. Let's. So there was something you mentioned.
Chaos Testing and Deployment
---
Leo Dion (host): I really picked my interest, speaking of listening to issues and errors and things, is this thing called a chaos engine. What, what is that exactly? What, what's going on there?
Vojtěch Rylko (guest): Yeah, I do. I think the best testing is testing on production. So, so the chaos testing is a way how to test the infrastructure and little bit also the code on production. [00:37:00] It's about introducing virus faults, injecting faults into, into the code or into infrastructure. So, for example, every morning the chaos engine we have inject some fault into into cluster.
It can be. That some of instances crashes. Like we've, we force instance to crash and we observe if it auto heals and cluster like gas into the health state again, or we, we have some part in the code where we force unwrap nil, and we do that ev every now and then and see. If we are properly notified about this happening and that no no issue happens, like to data or like everything gracefully recovers again, so,
Leo Dion (host): So is this like, is this like CR job and is this all written in [00:38:00] Swift or what exactly.
Vojtěch Rylko (guest): So it's quite, is quiet. Weird beast. So it's powered Byron. So every, every day it starts and it's like a shell script with shell scripts. And we are using various ways. There's for example AWS service, which helps you to inject particular SubT errors. For example, networking errors. And that's very useful to test.
Like you can test the split brain scenario. Like what if you disconnect your cluster to two parts, what, what will happen? So all of that is super, super interesting.
Werner Jainek (guest): And maybe I should, I I, I'd like to add to that like before we said Google App Engine was a, a black box for us. That was the downside. The upside is that they did a pretty good job, at having a great uptime of the things cloud service. And so throughout the years we've been using that the [00:39:00] uptime.
Uptime was phenomenal. And so when switching over to this entirely new cloud where we wanted to have more control, one of our worries was. Well, we better make sure that we have the same level of quality that we're delivering to our users. And we, we went at great lengths to ensuring that the chaos testing is part of it.
Like, how do you know how your system behaves if things go wrong, if you don't make them go wrong? Right? So we try to do that. And then the other part, of course is it's an entirely new stack, entirely new code base new technologies everywhere. How do we know, that it, that it works well, you know, that we're not constantly having some crashes and so forth.
And that was a big challenge when developing. And we can get into that if you like but we, we went at quite some lengths to ensure that on day one the new version we're shipping is already tested in production. So yeah, let, just as a small side that, that we basically ran both versions side by [00:40:00] side.
That was the, the trick there.
Leo Dion (host): How did that work out? Like, or how did you keep both in sync and stuff?
Werner Jainek (guest): Yeah, it's quite, quite cool. We think that's quite cool. So.
Vojtěch Rylko (guest): Yeah, so when we're developing new system we, our goal was to not disrupt users on day one. Like we, we, we needed the new system to be very robust and proven, already proven before releasing. And we found that the only way how to make make it is to to run it actually. And so we. We had a legacy system running and we started to run a new system alongside and we piped all the traffic through the new system where it executed its own logic, its own database access, its stored data, performed all the Swift code there, but it also forwarded the requests to the legacy system and that [00:41:00] were.
This was authoritative. So the response from legacy system got back to user. We, it is impossible to keep this two system completely in sync. So they they diverged in the data, like was in database, but that was not really, that was not really important. Like to the correctness. We tested the correctness and behavior manually and by other ways.
But the fact that it worked all the time, like for a year without downtime, without crashing, without performance issues that gave us confidence that once we turn off legacy system, it will work fine. And to make even more sure, we employ the chaos testing where we not only. Led the new system running, but we were also poking into that and seeing like if it auto recovers, how does it behave under extra load.
And similarly, and [00:42:00] we're observing the system for months and months before we got enough confidence to actually switch over to it.
Leo Dion (host): Okay. Yeah. That's awesome. So before we close out, I had one more question was how do you actually deploy. Your server app. So those of us who are iOS developers are quite familiar with, dealing with the app store server side is a bit different. And I know you talked about using GitHub actions and things like that.
You wanna explain how deployment works when you want to update the server.
Vojtěch Rylko (guest): It is actually pretty boring. So, GitHub actions, they
Leo Dion (host): a good thing sometimes, right?
Vojtěch Rylko (guest): right. So we compile our Swift code or using GitHub actions. Of course it could be faster. That would be very nice, but it's about 10 minutes and that
Leo Dion (host): you run their server? Do you run their, well, no. I guess you run, you run Linux, so you don't [00:43:00] even, like, you don't even need a Mac. Right.
Vojtěch Rylko (guest): like, GitHub provides you with runtime. So you just like create small YAML file where you specify a few steps and they are executed. It's very convenient. And so we build our Swift code base there inside the docker. So that creates a Docker image. And that got. Upload it in. So like in a fancy storage like Docker registry, but it's just like you put it somewhere and it has a tech attached, like some name, some tech.
And then when we want to deploy that thing, we basically tell Kubernetes to, to use a new tech, to this new tech. And Kubernetes tags the image. Starts down the old running service and starts a new one, basically. Or you can do, it can make it also [00:44:00] gradually, it can like keep old stuff running. Starts also new stuff and starts switching the traffic so you have no downtime.
That's also sometimes useful.
Leo Dion (host): That's awesome. Before we close out, was there anything else you wanted to talk about?
Werner Jainek (guest): I mean, maybe, just throwing a top level light on this whole endeavor, right? Like, why are we here? We, we feel that back when we started this project, as Tex said in the beginning, it was quite uncertain thing. Like, will it work out? Will it not work
Leo Dion (host): Right.
Werner Jainek (guest): And I guess along the way we grew in confidence.
Like there was no roadblock along the way. All the, there were no problems to speak of. And now even in production, it's been in production for a year. We haven't said that yet. Like we switched early last year. And it's been running great. So we considered this a great success story for Swift on the server.
And we felt that it's important to get the word out 'cause there's not that many [00:45:00] still. Great use cases out that people can listen to, right? So that's why we went to the Swift on server conference and had a short talk there to tell people, it's like, Hey, it's working. Like
Leo Dion (host): Yeah. Yeah.
Werner Jainek (guest): if you find yourself in a similar position as us maybe you already have a, a Swift code base and you need a server component, this is definitely an option.
Go try it. Right? Like, or at least consider it.
Leo Dion (host): And we will have links to your talk. And my, my success story from Swifton Server as well and some other folks. Definitely check that out.
Werner Jainek (guest): Are some. Of course. Yeah.
Leo Dion (host): Yeah, yeah, yeah. Alright. But everybody knows things, so that's, I think that's a big, big defining feature. For your, the part of your story is, is that I don't know how many users, gazillion users use things.
So we know it works. We know it
Werner Jainek (guest): That's about the number. Yeah.
Leo Dion (host): Yeah. Yeah. It's a scientific number.
Werner Jainek (guest): Yeah. Yeah.
Leo Dion (host): For someone who is an iOS developer, which is the vast majority of Swift [00:46:00] developers, they want to. Maybe they won't need a server. Maybe they're just kinda curious about having a server. How, how, why should they get started on Server side Swift, and how should they get started on server side?
Swift,
Vojtěch Rylko (guest): that is interesting. Like, I'm not sure that we are in position that we could give good advice here because we are the quite a backend people like me and my colleague, so who, who just know Swift. So we use Kubernetes and everything, but I. And recommend it for anyone who just want to try. It's pretty complex and it took us lots of time to set up it's like full blown, like full in solution, full in approach.
For somebody who just starts out, I don't know, maybe you already said about something about the, lambda now, now you can run Swift on Lambda. So maybe for some small minor server processing that makes [00:47:00] sense with some managed database, some key value store that could be maybe interesting. But I would definitely, if I would be just, I would, I would be, I always developer who just want to try out, I would be looking for some.
Managed solution where I don't need to care about easy two instances and Kubernetes and like no,
Leo Dion (host): big, I'm, I like Heroku. We use, I've used Heroku for most of my stuff. You just get a Vapor app, put it, put it up. It's pretty easy to get started. Vapor is super, a lot of handholding. Great documentation. Yeah,
Werner Jainek (guest): Yeah, definitely. I mean, we, we, that's how we got started as well. And it was great
Leo Dion (host): right before d before going down the rabbit hole of Docker and Kubernetes and all that
Vojtěch Rylko (guest): Yeah, I think there's a great
Leo Dion (host): And I do think like having some experience with server side development is helpful. Just getting that perspective as far as like what, what the other side does as you know, besides doing just table view controllers all the time. There's, there's [00:48:00] more to it on outside of that.
Gentlemen, thank you so much for coming on. I really appreciate it. This has been fantastic to hear your story, your success story with Server Side, Swift where can people find you online?
Werner Jainek (guest): Good question. Cultured code.com. I guess us personally ek, I dunno,
Vojtěch Rylko (guest): Best site com.
Leo Dion (host): Awesome. And we'll put links to your social stuff in the notes too. So, yeah.
Werner Jainek (guest): I mean, you should follow the, the company. That's where we communicate. Like I, I technically have a social account, but I'm not really using that much. So
Leo Dion (host): Thank you so much, gentlemen. It's been fantastic.
Werner Jainek (guest): you. Thanks for having us.
Leo Dion (host): People can find me@brightdigit.com. My social is at Leo g Dion on all the socials at Leo g Dion at C dot. I am on Mastodon. Thank you again. I look forward to talking to you in the next episode. Bye everybody.
Vojtěch Rylko (guest): Bye.
Leo Dion (host): [00:49:00] But
