Jacek Galowicz (Nixcademy)

A conversation with Jacek Galowicz (https://nixcademy.com/).

Creators & Guests

Host

Shahar 'Dawn' Or

Mob programming advocate, Rust enthusiast, Nix user.

Guest

Jacek Galowicz

Freelance Software Engineering Consultant with Nixcademy

What is Full Time Nix?

Long conversations with clever Nixers

Intro: Hello and welcome to Full-Time Nix number 2. Our guest today is Jacek Galowicz. Jacek is the founder of Nixcademy. Nixcademy is a company that offers Nix training and education. Our host is Shahar “Dawn” Or.

Dawn: Jacek Galowicz.

Jacek: Yes. Hello.

Dawn: Hi. We did one of these before, we did a whole recording, but I apologize: it didn't go through. It was my attempt to use local tooling but, with all due respect to all the benefit we get from open source, this time we'll be using a proprietary platform to record this. Maybe if anyone in the audience has better suggestions, they can send them our way. Please, introduce yourself.

Jacek: Yes. Thank you for having me here first. I'm Jacek, I'm founder of the Nixcademy. So for a little bit more than a year, I've been giving classes and consulting on Nix and NixOS. I specialize so far in companies who started using Nix. I mean, it's always the same situation or very often similar. You have like one, two, three people who are very enthusiastic about new technology and also invest time regularly into new stuff. And then at some point they stumble upon Nix and NixOS, and we know what a rabbit hole that is. And at some point you might have a borehole. However, at some point they kind of convince a few colleagues to use it too, at work, in their projects. And of course they will start with useful development shells, faster builds, caching effects and whatnot. I don't need to tell you the value of Nix and NixOS, of course. But the problem is usually that when you have only a few individuals starting with this in companies that they sometimes run into like burnout-like situations or at some point the adoption of Nix stalls, meaning that one or two people are hacking all the Nix expressions for everyone, everyone else just uses them, they are happy that it just works, and when it doesn't work, then it's the problem of the little starter group again. And that obviously doesn't scale in companies or corporations.

Dawn: That's what you call a knowledge silo.

Jacek: Yeah, yeah, that's true. I mean, the other people are not bad or anything. They are just like, look, I have all my deadlines. I'm taking everything for free that makes my life easier and I'm thankful, but if it doesn't work, I need to push it back because I don't have a manager who told me “you can spend some time on learning Nix” or stuff like that. Or maybe they're not interested if it's not for free, stuff like that. However, I mean, it's not bad people. It's just everyone's under pressure and the companies, right? And my typical customer is their manager who says, “okay, look, I have these few guys who are really enthusiastic about it. And I think they're onto something. And we really need to do something to ramp this up in the company. There's so many strategic goals that we could achieve with Nix. But I need to ramp up 10, 20, 30, 40, 50 engineers with this. Please help us.” And then typically, we make this official in the company by giving classes, right? So you order the Nixcademy class, a Nixcademy teacher comes to you and shows you what's the value of this. Every class always contains one or two people at least who are not convinced of Nix yet so we help them understand how it helps them. Then we go through the functional programming basics and everything that is alien about Nix to newcomers, and go through the concepts and I try to help people like in three or five days of Nix class depending on how much content the customer orders, to help them through the part that I found really difficult to get into, right? So I'm helping people reduce months of fighting against Nix into one week that is very intense. But after that, they are very empowered to know what is there, which documentation to look at, what are the basic concepts, how to debug stuff when it doesn't work, and so on and so on. Yeah, so far the feedback from the customers is great. First, by ordering a class and paying for this, it becomes pretty much official that Nix is an officially used technology. And then everyone gets empowered with corporate help to really get on it, right? Apart from that, I'm also doing consulting, like building some proof of concept that proves that Nix is feasible for this or that company. Or maybe building some existing project, like making it a Nixified project and then doing the knowledge transfer on how to maintain it yourself. All the consulting aims towards making the team independent of me, like how to get rid of me again. I'm not trying to sell more and more consulting. I'm trying to make the consulting stick so that companies are really self-dependent with Nix. Okay, so that's my summary. Maybe you have a question.

Dawn: A good question would be, how is it being a Nix instructor in 2024?

Jacek: Yes, the demand is rising. There are more and more companies. I have American customers, German customers, customers in Hungary, in Portugal, in England, different countries. And they all would like to do more with less people and less time or with the same amount of people do even more. Right. And some people are already interested in Nix so that you only have to show them how to use this, they are very eager to be thrown the complex, but really useful concepts of Nix. That's a lot of fun because you can show like already experienced experts how to do even nicer things with Nix, right? And then you also have people who are not convinced of Nix yet. Of course, by having a lot of experience with questions from both kinds of perspectives, I basically have all the answers or most of them. So questions that might have been hard to answer if you don't have an expert in front of you, can be answered in minutes. And then you can discuss this on their projects. So some customers let me also sign their NDAs so I can see their projects and then also give the classes in relation to what I learned the company needs, which makes it even more relevant. And for me, that is really interesting because I see different teams, different projects every week. Most of these teams are great. They are really motivated, have a nice learning culture, and it's a lot of fun to help here as a teacher. Some classes, in some classes we don't really get through the whole curriculum because they have so many good questions. But of course, answering these makes the experience even more relevant to them, which is what I'm aiming for, right? So in one word, it's fun. In two words, it's fun discovering what all the companies do.

Dawn: Well, I bet it's fun if you're a real expert. If I was placed in that position, I wouldn't have been able to do this work.

Jacek: Yeah, I have experience in that regard already. Like, two years ago, I wrote a book on C++ and was also teaching C++ in the past. I also gave a university lecture at the FH Münster in Germany, which was about software development practices and/or processes that ensure quality. So I was showing students how to do coding in teams with merge requests, with Git, with testing on the unit and integration test level, how to do fuzzing, stuff like that. And then also how to do this in reproducible ways with Nix. And yeah, for that reason, it's like just another lecture, which is simply specialized on Nix. So of course, when I'm doing this, 90 % of the skills that I need are pedagogical and of non-technical nature. You need to read the room, how the people are feeling, see if what I just said was understandable to everyone, especially when you have managers and people who are managed in the same team, in the same class. Sometimes people won't ask because they don't want to look stupid. If I just then answer the question without it being asked, people are very helpful, thankful very often. So stuff like this, you really need to sense out of the room to improve the experience for the participants. And 10 % of the time you need to have a lot of experience with the technology stack because some questions are extremely special and of course, it increases the trust in you as a teacher if you can answer them out of the box, right?

Dawn: Is there something that you find in common with the companies who take interest, who contact you or who end up hiring you for teaching Nix, for supporting Nix?

Jacek: So they are always similar in specific regards. So for example, there's like one cluster of companies, typically bigger companies have like one team or let's say roughly five people, including the manager to decide that they want to use Nix. And they typically run into the difficulty that the whole company structures are fighting against change. When you want to discuss with the administrators, “hey, please, we would like to have some Nix infrastructure, a cache or CI builders”, they are like, no, we are not doing something like this here. Or you need to prove that it doesn't have some of these problems or whatever. And they are typically not presented with a to-do list that when they really assess that, that then the admins are finally helpful. But it's simply people fighting against change and I've seen this in my own company that I founded a few years ago too. It starts with small companies that people find change when they're presented with it, either stressful or frightening or whatever because as already stated, most people are not bad, they are just deadline-pressure-driven, and then they are not really thankful if we give them more tasks to really organize this. But then really organizing something that is new and changing to something that is new, where not all the risks and potentials are completely evaluated because you are just starting with it, that is kind of the situation that they all have in common. And here is where I can really help, right? I mean, I've seen so many companies starting with Nix and using it over the years. I've seen winners and people who were not winning with Nix, where their change didn't really work out. And so I can help companies to assess the risks that I really have after talking to them about what they need and how to start, how not to start, what are the anti-patterns that some companies develop when they start using it and how to avoid them… So this is kind of common to all the companies. The lack of experience with how will this work at scale, right? I mean, it's not trivial even if there is not this fighting-against-change problem, because of course it's completely different from what you've been using before. But yeah, that's the typical thing they have in common, the lack of experience with how will this develop when we start using it with like 100 people and new infrastructure? Where will be the bottlenecks in the future? What problems will we have? Which risks do we try to avoid, right?

Dawn: When you're being contacted at that stage, and I realize it probably varies significantly, but at that stage, are those prospects typically at a point where they're just looking into Nix as an option, or maybe they are convinced already that they really want it. Where do they typically stand?

Jacek: So the majority is already convinced of Nix. They say we want to use Nix, but we want to use it professionally as quickly as possible, please help us. The majority of this majority just orders the class, because of course managers feel best with the technology when their people are educated on how to use this so that they are not depending on the consultant because when the consultant comes in, shows them how to do it, and then it's their problem when he's gone, that's not what everyone wants to have; so classes are the best way to build up this knowledge internally. Apart from this majority I also have a few customers calling me like, “okay, look, this next stuff looks really great. But before we start throwing big money into that direction, we want to have some proof of concepts. I have a few senior engineers who say it's not possible in our company. Can you please talk with them and maybe propose something so we can work out together if it works or not”. So you can start with a workshop next week, Friday, I'll be at a smaller company. We just do a one day workshop. I look at their projects, point out what will be easy to Nixify, what will be difficult to Nixify, talk with them about the strategic goals that they could expect, like why would you even use it for exactly? What would a Nix expert say? Where can you shrink costs or accelerate build times or whatever? Let's say what is useful to them and how that resonates and then I will just... they pay me basically to be there for one day and try to report and then they decide themselves if they want to do something with that report with me or without me. So that's also typical. And sometimes companies say, OK, we know that you want to use Nix, but I need someone to Nixify one of our projects after selecting together which one is best suited for that, demonstrate the value, and after it is demonstrated, I will order some knowledge transfer, and then we will use this internally. So they kind of pay for the evaluations, I would say. These are kind of the most typical projects.

Dawn: How do you develop your clientele?

Jacek: Basically, it's marketing. So I started my own home office, had one customer asking me for the training already. And I thought, OK, let's make a brand out of it, because the content was not specific to this company. So it was specific about Nix and everything that was publicly available. And after giving this training for one, two times, we realized that the challenges are mostly Nix-specific and not company-specific. So I thought, okay, let's make this a brand, call it Nix Academy, ask some designer to create a logo, create content online that shows how this stuff is helpful. Yeah, and a few months later, the first customers contacted me and since then I'm learning more and more what is interesting to people out there and how can I give a value to them.

Dawn: I suppose it also involves some semi-frequent stream of blogging, some kind of exposure.

Jacek: Yeah, true. So what my marketing really does is providing value to specific kind of customers especially, and that is a manager who wants to buy training from some foreign company that is completely unknown to his colleagues and his other management. When they buy such a training then this better be good, right? Otherwise they look like clowns. And with my marketing I can show with all the content that I provide. So when he shows them that to his senior engineers, they say, OK, the stuff that this company says does make sense. Let's look at the marketing page. They already had other customers. People are even giving out their faces for testimonials and say, wow, this training was great. So with this, I'm kind of helping the customer to build trust and get a little bit more content that when they buy this training, that it will be really good, right?

Dawn: That is an honorable strategy.

Jacek: It's all about trust, right?

Dawn: It's all about trust, building relationships and you are providing value. So you're adding to the world already without even making any sales. That's a very good thing. What are the common sales resistance points?

Jacek: Yeah, so there was this quote, I don't know where it’s from but it says “people love the sale or hate the salesman,” I'm not sure how exact it is, but it's roughly like this and yeah, it's really true: people really don't like to talk to a salesman who tries to push and oversell whatever deal, people would like to stay in control and think offline without pressure on what they need. And I mean, selling trainings and consulting to big companies is essentially a B2B market, right? And B2B means that the decisions are not done by an individual. If the manager says, I want to buy this training, but his technical people don't want it, then the decision was for nothing. If the technical people want to have a training on this topic, but they cannot convince their manager, then there won't be a decision. And people are also not eager to collect like 10 people, all the decision makers and technical people to talk to random salespeople. So I am basically not there when they make their decision. For them to make a decision that is in my favor, so that I can help them, the whole marketing thingy does kind of help them find out about me, checking if this training is worth something with all the content and customer testimonials and stuff like that. And then for me, that is behind closed doors when they decide, okay, let's call this company and buy some training from them, right? So that's kind of the biggest difficulty, to have to passively find out what people really care for: how does this have to be so that they would like to order it and so that it actually sticks for them, that it works for them, that it’s worth the money for them, right? No one really, not really no one, but seldomly people call you and tell you, “okay, I need this and this and this like this and not like that, so that I can order it”. If you put it on the homepage exactly described as they need it they're like “wow oh my god, finally I can order something like this” but people seldomly call you to talk to you about like “okay we need it in a completely different shape so that it works in our company, can you do it” so I have to find this out myself and that is what marketing is essentially for, right? I write blog articles, give talks about different topics, I find out how people resonate with that. And then I see, okay, from this topic, you do more, for this topic, you do less. And that is basically it.

Dawn: I realize not everyone would answer such a question, but are you willing to talk about your pricing?

Jacek: It really depends. So when companies order a training from us, it's basically never really the same. Especially because the manager fears that when they pull like 10 people from a deadline, then this really must have good impact after the training, right? So because I want to help them having this be as effective as possible, we talk a lot about what they are really doing there, what programming languages and text text they use, what other requirements they have. And then we put together a training that is like the content that I have from the shelf adapted to their needs. And that typically leads to a different price, of course. Some people also say, okay, we don't have five days for the training. The full training is five days, three days Nix and two days NixOS, but we need a specific mix. I'm preparing that for them. Also, in-person trainings at their company typically work best. You get the most value for your money when I'm there and training people, physically in front of them. And that, then it really depends. I would be typically the one person who travels. So you save money and time by having your teammates not having to travel. And of course that also increases the price if I have to jump on the plane and pay for the hotel. When we do this online, it also typically doesn't get cheaper because the deal that allows me to provide the best quality should have its best value and best deal. So for that reason the online trainings are not really cheaper. It's really complicated also. I mean, most of it is controlled by the content that you really need. If you just want the cheapest standard stock training, like from the shelf, I can go down with the price. But most companies say, okay, I don't really care about the price as long as you don't go too high. Because what is most costly to us is actually getting eight, nine or ten people away from their deadlines, right? that is really expensive and if you go down like 2000 euros and the result of this is that the training doesn't really stick then this was a bad deal even if it was cheaper, right? And very often when I'm talking to a manager I ask them if it is possible to talk with some of their technicians, and together with their technicians we dig up very interesting questions that expose that we should really cover this and this and that topic and then of course the length of the training goes up again, and with it the price a little bit but then the technical people say
“Yeah, we should really have this and that in the training and that will make it really much more effective”. And yeah, so it's really a price discussion. It's really a discussion about how important is this technology to you? How important is it that your people get more effective? And a little bit of price variation up and down doesn't really change anything for them.

Dawn: Is there anything that you can share about the teaching itself? How is it to teach Nix? Do you have any tips, any strategies, what's a good place to start and so on?

Jacek: Yeah, when you do want to do it alone, you do it like I did, eight years ago. And that was really hard. So what I did in the beginning was I kind of actually started with Nix by a strange path. So I was doing a lot of C++ programming 10 years ago. And I also used the template mechanism of its type system to do metaprogramming. And metaprogramming C++ is like a language in the language, which is purely functional. And then I asked a few colleagues who did formal methods and formal verification of code (that was in the security industry): “What kind of language should I learn to get the hang of this?” They introduced me to Haskell. So I read a book about it. And even before understanding how monads work, I was able to write better meta code in C++. And then I've seen in the Twitter sphere of all the Haskellers that most projects or many projects there had shell.nix files. I was like, okay, what is this? And I realized, wow, any project that has a shell.nix file, if you have Nix installed, you clone it, you run “nix shell” and half an hour or a few minutes later, depending on the download size, you can work. There's no more, oh, you read the 300 steps of README or install some Docker image that doesn't build any longer if it's too old or something and then you're on your own. That really did it for me. And then I said, okay, let's study what else this does. And I was completely overwhelmed by the value proposition of Nix and NixOS. So I ironed NixOS over my laptop's disk and I was unable to work for two weeks to full extent. But after that, after I was like gone out of this hole again, I was like, okay, I will never switch back and other Linux distributions are declared legacy now. What really did not help me at this time was that the Nix documentation and NixOS documentation—I mean, it is very good. People often claim that it's not good, but I would say that it is good, that the documentation team does a good job, even in the last years, this improved a lot, but it still has a lot of a reference character. In addition to this reference character, references are never books or libraries that you can Google or search through, like, “I want to do this, how to do that?”. They simply explain to you what's there and not how to hold it or what question to ask. For this, you need tutorials. And then tutorials are never more than helping you in a specific regard. There's nothing that connects the tutorials in a way that you get an overarching picture, right? That was missing for many years. Right now you have like nix.dev, which is then a very good tutorial page that concentrates on non-flakes Nix. So there's another topic, Flakes vs. Non-Flakes. Everything that is official from the NixOS community basically documents non-flakes Nix, but then customers are like, “oh, this Flake stuff, it sounds new. Can you show us that?” And then you are looking into unofficial tutorials and documentation. And that doesn't really make it easier because you have this “scatter” character of documentation here, documentation there. I mean, what I did in the beginning was I read the whole Nix documentation in one go, the whole NixOS documentation and the whole Nixpkgs documentation in one go. I understood less than half of it, but after that, when I stumbled upon new challenges, I knew where to read what. Of course, that took me like, and also the Nix Pills articles, I really like them. They are really time consuming to work through, but they got me a lot of comprehension about this stuff. This alone would take you weeks if you do this already, right? And then you are still not, yeah, experienced with how to start bigger projects. So it's really difficult getting into it. Again, I would say that documentation is good, but it has these two problems that it's a reference documentation and not like a book that you read from beginning to end and then it helps you understand the big picture. There's a lot of description of the big picture in it, but it's for people who want to look up details. That is what a reference is for. And then people really want to use Flakes. And then the official documentation doesn't really talk about Flakes. In the Nix docs, you see the Flake commands and stuff like that, but Nixpkgs and NixOS try to avoid Flakes so far, which makes sense as long as it's not stable. But it doesn't help people who really want to use Flakes. If you talk to the documentation team and other people, they patiently explain to you why this is important for them. But then you see that this has kind of a slightly political character. And newbies who just start with this, they simply don't care about the politics. They want the Nix value, right? And in my classes, I also explain to people when to use flakes and when not to. If you do it, what are the risks of it? If you don't, what can you still do without Flakes? You can do everything that you want without Flakes. Flakes just add some nice UX here and there, and some disadvantages here and there, too, because of this experimentality. And there is basically no one who really gives you all this information all at one hand, right? I mean, if I had a class like the ones that my company provides when I began, I could have been shrinking down learning time by at least three months just down to this week. And that is, I mean, I really can't say anything negative about the Nix docs other than they have a certain perspective from which they work where it really makes sense that how everything is written and structured. And it's similar to other technologies, right? When you want to start with Docker or Kubernetes, everyone who writes documentation about this has their perspective. And when you have “how to start” documentation that is tutorial-like, then you will end up reading many, many different tutorials that are written by different persons. And they don't really provide you the same style and oversight from one hand, right? That is one of the problems in Nix. It's simply technology that's even more complicated than Kubernetes.

Dawn: What are some of the interesting stories that you can share from your experience with Nix? Clients, either teaching or consulting, development.

Jacek: My favorite story about reproducibility with Nix is one that I didn't experience directly, but my colleagues taught me about this at Cyberus. So I am one of the founders of Cyberus Technology and they did a lot of Nix pioneering and building secure hypervisors and stuff like that. And we had this one situation where a colleague was building a whole Linux image locally and the Linux image was huge for testing. So they begin with building a micro hypervisor kernel: we do microkernels and we had our own hypervisor; that had to be built. And when you boot it on the machine, you boot this binary, then you boot the userland of this microkernel operating system. Then you boot the Linux kernel from an image, and that image contains all the Linux userland. The Linux kernel was typically patched. Then we had other patched packages; NixOS was really really good for providing very complex images like that. Those images also contained VirtualBox as a package that we also heavily patched so that it doesn't talk to the VirtualBox backend but against our micro hypervisor backend, that was for higher security and better performance. And then VM stuff on top of that. So people would for testing build a huge pile of tech stack themselves. And one colleague did that locally, tested it, and it worked. And then he pushed it to the GitLab and then the CI built it again. Of course, the CI would get the same result like the local build because that's how Nix works, right? Until it didn't on that day. And the thing was the CI was crashing on the VirtualBox build and locally it just worked. And they were like, wait, is Nix broken now or what?

Dawn: What do you mean by crashing?

Jacek: The build was crashing, the compiler was simply saying, sorry, I can't build this, there's a problem. I don't remember the error message, but it was, yeah, the compiler, the compiler complaining that it can't compile. And of course that shouldn't happen because that would mean that he kind of uploaded or committed broken code, but then it wouldn't have built locally, right? Okay, so he was like, okay, let's look at the top-level derivation file that he was building. Nix hashes all the content and then if it is really the same, it should have the same hash, and it would really be a Nix problem, or it might already have a different hash, then somehow the evaluation of all the Nix expressions on the CI server might have been different than on the local machine. He found out that actually from the top-level of the package that he was building there, the derivation hash was different. So that was an interesting pointer. Indeed, the CI would then build something different than he did locally. Okay, that's a good start. Then, because Nix can calculate the whole tree of dependencies and their dependencies’ dependencies up to the bootstrap build and all the source code packages, he was comparing the local one with the CI one and realized “oh, there's one source code package that is different”. It was indeed, I think, the VirtualBox package that's like 600 megabytes of source code and then he downloaded it from the CI which was really easy and you can simply get it from the CI server’s Nix store right? He diff’ed it and realized “oh there's one bit flip!”. There's only one bit flip in the whole VirtualBox source code. And then, of course, one bit flip means it really smells like broken RAM. And then they put this build machine offline, booted some RAM tester, and realized, OK, actually, the RAM is really broken. And they fixed it. And the whole story, like the whole development, from the first time the CI didn't build something until the server was up again with fixed RAM, took like three hours. And that was even in a time where they were not really experienced with all these commands that you would have used to find out what's different. So it might have gone even faster with a little more Nix experience. And imagine what would have happened without Nix. They would simply click Rebuild, right?

Dawn: Rebuild. And maybe it would have rebuilt.

Jacek: Yeah, exactly. Or maybe you could have shipped the binary to a customer with a bit flip inside and it would have been compiled, but then have problems somewhere else, right? So it's really a good feeling to use Nix like this and have problems like this pop up early and loudly in front of you. And when you fix them, they are gone. So that was a really nice experience with Nix.

Dawn: It's interesting. The problem would occur… First, there is no guarantee that a bit flip, even using Nix, would cause such an error or any error at all. Very much depends on which bit flipped and how many bits flipped right? But in this particular case, a bit flipped in the source of a software. And that means that the derivation that was created, the realized store path for that derivation was in the store. It ended up in the store and it happened probably the first, the very first time that that source has been realized into a store path. And any subsequent attempts to realize it would have been, by default at least, fetched from, or as we say, Nix substituted from the cache, from the store. It wouldn't have… that's not correct?

Jacek: So what happened the first time was, the CI server evaluated the Nix expression, which of course leads to 100 source code downloads or whatever you have there. And it, of course, downloaded the VirtualBox source code from our internal GitLab with our internal patches. And this download went through the RAM to the server's disk. While it went through the RAM, that was where the bit flip was introduced. But then the rest of it led to different hashes. I mean, wait. After downloading it and unpacking it, of course, Nix checks against the output hash that typically comes with the fetch source command or instruction. Must have happened somewhere else then. However, it's years ago already. I'm missing the details today. Somehow, some bit flip got into it. And when the server evaluated the Nix expression, it kind of led to a different derivation file than on the other computers. And that's how we found out, right?

Dawn: It may have ended up during the writing of it to disk. When Nix gives you something from the store, does it check the hash at that time? Probably not.

Jacek: That's of course implementation detail. But when it writes it to the store for the first time, of course, Nix also maintains an internal database. And yeah, that's where, I don't know. I mean, maybe the bit flip was in the database. I don't know. It's so many years ago already. The only part that I remember is that the top-level derivation, of course, had a different hash because one of its dependents had a different hash, which is how we found out. And after fixing the RAM it was gone.

Dawn: So that's an excellent example of how Nix can increase the reliability of your systems.

Jacek: Yeah. The good thing is with Nix, all the builds and even all the big integration tests, which are really typically very brittle pieces of software when you do something like this without Nix. And on Nix stuff you can really rely on. I mean, if something like this happens ever again, that something doesn't build on the CI, that built locally, this time I will be checking the RAM even earlier, because I would be so sure that something else must be broken. In eight years of Nix, of course, there have been a few Nix bugs here and there, especially when Flakes were new. But on the level of technical project management, Nix never let me down.

Dawn: Were there any stories of people who got turned completely around from, “I don't want to hear about this Nix thing,” to, “Wow, I'm not going anywhere without it.”?

Jacek: Yeah, so next week I'll be giving a workshop at a customer who wants to evaluate Nix for their project and I got to know them a few months ago and another freelancer here from my city. He's working for them now for a project and he told them, “you need to get to know Jacek because you and him, there's a lot of overlap on what you're doing so maybe I should just connect you,” so I went over there, I think it was in January or something. And we just had a coffee, sitting there for like two hours getting to know each other. And at some point I said, okay, maybe you should really have a look into Nix because what you're doing fits really well and you will profit from doing that with or without my help, of course. I can only give you an offer, but generally, what I heard, what you're doing, fits very well into the whole Nix ecosystem or the Nix ecosystem fits very well in what you're doing. And then the CEO said, yeah, sounds great. I mean, the big picture value add is clearly translatable into dollars and hours. So I would be a fan of that, but you need to convince my technical people because they have the last word in if it's possible for us, right? I had another two hours of talk with one of the colleagues from a technical perspective. I gave him some demos. Also my favorite demo is the nixos-anywhere demo. So clicking a new VM at Hetzner or whatever cloud provider you have, hacking together some example configuration from a server, and then iron it over the VM two minutes after you bought it there, in like three minutes. And then it's running as if there was NixOS on it forever. So it seems like I convinced them. So I connected via LinkedIn with all the people and two weeks later, one of the technical guys told me like, okay, look, I have been delving into this Nix stuff since you've shown it to me. And if I had known before that I would question my technical career after having seen this, maybe I shouldn't have looked into it. Because he was like, he was like, so completely overwhelmed by how useful this is, at least both in his private life, when as a hobby hacker, but also in the company. And then another two weeks later, I asked like, look, are you using it still only private or also in the company already? Because I would be really interested to see how it develops for you and how it unfolds for you. And he was like, yeah, so in the projects in the company it’s like really difficult for this and that reason. And I was like, yeah, I know these reasons because I've been doing this for years now, helping companies to get into Nix. Maybe I could just come over and do some workshop. You just pay me for the day and nothing else. And we could look into it. He said yeah, totally, let's do that. And this is kind of how this developed. There have also been people who were like, as you asked initially, who were completely against it and then started to use it.

Dawn: That was a good story by the way.

Jacek: It's fun to show it to people and see them like all their enlightening in their eyes like wow how useful is this, right? Yeah however there are people who like extremely against it but as a consultant I'm trying really hard to avoid having people tell me “No this will not work for us I'm pretty sure” and that in front of their other colleagues because if I prove them wrong by for example whooping out some example that shows their specific use case working with Nix and providing value that this is a problem because then one week later they would have to say that they were wrong and that in front of their people is embarrassing. So I'm trying to talk with people in a way that I ask more questions and giving answers to also really understand what they're doing there and what their requirements are. And then when I'm showing some examples for the first time, and this is already without having them expressed that they don't want it. So it would be fun to have people say something like this and then prove them wrong and then saying, “Okay, thank you, you completely turned me around,” but it doesn't work this way. I have to be very sensitive about foreseeing who might say something against this and then talk with those people even more, especially asking questions. And then a week later provide something. Also like psychologically, people fight for their own ideas. And if I give you an idea and ask you to please fight for it, it will not be the same as having you developing this idea yourself, right? That's also part of my job as a consultant and as a teacher to use this Socratic method of mostly asking questions and it's like hinting you the answers and when you find them they are yours, right? So I'm also providing this value to managers to help people find out that this is for them, which is of course, mostly a pedagogical and psychological thing, not a technical one, right? It's less fun for me, but it's more effective for the customer.

Dawn: It's a character developing exercise for you. I would assume.

Jacek: That's true. Yeah. I had to get more patient.

Dawn: Teaching is one of the most challenging professions, they say.

Jacek: Yeah, but it's fun. I mean, having someone saying you thank you, this really helped me is of course more rewarding than looking at the computer program that doesn't crash any longer or something when you compare it to like purely technical work. Okay, some programs when they crash until you're done and then they finally do their work are really rewarding. But yeah, having people smile and say thank you for talking to them six months later and telling you what they are doing with this now is very rewarding.

Dawn: Thank you for sharing these stories. What about ecosystem, open source contributions of yours, perhaps more recent ones that you would like to share?

Jacek: Yeah, so generally in the last year, I did less open source contributions, especially to the Nix and NixOS projects than I would have liked. I did, I filled most of my time with marketing. If I had more partners or customers asking me to please program something and ideally open source that, I would love it. But last year I had to do a lot of marketing. However, when I do open source contributions, it's mostly for the customers. So sometimes I help people get rid of their local patches to upstream that to their open source projects. Sometimes that's for Nixpkgs, package definitions, or Nix libraries. Sometimes that's for completely unrelated projects that they use with Nix. My biggest contribution to NixOS was so far porting the Perl implementation of the NixOS Integration Test Driver to Python, that was in 2019.

Dawn: Oh, that was you!

Jacek: Yeah, that was me. My personal impression is that more people started using it because Python is less frightening than Perl, I think. There's also many more contributions to improving the driver, which is great. And it seems like people trust this Python thing more than the Perl version of it before. Although it basically does the same thing as it did as the Perl implementation. It's just that more companies have the feeling that they understand this. They could improve this if they need to. And yeah, I think the Nixos Integration Test Driver with or without my contributions. I mean, even before my contributions was the biggest single biggest killer feature of NixOS, right? When you look at NixOS, it's the biggest open source package distribution in the world, that is at the same time, the freshest one. When you look at repology.org, the graph, it says everything.

Dawn: Yeah, that's a cool website. I recommend our listeners check out repology.org.

Jacek: And so it's very obvious that NixOS is among all the Linux distributions, the one with the highest package per maintainer ratio, because the community started out tiny and made a huge package distribution, right? And I think this is due to automation in general, the power of Nix and Nixpkgs. And also due to the tests. So for every release, they run so many integration tests. They test so many different things at the same time with high speed, high reliability, and so on. And that is something that companies should really use too. So that is why I contribute to this because I found this especially valuable for small companies.

Dawn: Good. Thank you for that.

Jacek: Apart from that, I had some minor contributions here and there, but nothing really special. Like many over time, but I saw kind of nagging people with Nix C++ code style changes and making more stuff const’s and so on. Whenever I read code, I try to improve it, but there hasn't been any bigger project so far in the last year.

Dawn: Let's see. What more should we talk about? What should we talk about?

Jacek: I'd love to talk about the changes that are coming this year with the Nixcademy company. So as of this year, I'm going to provide classes where you can buy individual seats. That would be online. So far, you could only buy classes for the whole team, and then I would come over or do it remotely, whatever. Generally, the customer would be the company and their whole team or teams. And in the future, you can buy an online training seat, just for you. And then you will be part of a class where we will do it all online. It will not be video-based, but everything in person. So I can, so I or another Nixcademy teacher will lead you through all the examples. And if they don't work for you, we can help you at the same time so that you will go through this with less frustration than learning yourself and also provide the whole content in exactly the order that we would have wished it ourselves to learn it as quickly as possible, right? Also, we try to provide packages, like offers that fit into the typical learning budget of individual customers and stuff like this. And I was also thinking about starting on a Nix and NixOS book. So far, there hasn't been much on that yet and many people asked me if there will be a book because I wrote a book before on C++ which they really liked and having something a similar style. So I'm kind of evaluating when to start with and that's kind of the big changes this year. Also because of the demand that is increasing, I realized working with some other colleagues together, that can specifically help specific customers better because they are experts at different fields. It's also valuable. So we will provide more value in different models so that we can help even more people. I'm really looking forward to that.

Dawn: Do you have anything to share about current events in Nix, the direction various projects are going, any feedback to the authors of these projects, anything to add to the discussion?

Jacek: So, I mean, the Nix project is huge, right? So there's something for everyone, no matter what the hell you're doing, there's some part of Nix that might help you with that. For that reason, I kind of see the stream of projects going on, being started, whatever is there, it's scattered a lot. So it's kind of hard to connect all the different projects. If you say, okay, I'm on project who does this and that. For example, there are different projects which help you calculate an automatic software bill of materials. So maybe you want to do this to create a PDF and put it next to your project product to show all the licenses involved or to do some CVE scanning or something like this. So far you can use all this, but it's most of the time with tinkering involved. The good thing is after you made it work, it will work forever because that's how Nix works, at least until the next upgrade, right? So because you can lock it down and have it very reliable, it is really worth tinkering in that regard, but still there's nothing where you can just push a button and get this additional value add and that additional value add. So I'm really looking forward to what companies like Flox, Determined Systems and so on are coming up with because they obviously try to create more integrated experiences with Nix for corporate users. And yeah, I hope there will be a lot of free stuff coming too for all the non-corporate users because that would help the whole community. But so far I don't really know what's coming there. I hope there won't be any big lock-in like solutions, right? And so that it keeps its open source character, although more and more commercial players are entering the market. And I also really hope that the Flakes situation is resolved soon, right? I mean, in 2022 I asked Eelco what's his opinion on when Flakes will be stable and integrated into the normal default installation and so on. He answered something like “months”, of course, that didn't happen like this because it's still very complicated. And I mean, I don't think it's his fault. It's just that the whole community is huge. And when you talk to five people, you get six different opinions about things. And yeah, the question is how to govern this all, right? And then you, when you look at the NixOS Foundation, have an official chair, like, or committee members there, and it should be able to decide things and move things into some direction. Of course, there's kind of, they're still building up structure there. They're really looking forward to see that this foundation will be kind of a strong authority to make the right decisions to move this up onto new levels. At the same time, of course, if you make decisions, you make these decisions in favor of one party and against the favor of another party and we will see how this develops. But I think Nix could use some more decision-making in some regards. I mean, having flakes and non-flakes and with all the complications is kind of an example of what happens when you can't really decide something, right? It seems like when flakes will be default is a decision that cannot be made.

Dawn: Do you have input on this? What would you like to have happen with flakes and their integration into stable?

Jacek: I mean, if you would say, okay, we ditch flakes and then everyone concentrates on the non-flakes thing or we say okay let's make flakes official now, as imperfect as they still are, but everyone concentrating at least into going the same direction, would already be of help, right? I mean, maintaining two different worlds might not be better than simply deciding for one and accepting that it has a few drawbacks that could be fixed over time. So simply accelerating the decision towards flakes or against, would already be better than not deciding.

Dawn: Do you think that's what's happening? Do you think there's a lack of decision or is it the pace of a certain approach towards development?

Jacek: Some parts of flakes don't seem to be thought to the end yet. I mean, they don't have to because it's not stable, right? So they are still changing things to the better. Then there are a few things that are very nice with flakes, and a few drawbacks with flakes that might have been better before, or are simply unergonomic to use yet or don't scale well, stuff like this. So I think there are many people who are right in saying we shouldn't decide yet, we should fix the bad parts first and then declare it standard and default when it's done. But then at the same time software is never done right? I think at some point we should decide okay it's good but not perfect. Let's make it default now, live with the drawbacks but then at the same time finally be able to officially document flakes as they should be, into all the workflows of NixOS, Nix packages and so on. Except that some people will not like it. And then go on, right?

Dawn: I don't know enough about the... I'm using Flakes, mostly Flakes, for most of the projects I'm involved in. But on my personal machines, I don't use Flakes. I use channels. And I don't know… One of the drawbacks I've experienced with non-flakes is that I cannot easily automate the bumping. What I'm using for flakes is the Determinate Systems GitHub action that bumps the inputs. I don't know how to do that for non-flakes. Is there a tool for that?

Jacek: Yeah, before Flakes, I was using the tool niv. And that does similar things like Flake input locking. It simply creates a JSON file. And then you can use niv show, niv update, niv add, and stuff like that. It works really similar to Flake inputs. The biggest difference is that niv uses just a list of inputs while Flake locks are a tree, so you can also lock down the inputs’ inputs. That makes many things better and easier. You can override the flakes’ inputs’ inputs so that if you say, okay, I'm importing my own nixpkgs and some other flake that has mixed packages too, I can tell the other flake to take my nixpkgs instead of theirs so I can do stuff like that. That is all very easy and ergonomic with Nix flakes I mean. But the other thing is when I'm telling my customers about the drawbacks and disadvantages of Flakes. I'm using Flakes all the time and all the projects where I'm alone too. But if something changes, if some experimental interface changes or something, it's not really a hassle for me to just fix it. When you have like 100 people working with Nix and Nix hasn't been really officially established yet and accepted, and then something changes, which is breaking so that you have to fix things over three projects at the same time, then all the people who were against Nix in the beginning will again get another argument against it. In those companies where change is very political, then flakes are sometimes not really helpful. Also, what I've seen, many people who jump onto flakes do that with the impression that with flakes you get so much more, many more features and it's all much better. I mean, the evaluation is hermetic now and can be cached, which is great. On very large flakes, you don't really see that. But then also, people look at how to write flakes and then they completely forget that there's like overlays and stuff like this and, for example, when I import another flake and I get a package from there and I want to override their OpenSSL or libc or whatever because my whole system uses very specific versions of that, then I can only so far override their flake inputs but not their packet definitions. And then it would be great if every Flake was at the same time emitting overlays and packages from this, so that the user can choose. And people very often don't even know about this because they only concentrate on Flakes and not what was even there before. I'm very often trying to teach people first all the features of Nix before Flakes and then we tack on Flakes and also see how you can use it with and without flakes and what are the advantages and disadvantages. To give people like all the facts and features on the table before they decide how they want to use it. And very often people are surprised how much is possible even without flakes. And then they are like, okay, maybe we don't really need it because some specific disadvantage of flakes might be a bigger problem in the company than thought before. And then I can help them how to use Nix in a way that when Flakes becomes stable and default and fix this and that flaw, that you can switch in a very short time because it's all already prepared, right? There's no guidance from the community on this because first it's kind of a corporate problem, and then the official documentation mostly ignores flakes for their good reasons, right? And then all the pro-flake people don't want to hear anything about what was before flakes and it's a little bit, yeah, that's, I don't know. It would be easier if there was some decision to finally, or at least a better explained path towards getting it stable and default.

Dawn: I kind of wish that with your attitude towards this, which I perceive as mature and user centric, I kind of wish that you were participating in the Nix team, helping drive forward, adopting, contributing your perspective.

Jacek: I tried to get more into the community teams, for marketing, Nix architecture and stuff like that. Somehow I didn't really get the hang of it. They are all nice people and do good work and me participating in those teams somehow didn't really work because, at least last year I was like randomly stressed out with my projects and stuff like that and I cannot really allocate some static portion of my time to recurring meetings and project work like this. At the last NixCon in Germany I asked Rok Garbas from the marketing team if I could participate with the homepage work because they do a lot of restructuring now. Then I halfway dropped out of it again because it simply didn't work for me to participate in the way how this project would really be worked on. So I respect the other guys to continue this work. This didn't work for me. I'd love to, when I scale this business up, to simply have people with, I mean, I'm doing some entrepreneurial jumps here and there and I think it will stay like this for the next years. But with this company scaling up, I hope that I will be able to give people jobs where they, like at some portions of their time, work for our paying customers, and the rest of the time contributes to open source stuff and also contribute to the NixOS community. Because that is kind of just a good thing to do, right?

Dawn: It's a generous thing to do and it's also contributing to this shared resource that we all benefit from and to varied extents build our careers on.

Jacek: Yeah, but well, I'm really not good at all these political discussions. Like, I really admire the people who have the time and patience that they put into writing an RFC, going through all the thousand opinions that come together, trying to fiddle them apart, find some common path through them so that everyone gets their share of happiness from it and all the arguments and discussions. Not all the people are nice there too, right? And I don't know, I don't have the patience for that. I became a really patient teacher over time, but I don't have enough patience for RFCs. It's admirable work that they do there. All the people who really work on the RFCs. I'm not the right person for that, I think.

Dawn: Recently, on an issue in one of my projects, I received what I perceived to be a nasty comment. And I said, and I didn't like the issue anyway. It was asking me to change the name of the project because they didn't like it. And my response to that was, I'm done with this issue. And I was mocked for doing that, in the issue, so I get your… it's like the nerve of some people to be unkind, to be impolite, publicly in such contexts… How do we get past these things and continue collaborating?

Jacek: Yeah, I remember for example there was this... I forgot his name. But I don't know him personally, I just chatted with him a few times on some technical things. He seems to be someone with a polarizing character too, but not a bad person. And then he was kind of blocked from the community and there have been comments like, okay, that he was kicked out for having sizzling stakes as avatar images or something like that. Okay, however, I also didn't have the full picture. I thought I had the full picture because I looked at all the Twitter comments and moderator comments and stuff like that. I was like, Oh, this is an unfair reason to be kicked out for it seems like the woke people are like really turning the community around now. And then just a day later more details popped up and this was like, okay, maybe the temporary block might have been fair for what was what was kind of exposed then and I felt embarrassed to have jumped on the comment train so early. So these are like community dynamics that are really hard I think especially when you're moderator and then you finally decide to kick someone because he or she did the wrong things. If you don't explain yourself perfectly from second number one, then all the shitstorm is on you too and it really has some unthankful dynamics. I think also like on the last NixCon in Darmstadt, how it developed that Anduril was one of the sponsors. Many people in the community were really extremely against that and then they already had the money but then had to remove the sponsoring stuff and the pressure that was on other people like Ron and so on from the Nixos chair. All the discussions that they had to do like internally and externally, then decide on how to communicate that. It has like nothing to do with technical progress, right? But it's huge overhead for everyone involved. And I think no one asked for that. Another kind of work that I would really not like to do, because it's kind of the opposite of being productive, right? But then at the same time, the community is huge and you don't want to piss off people for random reasons. So. Yeah, that's how it is now, I guess.

Dawn: But isn't it that the world became more sensitive? I'm 37. And I don't remember throughout my life, the world being and the internet being this reactive. You described that you said maybe not the wrong thing, but you said the right thing, even wrongly. And, this kind of thing can happen in person as well, but in person, at least you'd probably have the chance to realize that you said something in a suboptimal manner and then you rephrase yourself immediately, you'd have that opportunity, but on the internet you make a comment, you go to sleep or you make even you make a genuine mistake, right? You make an error, you do something that you would perceive as regrettable and you'd like to fix it. Like, “Sorry, I was wrong.” Is that enough these days?

Jacek: Yeah, I think that has different reasons. When you talk with people in person and you get to know more context of them, like their character and stuff like that. Imagine you are working with someone for a year already and you know he or she is a very nice person, maybe sometimes forgetful or not enough time to obey all the details, whatever. And then that person does something wrong. With all the context that you have about this person, you will give them the benefit of a doubt, right? So maybe they were stressed out and did something wrong. That happens. So you just go to them, tell them, look, this was not optimal. Let's fix it. And then everything's good, right? When you are starting to depend on work, and you have a lot of pressure to provide things to customers and so on. And then someone, by clicking the merge button too early on some merge request, on some foreign repository in the world, and then it affects you. And then someone's work affects you who you really don't know. And then when you look at the code, you don't know their perspective. You could maybe sometimes think, “Okay, only an idiot would merge this code. What, what the hell?” And then, under the pressure that you have, you might be commenting like, “You idiot, why did you merge this stuff?” You don't know them anyway, right? And I think that this polarizes someone who gets messages like this will not say, “Oh, I'm sorry, I'm really an idiot” and blah, blah, blah. It's just like, okay, maybe we forgot something and here's the fix, right? And then when people were very aggressive in their comment from the beginning, you are kind of asking for an aggressive answer back and this is not the right resolution right. At the same time I also worked myself with people, even personally, that are very, let's say, avoiding confrontation. So they would never say, oh you did something wrong here or it seems like you didn't think of me here or there, but what they would do instead is they go to their friends, talk about what a bad person you are and then maybe they just steam off a little bit. Like maybe they need to relieve the pressure from their soul and after that they can be nice again. But then people take that literally come back to you three months later and tell you what an asshole you have been to someone else. I also had that sometimes and that really doesn't work because this confrontation avoidance in the beginning creates even bigger confrontation later. That quickly gets personal, right? So without the internet, communication has always been hard, but with the asynchronicity of internet communication, it gets even harder because if you don't formulate everything perfectly from the beginning, which is basically impossible, right, people will use this to have offline discussions on how wrong this is and then the way it comes back to you is like a huge wave that you can't fix anymore, right? I think sometimes the communication on the internet forums and pull requests and RFCs, sometimes it really looks like this: “You said something like this and that and I interpreted it like this and that's why I'm saying you're a bad person.” No, but that's not what I meant. “But you wrote it like this. I'm right. You wrote it like this.” Yeah, but I didn't intend to say what you received with this. And if I knew how you would interpret it, I would have written it differently. “But you didn't.” Right?

Dawn: I do wish that we learn from these occurrences, these instances, and try to give each other our best. And try to have faith that maybe it was a bad phrasing or even a bad day. It could have been a bad day.

Jacek: Yeah, the only thing that you can do now yourself is simply putting away your own ego and having more patience with people who appear to be dicks, but whatever they write, just try to be nice and not be triggerable, right? And yeah, it doesn't fix the world, but it might be a good step into the right direction. Also keep in mind that, I mean, the people who complain about parties the most are usually people who never organized the party themselves. Just try to keep off this whenever you do something that someone else complains about. Try to keep calm.

Dawn: Keep calm.