Subscribe
Share
Share
Embed
https://nix.dev/manual/nix/2.23/release-notes/rl-2.23
https://galowicz.de/
https://galowicz.de/
Creators & Guests
What is Full Time Nix?
Long conversations with clever Nixers
Shahar "Dawn" Or (mightyiam) (00:00)
Hello and welcome to Full Time Nix. This is an episode about release notes of Nix 2.23.0 with Jacek Galowicz.
Hi, Jacek.
Jacek Galowicz (00:14)
Hey, Shahar how are you?
Shahar "Dawn" Or (mightyiam) (00:15)
Pretty good, pretty good. Well, let's talk about it.
Jacek Galowicz (00:20)
Yes, let's go.
Shahar "Dawn" Or (mightyiam) (00:20)
There are two new features. One is a new built-in. and as all built-ins it can be found under the global scope `builtins` attrset and it's called warn. So it'll be `builtins.warn`. What does it do?
Jacek Galowicz (00:43)
Yeah, so when you use `builtins.warn`, it has a very similar semantics to `builtins.trace`. Basically, `builtins.trace` is a function that takes two parameters. The first one is a string that will be printed on the user's shell. It will not go into the program itself. It will not change how your program behaves.
Shahar "Dawn" Or (mightyiam) (01:00)
Mm -hmm.
Jacek Galowicz (01:07)
and the second parameter to it will be some kind of value that goes from somewhere somewhere else and will just be forwarded while your message is printed on the terminal only visible to the user. Developers use this for deprecation warnings or just printf style debugging or whatever else. Generally, traces don't really do anything and it's
relatively difficult for the end user if you are triggering a warning somewhere that is disguised as a trace kind of to find out where it comes from and we kind of lately got this new debugging features in Nix and you can start a debugger on traces but then the problem is if you're
Shahar "Dawn" Or (mightyiam) (01:59)
.
Jacek Galowicz (01:59)
code is full of traces and you just want to catch the one specific trace that is a warning about something that you're doing wrong because of deprecation or whatever. Yeah, you get in trouble distinguishing that. So they introduced `builtins.warn`, which is kind of a special trace, which is meant to be only used for actual warnings, warnings that users ought to fix.
Shahar "Dawn" Or (mightyiam) (02:08)
Thanks for watching!
Jacek Galowicz (02:27)
it is now possible to tell the debugger that it should only trigger not on traces but only on warnings or even abort on warnings. So you could either as a developer say, OK, you activate the option debugger on warn and then your debugger will jump into the code and give you the context as soon as it sees a warning being triggered. You could also in your CI,
Shahar "Dawn" Or (mightyiam) (02:29)
So we'll tell the bugger that it should only trigger a lot of traces, a lot of warnings, or more warnings than you could. Either as a developer saying, OK, activate the launch.
Jacek Galowicz (02:55)
or whatever else, set your Nix builds with the option `--abort-on-warn` and then it will go red in front of your face when your code triggers a warning. So this is very nice because all the traces could still be in there. They wouldn't produce any noise because you can already distinguish between tracing and warning, right? Also, the warning messages have a little bit of a different
Shahar "Dawn" Or (mightyiam) (03:08)
very nice because in there they don't use any voice because you cannot be interested in what you are tracing. Also warning messages have a different route of character.
Jacek Galowicz (03:24)
output character so it's easier to grab for warnings or whatever because of their fixed format. So this is in general useful for the debugging mix developer and also for people who simply want to get rid of all warnings for example after an update or similar.
Shahar "Dawn" Or (mightyiam) (03:46)
That is great. Sorry, what is output character? Mean.
Jacek Galowicz (03:51)
I mean, not output character as a character, but the general character of the outputs being printed. Before that, you had `builtins.trace` with maybe a warning prefix, maybe not. So there was no real guarantee that all the warning output that comes from faces being used would look similar. There was no guarantee, right? Now...
Shahar "Dawn" Or (mightyiam) (03:57)
okay.
Right.
Jacek Galowicz (04:20)
with `builtins.warn` being a built-in primitive, they will have exactly the same output formatting. So you will have a nicer time simply distinguishing that.
Shahar "Dawn" Or (mightyiam) (04:29)
Beautiful.
Jacek Galowicz (04:34)
from.
This feature certainly makes it even easier because you don't get false positives. Everything that is not a warning is not being triggered any longer as long as developers use this warn feature.
Shahar "Dawn" Or (mightyiam) (04:48)
I asked for this on this course.
Jacek Galowicz (04:51)
nice.
Shahar "Dawn" Or (mightyiam) (04:52)
Anyway, thank you, who did this?
Robert H. Hensing. Thank you Robert.
Jacek Galowicz (04:59)
about hensing, yeah.
Shahar "Dawn" Or (mightyiam) (05:02)
new command nix env shell as the new nix shell so what is a nix space shell what is that the existing command
Jacek Galowicz (05:17)
Yeah, yo, if you generally want to start a new shell with specific packages installed, you could write like, Nix space, shell space, and then nixpkgs, hashtag, some package name. Then you would be launched into a new shell that seems like this package is installed there. And when you exit that shell, it's gone again. It's one of the very nice killer features of Nix, right?
Shahar "Dawn" Or (mightyiam) (05:35)
Mm -hmm.
Jacek Galowicz (05:46)
I mean, people who don't use Nix, they do Docker run something, something, and then they simply start Docker images where one or multiple tools are installed in. If you want to customize that, you have to write your own Docker image. And that is not really composable. You cannot combine two Docker images easily that have all the tools that you like. With the Nix shell, this is different. You simply line up all the...
Shahar "Dawn" Or (mightyiam) (05:46)
Yes.
Jacek Galowicz (06:15)
packages that you would like to have in your shell. Launch the command. Boom, you're in a shell now that has all these packages installed, right? Also, not to underestimate that or not to forget, Nix. I mean, you could do this from any Linux distro, right? Many people are on Fedora or Ubuntu or whatever. Maybe they don't have this package, but Nix gives you access to the biggest and freshest open source package repository in the world, right?
Shahar "Dawn" Or (mightyiam) (06:19)
Mm -hmm.
Correct. Yes.
Jacek Galowicz (06:44)
This is two features at the same time. One is these ephemeral shells. You go into this shell, have this package, you exit the shell and it's gone again, right? At least it looks like this. And access to this huge package collection, right? So that is what nix shell generally does.
Shahar "Dawn" Or (mightyiam) (06:56)
access to this huge package of actual... Yes. And it's not only for ad hoc invocations,
Right?
Jacek Galowicz (07:06)
Generally, when we look at the old commands, nix-shell, not nix -space -shell, but nix-shell, it was kind of meant to be used in two different scenarios. One is with the list of packages on the command line, so you get an ad hoc shell that you just define on the command line. The other was to source a shell.nix file or similar.
Shahar "Dawn" Or (mightyiam) (07:21)
Mm -hmm.
Jacek Galowicz (07:34)
where someone already prepared all the projects packages that you want to work on, then you wouldn't have any other command -line parameters. With Flakes, this developed a little bit in a different direction. So we have two different commands for doing these two things. One is for an ad hoc shell, where you simply line up all the package names on the command line, you use `nix shell` for that.
If you want to start a development environment from a Flake project, you would go with the next develop command.
So, Nix Develop will look into the Dev Shells section of your Flake and without any parameters, select the default one. Otherwise, you could also select a specific one on the command line so that you can have different developer environments per project if that is useful. And then it would simply launch that. And that's two separate commands now, right?
Shahar "Dawn" Or (mightyiam) (08:32)
Correct, I wonder why it was renamed, but I also, right, but there is a Nix, so there's the old Nix hyphen shell command, which does not work with Flakes, and it works with ad hoc invocations, and otherwise it looks for, by default, a shell.nix, but it can be arbitrary file. And with Flakes or,
Jacek Galowicz (08:45)
Mm -hmm. Mm -hmm.
Shahar "Dawn" Or (mightyiam) (09:00)
Flakes and nix-command experimental features. There is Nix space shell, which is only equivalent to the Nix hyphen shell ad hoc invocation.
Jacek Galowicz (09:17)
Accepted.
Shahar "Dawn" Or (mightyiam) (09:18)
The equivalent to the shell.nix usage of the classic nix-shell is nix develop.
Jacek Galowicz (09:18)
Yeah, the -
Yeah, so when you look at how `nix-shell`'s feature with ad -hoc shells was implemented, I mean, you use the nix-shell command and then dash p and then list of packages. It would kind of take all your package names, put them into a bigger Nix expression and launch that as it's kind of ad -hoc Nix shell or shell.nix. So it would kind of create one on the fly and then simulate this behavior.
The problem was there that it selects the nixpkgs for you where the packages come from. You could not really easily change that without tampering with environment variables. And the new Nix shell, Nix space shell, it waits for Flake input and then the package input from the Flake. You write Nix shell.
nixpkgs, hashtag, some package name. And then this command line defines not only the package that you want, but also the policy decision, where should this package come from? So you could mix up different nixpkgs commits on the command line. You want package A in very new and package B in very old. Or you could simply access completely
Shahar "Dawn" Or (mightyiam) (10:50)
you
Jacek Galowicz (10:54)
different third -party flakes. So you have some hobby repository somewhere and tell someone, hey, you can simply run my package from the command line without cloning my flake. So you would write nix shell github colon shahar slash saharsproject hashtag shahar's package name. And boom, that's it, right? So the old nix shell command couldn't do this. Or not without a lot of hassle.
It was trying to do this wouldn't be a nice user experience with the old command. And with the new command, it's really easy, right? So it's more powerful, actually, much more powerful.
Shahar "Dawn" Or (mightyiam) (11:35)
Yes. And what is this `nix env shell`? Is it supposed to replace nix space shell? So it's nix space env space shell.
Jacek Galowicz (11:48)
Exactly. So if you look at this alone in isolation, it looks like a completely random rename, but that's not what it is. There's a concept behind that. And the whole idea is that we have so many different Nix commands, right? Nix shell this, sorry, Nix space this, Nix space that. The developers tried to make it more intuitive by...
giving these subcommands a common scheme where you know, okay, I want to do something with development environments. So there are multiple Nix, space, dev, space, something commands, or I want to do something with package installations or building or running. Then there's a Nix space, package space, something, something subcategory. So they are kind of subdividing the commands which kind of randomly piled up over the years.
Shahar "Dawn" Or (mightyiam) (12:37)
Mm -hmm.
Jacek Galowicz (12:44)
into new categories that make more sense because they're more coherent. So for example, there will be `nix dev shell` and `nix dev run` and `nix dev print-env`. So these things all have to do something with development shells that are from a flake. But you can also run `nix env shell` in contrast to `nix dev shell`.
Shahar "Dawn" Or (mightyiam) (12:57)
Mm -hmm.
That's better
Jacek Galowicz (13:15)
And `nix env shell` will be kind of ad hoc shells. Everything with `nix env` will be something like an ad hoc thingy. Everything with `nix dev` will be something with prepared development environments from Flakes, right? So this totally makes sense. And the other commands will be like NixPackageBuild, NixPackageRun, NixPackageInfo, NixPackageEdit. And that will be something that will be like per package do something with one specific package.
And yeah, so they're kind of there's a lot of discussion going on. So I went through all the GitHub issues where they were discussing how stuff should be named. Of course, naming is one of the hard problems in IT. People put like five people in a room to discuss how something should be named and you get 10 different opinions out of it. So I think we will.
have more discussion about that in the GitHub issues and forums, how this should be. Some people will like it, some people won't, but in general, I'm looking forward to this improved structure, right?
Shahar "Dawn" Or (mightyiam) (14:24)
Yes, it sounds right. It sounds right to me and I also trust that the intelligent beings who tend to this project are making reasonable decisions.
Alright, so that's a small and yet part of a greater improvement change.
Jacek Galowicz (14:45)
Yeah, absolutely. So one other note. They will not remove the `nix shell` command because of `nix env shell`. So both will coexist for a while, I guess. So this is not a breaking change where everyone who updates to Nix 23.0 will have to rewrite all their commands. That is not the case. They are simply added at this point.
Shahar "Dawn" Or (mightyiam) (14:58)
Mm -hmm.
and I hope that it will eventually be removed because this is the time to do this all of these are supposed to be experimental it's just that we're we took our time
Jacek Galowicz (15:29)
Yeah, of course. But I mean, commands like nix develop, nix shell, nix build with the spaces in between and not the dashes. Yeah, so many people use it already. And of course, this is officially experimental, but it's experimental for so long already that people ignore that. So our experimental semantics are not really not equally useful everywhere right now.
Shahar "Dawn" Or (mightyiam) (15:58)
Mm -hmm.
Jacek Galowicz (15:59)
It totally makes sense that they are not deprecating the nix shell, nix develop, nix build and nix run commands too quickly. But eventually you are right, eventually the whole interface will look chaotic and noisy if you keep all of this, right?
Shahar "Dawn" Or (mightyiam) (16:17)
Yeah, I'll leave it up to the Nix team.
Jacek Galowicz (16:23)
Absolutely. Yeah, they will figure it out and do it well.
Shahar "Dawn" Or (mightyiam) (16:27)
So there are some more improvements. One is to the consistency.
It's about making a nix build with the keep going flag consistent with the nix-build with the keep going flag. How is it not consistent?
Jacek Galowicz (16:47)
Mm -hmm.
So first, the use case for the keep going flag is usually when you're building something on your laptop and that rebuilds many, many different derivations at the same time. For example, if you updated your inputs and then rebuilt, of course, everything is going to be rebuilt usually. And then without the keep going flag, the whole build will simply abort on the first error.
Shahar "Dawn" Or (mightyiam) (17:19)
Mm -hmm.
Jacek Galowicz (17:20)
And maybe there are 10 errors and you want different packages and you want to see them all. Typically what's also the case when you are, let's say you have like two packages building at the same time and one builds for an hour and will be completely fine. And the other one builds for like 10 minutes until it breaks. And without keep going the
whole build will break after 10 minutes after the first error, while the other package, which you could simply let continue, is aborted too. So on the next nix build, the longer package build will be completely restarted, and that might not be necessary. So if you run Nix build with the keep going parameter, it will simply print all the errors that it finds on the way, so you can on your other screen start to
fix problems and bugs and at the same time all the builds that are completely fine will continue to build, right? So you're kind of not wasting time. While you're waiting anyway, you can fix the bugs of the builds that don't work, right? So that's very useful of a feature. Typically, I don't enable it in CIs because I want CIs to get red pretty quickly and then abort so they have the resources for other builds that are not going red.
while at home while debugging stuff, I'm using the keep going flag to just keep the builds that work building while I fix the problems, right? But now it was like this feature existed in old times already with the old `nix-build` command and with the new `nix build` command, they still have it, but they are slightly, slightly different. So for example, if you have like a...
Shahar "Dawn" Or (mightyiam) (19:08)
Mm -hmm.
Jacek Galowicz (19:12)
fixed output derivations that fail to build. Nix is basically comparing hashes and hash mismatches are displayed in terms of the error message. And maybe you have like three, four, five different hash mismatches that you want to fix quickly. But then you can't because it's already aborting after the first one. So the...
Shahar "Dawn" Or (mightyiam) (19:38)
Mm -hmm.
Jacek Galowicz (19:41)
old `nix-build` command showed you all of them, the new one didn't and now this is fixed so this will be as useful as the old one.
Shahar "Dawn" Or (mightyiam) (19:50)
Sorry, so what was the difference? There was a difference in what it keeps going and what it fails on with regards to this particular hash mismatch?
Jacek Galowicz (19:58)
So...
In case of hash mismatches, there were cases where not all of them are displayed that were found.
Shahar "Dawn" Or (mightyiam) (20:10)
It's only with regards to what is displayed, you say.
Jacek Galowicz (20:17)
Yeah, kind of. In general, the whole idea of keep going is simply print all the errors that you see, but also keep going with everything else. And this promise of really keep going during a build was, so to say, not completely followed by the new nix build command.
So sometimes it was the case that you run nix space build keep going, then it displays you some kind of hash mismatch. And then, okay, then you fix it. Then you run it again, and then it will come up with a new hash mismatch that could have been shown on the first run already, but didn't. Of course, that's irritating, that's stupid, but now they fixed that.
Shahar "Dawn" Or (mightyiam) (20:43)
Mm -hmm.
huh.
Okay, so some things were...
triggering a complete failure short -circuiting everything in spite of the flag keep going and now it's fixed now they will not cause the whole build to fail it will keep going
Jacek Galowicz (21:26)
Yeah, so to say keep going now, like really keep going. The promise wasn't met completely right now it is. And it's kind of, it's not forgetting to print all of the errors that I've seen.
Shahar "Dawn" Or (mightyiam) (21:34)
Mm -hmm.
Jacek Galowicz (21:40)
There were simply corner cases in which this was the case and some cases in which it wasn't. I didn't really experience this myself, so I'm just reading from the issue description on GitHub and the pull requests what was the case. And it really seems like there were simply a few special cases, like with fixed output derivations, where it didn't keep up the promise.
Shahar "Dawn" Or (mightyiam) (22:05)
Mm -hmm.
It's good to know.
Jacek Galowicz (22:10)
I mean, if you don't know, then you can already forget. Because it's now in parity now, right?
Shahar "Dawn" Or (mightyiam) (22:15)
Yes.
I just would like to thank the author of this fix. And so whoever you are, thank you.
Okay, by the way with this feature flag keep going, this keep going flag, errors are being printed, there is no summary, right? It's not like you get a summary of errors at the end, right?
Jacek Galowicz (22:41)
Mm -hmm.
Yeah, that's true. I mean you get a summary of all the, I mean a list of packages that failed obviously. And for each of them you can run the show log command and then it will give you the whole log of this specific package built in isolation, which is very useful because typically they are just mixed on the terminal, right? But if you really want to understand a more complex error
Shahar "Dawn" Or (mightyiam) (22:53)
Mm -hmm
Jacek Galowicz (23:14)
then you want to see the package build log in isolation which you always could. It's just another command.
Shahar "Dawn" Or (mightyiam) (23:23)
okay great JSON format update modify `nix derivation add` and `show` `nix derivation add`, `nix derivation show` JSON format. What is `nix derivation add` and `nix derivation show`?
Jacek Galowicz (23:45)
So, `nix derivation show` is a command which allows you to get a more human readable form of a Nix Derivation, right? When you evaluate a Nix expression, it will kind of, yeah, interpret your Nix script and create a data structure. This data structure is called a Derivation. And when you build a package, this Derivation is actually
the thing that is being built, right? Your Nix expression is full of variables and whatever, you can update the nixpkgs inputs and whatnot, but the derivation is completely fixed, completely static, contains no variables any longer whatsoever. All the Nix paths are set there. And this is the format that kind of doesn't know about the Nix language any longer. It's just the pure static data structure.
Shahar "Dawn" Or (mightyiam) (24:17)
Yes.
Jacek Galowicz (24:42)
And this data structure can be sent to other remote builders, for example, and they will be able to build stuff. So you could, in theory, even create different programming, use different programming languages to create a derivation structure, right? You wouldn't completely have to use Nix. So this is a useful abstraction layer.
Now the question is in what format are these derivations defined? How to read them, how to create them and so on, right? And below the code, when you look at the derivation file, it is implemented in so -called ATerm format. ATerm is a very old serialization format. It is halfway human readable. So it's not really binary, it's ASCII.
But nowadays people use JSON to describe all kinds of data structures because that's a very nice language agnostic thing. All the programming languages have JSON serialization and deserialization libraries and so on. So the `nix derivation show` command gives you a very nice JSON object from a derivation. The thing is that this
feature has been used in the past by humans to simply read format and search derivation structures. So it was fine if this was not completely valid JSON. But they are going to change this. So they want to have it like real JSON by following the spec. And in this case, what they did is they revised the output
to better conform their own JSON guidelines. For example, when you have content address derivations, the content address derivations specify the output hash that comes there and the used hash algorithm and also the kind of serialization that they did. So that's very much content addressable.
storage details here. The thing is that they had like their own format of describing a content addressable derivation and they're kind of pulling this apart to be more composed. In the past they put multiple different kinds of information into the same string and separated them by columns and now they are trying to make this individual JSON fields. So it becomes easier to programmatically
do something with the derivation files. This is a pretty much deep below the hood change. So normal users who don't really play around with derivation files will not notice this. But it will enable tooling that reads derivation files to do something else with that to have easier implementations because of this improved format, right?
Shahar "Dawn" Or (mightyiam) (27:58)
Sounds like it.
So thank you.
John Ericson.
for this change. Warnings, warn on unknown settings anywhere in the command line.
Jacek Galowicz (28:14)
Right, so this is extremely useful.
Shahar "Dawn" Or (mightyiam) (28:14)
Mhm
Jacek Galowicz (28:18)
With Flakes this is not really relevant because Flakes don't really take command line like options on the command line. Flakes are so hermetic that you cannot really change the output by something on the command line. But this is relevant for the old non -flakes commands. So for example when you run nix-build or nix-instantiate on something
on a Nix expression that takes a parameter on the top level, you could provide this parameter on the command line, like a flag, a string, or whatever. And you could simply run nix-build and then your Nix expression file, and then `--option`, and parameter name and parameter value, right? And it would simply inject that into your Nix script.
and then evaluate it and that's a very useful feature in general that kind of got lost with Flakes. I hope Flakes will have this in the future too. However, if your option has a typo in it, Nix would simply do nothing with that, right? So it's like with JavaScript or similar stuff when you have a typo in a variable name, it will simply invent a new one.
and you have like undefined values and so on. And with this change, Nix actually warns you. So it sees, sorry, I only have parameters A and B and C. And the one that you provided is unknown to this set. So this is an error now. And this way you will never ever struggle any longer with expressions that for some magic reasons don't work while you for hours just had a typo somewhere and didn't notice it.
Shahar "Dawn" Or (mightyiam) (30:13)
nice i guess Cole had such an incident enough something to make this PR so thank you Cole Helbling for fixing this for us.
Jacek Galowicz (30:36)
Hehehehe
Shahar "Dawn" Or (mightyiam) (30:28)
In the category of Flake Operations, print the failing value and its type.
What does that mean?
Jacek Galowicz (30:38)
Right, so for example in flakes, flakes have a very specific format, right? You have output categories like packages, devShells, nixosConfigurations, nixosModules, overlays, blah blah blah. That list will get longer over time, I guess. However, for example, if you run... yeah, so schema will make them even user -definable.
Shahar "Dawn" Or (mightyiam) (30:53)
Mm -hmm.
well there is the schema like schemas
Jacek Galowicz (31:07)
So the list will grow longer. However, for example, let's stay on the basic case when you run nix build myPackageName. And your package is on the nix level, not really a derivation. Then, yeah, it won't work. Nix will tell you, sorry, this is not a derivation. And if you ask, OK, what is it else? Because you're
Shahar "Dawn" Or (mightyiam) (31:35)
Mm -hmm.
Jacek Galowicz (31:36)
Of course, you didn't do this on purpose. It's kind of a mistake, right? Then it wasn't really easy to find out what it was. Maybe you are like pointing at an attribute set and forgot to add the sub attribute or something like this. Then nix won't really tell you. Even if you open your flake in the REPL just to browse on the command line to your package path, then it will tell you, sorry, this is not a derivation. And you're like, okay.
Shahar "Dawn" Or (mightyiam) (31:40)
Yeah
Jacek Galowicz (32:05)
How the hell am I supposed to debug this now?
Shahar "Dawn" Or (mightyiam) (32:08)
Well, you have to put it outside of the packages into some undefined, unspecified, and then it'll tell you.
Jacek Galowicz (32:18)
Of course, that's an idea. However, even without doing that, the flake command will now print you that this is not a derivation, but it will also print you what it found otherwise. So you will see, this is not a path, but for example, a set or a list or whatever.
And then you will most probably recognize what you did wrong when you see this output. So this is another improvement that will make the next programmer's life easier.
Shahar "Dawn" Or (mightyiam) (32:53)
Mm -hmm.
That's very nice.
So thank you to whomever fixed this quality of life improvement. Thank you.
fetchTree now fetches git repositories shallowly by default. I'm not familiar with fetchTree. What is fetchTree?
Jacek Galowicz (33:21)
Yeah, so fetchTree basically is able to clone Git repositories. The thing is that Git repositories can be regarded as a tree of Git repositories again, but then also as a tree of commits. So this deep or shallow cloning of Git repositories can have two different meanings on two different axes.
Shahar "Dawn" Or (mightyiam) (33:47)
Mm -hmm.
Jacek Galowicz (33:48)
Meaning is, if you check out a Git repository that, for example, has sub -modules, then you maybe want to check out the sub -modules too. So this is a yes -no decision. And that's useful in different scenarios. The other thing is, when you check out a Git repository, you may want to check it out with all its branches and all its history, which is typically the case when you Git clone something, like without Nix even. If you just use Git.
And do `git clone `, it will download the whole history. For example, on nixpkgs, this takes a very long time because it's such a huge repository. Same with the Linux kernel, for example. And the thing is, you can tell Nix to please check out all the sub -modules, yes or no. But you weren't really able to tell Nix or...
it was was wasn't told by default to check out the whole history yes or not it was checking out the whole history so typically all the nix nixpkgs and so on which reference some github repository they wouldn't even use the git protocol but download via HTTPS
And GitHub has this nice feature that when you provide a certain URL to a repository and a commit, it will give you a tarball which doesn't even contain all the Git information any longer. So they basically used this GitHub feature as an easy way to have a shallow clone of just the latest commit of a Git repository. And the whole idea of
this change in fetchTree is now that you basically get the same very quick checkout of just the latest commit with Git by default, which will accelerate many derivations that already reference Git repositories because all the history that happened before the latest commit is most of the time not relevant to the build, right?
Shahar "Dawn" Or (mightyiam) (36:04)
That's correct.
for this podcast, FullTimeNix for a different format episode. In fact, it's very similar. It's news, but it's news for nixpkgs instead of Nix.
We have been working on a program that will generate the script, or at least a list of packages that are new since last week in nixpkgs and their descriptions and their maintainers.
Jacek Galowicz (36:47)
Nice.
Shahar "Dawn" Or (mightyiam) (36:54)
And for that, we want indeed to, well, we want to test the, both in testing and in implementation, we want to clone nixpkgs.
and we don't want to clone the entire history.
But also we don't want only a depth of one latest commit. We want a depth that is possibly by date because we know the date, but that would have to kind of be resolved. You just can't know that. It's not exposed. You have to get the commits and then examine them. But details aside, we would need an arbitrary
depth.
Jacek Galowicz (37:50)
So it's just that this little change in the default can have a huge impact on the checkout performance of your repositories or your Nix builds. But it's actually not really a revolution going on.
Shahar "Dawn" Or (mightyiam) (38:06)
Yes. And my point is only adjacent and not really related, but I thought it would be interesting. So thank you.
Jacek Galowicz (38:19)
David Hauer
Shahar "Dawn" Or (mightyiam) (38:22)
David Hauer for optimizing this for us.
should save AWS a lot of money.
Jacek Galowicz (38:36)
Mm -hmm.
Shahar "Dawn" Or (mightyiam) (38:44)
Store Object Info JSON Format now uses null rather than omitting fields.
Jacek Galowicz (38:53)
Yeah, so this is a general thing in JSON parsing or deserializing JSON into your data structures. Generally, when you have some kind of optional field that is set, yes or no, for example, I don't know, some extra information in your JSON object, there are two ways to provide this when there is no information. So if you have some field that is called, for example, extraField,
and it contains a string. You can either, when you don't have this information and you create a JSON object, say extraField, colon, null. Then you say, okay, this field has no content. Or you simply don't provide the extra fields. So that's from, if your JSON structure generally has like 10 fields, you leave one of them empty, then you have just nine fields and you don't provide this field at all.
This is an annoyance for the programmer who is going to deserialize your JSON. Because the question is, when this extra field is not there, is this an error because someone provided a different kind of JSON object? Or would you just say, okay, because this field is optional, we ignore the missing existence of this field?
And the next team decided that wherever they create JSON objects, every field that has no value will exist as a field and its value will be null. This is not a huge change, but it will make all the code that will later deserialize JSON again simpler to deal with because you can assume that all the fields will be there.
It's just that their value will be distinguished if it's usefully set or not.
So this is something that the users will not really find relevant most of the time, but everyone who is programmatically involved with dealing with JSON objects from Nix tooling, they will experience a quality of life improvement.
Shahar "Dawn" Or (mightyiam) (40:54)
Good, that's a good change.
This PR is by John Ericson. Thank you, John Ericson
And the last one is large path warnings. Introduce warnings for large paths. What are large paths?
Jacek Galowicz (41:32)
Yeah, so generally this feature is also useful, especially for beginners. For example, something that I experience a lot when I'm helping organizations to adopt Nix is that many people will try to Nixify huge projects and then many projects are, I don't know, I mean, one gigabyte of source code is something that happens in real life.
Shahar "Dawn" Or (mightyiam) (41:50)
Mm -hmm.
Jacek Galowicz (41:58)
And then, of course, in 1 gigabyte of source code, you will have multiple packages that simply live in the same repository. And then when you want to build these packages, what happens very often is people use the src attribute and make derivation and just put in huge folders that contain much, much more than the actual package source code.
Shahar "Dawn" Or (mightyiam) (42:07)
Mm -hmm.
Jacek Galowicz (42:25)
Imagine you have a Nix package whose source attribute points to 1GB source folder and you change the readme for example.
Then you have one gigabyte of changed source folder. And if you rebuild this Nix derivation, it will copy the whole one gigabyte of source code into your Nix store again and rebuild it all due to relevant or irrelevant changes. So this is generally a code smell in projects where the answer is better project structure with subfolders and or source filtering.
But generally, what you see as a user is your builds are extremely slow. And then the question is, why are they so slow? Even without knowing the project structure, you want to know why they are so slow. And what you can do now is there's a new option. It's called `--warn-large-path-threshold`. And it takes one parameter, the size of the...
megabytes to warn after and if you say for example warn large path threshold 100m, it Nix will warn you whenever it copies more than 100 megabytes of folder content into the Nix store and then you can see I didn't intend to copy so much. Thank you for this warning line right now and then you will see which derivation copies so much stuff into the store. So this is
Imagine for example you have a repository with, I don't know, for example a video folder with 10 gigabytes of videos and somehow you are accidentally copying this all to the store, right? There was... You could look into your nix store paths if they contain stuff that you wouldn't suspect to be useful but that would be very laborious and with this new warning it will become very easy to check if you are accidentally copying too much stuff, right?
Shahar "Dawn" Or (mightyiam) (44:33)
Yes, is there a default for this or is it off?
Jacek Galowicz (44:39)
It's off by default.
Shahar "Dawn" Or (mightyiam) (44:41)
Hmm.
Jacek Galowicz (44:42)
I mean, if you really intend to copy so much stuff into the nix store, then it's perfectly fine to do that, right?
Shahar "Dawn" Or (mightyiam) (44:46)
Thanks.
Rarely. Rarely is.
Jacek Galowicz (44:53)
Really, but legitimate and it's completely thinkable that this is legitimate in specific cases, right? Most of the time, of course, it's not and now it's I mean, it's easy to simply add this warning flag, get a few warnings printed and then you tell people like look We are not filtering in this package now is copying 500 megabytes of source code into the nix store and after 10 rebuilds with
Shahar "Dawn" Or (mightyiam) (45:03)
Yeah.
Jacek Galowicz (45:23)
little irrelevant changes you have five gigabytes of wasted space on your disk and that's unnecessary let's fix that right so it's easier to find out which is the package that actually does that
Shahar "Dawn" Or (mightyiam) (45:40)
I kind of am like fantasizing at the moment that Nix would...
read somehow read all the files
and try to somehow intelligently figure out that they're all source code or maybe otherwise tiny. To help with preventing this, I know that we're supposed to be engineers and we're supposed to be careful about this.
Jacek Galowicz (46:12)
Mmm.
Shahar "Dawn" Or (mightyiam) (46:30)
But...
Sometimes you don't even know what exactly you have in your tree because projects are so big.
Jacek Galowicz (46:40)
Mm.
Shahar "Dawn" Or (mightyiam) (46:42)
Maybe there are assets hidden where you didn't know that they are. Maybe Nix can help you and say, you know what? I noticed you have videos in your source.
and warn you for that. So that would be a warning by file type instead.
Jacek Galowicz (47:04)
Yeah.
That would be most probably very useful to many, but I'm not sure if this is the right layer to do this. I mean, if you are not filtering your source code at all, Nix will simply take it all and use it. And if you would add intelligence to Nix, that actually looks through the whole folder to inspect content of each file. This could be extremely slow then, because this would be a very file system intensive.
workload. Of course, you could have extra tooling that simply evaluates all your Nix expressions and then goes through the list of derivations which lists up all the paths that they're going to use. And then it could kind of scan these paths and see what they contain and then warn you in specific cases. But I guess even this
I mean, you would need some kind of heuristic, right? Because if you are not the build system which actually uses this, it will be hard to distinguish what will be useful and what not. Of course, the build system could tell you, hey, look, I used the following 100 files and the rest of the 1 ,000 files in this repository hasn't been touched by anyone. But that would also not be trivial. You would have to kind of trace every processes.
file system operations to see what's actually used. So something like this exists, but this is very invasive tooling. I think it's a... Okay.
Shahar "Dawn" Or (mightyiam) (48:39)
I'm not... I'm not...
I'm not sure that you'd have to do it this way. Can't you track, read the whole tree after the build is done and say, well, because file systems can keep track of what file has been read, when's the last time it's been read.
Jacek Galowicz (49:03)
I see. Yeah. Okay. True. However, just because something hasn't really been used already, the question is, if you change some kind of build flag, will the file then be used or not? So this is something that is, they're like, from the outside domain, what the build system does with which file could look like completely random.
And then the question is, if the tooling has too many false positives, then it will be annoying to use, right?
And then all the nice ideas and intentions don't lead to a nice user experience, while maybe potentially making all the builds slower, right? So that's not too easy to solve.
Shahar "Dawn" Or (mightyiam) (49:59)
Yes, so this PR adds a simple tool that we can explicitly choose to use and this is by Elco Dolstra. Thank you, Elco.
Jacek Galowicz (50:19)
Yeah, this is nice and easy to use, right? You can say, OK, 6 megabytes are too much, 600 megabytes are too much, 6 gigabytes are too much.
For example, when you work with VirtualBox, VirtualBox has a huge source code folder. And the build system doesn't make it easy to just build portions of that. So you typically end up building the whole thing. And it simply has huge source closures that you're going to live with. With other projects, you could define completely different thresholds.
Shahar "Dawn" Or (mightyiam) (50:36)
Mm -hmm.
Jacek Galowicz (50:58)
And you as the developer typically know what threshold you want to use on which project. And that's simply easy to do. So they are only providing the mechanics of warning after a specific threshold. But this policy has to be defined by you ad hoc on each project, which is completely fine, right?
Shahar "Dawn" Or (mightyiam) (51:25)
Yeah, well, Nix is very useful and there are always ways to make it more useful and more pleasant to use.
Jacek Galowicz (51:36)
Right.
Shahar "Dawn" Or (mightyiam) (51:37)
That's all, isn't it?
Jacek Galowicz (51:38)
Yes, that's all.
Shahar "Dawn" Or (mightyiam) (51:39)
So I'd like to thank all the Nix developers and you, Jacek, for sharing these updates and explaining them to us. And I'm looking forward to the next Nix release.
Jacek Galowicz (51:59)
Yes, thank you for having me. It was a pleasure again.
Shahar "Dawn" Or (mightyiam) (52:03)
Be well, Jacek. Everyone, thank you for listening.
Jacek Galowicz (52:06)
Have a nice day.