1 00:00:03,096 --> 00:00:06,476 Hello and welcome to Full Time Nix. 2 00:00:06,736 --> 00:00:18,520 This is an episode about release notes of Nix 2.23.0 with Jacek Galowicz. 3 00:00:18,520 --> 00:00:19,612 Hi, Jacek. 4 00:00:20,590 --> 00:00:23,096 Hey, Shahar how are you? 5 00:00:23,096 --> 00:00:25,336 Pretty good, pretty good. 6 00:00:25,916 --> 00:00:29,614 Well, let's talk about it. 7 00:00:29,614 --> 00:00:35,427 Yes, let's go. 8 00:00:35,427 --> 00:00:40,308 There are two new features. 9 00:00:40,308 --> 00:00:43,501 One is a new built-in. 10 00:00:43,501 --> 00:00:53,368 and as all built-ins it can be found under the global scope `builtins` attrset and 11 00:00:53,368 --> 00:00:54,808 it's called warn. 12 00:00:54,808 --> 00:00:56,928 So it'll be `builtins.warn`. 13 00:00:57,328 --> 00:01:00,206 What does it do? 14 00:01:00,206 --> 00:01:06,386 Yeah, so when you use `builtins.warn`, it has a very similar semantics to 15 00:01:06,386 --> 00:01:07,706 `builtins.trace`. 16 00:01:07,706 --> 00:01:11,726 Basically, `builtins.trace` is a function that takes two parameters. 17 00:01:11,726 --> 00:01:16,306 The first one is a string that will be printed on the user's shell. 18 00:01:16,306 --> 00:01:19,796 It will not go into the program itself. 19 00:01:19,796 --> 00:01:22,960 It will not change how your program behaves. 20 00:01:23,470 --> 00:01:27,510 and the second parameter to it will be some kind of value that goes from 21 00:01:27,510 --> 00:01:32,850 somewhere somewhere else and will just be forwarded while your message is printed on 22 00:01:32,850 --> 00:01:36,070 the terminal only visible to the user. 23 00:01:37,030 --> 00:01:43,650 Developers use this for deprecation warnings or just printf style debugging or 24 00:01:43,650 --> 00:01:45,710 whatever else. 25 00:01:46,250 --> 00:01:50,560 Generally, traces don't really do anything and it's 26 00:01:50,670 --> 00:01:55,670 relatively difficult for the end user if you are triggering a warning somewhere 27 00:01:55,670 --> 00:02:05,590 that is disguised as a trace kind of to find out where it comes from and we kind 28 00:02:05,590 --> 00:02:12,270 of lately got this new debugging features in Nix and you can start a debugger on 29 00:02:12,270 --> 00:02:15,668 traces but then the problem is if you're 30 00:02:15,668 --> 00:02:17,414 . 31 00:02:17,414 --> 00:02:22,514 you just want to catch the one specific trace that is a warning about something 32 00:02:22,514 --> 00:02:25,814 that you're doing wrong because of deprecation or whatever. 33 00:02:27,194 --> 00:02:30,454 Yeah, you get in trouble distinguishing that. 34 00:02:30,454 --> 00:02:36,374 So they introduced `builtins.warn`, which is kind of a special trace, which is meant 35 00:02:36,374 --> 00:02:41,934 to be only used for actual warnings, warnings that users ought to fix. 36 00:02:44,174 --> 00:02:49,694 it is now possible to tell the debugger that it should only trigger not on traces 37 00:02:49,694 --> 00:02:53,514 but only on warnings or even abort on warnings. 38 00:02:53,514 --> 00:03:00,854 So you could either as a developer say, OK, you activate the option debugger on 39 00:03:00,854 --> 00:03:06,754 warn and then your debugger will jump into the code and give you the context as soon 40 00:03:06,754 --> 00:03:09,534 as it sees a warning being triggered. 41 00:03:09,774 --> 00:03:11,726 You could also in your CI, 42 00:03:11,726 --> 00:03:19,366 or whatever else, set your Nix builds with the option `--abort-on-warn` and then it 43 00:03:19,366 --> 00:03:23,946 will go red in front of your face when your code triggers a warning. 44 00:03:23,946 --> 00:03:27,386 So this is very nice because all the traces could still be in there. 45 00:03:27,386 --> 00:03:32,566 They wouldn't produce any noise because you can already distinguish between 46 00:03:32,566 --> 00:03:35,046 tracing and warning, right? 47 00:03:35,126 --> 00:03:41,166 Also, the warning messages have a little bit of a different 48 00:03:41,568 --> 00:03:43,070 route of character. 49 00:03:44,202 --> 00:03:50,302 it's easier to grab for warnings or whatever because of their fixed format. 50 00:03:50,302 --> 00:03:55,562 So this is in general useful for the debugging mix developer and also for 51 00:03:55,562 --> 00:04:02,042 people who simply want to get rid of all warnings for example after an update or 52 00:04:02,042 --> 00:04:04,088 similar. 53 00:04:04,088 --> 00:04:04,928 That is great. 54 00:04:04,928 --> 00:04:08,607 Sorry, what is output character? 55 00:04:09,168 --> 00:04:09,646 Mean. 56 00:04:09,646 --> 00:04:14,566 I mean, not output character as a character, but the general character of 57 00:04:14,566 --> 00:04:16,846 the outputs being printed. 58 00:04:18,146 --> 00:04:24,086 Before that, you had `builtins.trace` with maybe a warning prefix, maybe not. 59 00:04:24,086 --> 00:04:32,886 So there was no real guarantee that all the warning output that comes from faces 60 00:04:32,886 --> 00:04:35,226 being used would look similar. 61 00:04:35,226 --> 00:04:36,966 There was no guarantee, right? 62 00:04:36,966 --> 00:04:37,902 Now... 63 00:04:37,902 --> 00:04:45,782 with `builtins.warn` being a built-in primitive, they will have exactly the same 64 00:04:45,782 --> 00:04:47,142 output formatting. 65 00:04:47,682 --> 00:05:26,834 So you will have a nicer time simply distinguishing that. 66 00:05:26,834 --> 00:05:27,914 from. 67 00:05:28,718 --> 00:05:33,758 This feature certainly makes it even easier because you don't get false 68 00:05:33,758 --> 00:05:34,438 positives. 69 00:05:34,438 --> 00:05:38,178 Everything that is not a warning is not being triggered any longer as long as 70 00:05:38,178 --> 00:05:48,477 developers use this warn feature. 71 00:05:53,048 --> 00:05:57,802 I asked for this on this course. 72 00:05:57,802 --> 00:06:05,942 nice. 73 00:06:06,156 --> 00:06:17,272 Anyway, thank you, who did this? 74 00:06:17,272 --> 00:06:21,032 Robert H. 75 00:06:21,032 --> 00:06:22,172 Hensing. 76 00:06:23,672 --> 00:06:28,248 Thank you Robert. 77 00:06:28,248 --> 00:06:41,008 new command nix env shell as the new nix shell so what is a nix space shell what is 78 00:06:41,008 --> 00:06:45,048 that the existing command 79 00:06:45,330 --> 00:06:52,210 generally want to start a new shell with specific packages installed, you could 80 00:06:52,210 --> 00:07:00,230 write like, Nix space, shell space, and then nixpkgs, hashtag, some package name. 81 00:07:00,910 --> 00:07:05,790 Then you would be launched into a new shell that seems like this package is 82 00:07:05,790 --> 00:07:06,620 installed there. 83 00:07:06,620 --> 00:07:08,760 And when you exit that shell, it's gone again. 84 00:07:08,760 --> 00:07:12,302 It's one of the very nice killer features of Nix, right? 85 00:07:12,302 --> 00:07:19,002 I mean, people who don't use Nix, they do Docker run something, something, and then 86 00:07:19,002 --> 00:07:24,332 they simply start Docker images where one or multiple tools are installed in. 87 00:07:24,332 --> 00:07:28,582 If you want to customize that, you have to write your own Docker image. 88 00:07:30,302 --> 00:07:32,272 And that is not really composable. 89 00:07:32,272 --> 00:07:36,822 You cannot combine two Docker images easily that have all the tools that you 90 00:07:36,822 --> 00:07:37,462 like. 91 00:07:37,462 --> 00:07:39,132 With the Nix shell, this is different. 92 00:07:39,132 --> 00:07:41,390 You simply line up all the... 93 00:07:41,390 --> 00:07:44,550 packages that you would like to have in your shell. 94 00:07:44,550 --> 00:07:45,710 Launch the command. 95 00:07:45,710 --> 00:07:50,150 Boom, you're in a shell now that has all these packages installed, right? 96 00:07:50,150 --> 00:07:56,330 Also, not to underestimate that or not to forget, Nix. 97 00:07:56,330 --> 00:07:59,040 I mean, you could do this from any Linux distro, right? 98 00:07:59,040 --> 00:08:01,650 Many people are on Fedora or Ubuntu or whatever. 99 00:08:01,650 --> 00:08:06,170 Maybe they don't have this package, but Nix gives you access to the biggest and 100 00:08:06,170 --> 00:08:09,870 freshest open source package repository in the world, right? 101 00:08:09,966 --> 00:08:11,976 This is two features at the same time. 102 00:08:11,976 --> 00:08:15,206 One is these ephemeral shells. 103 00:08:15,206 --> 00:08:19,286 You go into this shell, have this package, you exit the shell and it's gone again, 104 00:08:19,286 --> 00:08:19,586 right? 105 00:08:19,586 --> 00:08:21,666 At least it looks like this. 106 00:08:21,666 --> 00:08:25,666 And access to this huge package collection, right? 107 00:08:25,666 --> 00:08:29,952 So that is what nix shell generally does. 108 00:08:29,952 --> 00:08:43,576 And it's not only for ad hoc invocations, 109 00:08:43,576 --> 00:08:44,456 Right? 110 00:08:45,525 --> 00:08:52,026 Generally, when we look at the old commands, nix-shell, not nix -space 111 00:08:52,026 --> 00:08:56,426 -shell, but nix-shell, it was kind of meant to be used in two different 112 00:08:56,426 --> 00:08:57,546 scenarios. 113 00:08:57,626 --> 00:09:02,826 One is with the list of packages on the command line, so you get an ad hoc shell 114 00:09:02,826 --> 00:09:05,646 that you just define on the command line. 115 00:09:05,866 --> 00:09:10,886 The other was to source a shell.nix file or similar. 116 00:09:11,022 --> 00:09:15,942 where someone already prepared all the projects packages that you want to work 117 00:09:15,942 --> 00:09:19,822 on, then you wouldn't have any other command -line parameters. 118 00:09:20,002 --> 00:09:24,552 With Flakes, this developed a little bit in a different direction. 119 00:09:24,552 --> 00:09:28,102 So we have two different commands for doing these two things. 120 00:09:28,262 --> 00:09:33,222 One is for an ad hoc shell, where you simply line up all the package names on 121 00:09:33,222 --> 00:09:37,186 the command line, you use `nix shell` for that. 122 00:09:37,358 --> 00:09:41,878 If you want to start a development environment from a Flake project, you 123 00:09:41,878 --> 00:09:46,798 would go with the next develop command. 124 00:09:46,798 --> 00:09:53,538 So, Nix Develop will look into the Dev Shells section of your Flake and without 125 00:09:53,538 --> 00:09:56,388 any parameters, select the default one. 126 00:09:56,388 --> 00:10:00,978 Otherwise, you could also select a specific one on the command line so that 127 00:10:00,978 --> 00:10:04,278 you can have different developer environments per project if that is 128 00:10:04,278 --> 00:10:05,118 useful. 129 00:10:05,118 --> 00:10:07,538 And then it would simply launch that. 130 00:10:07,538 --> 00:10:12,824 And that's two separate commands now, right? 131 00:10:12,824 --> 00:10:20,044 Correct, I wonder why it was renamed, but I also, right, but there is a Nix, so 132 00:10:20,044 --> 00:10:28,224 there's the old Nix hyphen shell command, which does not work with Flakes, and it 133 00:10:28,224 --> 00:10:33,204 works with ad hoc invocations, and otherwise it looks for, by default, a 134 00:10:33,204 --> 00:10:37,804 shell.nix, but it can be arbitrary file. 135 00:10:38,444 --> 00:10:42,648 And with Flakes or, 136 00:10:42,648 --> 00:10:45,988 Flakes and nix-command experimental features. 137 00:10:45,988 --> 00:10:59,628 There is Nix space shell, which is only equivalent to the Nix hyphen shell ad hoc 138 00:10:59,628 --> 00:11:00,892 invocation. 139 00:11:01,262 --> 00:11:03,736 Accepted. 140 00:11:03,736 --> 00:11:16,236 The equivalent to the shell.nix usage of the classic nix-shell is nix develop. 141 00:11:16,910 --> 00:11:24,430 Yeah, so when you look at how `nix-shell`'s feature with ad -hoc shells 142 00:11:24,430 --> 00:11:30,090 was implemented, I mean, you use the nix-shell command and then dash p and then 143 00:11:30,090 --> 00:11:31,310 list of packages. 144 00:11:31,310 --> 00:11:35,810 It would kind of take all your package names, put them into a bigger Nix 145 00:11:35,810 --> 00:11:40,900 expression and launch that as it's kind of ad -hoc Nix shell or shell.nix. 146 00:11:40,900 --> 00:11:45,898 So it would kind of create one on the fly and then simulate this behavior. 147 00:11:46,606 --> 00:11:56,146 The problem was there that it selects the nixpkgs for you where the packages come 148 00:11:56,146 --> 00:11:57,226 from. 149 00:11:57,486 --> 00:12:02,866 You could not really easily change that without tampering with environment 150 00:12:02,866 --> 00:12:03,986 variables. 151 00:12:04,946 --> 00:12:13,166 And the new Nix shell, Nix space shell, it waits for Flake input and then the package 152 00:12:13,166 --> 00:12:14,246 input from the Flake. 153 00:12:14,246 --> 00:12:16,410 You write Nix shell. 154 00:12:17,133 --> 00:12:20,914 nixpkgs, hashtag, some package name. 155 00:12:20,914 --> 00:12:25,354 And then this command line defines not only the package that you want, but also 156 00:12:25,354 --> 00:12:28,674 the policy decision, where should this package come from? 157 00:12:28,674 --> 00:12:33,294 So you could mix up different nixpkgs commits on the command line. 158 00:12:33,294 --> 00:12:37,494 You want package A in very new and package B in very old. 159 00:12:37,714 --> 00:12:41,070 Or you could simply access completely 160 00:12:41,070 --> 00:12:43,070 different third -party flakes. 161 00:12:43,070 --> 00:12:46,930 So you have some hobby repository somewhere and tell someone, hey, you can 162 00:12:46,930 --> 00:12:51,850 simply run my package from the command line without cloning my flake. 163 00:12:51,850 --> 00:13:00,010 So you would write nix shell github colon shahar slash saharsproject hashtag 164 00:13:00,010 --> 00:13:02,070 shahar's package name. 165 00:13:02,630 --> 00:13:03,830 And boom, that's it, right? 166 00:13:03,830 --> 00:13:07,950 So the old nix shell command couldn't do this. 167 00:13:08,570 --> 00:13:11,054 Or not without a lot of hassle. 168 00:13:11,054 --> 00:13:16,934 It was trying to do this wouldn't be a nice user experience with the old command. 169 00:13:16,934 --> 00:13:19,814 And with the new command, it's really easy, right? 170 00:13:19,894 --> 00:13:26,104 So it's more powerful, actually, much more powerful. 171 00:13:26,104 --> 00:13:27,304 Yes. 172 00:13:27,564 --> 00:13:32,464 And what is this `nix env shell`? 173 00:13:32,464 --> 00:13:36,064 Is it supposed to replace nix space shell? 174 00:13:36,064 --> 00:13:41,294 So it's nix space env space shell. 175 00:13:41,294 --> 00:13:41,854 Exactly. 176 00:13:41,854 --> 00:13:48,014 So if you look at this alone in isolation, it looks like a completely random rename, 177 00:13:48,014 --> 00:13:50,124 but that's not what it is. 178 00:13:50,124 --> 00:13:52,094 There's a concept behind that. 179 00:13:52,094 --> 00:13:57,454 And the whole idea is that we have so many different Nix commands, right? 180 00:13:57,454 --> 00:14:01,874 Nix shell this, sorry, Nix space this, Nix space that. 181 00:14:02,334 --> 00:14:08,718 The developers tried to make it more intuitive by... 182 00:14:08,718 --> 00:14:13,918 giving these subcommands a common scheme where you know, okay, I want to do 183 00:14:13,918 --> 00:14:16,168 something with development environments. 184 00:14:16,168 --> 00:14:21,538 So there are multiple Nix, space, dev, space, something commands, or I want to do 185 00:14:21,538 --> 00:14:25,898 something with package installations or building or running. 186 00:14:25,898 --> 00:14:31,698 Then there's a Nix space, package space, something, something subcategory. 187 00:14:31,758 --> 00:14:37,638 So they are kind of subdividing the commands which kind of randomly piled up 188 00:14:37,638 --> 00:14:38,830 over the years. 189 00:14:38,830 --> 00:14:42,970 into new categories that make more sense because they're more coherent. 190 00:14:42,970 --> 00:14:52,502 So for example, there will be `nix dev shell` and `nix dev run` and `nix dev 191 00:14:52,502 --> 00:14:53,190 print-env`. 192 00:14:53,190 --> 00:14:59,650 So these things all have to do something with development shells that are from a 193 00:14:59,650 --> 00:15:00,510 flake. 194 00:15:02,090 --> 00:15:10,311 But you can also run `nix env shell` in contrast to `nix dev shell`. 195 00:15:10,311 --> 00:15:12,372 That's better 196 00:15:12,372 --> 00:15:15,266 shell` will be kind of ad hoc shells. 197 00:15:15,266 --> 00:15:19,126 Everything with `nix env` will be something like an ad hoc thingy. 198 00:15:19,126 --> 00:15:22,826 Everything with `nix dev` will be something with prepared development 199 00:15:22,826 --> 00:15:24,226 environments from Flakes, right? 200 00:15:24,226 --> 00:15:26,166 So this totally makes sense. 201 00:15:27,006 --> 00:15:31,706 And the other commands will be like NixPackageBuild, NixPackageRun, 202 00:15:31,706 --> 00:15:34,846 NixPackageInfo, NixPackageEdit. 203 00:15:35,226 --> 00:15:39,766 And that will be something that will be like per package do something with one 204 00:15:39,766 --> 00:15:41,190 specific package. 205 00:15:41,934 --> 00:15:45,834 And yeah, so they're kind of there's a lot of discussion going on. 206 00:15:45,834 --> 00:15:50,094 So I went through all the GitHub issues where they were discussing how stuff 207 00:15:50,094 --> 00:15:51,514 should be named. 208 00:15:51,514 --> 00:15:56,854 Of course, naming is one of the hard problems in IT. 209 00:15:56,854 --> 00:16:02,654 People put like five people in a room to discuss how something should be named and 210 00:16:02,654 --> 00:16:05,274 you get 10 different opinions out of it. 211 00:16:05,314 --> 00:16:08,398 So I think we will. 212 00:16:09,230 --> 00:16:13,490 have more discussion about that in the GitHub issues and forums, how this should 213 00:16:13,490 --> 00:16:13,890 be. 214 00:16:13,890 --> 00:16:17,910 Some people will like it, some people won't, but in general, I'm looking forward 215 00:16:17,910 --> 00:16:27,536 to this improved structure, right? 216 00:16:27,536 --> 00:16:30,936 Yes, it sounds right. 217 00:16:30,996 --> 00:16:39,016 It sounds right to me and I also trust that the intelligent beings who tend to 218 00:16:39,016 --> 00:16:47,544 this project are making reasonable decisions. 219 00:16:47,544 --> 00:16:54,944 Alright, so that's a small and yet part of a greater improvement change. 220 00:16:55,894 --> 00:16:56,854 absolutely. 221 00:16:56,854 --> 00:16:58,634 So one other note. 222 00:16:58,634 --> 00:17:03,554 They will not remove the `nix shell` command because of `nix env shell`. 223 00:17:03,554 --> 00:17:06,914 So both will coexist for a while, I guess. 224 00:17:07,834 --> 00:17:15,614 So this is not a breaking change where everyone who updates to Nix 23.0 will have 225 00:17:15,614 --> 00:17:17,654 to rewrite all their commands. 226 00:17:17,654 --> 00:17:19,334 That is not the case. 227 00:17:20,194 --> 00:17:26,136 They are simply added at this point. 228 00:17:26,136 --> 00:17:32,496 and I hope that it will eventually be removed because this is the time to do 229 00:17:32,496 --> 00:17:41,036 this all of these are supposed to be experimental it's just that we're we took 230 00:17:41,036 --> 00:17:42,516 our time 231 00:17:42,830 --> 00:17:43,810 Yeah, of course. 232 00:17:43,810 --> 00:17:48,250 But I mean, commands like nix develop, nix shell, nix build with the spaces in 233 00:17:48,250 --> 00:17:50,430 between and not the dashes. 234 00:17:51,030 --> 00:17:53,310 Yeah, so many people use it already. 235 00:17:53,310 --> 00:17:59,990 And of course, this is officially experimental, but it's experimental for so 236 00:17:59,990 --> 00:18:02,030 long already that people ignore that. 237 00:18:02,030 --> 00:18:09,670 So our experimental semantics are not really not equally useful everywhere right 238 00:18:09,670 --> 00:18:10,410 now. 239 00:18:11,352 --> 00:18:12,418 Mm -hmm. 240 00:18:12,782 --> 00:18:17,102 It totally makes sense that they are not deprecating the nix shell, nix develop, 241 00:18:17,102 --> 00:18:20,642 nix build and nix run commands too quickly. 242 00:18:21,002 --> 00:18:27,982 But eventually you are right, eventually the whole interface will look chaotic and 243 00:18:27,982 --> 00:18:32,568 noisy if you keep all of this, right? 244 00:18:32,568 --> 00:18:38,268 Yeah, I'll leave it up to the Nix team. 245 00:18:38,670 --> 00:18:39,450 Absolutely. 246 00:18:39,450 --> 00:18:46,136 Yeah, they will figure it out and do it well. 247 00:18:46,136 --> 00:18:48,516 So there are some more improvements. 248 00:18:48,516 --> 00:18:55,128 One is to the consistency. 249 00:18:55,128 --> 00:19:03,968 It's about making a nix build with the keep going flag consistent with the 250 00:19:03,968 --> 00:19:08,168 nix-build with the keep going flag. 251 00:19:08,168 --> 00:19:10,928 How is it not consistent? 252 00:19:11,918 --> 00:19:19,598 So first, the use case for the keep going flag is usually when you're building 253 00:19:19,598 --> 00:19:25,358 something on your laptop and that rebuilds many, many different derivations at the 254 00:19:25,358 --> 00:19:26,238 same time. 255 00:19:26,238 --> 00:19:31,578 For example, if you updated your inputs and then rebuilt, of course, everything is 256 00:19:31,578 --> 00:19:33,558 going to be rebuilt usually. 257 00:19:33,558 --> 00:19:39,818 And then without the keep going flag, the whole build will simply abort on the first 258 00:19:39,818 --> 00:19:40,622 error. 259 00:19:41,400 --> 00:19:41,998 Mm -hmm. 260 00:19:41,998 --> 00:19:46,618 And maybe there are 10 errors and you want different packages and you want to see 261 00:19:46,618 --> 00:19:48,058 them all. 262 00:19:48,058 --> 00:19:52,038 Typically what's also the case when you are, let's say you have like two packages 263 00:19:52,038 --> 00:20:00,778 building at the same time and one builds for an hour and will be completely fine. 264 00:20:00,838 --> 00:20:06,378 And the other one builds for like 10 minutes until it breaks. 265 00:20:06,858 --> 00:20:09,318 And without keep going the 266 00:20:09,422 --> 00:20:13,562 whole build will break after 10 minutes after the first error, while the other 267 00:20:13,562 --> 00:20:18,342 package, which you could simply let continue, is aborted too. 268 00:20:18,342 --> 00:20:23,142 So on the next nix build, the longer package build will be completely 269 00:20:23,142 --> 00:20:27,692 restarted, and that might not be necessary. 270 00:20:27,692 --> 00:20:32,302 So if you run Nix build with the keep going parameter, it will simply print all 271 00:20:32,302 --> 00:20:37,806 the errors that it finds on the way, so you can on your other screen start to 272 00:20:37,806 --> 00:20:43,186 fix problems and bugs and at the same time all the builds that are completely fine 273 00:20:43,186 --> 00:20:44,976 will continue to build, right? 274 00:20:44,976 --> 00:20:48,086 So you're kind of not wasting time. 275 00:20:48,126 --> 00:20:52,176 While you're waiting anyway, you can fix the bugs of the builds that don't work, 276 00:20:52,176 --> 00:20:52,386 right? 277 00:20:52,386 --> 00:20:54,886 So that's very useful of a feature. 278 00:20:55,006 --> 00:21:00,726 Typically, I don't enable it in CIs because I want CIs to get red pretty 279 00:21:00,726 --> 00:21:04,686 quickly and then abort so they have the resources for other builds that are not 280 00:21:04,686 --> 00:21:05,746 going red. 281 00:21:05,806 --> 00:21:11,606 while at home while debugging stuff, I'm using the keep going flag to just keep the 282 00:21:11,606 --> 00:21:15,486 builds that work building while I fix the problems, right? 283 00:21:15,486 --> 00:21:19,946 But now it was like this feature existed in old times already with the old 284 00:21:19,946 --> 00:21:26,606 `nix-build` command and with the new `nix build` command, they still have it, but 285 00:21:26,606 --> 00:21:28,506 they are slightly, slightly different. 286 00:21:28,506 --> 00:21:34,374 So for example, if you have like a... 287 00:21:34,374 --> 00:21:37,514 output derivations that fail to build. 288 00:21:38,394 --> 00:21:45,874 Nix is basically comparing hashes and hash mismatches are displayed in terms of the 289 00:21:45,874 --> 00:21:47,114 error message. 290 00:21:47,254 --> 00:21:52,434 And maybe you have like three, four, five different hash mismatches that you want to 291 00:21:52,434 --> 00:21:54,074 fix quickly. 292 00:21:55,554 --> 00:21:59,914 But then you can't because it's already aborting after the first one. 293 00:21:59,934 --> 00:22:02,702 So the... 294 00:22:02,702 --> 00:22:08,402 old `nix-build` command showed you all of them, the new one didn't and now this is 295 00:22:08,402 --> 00:22:16,312 fixed so this will be as useful as the old one. 296 00:22:16,312 --> 00:22:19,332 Sorry, so what was the difference? 297 00:22:19,332 --> 00:22:26,052 There was a difference in what it keeps going and what it fails on with regards to 298 00:22:26,052 --> 00:22:29,662 this particular hash mismatch? 299 00:22:30,382 --> 00:22:34,401 In case of hash mismatches, there were cases where not all of them are displayed 300 00:22:34,401 --> 00:22:35,778 that were found. 301 00:22:36,312 --> 00:22:42,912 It's only with regards to what is displayed, you say. 302 00:22:43,118 --> 00:22:44,758 Yeah, kind of. 303 00:22:46,278 --> 00:22:54,078 In general, the whole idea of keep going is simply print all the errors that you 304 00:22:54,078 --> 00:22:57,258 see, but also keep going with everything else. 305 00:22:57,338 --> 00:23:04,418 And this promise of really keep going during a build was, so to say, not 306 00:23:04,418 --> 00:23:08,514 completely followed by the new nix build command. 307 00:23:08,526 --> 00:23:12,866 So sometimes it was the case that you run nix space build keep going, then it 308 00:23:12,866 --> 00:23:15,486 displays you some kind of hash mismatch. 309 00:23:15,786 --> 00:23:17,526 And then, okay, then you fix it. 310 00:23:17,526 --> 00:23:21,186 Then you run it again, and then it will come up with a new hash mismatch that 311 00:23:21,186 --> 00:23:24,486 could have been shown on the first run already, but didn't. 312 00:23:24,665 --> 00:23:30,600 Of course, that's irritating, that's stupid, but now they fixed that. 313 00:23:30,600 --> 00:23:37,944 so some things were... 314 00:23:37,944 --> 00:23:46,504 triggering a complete failure short -circuiting everything in spite of the 315 00:23:46,504 --> 00:23:53,404 flag keep going and now it's fixed now they will not cause the whole build to 316 00:23:53,404 --> 00:23:55,658 fail it will keep going 317 00:23:55,658 --> 00:23:59,838 say keep going now, like really keep going. 318 00:23:59,998 --> 00:24:03,338 The promise wasn't met completely right now it is. 319 00:24:03,418 --> 00:24:11,470 And it's kind of, it's not forgetting to print all of the errors that I've seen. 320 00:24:11,470 --> 00:24:15,670 There were simply corner cases in which this was the case and some cases in which 321 00:24:15,670 --> 00:24:17,090 it wasn't. 322 00:24:17,410 --> 00:24:25,510 I didn't really experience this myself, so I'm just reading from the issue 323 00:24:25,510 --> 00:24:28,990 description on GitHub and the pull requests what was the case. 324 00:24:28,990 --> 00:24:34,530 And it really seems like there were simply a few special cases, like with fixed 325 00:24:34,530 --> 00:24:41,628 output derivations, where it didn't keep up the promise. 326 00:24:41,628 --> 00:24:47,406 It's good to know. 327 00:24:47,406 --> 00:24:50,846 I mean, if you don't know, then you can already forget. 328 00:24:52,086 --> 00:24:57,528 Because it's now in parity now, right? 329 00:24:57,528 --> 00:25:04,388 I just would like to thank the author of this fix. 330 00:25:04,388 --> 00:26:00,216 And so whoever you are, thank you. 331 00:26:00,216 --> 00:26:11,656 Okay, by the way with this feature flag keep going, this keep going flag, errors 332 00:26:11,656 --> 00:26:15,366 are being printed, there is no summary, right? 333 00:26:15,366 --> 00:26:19,016 It's not like you get a summary of errors at the end, right? 334 00:26:20,465 --> 00:26:21,916 Yeah, that's true. 335 00:26:21,916 --> 00:26:26,726 I mean you get a summary of all the, I mean a list of packages that failed 336 00:26:26,726 --> 00:26:27,946 obviously. 337 00:26:29,026 --> 00:26:36,886 And for each of them you can run the show log command and then it will give you the 338 00:26:36,886 --> 00:26:41,706 whole log of this specific package built in isolation, which is very useful because 339 00:26:41,706 --> 00:26:44,786 typically they are just mixed on the terminal, right? 340 00:26:45,346 --> 00:26:49,230 But if you really want to understand a more complex error 341 00:26:49,230 --> 00:26:55,230 then you want to see the package build log in isolation which you always could. 342 00:26:55,230 --> 00:26:57,196 It's just another command. 343 00:26:57,880 --> 00:27:09,900 okay great JSON format update modify `nix derivation add` and `show` `nix derivation 344 00:27:09,900 --> 00:27:13,610 add`, `nix derivation show` JSON format. 345 00:27:13,610 --> 00:27:18,780 What is `nix derivation add` and `nix derivation show`? 346 00:27:19,662 --> 00:27:28,102 So, `nix derivation show` is a command which allows you to get a more human 347 00:27:28,102 --> 00:27:31,352 readable form of a Nix Derivation, right? 348 00:27:31,352 --> 00:27:37,722 When you evaluate a Nix expression, it will kind of, yeah, interpret your Nix 349 00:27:37,722 --> 00:27:40,762 script and create a data structure. 350 00:27:41,422 --> 00:27:44,082 This data structure is called a Derivation. 351 00:27:44,222 --> 00:27:49,198 And when you build a package, this Derivation is actually 352 00:27:49,198 --> 00:27:51,168 the thing that is being built, right? 353 00:27:51,168 --> 00:27:55,958 Your Nix expression is full of variables and whatever, you can update the nixpkgs 354 00:27:55,958 --> 00:28:00,798 inputs and whatnot, but the derivation is completely fixed, completely static, 355 00:28:00,798 --> 00:28:04,538 contains no variables any longer whatsoever. 356 00:28:05,018 --> 00:28:07,938 All the Nix paths are set there. 357 00:28:07,938 --> 00:28:13,458 And this is the format that kind of doesn't know about the Nix language any 358 00:28:13,458 --> 00:28:14,178 longer. 359 00:28:14,238 --> 00:28:16,846 It's just the pure static data structure. 360 00:28:16,846 --> 00:28:21,806 And this data structure can be sent to other remote builders, for example, and 361 00:28:21,806 --> 00:28:24,266 they will be able to build stuff. 362 00:28:24,686 --> 00:28:32,846 So you could, in theory, even create different programming, use different 363 00:28:32,846 --> 00:28:35,416 programming languages to create a derivation structure, right? 364 00:28:35,416 --> 00:28:37,806 You wouldn't completely have to use Nix. 365 00:28:37,806 --> 00:28:41,866 So this is a useful abstraction layer. 366 00:28:43,406 --> 00:28:47,886 Now the question is in what format are these derivations defined? 367 00:28:48,026 --> 00:28:51,926 How to read them, how to create them and so on, right? 368 00:28:52,046 --> 00:29:00,346 And below the code, when you look at the derivation file, it is implemented in so 369 00:29:00,346 --> 00:29:01,806 -called ATerm format. 370 00:29:01,806 --> 00:29:06,426 ATerm is a very old serialization format. 371 00:29:06,426 --> 00:29:09,206 It is halfway human readable. 372 00:29:09,726 --> 00:29:12,346 So it's not really binary, it's ASCII. 373 00:29:13,710 --> 00:29:19,630 But nowadays people use JSON to describe all kinds of data structures because 374 00:29:19,630 --> 00:29:22,070 that's a very nice language agnostic thing. 375 00:29:22,070 --> 00:29:25,490 All the programming languages have JSON serialization and deserialization 376 00:29:25,490 --> 00:29:26,910 libraries and so on. 377 00:29:26,910 --> 00:29:33,230 So the `nix derivation show` command gives you a very nice JSON object from a 378 00:29:33,230 --> 00:29:34,530 derivation. 379 00:29:35,490 --> 00:29:37,614 The thing is that this 380 00:29:37,614 --> 00:29:43,994 feature has been used in the past by humans to simply read format and search 381 00:29:43,994 --> 00:29:45,594 derivation structures. 382 00:29:45,794 --> 00:29:49,574 So it was fine if this was not completely valid JSON. 383 00:29:51,174 --> 00:29:53,514 But they are going to change this. 384 00:29:53,514 --> 00:29:59,534 So they want to have it like real JSON by following the spec. 385 00:29:59,534 --> 00:30:07,470 And in this case, what they did is they revised the output 386 00:30:07,470 --> 00:30:10,870 to better conform their own JSON guidelines. 387 00:30:11,490 --> 00:30:19,010 For example, when you have content address derivations, the content address 388 00:30:19,010 --> 00:30:27,110 derivations specify the output hash that comes there and the used hash algorithm 389 00:30:27,110 --> 00:30:32,030 and also the kind of serialization that they did. 390 00:30:32,030 --> 00:30:36,122 So that's very much content addressable. 391 00:30:36,334 --> 00:30:39,384 storage details here. 392 00:30:39,384 --> 00:30:44,074 The thing is that they had like their own format of describing a content addressable 393 00:30:44,074 --> 00:30:51,074 derivation and they're kind of pulling this apart to be more composed. 394 00:30:51,074 --> 00:30:56,654 In the past they put multiple different kinds of information into the same string 395 00:30:56,654 --> 00:31:01,654 and separated them by columns and now they are trying to make this individual JSON 396 00:31:01,654 --> 00:31:02,614 fields. 397 00:31:02,694 --> 00:31:05,678 So it becomes easier to programmatically 398 00:31:05,678 --> 00:31:08,138 do something with the derivation files. 399 00:31:08,138 --> 00:31:14,838 This is a pretty much deep below the hood change. 400 00:31:14,838 --> 00:31:19,918 So normal users who don't really play around with derivation files will not 401 00:31:19,918 --> 00:31:21,278 notice this. 402 00:31:21,458 --> 00:31:28,358 But it will enable tooling that reads derivation files to do something else with 403 00:31:28,358 --> 00:31:37,144 that to have easier implementations because of this improved format, right? 404 00:31:37,144 --> 00:31:40,824 Sounds like it. 405 00:31:40,824 --> 00:31:47,480 So thank you. 406 00:31:47,480 --> 00:31:52,088 John Ericson. 407 00:31:52,088 --> 00:31:53,768 for this change. 408 00:31:55,728 --> 00:32:06,062 Warnings, warn on unknown settings anywhere in the command line. 409 00:32:06,062 --> 00:32:09,462 Right, so this is extremely useful. 410 00:32:12,270 --> 00:32:18,590 With Flakes this is not really relevant because Flakes don't really take command 411 00:32:18,590 --> 00:32:21,750 line like options on the command line. 412 00:32:21,750 --> 00:32:27,250 Flakes are so hermetic that you cannot really change the output by something on 413 00:32:27,250 --> 00:32:28,490 the command line. 414 00:32:28,490 --> 00:32:32,130 But this is relevant for the old non -flakes commands. 415 00:32:32,130 --> 00:32:41,046 So for example when you run nix-build or nix-instantiate on something 416 00:32:41,198 --> 00:32:48,398 on a Nix expression that takes a parameter on the top level, you could provide this 417 00:32:48,398 --> 00:32:54,818 parameter on the command line, like a flag, a string, or whatever. 418 00:32:54,818 --> 00:33:00,598 And you could simply run nix-build and then your Nix expression file, and then 419 00:33:00,598 --> 00:33:05,898 `--option`, and parameter name and parameter value, right? 420 00:33:06,058 --> 00:33:09,262 And it would simply inject that into your Nix script. 421 00:33:09,262 --> 00:33:14,162 and then evaluate it and that's a very useful feature in general that kind of got 422 00:33:14,162 --> 00:33:15,202 lost with Flakes. 423 00:33:15,202 --> 00:33:19,102 I hope Flakes will have this in the future too. 424 00:33:19,102 --> 00:33:30,752 However, if your option has a typo in it, Nix would simply do nothing with that, 425 00:33:30,752 --> 00:33:31,132 right? 426 00:33:31,132 --> 00:33:35,782 So it's like with JavaScript or similar stuff when you have a typo in a variable 427 00:33:35,782 --> 00:33:38,182 name, it will simply invent a new one. 428 00:33:38,478 --> 00:33:41,318 and you have like undefined values and so on. 429 00:33:41,918 --> 00:33:44,878 And with this change, Nix actually warns you. 430 00:33:44,878 --> 00:33:48,798 So it sees, sorry, I only have parameters A and B and C. 431 00:33:49,578 --> 00:33:53,178 And the one that you provided is unknown to this set. 432 00:33:53,178 --> 00:33:55,138 So this is an error now. 433 00:33:55,138 --> 00:34:01,598 And this way you will never ever struggle any longer with expressions that for some 434 00:34:01,598 --> 00:34:06,798 magic reasons don't work while you for hours just had a typo somewhere and didn't 435 00:34:06,798 --> 00:34:10,616 notice it. 436 00:34:10,616 --> 00:34:22,276 nice i guess Cole had such an incident enough something to make this PR so thank 437 00:34:22,276 --> 00:34:32,124 you Cole Helbling for fixing this for us. 438 00:34:39,946 --> 00:34:44,920 Hehehehe 439 00:34:44,920 --> 00:34:45,806 What does that mean? 440 00:34:45,806 --> 00:34:53,496 Right, so for example in flakes, flakes have a very specific format, right? 441 00:34:53,496 --> 00:34:58,546 You have output categories like packages, devShells, nixosConfigurations, 442 00:34:58,546 --> 00:35:02,126 nixosModules, overlays, blah blah blah. 443 00:35:02,126 --> 00:35:05,246 That list will get longer over time, I guess. 444 00:35:06,326 --> 00:35:11,326 However, for example, if you run... 445 00:35:11,326 --> 00:35:14,798 yeah, so schema will make them even user -definable. 446 00:35:14,798 --> 00:35:17,058 So the list will grow longer. 447 00:35:17,058 --> 00:35:21,878 However, for example, let's stay on the basic case when you run nix build 448 00:35:21,878 --> 00:35:24,878 myPackageName. 449 00:35:25,458 --> 00:35:31,058 And your package is on the nix level, not really a derivation. 450 00:35:31,278 --> 00:35:33,598 Then, yeah, it won't work. 451 00:35:33,598 --> 00:35:37,138 Nix will tell you, sorry, this is not a derivation. 452 00:35:37,478 --> 00:35:41,238 And if you ask, OK, what is it else? 453 00:35:42,718 --> 00:35:43,950 Because you're 454 00:35:43,950 --> 00:35:45,950 Of course, you didn't do this on purpose. 455 00:35:45,950 --> 00:35:47,790 It's kind of a mistake, right? 456 00:35:47,790 --> 00:35:50,770 Then it wasn't really easy to find out what it was. 457 00:35:50,770 --> 00:35:57,430 Maybe you are like pointing at an attribute set and forgot to add the sub 458 00:35:57,430 --> 00:35:59,310 attribute or something like this. 459 00:35:59,310 --> 00:36:01,650 Then nix won't really tell you. 460 00:36:01,650 --> 00:36:07,570 Even if you open your flake in the REPL just to browse on the command line to your 461 00:36:07,570 --> 00:36:10,960 package path, then it will tell you, sorry, this is not a derivation. 462 00:36:10,960 --> 00:36:12,314 And you're like, okay. 463 00:36:12,366 --> 00:36:15,246 How the hell am I supposed to debug this now? 464 00:36:15,512 --> 00:36:24,322 Well, you have to put it outside of the packages into some undefined, unspecified, 465 00:36:24,322 --> 00:36:25,998 and then it'll tell you. 466 00:36:25,998 --> 00:36:28,738 Of course, that's an idea. 467 00:36:29,957 --> 00:36:38,098 However, even without doing that, the flake command will now print you that this 468 00:36:38,098 --> 00:36:44,778 is not a derivation, but it will also print you what it found otherwise. 469 00:36:44,778 --> 00:36:50,052 So you will see, this is not a path, but for example, a set or a list or whatever. 470 00:36:50,478 --> 00:36:55,038 And then you will most probably recognize what you did wrong when you see this 471 00:36:55,038 --> 00:36:55,528 output. 472 00:36:55,528 --> 00:37:02,904 So this is another improvement that will make the next programmer's life easier. 473 00:37:02,904 --> 00:37:05,656 Mm -hmm. 474 00:37:05,656 --> 00:39:30,968 That's very nice. 475 00:39:30,968 --> 00:39:39,188 So thank you to whomever fixed this quality of life improvement. 476 00:39:39,188 --> 00:39:40,478 Thank you. 477 00:39:42,552 --> 00:39:51,132 fetchTree now fetches git repositories shallowly by default. 478 00:39:51,332 --> 00:39:53,852 I'm not familiar with fetchTree. 479 00:39:53,852 --> 00:39:55,512 What is fetchTree? 480 00:39:56,750 --> 00:40:03,330 Yeah, so fetchTree basically is able to clone Git repositories. 481 00:40:03,330 --> 00:40:12,050 The thing is that Git repositories can be regarded as a tree of Git repositories 482 00:40:12,050 --> 00:40:14,790 again, but then also as a tree of commits. 483 00:40:14,790 --> 00:40:20,330 So this deep or shallow cloning of Git repositories can have two different 484 00:40:20,330 --> 00:40:22,550 meanings on two different axes. 485 00:40:23,224 --> 00:40:23,870 Mm -hmm. 486 00:40:23,870 --> 00:40:28,990 is, if you check out a Git repository that, for example, has sub -modules, then 487 00:40:28,990 --> 00:40:31,970 you maybe want to check out the sub -modules too. 488 00:40:31,970 --> 00:40:34,010 So this is a yes -no decision. 489 00:40:35,050 --> 00:40:38,030 And that's useful in different scenarios. 490 00:40:38,470 --> 00:40:44,110 The other thing is, when you check out a Git repository, you may want to check it 491 00:40:44,110 --> 00:40:48,170 out with all its branches and all its history, which is typically the case when 492 00:40:48,170 --> 00:40:50,570 you Git clone something, like without Nix even. 493 00:40:50,570 --> 00:40:52,046 If you just use Git. 494 00:40:52,046 --> 00:40:55,786 And do `git clone `, it will download the whole history. 495 00:40:55,786 --> 00:41:00,926 For example, on nixpkgs, this takes a very long time because it's such a huge 496 00:41:00,926 --> 00:41:01,866 repository. 497 00:41:01,866 --> 00:41:04,266 Same with the Linux kernel, for example. 498 00:41:05,446 --> 00:41:11,746 And the thing is, you can tell Nix to please check out all the sub -modules, yes 499 00:41:11,746 --> 00:41:13,006 or no. 500 00:41:13,006 --> 00:41:17,038 But you weren't really able to tell Nix or... 501 00:41:17,038 --> 00:41:22,598 it was was wasn't told by default to check out the whole history yes or not it was 502 00:41:22,598 --> 00:41:30,058 checking out the whole history so typically all the nix nixpkgs and so on 503 00:41:30,058 --> 00:41:38,778 which reference some github repository they wouldn't even use the git protocol 504 00:41:38,778 --> 00:41:42,358 but download via HTTPS 505 00:41:42,734 --> 00:41:48,454 And GitHub has this nice feature that when you provide a certain URL to a repository 506 00:41:48,454 --> 00:41:54,094 and a commit, it will give you a tarball which doesn't even contain all the Git 507 00:41:54,094 --> 00:41:55,494 information any longer. 508 00:41:55,494 --> 00:42:02,254 So they basically used this GitHub feature as an easy way to have a shallow clone of 509 00:42:02,254 --> 00:42:05,574 just the latest commit of a Git repository. 510 00:42:05,814 --> 00:42:08,554 And the whole idea of 511 00:42:08,558 --> 00:42:13,958 this change in fetchTree is now that you basically get the same very quick checkout 512 00:42:13,958 --> 00:42:22,218 of just the latest commit with Git by default, which will accelerate many 513 00:42:22,218 --> 00:42:31,778 derivations that already reference Git repositories because all the history that 514 00:42:31,778 --> 00:42:36,198 happened before the latest commit is most of the time not relevant to the build, 515 00:42:36,198 --> 00:42:37,060 right? 516 00:42:40,056 --> 00:42:41,330 That's correct. 517 00:42:45,688 --> 00:42:56,308 for this podcast, FullTimeNix for a different format episode. 518 00:42:58,988 --> 00:43:00,338 In fact, it's very similar. 519 00:43:00,338 --> 00:43:05,656 It's news, but it's news for nixpkgs instead of Nix. 520 00:43:08,056 --> 00:43:18,736 We have been working on a program that will generate the script, or at least a 521 00:43:18,736 --> 00:43:25,316 list of packages that are new since last week in nixpkgs and their descriptions and 522 00:43:25,316 --> 00:43:27,856 their maintainers. 523 00:43:30,168 --> 00:43:38,748 And for that, we want indeed to, well, we want to test the, both in testing and in 524 00:43:38,748 --> 00:43:44,448 implementation, we want to clone nixpkgs. 525 00:43:48,632 --> 00:43:51,332 and we don't want to clone the entire history. 526 00:43:53,048 --> 00:43:59,158 But also we don't want only a depth of one latest commit. 527 00:43:59,158 --> 00:44:08,828 We want a depth that is possibly by date because we know the date, but that would 528 00:44:08,828 --> 00:44:10,708 have to kind of be resolved. 529 00:44:10,708 --> 00:44:12,758 You just can't know that. 530 00:44:12,758 --> 00:44:13,698 It's not exposed. 531 00:44:13,698 --> 00:44:16,548 You have to get the commits and then examine them. 532 00:44:17,628 --> 00:44:23,178 But details aside, we would need an arbitrary 533 00:44:23,256 --> 00:44:24,836 depth. 534 00:45:04,438 --> 00:45:09,998 So it's just that this little change in the default can have a huge impact on the 535 00:45:09,998 --> 00:45:14,478 checkout performance of your repositories or your Nix builds. 536 00:45:14,998 --> 00:45:19,418 But it's actually not really a revolution going on. 537 00:45:19,960 --> 00:45:20,980 Yes. 538 00:45:20,980 --> 00:45:28,980 And my point is only adjacent and not really related, but I thought it would be 539 00:45:28,980 --> 00:45:30,100 interesting. 540 00:45:30,200 --> 00:45:32,860 So thank you. 541 00:45:33,742 --> 00:45:34,704 David Hauer 542 00:45:36,184 --> 00:45:44,326 David Hauer for optimizing this for us. 543 00:45:46,104 --> 00:45:49,296 should save AWS a lot of money. 544 00:45:50,094 --> 00:45:51,114 Mm -hmm. 545 00:45:58,488 --> 00:46:06,492 Store Object Info JSON Format now uses null rather than omitting fields. 546 00:46:09,042 --> 00:46:17,962 Yeah, so this is a general thing in JSON parsing or deserializing JSON into your 547 00:46:17,962 --> 00:46:19,282 data structures. 548 00:46:19,302 --> 00:46:24,342 Generally, when you have some kind of optional field that is set, yes or no, for 549 00:46:24,342 --> 00:46:29,242 example, I don't know, some extra information in your JSON object, there are 550 00:46:29,242 --> 00:46:33,242 two ways to provide this when there is no information. 551 00:46:33,242 --> 00:46:36,910 So if you have some field that is called, for example, extraField, 552 00:46:36,910 --> 00:46:38,670 and it contains a string. 553 00:46:39,170 --> 00:46:43,850 You can either, when you don't have this information and you create a JSON object, 554 00:46:43,850 --> 00:46:46,670 say extraField, colon, null. 555 00:46:46,670 --> 00:46:49,530 Then you say, okay, this field has no content. 556 00:46:50,470 --> 00:46:54,190 Or you simply don't provide the extra fields. 557 00:46:55,170 --> 00:47:01,990 So that's from, if your JSON structure generally has like 10 fields, you leave 558 00:47:01,990 --> 00:47:06,450 one of them empty, then you have just nine fields and you don't provide this field at 559 00:47:06,450 --> 00:47:07,064 all. 560 00:47:07,374 --> 00:47:14,034 This is an annoyance for the programmer who is going to deserialize your JSON. 561 00:47:14,034 --> 00:47:19,814 Because the question is, when this extra field is not there, is this an error 562 00:47:19,814 --> 00:47:23,674 because someone provided a different kind of JSON object? 563 00:47:23,674 --> 00:47:30,454 Or would you just say, okay, because this field is optional, we ignore the missing 564 00:47:30,454 --> 00:47:32,212 existence of this field? 565 00:47:32,750 --> 00:47:40,210 And the next team decided that wherever they create JSON objects, every field that 566 00:47:40,210 --> 00:47:46,350 has no value will exist as a field and its value will be null. 567 00:47:48,330 --> 00:47:53,210 This is not a huge change, but it will make all the code that will later 568 00:47:53,210 --> 00:47:59,190 deserialize JSON again simpler to deal with because you can assume that all the 569 00:47:59,190 --> 00:48:00,556 fields will be there. 570 00:48:00,622 --> 00:48:05,982 It's just that their value will be distinguished if it's usefully set or not. 571 00:48:08,078 --> 00:48:13,428 So this is something that the users will not really find relevant most of the time, 572 00:48:13,428 --> 00:48:18,818 but everyone who is programmatically involved with dealing with JSON objects 573 00:48:18,818 --> 00:48:25,282 from Nix tooling, they will experience a quality of life improvement. 574 00:48:28,088 --> 00:48:30,688 This PR is by John Ericson. 575 00:48:30,688 --> 00:48:32,608 Thank you, John Ericson 576 00:48:36,024 --> 00:48:40,404 And the last one is large path warnings. 577 00:48:40,404 --> 00:48:43,134 Introduce warnings for large paths. 578 00:48:43,134 --> 00:48:54,149 What are large paths? 579 00:48:54,149 --> 00:48:59,770 Yeah, so generally this feature is also useful, especially for beginners. 580 00:48:59,770 --> 00:49:06,230 For example, something that I experience a lot when I'm helping organizations to 581 00:49:06,230 --> 00:49:12,970 adopt Nix is that many people will try to Nixify huge projects and then many 582 00:49:12,970 --> 00:49:18,090 projects are, I don't know, I mean, one gigabyte of source code is something that 583 00:49:18,090 --> 00:49:19,812 happens in real life. 584 00:49:20,142 --> 00:49:24,502 And then, of course, in 1 gigabyte of source code, you will have multiple 585 00:49:24,502 --> 00:49:28,002 packages that simply live in the same repository. 586 00:49:29,122 --> 00:49:33,862 And then when you want to build these packages, what happens very often is 587 00:49:33,862 --> 00:49:40,762 people use the src attribute and make derivation and just put in huge folders 588 00:49:40,762 --> 00:49:44,922 that contain much, much more than the actual package source code. 589 00:49:47,118 --> 00:49:53,838 Imagine you have a Nix package whose source attribute points to 1GB source 590 00:49:53,838 --> 00:49:57,218 folder and you change the readme for example. 591 00:49:58,638 --> 00:50:01,278 Then you have one gigabyte of changed source folder. 592 00:50:01,278 --> 00:50:07,718 And if you rebuild this Nix derivation, it will copy the whole one gigabyte of source 593 00:50:07,718 --> 00:50:14,697 code into your Nix store again and rebuild it all due to relevant or irrelevant 594 00:50:14,697 --> 00:50:15,248 changes. 595 00:50:15,248 --> 00:50:21,698 So this is generally a code smell in projects where the answer is better 596 00:50:21,698 --> 00:50:26,258 project structure with subfolders and or source filtering. 597 00:50:27,886 --> 00:50:34,106 But generally, what you see as a user is your builds are extremely slow. 598 00:50:34,106 --> 00:50:36,666 And then the question is, why are they so slow? 599 00:50:36,666 --> 00:50:39,846 Even without knowing the project structure, you want to know why they are 600 00:50:39,846 --> 00:50:40,886 so slow. 601 00:50:41,766 --> 00:50:44,586 And what you can do now is there's a new option. 602 00:50:44,586 --> 00:50:49,686 It's called `--warn-large-path-threshold`. 603 00:50:50,546 --> 00:50:53,870 And it takes one parameter, the size of the... 604 00:50:53,870 --> 00:51:01,230 megabytes to warn after and if you say for example warn large path threshold 100m, it 605 00:51:01,230 --> 00:51:08,450 Nix will warn you whenever it copies more than 100 megabytes of folder content into 606 00:51:08,450 --> 00:51:13,570 the Nix store and then you can see I didn't intend to copy so much. 607 00:51:13,570 --> 00:51:19,010 Thank you for this warning line right now and then you will see which derivation 608 00:51:19,010 --> 00:51:21,990 copies so much stuff into the store. 609 00:51:22,370 --> 00:51:23,630 So this is 610 00:51:23,630 --> 00:51:28,270 Imagine for example you have a repository with, I don't know, for example a video 611 00:51:28,270 --> 00:51:34,230 folder with 10 gigabytes of videos and somehow you are accidentally copying this 612 00:51:34,230 --> 00:51:35,610 all to the store, right? 613 00:51:35,610 --> 00:51:37,270 There was... 614 00:51:37,270 --> 00:51:42,010 You could look into your nix store paths if they contain stuff that you wouldn't 615 00:51:42,010 --> 00:51:47,890 suspect to be useful but that would be very laborious and with this new warning 616 00:51:47,890 --> 00:51:52,030 it will become very easy to check if you are accidentally copying too much stuff, 617 00:51:52,030 --> 00:51:52,850 right? 618 00:51:54,776 --> 00:51:59,856 Yes, is there a default for this or is it off? 619 00:52:00,302 --> 00:52:01,966 It's off by default. 620 00:52:02,616 --> 00:52:03,502 Hmm. 621 00:52:03,502 --> 00:52:08,222 I mean, if you really intend to copy so much stuff into the nix store, then it's 622 00:52:08,222 --> 00:52:10,342 perfectly fine to do that, right? 623 00:52:12,344 --> 00:52:14,404 Rarely. 624 00:52:14,404 --> 00:52:15,498 Rarely is. 625 00:52:15,498 --> 00:52:21,318 but legitimate and it's completely thinkable that this is legitimate in 626 00:52:21,318 --> 00:52:23,118 specific cases, right? 627 00:52:23,118 --> 00:52:28,498 Most of the time, of course, it's not and now it's I mean, it's easy to simply add 628 00:52:28,498 --> 00:52:34,198 this warning flag, get a few warnings printed and then you tell people like look 629 00:52:34,198 --> 00:52:39,538 We are not filtering in this package now is copying 500 megabytes of source code 630 00:52:39,538 --> 00:52:44,558 into the nix store and after 10 rebuilds with 631 00:52:44,558 --> 00:52:51,158 little irrelevant changes you have five gigabytes of wasted space on your disk and 632 00:52:51,158 --> 00:52:58,178 that's unnecessary let's fix that right so it's easier to find out which is the 633 00:52:58,178 --> 00:53:01,038 package that actually does that 634 00:53:01,464 --> 00:53:11,224 I kind of am like fantasizing at the moment that Nix would... 635 00:53:13,528 --> 00:53:18,688 read somehow read all the files 636 00:53:22,168 --> 00:53:34,708 and try to somehow intelligently figure out that they're all source code or maybe 637 00:53:34,708 --> 00:53:39,408 otherwise tiny. 638 00:53:41,348 --> 00:53:49,348 To help with preventing this, I know that we're supposed to be engineers and we're 639 00:53:49,348 --> 00:53:51,640 supposed to be careful about this. 640 00:53:51,640 --> 00:53:53,200 But... 641 00:53:54,936 --> 00:54:00,416 Sometimes you don't even know what exactly you have in your tree because projects are 642 00:54:00,416 --> 00:54:01,644 so big. 643 00:54:01,998 --> 00:54:02,858 Mm. 644 00:54:03,383 --> 00:54:09,413 Maybe there are assets hidden where you didn't know that they are. 645 00:54:09,413 --> 00:54:14,893 Maybe Nix can help you and say, you know what? 646 00:54:14,893 --> 00:54:19,763 I noticed you have videos in your source. 647 00:54:23,608 --> 00:54:25,148 and warn you for that. 648 00:54:25,148 --> 00:54:28,654 So that would be a warning by file type instead. 649 00:54:29,550 --> 00:54:34,310 That would be most probably very useful to many, but I'm not sure if this is the 650 00:54:34,310 --> 00:54:35,970 right layer to do this. 651 00:54:35,970 --> 00:54:43,130 I mean, if you are not filtering your source code at all, Nix will simply take 652 00:54:43,130 --> 00:54:44,710 it all and use it. 653 00:54:44,790 --> 00:54:49,410 And if you would add intelligence to Nix, that actually looks through the whole 654 00:54:49,410 --> 00:54:52,370 folder to inspect content of each file. 655 00:54:52,690 --> 00:54:58,210 This could be extremely slow then, because this would be a very file system 656 00:54:58,210 --> 00:54:58,990 intensive. 657 00:54:58,990 --> 00:55:00,410 workload. 658 00:55:01,010 --> 00:55:05,590 Of course, you could have extra tooling that simply evaluates all your Nix 659 00:55:05,590 --> 00:55:10,870 expressions and then goes through the list of derivations which lists up all the 660 00:55:10,870 --> 00:55:13,150 paths that they're going to use. 661 00:55:13,330 --> 00:55:22,230 And then it could kind of scan these paths and see what they contain and then warn 662 00:55:22,230 --> 00:55:24,530 you in specific cases. 663 00:55:24,530 --> 00:55:26,558 But I guess even this 664 00:55:26,606 --> 00:55:28,846 I mean, you would need some kind of heuristic, right? 665 00:55:28,846 --> 00:55:34,566 Because if you are not the build system which actually uses this, it will be hard 666 00:55:34,566 --> 00:55:37,226 to distinguish what will be useful and what not. 667 00:55:37,826 --> 00:55:42,526 Of course, the build system could tell you, hey, look, I used the following 100 668 00:55:42,526 --> 00:55:47,106 files and the rest of the 1 ,000 files in this repository hasn't been touched by 669 00:55:47,106 --> 00:55:47,966 anyone. 670 00:55:48,106 --> 00:55:50,026 But that would also not be trivial. 671 00:55:50,026 --> 00:55:54,686 You would have to kind of trace every processes. 672 00:55:55,182 --> 00:55:59,942 file system operations to see what's actually used. 673 00:55:59,942 --> 00:56:04,262 So something like this exists, but this is very invasive tooling. 674 00:56:05,022 --> 00:56:07,642 I think it's a... 675 00:56:07,642 --> 00:56:08,228 Okay. 676 00:56:08,228 --> 00:56:17,308 track, read the whole tree after the build is done and say, well, because file 677 00:56:17,308 --> 00:56:23,588 systems can keep track of what file has been read, when's the last time it's been 678 00:56:23,588 --> 00:56:24,428 read. 679 00:56:25,218 --> 00:56:26,218 I see. 680 00:56:26,218 --> 00:56:26,378 Yeah. 681 00:56:26,378 --> 00:56:26,798 Okay. 682 00:56:26,798 --> 00:56:27,618 True. 683 00:56:28,458 --> 00:56:34,558 However, just because something hasn't really been used already, the question is, 684 00:56:34,558 --> 00:56:43,198 if you change some kind of build flag, will the file then be used or not? 685 00:56:43,198 --> 00:56:50,458 So this is something that is, they're like, from the outside domain, what the 686 00:56:50,458 --> 00:56:54,682 build system does with which file could look like completely random. 687 00:56:55,662 --> 00:57:02,482 And then the question is, if the tooling has too many false positives, then it will 688 00:57:02,482 --> 00:57:04,382 be annoying to use, right? 689 00:57:06,286 --> 00:57:11,206 And then all the nice ideas and intentions don't lead to a nice user experience, 690 00:57:11,206 --> 00:57:15,886 while maybe potentially making all the builds slower, right? 691 00:57:15,886 --> 00:57:19,940 So that's not too easy to solve. 692 00:57:21,208 --> 00:57:35,728 Yes, so this PR adds a simple tool that we can explicitly choose to use and this is 693 00:57:35,728 --> 00:57:37,788 by Elco Dolstra. 694 00:57:37,788 --> 00:57:39,208 Thank you, Elco. 695 00:57:41,006 --> 00:57:42,836 Yeah, this is nice and easy to use, right? 696 00:57:42,836 --> 00:57:49,426 You can say, OK, 6 megabytes are too much, 600 megabytes are too much, 6 gigabytes 697 00:57:49,426 --> 00:57:50,806 are too much. 698 00:57:52,238 --> 00:57:56,238 For example, when you work with VirtualBox, VirtualBox has a huge source 699 00:57:56,238 --> 00:57:57,778 code folder. 700 00:57:58,558 --> 00:58:04,128 And the build system doesn't make it easy to just build portions of that. 701 00:58:04,128 --> 00:58:07,838 So you typically end up building the whole thing. 702 00:58:08,058 --> 00:58:15,418 And it simply has huge source closures that you're going to live with. 703 00:58:15,778 --> 00:58:20,142 With other projects, you could define completely different thresholds. 704 00:58:20,142 --> 00:58:25,522 And you as the developer typically know what threshold you want to use on which 705 00:58:25,522 --> 00:58:25,982 project. 706 00:58:25,982 --> 00:58:29,362 And that's simply easy to do. 707 00:58:29,442 --> 00:58:35,442 So they are only providing the mechanics of warning after a specific threshold. 708 00:58:35,442 --> 00:58:41,122 But this policy has to be defined by you ad hoc on each project, which is 709 00:58:41,122 --> 00:58:42,822 completely fine, right? 710 00:58:46,887 --> 00:58:52,408 Yeah, well, Nix is very useful and there are always ways to make it more useful and 711 00:58:52,408 --> 00:58:57,050 more pleasant to use. 712 00:58:57,678 --> 00:59:03,256 Right. 713 00:59:03,256 --> 00:59:04,836 That's all, isn't it? 714 00:59:05,198 --> 00:59:10,360 Yes, that's all. 715 00:59:10,360 --> 00:59:20,820 So I'd like to thank all the Nix developers and you, Jacek, for sharing 716 00:59:20,820 --> 00:59:24,600 these updates and explaining them to us. 717 00:59:24,600 --> 00:59:29,590 And I'm looking forward to the next Nix release. 718 00:59:30,318 --> 00:59:32,068 Yes, thank you for having me. 719 00:59:32,068 --> 00:59:35,032 It was a pleasure again. 720 00:59:35,032 --> 00:59:36,232 Be well, Jacek. 721 00:59:36,232 --> 00:59:38,552 Everyone, thank you for listening. 722 00:59:38,958 --> 00:59:44,698 Have a nice day.