I am one of the core Nix developers, and hte one most involved with Dynamic Derivations. Yes, I'm aiming to massage the issue trick a bit on this, and write up a proper post going over what's left.
The short version is that dynamic derivations depends on content-addressing derivations, and content-addressing derivations has been in experimental-feature limbo for a while now. However (!), with a little push + judicious cutting scope, we can get the latter stabilized. And then the remaining road to stable dynamic derivations should be very smooth.
It's a more complicated story to tell because the work is a series of yak shaves, but it's not actually a huge amount of work.
Since there's a target that depends on a generated source file for Nix's bison parser. But other targets incrementally compile comparably with regular ninja. So far we observed that Nix's sandboxing overhead is neglible.
As a developer not of this, but of dynamic derivations in Nix itself, let me put it this way:
A long term end goal is that if you don't change headers and only change is single C/C++, anywhere in the build graph for your entire system you should get a quick recompile of just that file + relink just the executables/shared libraries the output object file is built in.
This will require boiling an oceans, to get all the packages' build systems using Ninja or similar like this. But that's hopefully
- less boiling that rewriting the whole world of open source in Bazel/Buck2.
- a far more incremental, crowd-source-able project, as you could convert packages one-by-one, starting with big builds like LLVM and Chromium as Edgar says.
No worries. I think Make vs Ninja is a whole another discussion but how it relates to this project is that Ninja has less features than Make (e.g. no globs, etc) which makes it attractive as a simple build graph format.
Nix provides the toolchain and dependency management (like where boost is from), as well as ability to remotely execute it which makes it attractive to scale out large builds like a `-j 999`. Nix also lets you do things like patching boost and then recompiling both boost and the downstreams (incrementally and with early cut-off if using nix-ninja) all in one build graph.
All in all, probably not useful if you're not already needing features from Nix. But if you are, this should speed up your builds by a significant amount.
Not doing nix.. I tried a few times.. maybe I'll adapt my docker-based tooling at a certain point.
Less features is probably good, because make originally was just the dependency thing, but got abused as a build system.
Any performance improvements such as incremental compilations / incremental builds / interpreters / copy-on-write / hot-code reloading are always welcome.
How long do we expect it to take dynamic derivations to get stabilized?
The GitHub RFC says it's only 80% complete
I am one of the core Nix developers, and hte one most involved with Dynamic Derivations. Yes, I'm aiming to massage the issue trick a bit on this, and write up a proper post going over what's left.
The short version is that dynamic derivations depends on content-addressing derivations, and content-addressing derivations has been in experimental-feature limbo for a while now. However (!), with a little push + judicious cutting scope, we can get the latter stabilized. And then the remaining road to stable dynamic derivations should be very smooth.
It's a more complicated story to tell because the work is a series of yak shaves, but it's not actually a huge amount of work.
I'm not sure, but we're in contact with some core Nix folks to see what we can do help move it along.
Is there a particular reason you chose to target ninja over other formats?
It made sense internally and we also thought it'll be compelling for people interested in Nix since Nix itself uses meson and outputs ninja.
What other formats do you think are worth targeting?
Not particularly, I was just curious thanks!
This is really exciting! What are the early performance gains looking like in nix compilation?
We're still missing this feature to fully incrementally compile nix: https://github.com/pdtpartners/nix-ninja/issues/19
Since there's a target that depends on a generated source file for Nix's bison parser. But other targets incrementally compile comparably with regular ninja. So far we observed that Nix's sandboxing overhead is neglible.
As a developer not of this, but of dynamic derivations in Nix itself, let me put it this way:
A long term end goal is that if you don't change headers and only change is single C/C++, anywhere in the build graph for your entire system you should get a quick recompile of just that file + relink just the executables/shared libraries the output object file is built in.
This will require boiling an oceans, to get all the packages' build systems using Ninja or similar like this. But that's hopefully
- less boiling that rewriting the whole world of open source in Bazel/Buck2.
- a far more incremental, crowd-source-able project, as you could convert packages one-by-one, starting with big builds like LLVM and Chromium as Edgar says.
Wow this is so cool! Looking forward to trying it out!
This is cool! I'll be staying tuned for updates
Very cool.
I'll probably get downvoted. But isn't that exactly what make does? specify how an output file should be made, and which input files it depends on.
No worries. I think Make vs Ninja is a whole another discussion but how it relates to this project is that Ninja has less features than Make (e.g. no globs, etc) which makes it attractive as a simple build graph format.
Nix provides the toolchain and dependency management (like where boost is from), as well as ability to remotely execute it which makes it attractive to scale out large builds like a `-j 999`. Nix also lets you do things like patching boost and then recompiling both boost and the downstreams (incrementally and with early cut-off if using nix-ninja) all in one build graph.
All in all, probably not useful if you're not already needing features from Nix. But if you are, this should speed up your builds by a significant amount.
Not doing nix.. I tried a few times.. maybe I'll adapt my docker-based tooling at a certain point.
Less features is probably good, because make originally was just the dependency thing, but got abused as a build system.
Any performance improvements such as incremental compilations / incremental builds / interpreters / copy-on-write / hot-code reloading are always welcome.