Having spent ~3 weeks of evenings and weekends on this, I was a bit burned out to keep trying to optimize especially because all of this code is just a dead end anyway...
The fix seemed to be to split up Bun's Zig module into ~100 pieces just like they've done on Rust but how representative that would have been on anything Bun might of shipped is very much in question!
The issue with splitting up Bun's Zig module is that Rust is designed for multiple crates compiling separate objects and linking them together in the end, but for Zig you would need to manually export and import C abi functions and you can't make use of language features like slices, generics, non-extern structs, and others anymore. In Zig, every module is compiled in the same compilation unit. Even the standard library is compiled in the same unit as your app.
Your post went into way more depth than I was expecting when I first opened up it up. You clearly spent a lot of time on it and I appreciated reading the tale so thank you. Given all the drama around AK / Bun, it just would have been so funny to see someone not involved in the project solve one of the main motivations for the move to rust.
I believe there is a fork of bun from 1.3 that a different community is maintaining. I wonder if they’ve been able to get the compile times down by modularizing the build?
Nice idea! I was only thinking about "how could I make Zig more efficient", I didn't even consider making Rust more similar to Zig.
I added a Python script wrapping rustc which serializes all rustc invocations for bun_* crates; you can find the recoridng at [1]. Turns out it only increased the build by 1 minute! So clearly my intuitition that it was just crate parallelism was not correct.
One more confounding factor though is that rustc is parallelised while Zig's compiler is not (yet). So it's still possible that might be the reason.
Because that would also cause the dependencies to be compiled in series also! The point was only to measure "what if the Bun crate was built in one shot" not any dependencies of Bun.
this! - the reason Zig defaults to one semantic analysis thread is due to current (to be improved) limitations; some ideas for Zig also will require one compilation unit - the restricted function pointers proposal being a good example: https://github.com/ziglang/zig/issues/23367
The bun team tried to parallelize Zig's semantic analysis a while back - and it DID compile about 4x as fast... YET the builds were no longer deterministic, which is a hard requirement for lots of things. More effort will need to be spent here than naively adding a thread pool. Hopefully Zig's Sema.zig gets faster, but Rust is also super slow at compiling and needs to improve. Oh - one last thing to note, Zig has very fast debug compilation and incremental compilation, it's a real nice thing to work with that you simply can't approach in most other languages right now.
> The bun team tried to parallelize Zig's semantic analysis a while back - and it DID compile about 4x as fast...
I actually collected a build with multi-threading turned on [1]. It helped quite a bit, cutting the Zig object time from 7m49s to 4m05s. But because of the reason you mentioned (and the post was already so long!) I decided not to bring it up.
> Oh - one last thing to note, Zig has very fast debug compilation and incremental compilation, it's a real nice thing to work with that you simply can't approach in most other languages right now.
Yes, in fact I also tested just running bun run build / bun run build:release and captured traces for both! The clean release build took 3m36s for Zig versus 5m50s for Rust [2][3]. Clean debug was closer: 3m07s versus 3m27s [4][5].
I also tried incremental debug builds: adding a comment to output.zig / output.rs took 48.6s / 65.4s to rebuild. Ordinary developer builds certainly gave a different picture from CI.
"Fairer" seems like a strange characterization if what you're actually telling to measure is real-world compile times. If two people were running a race and one had better, trying to make the race "fairer" by forcing that one to run with the same form as the other pretty silly.
It's like Electric Insight, which was a fantastic if proprietary tool.
There's an enormous amount of analysis you can do with tools like this from estimating how much faster the build would be with more cores (i.e. is it worth adding more) to things like a diff of one build against another: why is one build bad and the other good - what tasks had different parameters or weren't in both builds?
It's quite odd how everything gets re-invented isn't it? N years ago I didn't think that javascript would ever have a "build time" problem but I've already been asked if I can write a "filesystem" for one well known web platform and I wondered if they wanted something like EFS. I obviously said no because I didn't feel like trying to pretend that I write filesystems three times a year.
Excellent write-up, love to see profiling work on compile times. I have some rudimentary tooling I threw together myself but nothing that looks as comprehensive and insightfully presented as this, I'll be giving it a try!
Update: this tool allowed me to almost immediately diagnose a compile time mystery I encountered yesterday that I hadn't yet been able to identify with my own tools. Great work!
If you mean "how would I implement it without ptrace", there are a few options but likely the Endpoint Security API [1] would be the way to go. But it's not too great because it needs full disk access and root, neither of which I would be particularly happy accepting :(
It would be interesting to see what would be good input for LLMs to do the optimizations / trials on that automatically. Is the visual representation best or something else?
It's actually trivially easy to add a "automated report" feature to buildprof due to its architecture (just a few SQL queries on top of Perfetto's trace processor). And I'm sure AI could hill climb the build time based on this.
But I still think the visual view is invaluable for human understanding and for the "why is this even happening" problems which I think AI is still bad at seeing.
build time visualizers always reveal something embarrassing in the graph. did the bun team engage with your findings, curious if the bottlenecks were known internally
Good write up! I was hoping it was going to conclude with the author getting buns zig build way faster than Rusts but a good deep dive nevertheless
Having spent ~3 weeks of evenings and weekends on this, I was a bit burned out to keep trying to optimize especially because all of this code is just a dead end anyway...
The fix seemed to be to split up Bun's Zig module into ~100 pieces just like they've done on Rust but how representative that would have been on anything Bun might of shipped is very much in question!
The issue with splitting up Bun's Zig module is that Rust is designed for multiple crates compiling separate objects and linking them together in the end, but for Zig you would need to manually export and import C abi functions and you can't make use of language features like slices, generics, non-extern structs, and others anymore. In Zig, every module is compiled in the same compilation unit. Even the standard library is compiled in the same unit as your app.
Zig 0.17 should solve this with incremental compilation. What you said will stay but should only apply for the first compilation.
Your post went into way more depth than I was expecting when I first opened up it up. You clearly spent a lot of time on it and I appreciated reading the tale so thank you. Given all the drama around AK / Bun, it just would have been so funny to see someone not involved in the project solve one of the main motivations for the move to rust.
I believe there is a fork of bun from 1.3 that a different community is maintaining. I wonder if they’ve been able to get the compile times down by modularizing the build?
> I believe there is a fork of bun from 1.3 that a different community is maintaining.
There is Buz effort to do this which might be worked at, behind the scenes: https://github.com/jazzzooo/buz
(I'm looking forward to looking into Zig)
The next questions in my head are:
A) Is there any performance penalty on the compiled result due to being optimized in separate pieces
B) If not, why isn't the optimizer smart enough to split them up internally and optimize in parallel?
> A) Is there any performance penalty on the compiled result due to being optimized in separate pieces
AFAIK, the answer to this is no but only if LTO is doing a good job and actually optimizing well across object files.
> B) If not, why isn't the optimizer smart enough to split them up internally and optimize in parallel?
I think there was another comment by someone more knowledgable in Zig that this is something which is being discussed by the Zig compiler team.
Can you force a non-parallel build of those rust crates to make the comparison fairer?
Nice idea! I was only thinking about "how could I make Zig more efficient", I didn't even consider making Rust more similar to Zig.
I added a Python script wrapping rustc which serializes all rustc invocations for bun_* crates; you can find the recoridng at [1]. Turns out it only increased the build by 1 minute! So clearly my intuitition that it was just crate parallelism was not correct.
One more confounding factor though is that rustc is parallelised while Zig's compiler is not (yet). So it's still possible that might be the reason.
[1] https://buildprof.lalitm.com/#!/?url=https%3A%2F%2Fblogexamp...
Why not just pass `-jobs 1` to cargo?
Still won't be quite apples to apples because a multi-crate project will serialize more to disk.
Because that would also cause the dependencies to be compiled in series also! The point was only to measure "what if the Bun crate was built in one shot" not any dependencies of Bun.
this! - the reason Zig defaults to one semantic analysis thread is due to current (to be improved) limitations; some ideas for Zig also will require one compilation unit - the restricted function pointers proposal being a good example: https://github.com/ziglang/zig/issues/23367
The bun team tried to parallelize Zig's semantic analysis a while back - and it DID compile about 4x as fast... YET the builds were no longer deterministic, which is a hard requirement for lots of things. More effort will need to be spent here than naively adding a thread pool. Hopefully Zig's Sema.zig gets faster, but Rust is also super slow at compiling and needs to improve. Oh - one last thing to note, Zig has very fast debug compilation and incremental compilation, it's a real nice thing to work with that you simply can't approach in most other languages right now.
> The bun team tried to parallelize Zig's semantic analysis a while back - and it DID compile about 4x as fast...
I actually collected a build with multi-threading turned on [1]. It helped quite a bit, cutting the Zig object time from 7m49s to 4m05s. But because of the reason you mentioned (and the post was already so long!) I decided not to bring it up.
> Oh - one last thing to note, Zig has very fast debug compilation and incremental compilation, it's a real nice thing to work with that you simply can't approach in most other languages right now.
Yes, in fact I also tested just running bun run build / bun run build:release and captured traces for both! The clean release build took 3m36s for Zig versus 5m50s for Rust [2][3]. Clean debug was closer: 3m07s versus 3m27s [4][5].
I also tried incremental debug builds: adding a comment to output.zig / output.rs took 48.6s / 65.4s to rebuild. Ordinary developer builds certainly gave a different picture from CI.
[1] https://buildprof.lalitm.com/#!/?url=https%3A%2F%2Fblogexamp...
[2] https://buildprof.lalitm.com/#!/?url=https%3A%2F%2Fblogexamp...
[3] https://buildprof.lalitm.com/#!/?url=https%3A%2F%2Fblogexamp...
[4] https://buildprof.lalitm.com/#!/?url=https%3A%2F%2Fblogexamp...
[5] https://buildprof.lalitm.com/#!/?url=https%3A%2F%2Fblogexamp...
"Fairer" seems like a strange characterization if what you're actually telling to measure is real-world compile times. If two people were running a race and one had better, trying to make the race "fairer" by forcing that one to run with the same form as the other pretty silly.
It's like Electric Insight, which was a fantastic if proprietary tool.
There's an enormous amount of analysis you can do with tools like this from estimating how much faster the build would be with more cores (i.e. is it worth adding more) to things like a diff of one build against another: why is one build bad and the other good - what tasks had different parameters or weren't in both builds?
Diffing is something I definitely want to support. I've filed https://github.com/LalitMaganti/buildprof/issues/12 to add support for this, I have ideas!
Hey, I created ElectricInsight! It's nice to hear somebody besides me remembers it fondly.
I know you did!
It's quite odd how everything gets re-invented isn't it? N years ago I didn't think that javascript would ever have a "build time" problem but I've already been asked if I can write a "filesystem" for one well known web platform and I wondered if they wanted something like EFS. I obviously said no because I didn't feel like trying to pretend that I write filesystems three times a year.
Excellent write-up, love to see profiling work on compile times. I have some rudimentary tooling I threw together myself but nothing that looks as comprehensive and insightfully presented as this, I'll be giving it a try!
Update: this tool allowed me to almost immediately diagnose a compile time mystery I encountered yesterday that I hadn't yet been able to identify with my own tools. Great work!
just a comment to support post :) Incredible work!!!
Nice!
What would the macOS equivalent be?
There's an open bug for adding macOS support to buildprof: https://github.com/LalitMaganti/buildprof/issues/2
If you mean "how would I implement it without ptrace", there are a few options but likely the Endpoint Security API [1] would be the way to go. But it's not too great because it needs full disk access and root, neither of which I would be particularly happy accepting :(
[1] https://developer.apple.com/documentation/endpointsecurity
It would be interesting to see what would be good input for LLMs to do the optimizations / trials on that automatically. Is the visual representation best or something else?
It's actually trivially easy to add a "automated report" feature to buildprof due to its architecture (just a few SQL queries on top of Perfetto's trace processor). And I'm sure AI could hill climb the build time based on this.
But I still think the visual view is invaluable for human understanding and for the "why is this even happening" problems which I think AI is still bad at seeing.
This is just how hacker news is now, huh?
“I did this cool useful thing”
“Great! Please describe how I can get a machine to replace you.”
yh mate sorry for living in a capitalist society
Full LTO on the WebKit link is the serial part. Does your visualizer split link time from codegen?
Yes the whole article is about exactly that :)
You can also use --compiler-traces to understand the inside of the linking process as well; these annotations are provided by lld.
build time visualizers always reveal something embarrassing in the graph. did the bun team engage with your findings, curious if the bottlenecks were known internally
They did not, I did cc Jarred on X but not reached out anything beyond that!
[dead]