> “bzip3 is not yet listed on the large text compression benchmark” It is now
And it comes in relatively well, in my opinion. I'm a compression amateur, but bzip3 is the first entry I recognize as a general purpose compression program.
Yes, bzip3 compresses that 1GB text file into 170 MB while zstd needs 213 MB (SI units, from my own tests with the "enwik9" file of the GP's page, and the latest releases of both programs, same commands). But the decompression (memory and speed) is a different story.
On my desktop, decompressing with zstd requires 128 MB and 1 s, while bzip3 uses 3.2 GB and 90 s.
Without `-b 511`, bzip3's ratio decreases and decompression requires about the same memory as zstd. But the decompression is still two orders of magnitude slower.
Matt added bzip3 to the benchmark seemingly after the last year thread. If the zstd benchmark, last update at max 22 in 2016, were to be updated for today’s zstd (for long-range if nothing else) after coming across this thread, what would be the most compact possible list of option sets to benchmark to reflect the value of LR et al.?
The benchmarks are disingenuous, to the point of looking cherry-picked. The block size for bzip3 is set to 512MB, but the window size for zstd is left to its default (8MB I believe for high levels). So in this corpus, which is made up of all versions of Perl source code concatenated, the window is too small to see all the identical files and just match them.
Also corpora made out of very long repetitions are pretty much the best case scenario for BWT-based compressors.
If we match the window size of zstd to that of bzip3 we get dramatically different results:
% gzcat *.gz | time zstd -T8 -16 | wc -c # baseline
2819113884
zstd -T8 -16 2054.50s user 3.47s system 783% cpu 4:22.80 total
% gzcat *.gz | time zstd -T8 -16 --long=29 | wc -c
196405076
zstd -T8 -16 --long=29 1083.06s user 2.41s system 783% cpu 2:18.55 total
Almost 15x smaller than the baseline, and more than 2x smaller than bzip3, also CPU time halves (since long matches are found earlier, so there's less work to do).
(the baseline number is slightly different because I don't have the exact Perl version set used by the author)
Also, in the benchmarks using lrzip, which would make the window size less relevant, zstd is not even compared.
> That always seemed annoying to me. They couldn't allocate 5 more bits somewhere to let the decompressor autodetect longer window sizes?
I believe this is just to prevent the decompressor from arbitrarily blowing up memory usage based on the input; I think if you want to accept long windows you can just always decompress with --long=63 regardless of whether the input needs it? (you will run out of RAM decompressing a long=63 file though of course)
These command line parameters change the maximum the decompressor will allow. It's 128 MiB by default in the command line decompressor; other uses (like the "zstd" content coding for HTTP in web browsers) use a lower limit of 8 MiB (see https://www.rfc-editor.org/rfc/rfc9659.html).
2^27 is 128 megabytes. How much RAM do you want the decompressor to have to allocate for every file? Especially since you can't tell, by looking only at the file size of a compressed file, how many bytes it will decompress to. You could read the file header, but if it's a malicious "zip bomb" type of file, the header could be lying.
If the header says the file is smaller than it really is, you've already allocated a small widow by the time you realize it lied, so it doesn't harm you here.
If the header says the file is bigger than it really is, it can get you to allocate a pointlessly large window. But if a large allocation is the goal, they can make the file actually decompress that big without affecting the compressed size. So lying is pointless.
I just tried a tar file of a git clone of the linux kernel sources where the most recent commit is 72c395024dac5e215136cbff793455f065603b06 (early Feb of this year). zstd -19 got a slightly smaller size (3582930348 bytes vs bz3 -b 511's 3597411687 bytes or 0.4% advantage to zstd). More significantly 4-core zstd decompression was 2.05 seconds vs a whopping 297 seconds for bzip3 -dj4 - 145x or over 2 orders of magnitude slower (about as much time to decode as to encode in the first place). bzip3 1.5.3 compiled with gcc-16.1.0. Granted, the .git objects are all compressed already and uncompressed tar-ball was only 5426667520 bytes, but even so...A lot of people care about fast(-ish) decompression. Maybe I did something wrong? Maybe `rm -rf .git` first would be a better benchmark?
This was my point too, but related to LZMA(2?) / xz, as the exact parameters were not specified while it supports setting compression level up to -9e, and the dictionary size can be controlled directly as well (in addition to quite a lot of fine-tuning knobs), increasing it up to 1536 MiB.
Wow, that is widely disingenuous, I don't really think there is any excuse for that, I don't believe someone deep in compression algorithms wouldn't know they could adjust the block size, and 512GB is a huge block size for bzip3, as it needs to basically all be in memory so you can't pretend that's just 'the standard value'.
Can you tell me what the zstd invocation is that corresponds to the default invocation of bzip3, which uses block size 16 MiB (according to the man page)?
I got some really good results with bzip3 compression Wikipedia XML dumps, and I would like to check if it's actually better or if I was just calling zstd wrong.
If you want a 16MB window, use `--long=24` (2^24 is 16M). (I believe this is larger than the default window for zstd level 3, but smaller than the default window at higher compression levels.)
I was processing compressed .jsonl files recently (JSON lines format). I found that lzma gave a much better compression than gzip or bzip2, which helps for archival costs, but it's challenging to work with as software support is lacking. I do duckdb processing which supports gzip transparently. There's an extension for bzip2, but not for lzma or bzip3.
I ended up using gzip because it's best supported by the software I use and most likely to have support in software I adopt. But it gave the worst compression results of the options I tried. These bzip3 numbers certainly give me FOMO...
For that type of structured data (logs and such), a custom dictionary can be extremely effective. Zstd among others support generating a custom dictionary. You just run zstd --train over the data first, and then feed that in when you run zstd. For example: I found ~10 gigabytes of Usenet headers compress to ~700 MB using Zstd and 1 MB shared dictionary -- and that's with each header individually compressed, so o(1) lookup time.
Back in my data hoarder days, I downloaded one of those torrents that had all the world's books in it. It was dunno how many terabytes, but way more than I had HDDs.
So I stripped out formatting, got rid of dupes, and tried out zstd, which was the hot new thing, along with the dictionary feature you describe, figuring it'd help. It didn't. I tried having one per book, one per multiple books, one for the whole archive.
It didn't work, or the gains were so marginal that I ended up scrapping the approach.
So it's not impossible that it can work, but stuff like regular json already compresses extremely well, I haven't found a scenario where it's a major boon.
When I studied at school, I used ZFS with lz4 enabled on my working machine. During that times I had a task of parsing Wikipedia's data. I had enough brain cells to find compressed dumps and download them with aria2 but not enough to leave the file compressed. I ran a decompressor. It'd been taking longer than I expected so I went out to walk a dog.
Imagine how fast me and the dog ran back 30 minutes later when I realized how cooked I was. I only had 10 GB left on my disks after I downloaded that 20 GB file. This decompressed file would have blown the machine up. I was terrified to find a frozen system with no storage space left.
Instead, the process finished and `df -h` reported 8 GB of the free space left. Files were decompressed. I could `less` them! That made no sense! Only many many minutes later I finally figured out to run a `zfs get compressratio` command which showed ZFS successfully and transparently recompressed everything on the fly. That was too impressive for that teenager and he never switched to a different file system.
Solaris has had so many cool features, like ZFS or doors. What I liked about ZFS is you coul make snapshot, which is basically the solution to how to treat data files a single, cheap to access unit, yet still use standard apis for file management, great for containerizing apps, making copies for experiments, or shipping stuff. Node.js just received this as a bespoke, app-level feature. But these things are too many to count, and make a ton of sense if you know how filesystems actually work. useful Also copy-on-write, temp overlays.
Pretrained dictionaries have never been intended to help with book sized or bigger compression. zstd automatically learns the most efficient dictionary it can within a few kilobytes. Pretrained dictionaries are only useful when you're independently compressing very small records.
Note that the dictionary options are only needed to improve compression ratios when compressing lots of small messages. If you have a bigger file (eg a tar file of Usenet messages) the regular Zstd compression will build a good dictionary without additional options.
The thing zstd got really right is fast decompression. For write-once read-never data like backups lzma (aka xz/7zip/lzip) is great. But it takes forever to decompress. On zstd I can get good compression while decompressing the file only marginally slower than reading the uncompressed file from SSD
Writing your files directly into a compressed stream and decompressing on the fly has become almost a standard workflow for any files I'm going to read and write sequentially anyways. No need for the data to ever exist uncompressed on the file system. Previous formats never did that for me because they either had too much overhead or too little gain, often both
But zstd is super tunable. Where gzip gives you compression levels from 1 to 9, zstd gives you up to 22 for ultra compression and negative compression levels for ultra fast. The ultra fast options so fast that they are great as a substitute for memcpy if your CPU is already waiting for other things, like DRAM.
zstd has a built-in benchmark mode to compare different compression levels, e.g. `zstd -b1 -e9 [FILE]` to test levels 1 to 9 (try up to 22 if you have enough spare time)
Duckdb supports loading and saving to zstd for all it's base loading/saving formats csv/tsv/json/jsonlines, but, for good or bad, those are solid compression.
Under most r/w workloads, using parquet/lance/vortex/native-duckdb, with their built-in columnar compression will result in more performance AND space savings. Non-solid compression. Then, the query engine can push down your query predicate to a column row group level, instead of forcing it to decompress the entire dataset to operate.
Practical example: duckdb has syntax - https://duckdb.org/docs/lts/data/multiple_files/overview - to glob multiple files at once, but that really only works if you're applying push down query predicates instead of re-decompressing your entire data set per SELECT. I would say for most dataset, even 20%+ size is worth not having to decompress (or even download!) the entire dataset, to figure out if something fits the predicate.
After all, if you have to download and decompress the dataset back again to operate, then the "space savings" are gone.
It isn't really the go-to compression format, because it isn't ubiquitous like gzip and zip, there are a variety of compression tools out there for different purposes, and there is image/audio/video compression. There is also specialized compression like what git does with its rolling hashes. I think of it as there not being a go-to compression format.
A go-to thing means it's a sensible default choice and has no little to no downsides (versus not using compression), it doesn't mean it's the best for everything.
Until now the go-to has been DEFLATE (gzip and zip) but zstd is definitely competing against it because it is better in almost every way.
I think this should include benchmarks zstd with larger windows and long range mode. I wouldn't be surprised if the window they used is smaller than an individual version tar file, which would prevent useful compression.
> However, the complexity of the algorithms, and, in particular, the presence of various special cases in the code which occur with very low but non-zero probability make it impossible to rule out the possibility of bugs remaining in the program.
Sounds like perhaps a nice testcase for formalization + AI?
imo, partially because it's still not easy (in terms of code -> formal proof). With AI, I've been Lean-ifying a simpler (but non-trivial) algo. Pointing (current) AI at it only goes so far and in fact might go "too far" in certain cases, where a non-formalized argument would have sufficed. There's also "who watches the watcher" -- did it really prove what we're supposed to prove?
For something like these compression algos, though, I imagine it would be much easier since they already have actual proofs out there.
formal correctness proof is ridiculously hard, like, insanely hard, have you ever tried writing one for any real algorithm (not a platonic ideal quicksort)?
It's distasteful to use such a name when it's not created by the bzip authors. For some reason open source developers love using would be trademark infringing names instead of coming up with something unique.
Different people have different licence preferences. "Why this and not that" questions usually don't spark joy. (Might seem fine for the individual who asks it, but not really in a community.)
You're also entitled to not comment on discussions you're not interested in. License discussion is very much relevant for anything that might end up being used for data interchange.
That's like comparing apples to pears. OpenZL is not general purpose. You specify a format for data and it compresses that format. Specifying a general "could be anything" format would be interesting, but I doubt it would compress as well.
Is that all that different than the standard MIT License:
> THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE....
Practically, no. But that specific disclaimer could lead on to think that there may have been some observed data corruption in practice that isn't disclosed. I have no idea if there is, but I'm not keen to discover it myself.
Also practically, this isn't MIT. It is LGPL 3, which I believe includes the warranty terms of GPL 3, i.e., no warranty. So we're in the same place anyway.
Tool author here. bzip2 also has this clause, in fact it has been lifted from its dist tarball README verbatim. So does lzma, xz, or in practice any open source program that you use.
Hey, I am the creator of https://www.photopea.com and if a user decides to publish their work, it compresses PSD files (inside a browser) and sends them to the server for storing. Right now, we have about 2 TB of files.
I use the DEFLATE compression which I fully implemented myself (compressor and decompressor, inside a library called UZIP.js).
I wish I could use something better :D but it must be implemented in Javascript to run inside a browser, and the library should not be too large (mine is about 8 kB ZIPped). I would love to switch to ZSTD once someone implements a compressor in Javascript that is under 40 - 60 kB. Or do you think that bzip3 could be re-implemented easily in JS using some AI?
I really admire you for being so smart at such a young age :) I wish you best of luck in your scientific career! I visited Warsaw this year, but I am not sure if that is where you are from :D
This big warning gave me pause, too. Why not have a compression option that automatically checks the data after compression (by decompressing and checking against a hash of the original data), which would reduce the probability of undetected errors to that of a hash collision? i.e. like `7z a` followed by `7z t`, but in one command.
So apparently zamadatix's point wasn't well taken, but I think it could be an effective approach to store the same data twice over to increase the likelihood of successful recovery. Could be even be even ten times, or e.g. 6+4 with redundant recovery coding.
Although you do have a point that the compressed data might be more difficult to decipher, if it doesn't have sufficient redundancy to skip bad parts, or if it is essential that the data is aligned in a certain way (e.g. disk images, and probably many other formats) and the format doesn't take this into account. Shorter window sizes, window reset markers, and explicit offset information could mitigate those problems.
Yes, that's why I was wondering about any relations.
But later I've read that there was some kind of Anime called Iczelion (or it was the hero name, whatever). Author also is interested in anime, I guess that's the reason for nickname similarity. Mystery solved.
Tool author here. Sure, out of all bait in this thread I will bite this one. Curiously, it seems like Konstantinos has opened this ticket, i.e. https://github.com/iczelia/bzip3/issues/177, and open-sourced his algorithm here -- https://codeberg.org/kagiannis/gdcc-2021. As the allegation is very serious, I will also copy the edited version of this response to my website.
I have not seen his tool before, or even was aware of it. However, once you take a closer look, you notice that these two tools could not be more dissimilar. Agiannis' compressor uses a compact context representation to group bytes, followed by RLE and prefix coding. Bzip3 combines repetition removal (here via a run-length code -- prior to the BWT -- and LZP of Charles Bloom), a full Burrows–Wheeler transform, and a richer (thus slower) statistical arithmetic coder. The use of RLE for post-coding the BWT output dates as far back to Julian Seward, perhaps even further. The RLE and LZP are applied before the Burrows-Wheeler transform (as opposed to the implementation in `text', which makes a big difference). Bzip3 uses a proper SAIS library for the forward and backward transforms. Konstantinos' entropy coder seems to use FPC (bytewise prefix codes over adaptively selected subblocks), bzip3 uses an idea similar to this of bcm, which itself descends from Mahoney and ancient work of JS Vitter on arihtmetic coding, where a bitwise arihtmetic coder is input mixed probability estimates from exponential-moving averages with probability refinement.
You are welcome to conduct your own analysis, but this is the gist of it -- perhaps Konstantinos has convinced himself that he had invented run-length coding?
Depends - if parallel decompression of any bzip3 archive is possible I'd consider it fair, if it requires special flags on archive creation I'd consider it unfair. I didn't see any description about that on that page.
pbzip2 only can do parallel decompression on archives created with pbzip2, otherwise it'll fall back to single thread. There nowadays seems to be lbzip2, though, which claims to be able to add SMP support for standard bzip2 archives.
I'll need to try that next time I'm working with large archives - I learned about the pbzip2 limitations the hard way last time I was shuffling around a few multi-10GB archives, and was trying to speed things up fully utilising my 32 core threadripper.
It's a very strange algorithm completely different from most compression methods, that's what makes it interesting IMO. But it's probably not realistically competitive, since one needs to do more after the burrows-wheeler transform, and all that "more" has been ridiculously more optimized in zstd and other modern compression methods. Compressing bwt-transformed data is easier, but that doesn't mean it's easy to further than what's easy.
I saw bellard neural net compression benchmarks. If you have time and performant system, based on the semantic of the data to compress, neural net seems tough to beat.
It is a matter of good compromises: speed and efficacy. I even wonder if LZMA2 is worth replacing bzip2.
I think compression algorithms are ripe for significant improvement with LLMs. It’s an ideal candidate problem you can have in a closed loop evaluation, and you can just let agent try things.
I have not experimented with bzip3 recently, but more than a year ago I have done many tests with it.
Initially I was extremely impressed with it, because in a lot of tests it succeeded to compress hard-to-compress files, like movies, and in many cases it demonstrated a much better compromise between speed and compression ratio than zstd, i.e. depending on the command parameters I could make it either compress better than zstd at similar compression/decompression speed, or compress/decompress faster at a similar compression ratio.
Alas, the initial extremely favorable conclusion was short-lived, because trying later bzip3 on other data files gave worse results than zstd.
So the final conclusion was that the performance of bzip3 was somewhat unpredictable, being highly data dependent. For some files it provided outstanding compression ratio or speed, but for others it was inferior.
The problem was that without doing a compression there was no way to guess whether a file would be among those preferred by bzip3 or by zstd or by xz.
So now I would use it only for a file for which I want maximum compression and which I would compress once and decompress many times, so I can afford a very long compression time, during which I would test multiple compression algorithms, including bzip3 and zstd, with multiple parameter choices, and I would eventually choose the one that offers the best compromise between compression ratio and decompression time, for that particular file.
It certainly is a competitive compression algorithm, but unless it has changed since I last tested it, you cannot guess for which files it would win the compression competition.
Would a multi-stream archive format make sense at this point? I.e. store several compressed streams in the same file and use heuristics to decide where each file (or portion of file) goes.
But developing the heuristics for choosing the appropriate compression algorithm for a stream of data is likely to need a very long time for compression tests of a lot of diverse training data, similarly to the training of a specialized ML model that classifies patterns.
Such heuristics should provide not only algorithm selection, but also parameter selection, when given only some simple input, e.g. the relative importances of compression ratio, decompression speed and compression speed.
There is a comparison with "zstd -19" on the Silesia corpus, showing better compression ratio for bzip3 (47.2 vs 53MB) while being ~5 times faster (and using only half the memory).
Even if the examples are highly cherry-picked, it is quite suprising to me that such pareto-dominance is possible at all.
edit: Tested it myself and found that it often also does slightly worse than zstd -19 in compression ratio but faster (it was slower in one case on "uncompressible" input).
Compression performance vs "zstd -19" seems to depends a lot on actual input data in a very unpredictable way. I'd assume the benchmarks that they show are definitely somewhat cherry-picked.
having used zstd, it has terrible defaults, optimized for speed and low-memory. You need to change it's params (not just level and dict size) to get high performance.
probably somebody should use a coding agent to do auto-research to optimize params for each compression algo, while matching one fixed goal - time, memory or size
The benchmark is very rudimentary. It does not test different levels/settings apart from its own -b 256/512 (does it affect decompression?), it doesn't measure compression time and memory usage. It does not specify parallel vs single-threaded (it mentions parallel on the one decoding number but what about the others?).
The lrzip test is interesting but it omits for example zstd and doesn't even have (de-)compression timings.
A lot more numbers are needed to present a fair and informative comparison.
I don't want this to be a swipe against bzip3, I only want to point out the presented benchmarks could be a lot better.
Previously:
“Hi, tool author here.” A useful explanation of Burrows-Wheelers transform as used by bzip3: https://news.ycombinator.com/item?id=42902407
“bzip3 is not yet listed on the large text compression benchmark” It is now: https://mattmahoney.net/dc/text.html
(2 years ago, 176 comments) https://news.ycombinator.com/item?id=42899713
(4 years ago, 104 comments) https://news.ycombinator.com/item?id=31324439
> “bzip3 is not yet listed on the large text compression benchmark” It is now
And it comes in relatively well, in my opinion. I'm a compression amateur, but bzip3 is the first entry I recognize as a general purpose compression program.
Yes, bzip3 compresses that 1GB text file into 170 MB while zstd needs 213 MB (SI units, from my own tests with the "enwik9" file of the GP's page, and the latest releases of both programs, same commands). But the decompression (memory and speed) is a different story.
On my desktop, decompressing with zstd requires 128 MB and 1 s, while bzip3 uses 3.2 GB and 90 s.
Without `-b 511`, bzip3's ratio decreases and decompression requires about the same memory as zstd. But the decompression is still two orders of magnitude slower.
Bzip2 has also always be a pig on decompression so I was fearing that this was the catch here as well.
Might still be useful as an archival compression format even if it will be less useful for distribution.
This should get more attention - those decompression requirements are a massive impost.
> It is now:
Matt added bzip3 to the benchmark seemingly after the last year thread. If the zstd benchmark, last update at max 22 in 2016, were to be updated for today’s zstd (for long-range if nothing else) after coming across this thread, what would be the most compact possible list of option sets to benchmark to reflect the value of LR et al.?
> “Hi, tool author here.” A useful explanation of Burrows-Wheelers transform as used by bzip3
FWIW, the Burrows-Wheelers transform is also used by bzip2, so this isn’t a new feature even though that quote kinda sounds like it is.
They missed an opportunity to call it the Wheel-Burrow algorithm didn't they
It’s also used in this paper blending an aspect of LZ77 with BWT:
https://arxiv.org/abs/1502.05937
https://github.com/nicolaprezza/lz-rlbwt
Which leads me to wonder if someone did, in fact, solve how to ‘combine zstd and bzip’ (so to speak) since the last time.
The benchmarks are disingenuous, to the point of looking cherry-picked. The block size for bzip3 is set to 512MB, but the window size for zstd is left to its default (8MB I believe for high levels). So in this corpus, which is made up of all versions of Perl source code concatenated, the window is too small to see all the identical files and just match them. Also corpora made out of very long repetitions are pretty much the best case scenario for BWT-based compressors.
If we match the window size of zstd to that of bzip3 we get dramatically different results:
Almost 15x smaller than the baseline, and more than 2x smaller than bzip3, also CPU time halves (since long matches are found earlier, so there's less work to do).(the baseline number is slightly different because I don't have the exact Perl version set used by the author)
Also, in the benchmarks using lrzip, which would make the window size less relevant, zstd is not even compared.
> 8MB I believe for high levels
Yep, i found it in the source here:
- https://github.com/facebook/zstd/blob/d9c0c7e2cf8a8bf9fb98d3...
- https://github.com/facebook/zstd/blob/d9c0c7e2cf8a8bf9fb98d3...
Also, zstd docs say:
> Note: If windowLog is set to larger than 27, --long=windowLog or --memory=windowSize needs to be passed to the decompressor.
That always seemed annoying to me. They couldn't allocate 5 more bits somewhere to let the decompressor autodetect longer window sizes?
> That always seemed annoying to me. They couldn't allocate 5 more bits somewhere to let the decompressor autodetect longer window sizes?
I believe this is just to prevent the decompressor from arbitrarily blowing up memory usage based on the input; I think if you want to accept long windows you can just always decompress with --long=63 regardless of whether the input needs it? (you will run out of RAM decompressing a long=63 file though of course)
> They couldn't allocate 5 more bits somewhere to let the decompressor autodetect longer window sizes?
It's actually 8 bits: https://www.rfc-editor.org/rfc/rfc8878.html#name-window-desc...
These command line parameters change the maximum the decompressor will allow. It's 128 MiB by default in the command line decompressor; other uses (like the "zstd" content coding for HTTP in web browsers) use a lower limit of 8 MiB (see https://www.rfc-editor.org/rfc/rfc9659.html).
2^27 is 128 megabytes. How much RAM do you want the decompressor to have to allocate for every file? Especially since you can't tell, by looking only at the file size of a compressed file, how many bytes it will decompress to. You could read the file header, but if it's a malicious "zip bomb" type of file, the header could be lying.
If the header says the file is smaller than it really is, you've already allocated a small widow by the time you realize it lied, so it doesn't harm you here.
If the header says the file is bigger than it really is, it can get you to allocate a pointlessly large window. But if a large allocation is the goal, they can make the file actually decompress that big without affecting the compressed size. So lying is pointless.
I just tried a tar file of a git clone of the linux kernel sources where the most recent commit is 72c395024dac5e215136cbff793455f065603b06 (early Feb of this year). zstd -19 got a slightly smaller size (3582930348 bytes vs bz3 -b 511's 3597411687 bytes or 0.4% advantage to zstd). More significantly 4-core zstd decompression was 2.05 seconds vs a whopping 297 seconds for bzip3 -dj4 - 145x or over 2 orders of magnitude slower (about as much time to decode as to encode in the first place). bzip3 1.5.3 compiled with gcc-16.1.0. Granted, the .git objects are all compressed already and uncompressed tar-ball was only 5426667520 bytes, but even so...A lot of people care about fast(-ish) decompression. Maybe I did something wrong? Maybe `rm -rf .git` first would be a better benchmark?
How does memory usage compare between your two runs?
The benchmarks report 687M for their run of zstd, and 12178M and 18301M for the two runs of BZip3. Which itself is a bit eyebrow raising
Nice catch.
it has been a long time since: "lies, damn lies, benchmarks" failed to hold true. Sometimes I wonder why gaming benchmarks has become so common.
This was my point too, but related to LZMA(2?) / xz, as the exact parameters were not specified while it supports setting compression level up to -9e, and the dictionary size can be controlled directly as well (in addition to quite a lot of fine-tuning knobs), increasing it up to 1536 MiB.
Wow, that is widely disingenuous, I don't really think there is any excuse for that, I don't believe someone deep in compression algorithms wouldn't know they could adjust the block size, and 512GB is a huge block size for bzip3, as it needs to basically all be in memory so you can't pretend that's just 'the standard value'.
> 512GB is a huge block size for bzip3
Sorry! That was a typo, it should have been 512MB (now fixed). Still huge.
Can you tell me what the zstd invocation is that corresponds to the default invocation of bzip3, which uses block size 16 MiB (according to the man page)?
I got some really good results with bzip3 compression Wikipedia XML dumps, and I would like to check if it's actually better or if I was just calling zstd wrong.
If you want a 16MB window, use `--long=24` (2^24 is 16M). (I believe this is larger than the default window for zstd level 3, but smaller than the default window at higher compression levels.)
I was processing compressed .jsonl files recently (JSON lines format). I found that lzma gave a much better compression than gzip or bzip2, which helps for archival costs, but it's challenging to work with as software support is lacking. I do duckdb processing which supports gzip transparently. There's an extension for bzip2, but not for lzma or bzip3.
I ended up using gzip because it's best supported by the software I use and most likely to have support in software I adopt. But it gave the worst compression results of the options I tried. These bzip3 numbers certainly give me FOMO...
For that type of structured data (logs and such), a custom dictionary can be extremely effective. Zstd among others support generating a custom dictionary. You just run zstd --train over the data first, and then feed that in when you run zstd. For example: I found ~10 gigabytes of Usenet headers compress to ~700 MB using Zstd and 1 MB shared dictionary -- and that's with each header individually compressed, so o(1) lookup time.
Back in my data hoarder days, I downloaded one of those torrents that had all the world's books in it. It was dunno how many terabytes, but way more than I had HDDs.
So I stripped out formatting, got rid of dupes, and tried out zstd, which was the hot new thing, along with the dictionary feature you describe, figuring it'd help. It didn't. I tried having one per book, one per multiple books, one for the whole archive.
It didn't work, or the gains were so marginal that I ended up scrapping the approach.
So it's not impossible that it can work, but stuff like regular json already compresses extremely well, I haven't found a scenario where it's a major boon.
I have an anecdote about compressed data.
When I studied at school, I used ZFS with lz4 enabled on my working machine. During that times I had a task of parsing Wikipedia's data. I had enough brain cells to find compressed dumps and download them with aria2 but not enough to leave the file compressed. I ran a decompressor. It'd been taking longer than I expected so I went out to walk a dog.
Imagine how fast me and the dog ran back 30 minutes later when I realized how cooked I was. I only had 10 GB left on my disks after I downloaded that 20 GB file. This decompressed file would have blown the machine up. I was terrified to find a frozen system with no storage space left.
Instead, the process finished and `df -h` reported 8 GB of the free space left. Files were decompressed. I could `less` them! That made no sense! Only many many minutes later I finally figured out to run a `zfs get compressratio` command which showed ZFS successfully and transparently recompressed everything on the fly. That was too impressive for that teenager and he never switched to a different file system.
Solaris has had so many cool features, like ZFS or doors. What I liked about ZFS is you coul make snapshot, which is basically the solution to how to treat data files a single, cheap to access unit, yet still use standard apis for file management, great for containerizing apps, making copies for experiments, or shipping stuff. Node.js just received this as a bespoke, app-level feature. But these things are too many to count, and make a ton of sense if you know how filesystems actually work. useful Also copy-on-write, temp overlays.
Sun was a really cool company.
\[T]/
as oxide is nowadays ;-)
"Dictionary gains are mostly effective in the first few KB."
https://facebook.github.io/zstd/index.html
Pretrained dictionaries have never been intended to help with book sized or bigger compression. zstd automatically learns the most efficient dictionary it can within a few kilobytes. Pretrained dictionaries are only useful when you're independently compressing very small records.
JSONL files are likely to have a lot of the same words repeated MANY times. Same as with headers...
Because JSON is an inefficient text encoding, compression (with custom dictionary) are likely to really well on those.
Books have recurring words, but probably much less.
Note that the dictionary options are only needed to improve compression ratios when compressing lots of small messages. If you have a bigger file (eg a tar file of Usenet messages) the regular Zstd compression will build a good dictionary without additional options.
As I understand there is no advantage in using a custom dictionary to compress 1 file. It benefits compressing _several_ (small) files.
zstd is the go-to compression format these days. It's even supported in low-level software such as many linux filesystems.
I don't know much about duckdb but it looks like it supports zstd too: https://duckdb.org/docs/lts/data/json/loading_json
The thing zstd got really right is fast decompression. For write-once read-never data like backups lzma (aka xz/7zip/lzip) is great. But it takes forever to decompress. On zstd I can get good compression while decompressing the file only marginally slower than reading the uncompressed file from SSD
Writing your files directly into a compressed stream and decompressing on the fly has become almost a standard workflow for any files I'm going to read and write sequentially anyways. No need for the data to ever exist uncompressed on the file system. Previous formats never did that for me because they either had too much overhead or too little gain, often both
Agreed! Relatedly: https://news.ycombinator.com/item?id=49599953
How did I miss zstd?
Here are my benchmarks for 2.3 GB of jsonl, on a laptop. Compressed size, compress time, decompress time; using defaults.
At what levels? There’s no guarantee that the default compression level is comparable. You have to normalize by time spent compressing.
But zstd is super tunable. Where gzip gives you compression levels from 1 to 9, zstd gives you up to 22 for ultra compression and negative compression levels for ultra fast. The ultra fast options so fast that they are great as a substitute for memcpy if your CPU is already waiting for other things, like DRAM.
zstd has a built-in benchmark mode to compare different compression levels, e.g. `zstd -b1 -e9 [FILE]` to test levels 1 to 9 (try up to 22 if you have enough spare time)
zstd with better compression level would be nice - these numbers are not really comparable since both time and compression level are too different
Duckdb supports loading and saving to zstd for all it's base loading/saving formats csv/tsv/json/jsonlines, but, for good or bad, those are solid compression.
Under most r/w workloads, using parquet/lance/vortex/native-duckdb, with their built-in columnar compression will result in more performance AND space savings. Non-solid compression. Then, the query engine can push down your query predicate to a column row group level, instead of forcing it to decompress the entire dataset to operate.
Practical example: duckdb has syntax - https://duckdb.org/docs/lts/data/multiple_files/overview - to glob multiple files at once, but that really only works if you're applying push down query predicates instead of re-decompressing your entire data set per SELECT. I would say for most dataset, even 20%+ size is worth not having to decompress (or even download!) the entire dataset, to figure out if something fits the predicate.
After all, if you have to download and decompress the dataset back again to operate, then the "space savings" are gone.
all hail zstd, the one format to rule them all
Or lz4
Not really, it's a popular dictionary-based compression format.
I'm pretty new to choosing compression libraries - I started with zlib and was delighted at how much faster and smaller zstd made things.
Since we kind of need a default "Need to compress something? Use this!" setting - would you prefer zlib over zstd, or something else for that role?
The only reason to pick anything but zstd is that platform support might be better.
If your platform/sdk/browser/standard-library comes with zlib/gzip/.. then it's often easier to just pick that.
No new dependencies is always a win. App size. Security, etc.
Otherwise, if zstd is easy to add, IMO I would always prefer, zstd, lz4 or brotli.
zstd or lz4
”Not really" what?
It's hard to understand what point you're trying to make. Can you clarify?
It isn't really the go-to compression format, because it isn't ubiquitous like gzip and zip, there are a variety of compression tools out there for different purposes, and there is image/audio/video compression. There is also specialized compression like what git does with its rolling hashes. I think of it as there not being a go-to compression format.
I think you're being a bit pedantic.
A go-to thing means it's a sensible default choice and has no little to no downsides (versus not using compression), it doesn't mean it's the best for everything.
Until now the go-to has been DEFLATE (gzip and zip) but zstd is definitely competing against it because it is better in almost every way.
For structured logs and json I've found a lot of success with PPM-style schemes.
If your JSON file has many of the same object, you could see ratios in the single digits.
I'd recommend trying openzl for jsonl.
An interesting unintentional benchmark is to go to https://github.com/iczelia/bzip3/releases and see to what degree bzip3 compresses its own release archives; and go to https://github.com/iczelia/bzip3/blob/master/.github/workflo... to see what options have been chosen for the other compressors here.
I think this should include benchmarks zstd with larger windows and long range mode. I wouldn't be surprised if the window they used is smaller than an individual version tar file, which would prevent useful compression.
“Additional benchmarks on the same dataset” (2025) explores various compression levels with and without long range mode: https://news.ycombinator.com/item?id=42901476
> However, the complexity of the algorithms, and, in particular, the presence of various special cases in the code which occur with very low but non-zero probability make it impossible to rule out the possibility of bugs remaining in the program.
Sounds like perhaps a nice testcase for formalization + AI?
It's beyond me why such foundational libraries don't have formal correctness proofs attached these days.
imo, partially because it's still not easy (in terms of code -> formal proof). With AI, I've been Lean-ifying a simpler (but non-trivial) algo. Pointing (current) AI at it only goes so far and in fact might go "too far" in certain cases, where a non-formalized argument would have sufficed. There's also "who watches the watcher" -- did it really prove what we're supposed to prove?
For something like these compression algos, though, I imagine it would be much easier since they already have actual proofs out there.
formal correctness proof is ridiculously hard, like, insanely hard, have you ever tried writing one for any real algorithm (not a platonic ideal quicksort)?
I have. And it is ridiculously hard.
But having your decompression algorithm not produce what you compressed earlier is also ridiculously bad.
So, seems worth the effort, given that AI can do lots the heavy lifting these days.
It's distasteful to use such a name when it's not created by the bzip authors. For some reason open source developers love using would be trademark infringing names instead of coming up with something unique.
Other than the oppressive trademark law how isn't it bzip3?
Because it's not version 3.0 of bzip, but an entirely different program.
bzip2 isn't compatible with bzip, and the current maintainer of bzip2 made neither bzip nor bzip2.
Julian Seward created both bzip and bzip2.
Kamila Szewczyk created bzip3
Besides the oppressive trademark law how isn't it bzip3?
Why LGPL when the original license is more permissive? (https://sourceware.org/bzip2/manual/manual.html)
bzip3 is not made by the same author as bzip2, that's why there is a licence disparity.
Feels pretty odd to take something named foo2 and claim foo3 on top of it.
The author of bzip3 probably doesnt like pushover licenses so made user rights stronger
Please stop questioning people's choice of license.
If you don't like it, you're entitled to not using it.
Different people have different licence preferences. "Why this and not that" questions usually don't spark joy. (Might seem fine for the individual who asks it, but not really in a community.)
You're also entitled to not comment on discussions you're not interested in. License discussion is very much relevant for anything that might end up being used for data interchange.
Having been annoyed by people who asked me to change license several times, I have no sympathy for these requests.
I would be interrested in a comparison with openzl
That's like comparing apples to pears. OpenZL is not general purpose. You specify a format for data and it compresses that format. Specifying a general "could be anything" format would be interesting, but I doubt it would compress as well.
Not sure why this is being posted today or getting comments.
bzip3 not a terribly impressive or useful compression algo.
benchmark on enwik: https://www.mattmahoney.net/dc/text.html#1703
"DO NOT COMPRESS ANY DATA WITH THIS PROGRAM UNLESS YOU ARE PREPARED TO ACCEPT THE POSSIBILITY, HOWEVER SMALL, THAT THE DATA WILL NOT BE RECOVERABLE."
So why would anybody, hobbyist or enterprise, use this? Or does something older like 7zip also have this caveat that I've never experienced.
Is that all that different than the standard MIT License:
> THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE....
Practically, no. But that specific disclaimer could lead on to think that there may have been some observed data corruption in practice that isn't disclosed. I have no idea if there is, but I'm not keen to discover it myself.
Also practically, this isn't MIT. It is LGPL 3, which I believe includes the warranty terms of GPL 3, i.e., no warranty. So we're in the same place anyway.
Tool author here. bzip2 also has this clause, in fact it has been lifted from its dist tarball README verbatim. So does lzma, xz, or in practice any open source program that you use.
Hey, I am the creator of https://www.photopea.com and if a user decides to publish their work, it compresses PSD files (inside a browser) and sends them to the server for storing. Right now, we have about 2 TB of files. I use the DEFLATE compression which I fully implemented myself (compressor and decompressor, inside a library called UZIP.js).
I wish I could use something better :D but it must be implemented in Javascript to run inside a browser, and the library should not be too large (mine is about 8 kB ZIPped). I would love to switch to ZSTD once someone implements a compressor in Javascript that is under 40 - 60 kB. Or do you think that bzip3 could be re-implemented easily in JS using some AI?
I really admire you for being so smart at such a young age :) I wish you best of luck in your scientific career! I visited Warsaw this year, but I am not sure if that is where you are from :D
Hi Ivan,
Please send me an e-mail! Perhaps we can figure something out.
This big warning gave me pause, too. Why not have a compression option that automatically checks the data after compression (by decompressing and checking against a hash of the original data), which would reduce the probability of undetected errors to that of a hash collision? i.e. like `7z a` followed by `7z t`, but in one command.
faster than gzip now or still slower? (sorry I did not read readme)
For data recovery of archival stuff it's better to keep it uncompressed, no?
So apparently zamadatix's point wasn't well taken, but I think it could be an effective approach to store the same data twice over to increase the likelihood of successful recovery. Could be even be even ten times, or e.g. 6+4 with redundant recovery coding.
Although you do have a point that the compressed data might be more difficult to decipher, if it doesn't have sufficient redundancy to skip bad parts, or if it is essential that the data is aligned in a certain way (e.g. disk images, and probably many other formats) and the format doesn't take this into account. Shorter window sizes, window reset markers, and explicit offset information could mitigate those problems.
Error correction codes would be a much better use than simply duplicating the data.
Depends on the specific scenario. E.g. efficient compression can speed up recovery from slow media and provide space for increased parity.
Any relation to Iczelion from the masm32 tutorial?
Those tutorials are about as old as the author of the repository.
Yes, that's why I was wondering about any relations.
But later I've read that there was some kind of Anime called Iczelion (or it was the hero name, whatever). Author also is interested in anime, I guess that's the reason for nickname similarity. Mystery solved.
Bro just decompile agiannis_text
https://codeberg.org/kagiannis/gdcc-2021/src/branch/main/shu...
original source
How is this source code the original source and which parts? Please elaborate.
Source of agiannis_text. He probably decompiled it.
Tool author here. Sure, out of all bait in this thread I will bite this one. Curiously, it seems like Konstantinos has opened this ticket, i.e. https://github.com/iczelia/bzip3/issues/177, and open-sourced his algorithm here -- https://codeberg.org/kagiannis/gdcc-2021. As the allegation is very serious, I will also copy the edited version of this response to my website.
I have not seen his tool before, or even was aware of it. However, once you take a closer look, you notice that these two tools could not be more dissimilar. Agiannis' compressor uses a compact context representation to group bytes, followed by RLE and prefix coding. Bzip3 combines repetition removal (here via a run-length code -- prior to the BWT -- and LZP of Charles Bloom), a full Burrows–Wheeler transform, and a richer (thus slower) statistical arithmetic coder. The use of RLE for post-coding the BWT output dates as far back to Julian Seward, perhaps even further. The RLE and LZP are applied before the Burrows-Wheeler transform (as opposed to the implementation in `text', which makes a big difference). Bzip3 uses a proper SAIS library for the forward and backward transforms. Konstantinos' entropy coder seems to use FPC (bytewise prefix codes over adaptively selected subblocks), bzip3 uses an idea similar to this of bcm, which itself descends from Mahoney and ancient work of JS Vitter on arihtmetic coding, where a bitwise arihtmetic coder is input mixed probability estimates from exponential-moving averages with probability refinement.
You are welcome to conduct your own analysis, but this is the gist of it -- perhaps Konstantinos has convinced himself that he had invented run-length coding?
Imagine bzip4
The latest release is a year ago, the last commit is two months ago, and the build is failing.
The claim “stronger than bzip2” is strange. What does it even mean?
Also, comparing parallel decompression benchmarks with bzip2 instead of pbzip2 seems unfair.
Depends - if parallel decompression of any bzip3 archive is possible I'd consider it fair, if it requires special flags on archive creation I'd consider it unfair. I didn't see any description about that on that page.
pbzip2 only can do parallel decompression on archives created with pbzip2, otherwise it'll fall back to single thread. There nowadays seems to be lbzip2, though, which claims to be able to add SMP support for standard bzip2 archives.
I'll need to try that next time I'm working with large archives - I learned about the pbzip2 limitations the hard way last time I was shuffling around a few multi-10GB archives, and was trying to speed things up fully utilising my 32 core threadripper.
> pbzip2 only can do parallel decompression on archives created with pbzip2
Oh I didn’t know that.
“for fairness, the benchmarks have been performed using single thread mode” (2025) https://news.ycombinator.com/item?id=42902241
Please report any build errors in the GitHub issue tracker so that the isseu can be fixed for everyone. Thank you!
Isn't that XZ?
No. It is an unrelated algorithm.
It's a very strange algorithm completely different from most compression methods, that's what makes it interesting IMO. But it's probably not realistically competitive, since one needs to do more after the burrows-wheeler transform, and all that "more" has been ridiculously more optimized in zstd and other modern compression methods. Compressing bwt-transformed data is easier, but that doesn't mean it's easy to further than what's easy.
I saw bellard neural net compression benchmarks. If you have time and performant system, based on the semantic of the data to compress, neural net seems tough to beat.
It is a matter of good compromises: speed and efficacy. I even wonder if LZMA2 is worth replacing bzip2.
You're probably thinking of LZMA2.
I think compression algorithms are ripe for significant improvement with LLMs. It’s an ideal candidate problem you can have in a closed loop evaluation, and you can just let agent try things.
> Compression is prediction
(According to recent article on HN[1]. So LLMs _are_ compression!)
[1]: https://hn.leftium.com/i/49263497
Impressive compression benchmark. Four times smaller than z standard.
I have not experimented with bzip3 recently, but more than a year ago I have done many tests with it.
Initially I was extremely impressed with it, because in a lot of tests it succeeded to compress hard-to-compress files, like movies, and in many cases it demonstrated a much better compromise between speed and compression ratio than zstd, i.e. depending on the command parameters I could make it either compress better than zstd at similar compression/decompression speed, or compress/decompress faster at a similar compression ratio.
Alas, the initial extremely favorable conclusion was short-lived, because trying later bzip3 on other data files gave worse results than zstd.
So the final conclusion was that the performance of bzip3 was somewhat unpredictable, being highly data dependent. For some files it provided outstanding compression ratio or speed, but for others it was inferior.
The problem was that without doing a compression there was no way to guess whether a file would be among those preferred by bzip3 or by zstd or by xz.
So now I would use it only for a file for which I want maximum compression and which I would compress once and decompress many times, so I can afford a very long compression time, during which I would test multiple compression algorithms, including bzip3 and zstd, with multiple parameter choices, and I would eventually choose the one that offers the best compromise between compression ratio and decompression time, for that particular file.
It certainly is a competitive compression algorithm, but unless it has changed since I last tested it, you cannot guess for which files it would win the compression competition.
Would a multi-stream archive format make sense at this point? I.e. store several compressed streams in the same file and use heuristics to decide where each file (or portion of file) goes.
I think so.
But developing the heuristics for choosing the appropriate compression algorithm for a stream of data is likely to need a very long time for compression tests of a lot of diverse training data, similarly to the training of a specialized ML model that classifies patterns.
Such heuristics should provide not only algorithm selection, but also parameter selection, when given only some simple input, e.g. the relative importances of compression ratio, decompression speed and compression speed.
with zstd at level 16 with default params (dict size, ...). Serious compression starts at level 19 and with much higher dict sizes.
how is this an honest benchmark:
There is a comparison with "zstd -19" on the Silesia corpus, showing better compression ratio for bzip3 (47.2 vs 53MB) while being ~5 times faster (and using only half the memory).
Even if the examples are highly cherry-picked, it is quite suprising to me that such pareto-dominance is possible at all.
edit: Tested it myself and found that it often also does slightly worse than zstd -19 in compression ratio but faster (it was slower in one case on "uncompressible" input).
Compression performance vs "zstd -19" seems to depends a lot on actual input data in a very unpredictable way. I'd assume the benchmarks that they show are definitely somewhat cherry-picked.
having used zstd, it has terrible defaults, optimized for speed and low-memory. You need to change it's params (not just level and dict size) to get high performance.
probably somebody should use a coding agent to do auto-research to optimize params for each compression algo, while matching one fixed goal - time, memory or size
The benchmark is very rudimentary. It does not test different levels/settings apart from its own -b 256/512 (does it affect decompression?), it doesn't measure compression time and memory usage. It does not specify parallel vs single-threaded (it mentions parallel on the one decoding number but what about the others?).
The lrzip test is interesting but it omits for example zstd and doesn't even have (de-)compression timings.
A lot more numbers are needed to present a fair and informative comparison.
I don't want this to be a swipe against bzip3, I only want to point out the presented benchmarks could be a lot better.