One of my favorite moments in HN history was watching the authors of the various search tools decide on a common ".ignore" file as opposed to each having their own: https://news.ycombinator.com/item?id=12568245
I would argue that grep-like tools which read .gitignore violate the Principle of Least Astonishment (POLA). It would be fine if there were a --ignore flag to enable such functionality, but defaulting to it just feels wrong to me. Obviously smarter people than I disagree, but my dumdum head just feels that way.
Back in the day I would have agreed with you, but ever since there is js everywhere you end up with minified js that megabytes big and match everything. I still have muscle memory with `grep -r` and it almost always ends up with some js file, that I didn't know exists ruining the moment.
> Obviously smarter people than I disagree, but my dumdum head just feels that way.
That's absolutely not it. What you're describing is part of the UNIX philosophy: programs should do one thing and do it well, and they should function in a way that makes them very versatile and composable, etc.
And that part of the philosophy works GREAT when everything follows another part of the philosophy: everything should be based on flat text files.
But for a number of reasons, and regardless of whatever we all think of those reasons, we live in a world that has a lot of stuff that is NOT the kind of flat text file grep was made for. Binary formats, minified JS, etc. And so to make the tool more practical on a modern *nix workstation, suddenly more people want defaults that are going to work on their flat text files and transparently ignore things like .git.
It's just that you've showed up to an wildly unprincipled world armed with principles.
I have to agree here. I love ripgrep, but at times I've had to go back to regular grep because I couldn't figure out what it was ignoring and why, and there were far too many settings to figure it out.
It's good if they can share syntax. You use the same English words to ask Alice and Bob questions, but when you say "So, tell me, Alice…" you don't want Bob to answer you instead. Using another tool's config by default, making it difficult/impossible to use the dedicated config is the most annoying thing I can imagine. If that's what rg does, I guess that must be the reason I couldn't switch.
I’ve read this multiple times over the years and this post is still the most interesting and informative piece describing the problem of making a fast grep-like tool. I love that it doesn’t just describe how ripgrep works but also how all the other tools work and then compares the various techniques. It’s simultaneously a tutorial and an expert deep dive. Just a beautiful piece of writing. In a perfect world, all code would be similarly documented.
Such a good read. I actually went back though it the other day to steal the searching for the least common byte idea out to speed up my search tool https://github.com/boyter/cs which when coupled with the simd upper lower search technique from fzf cut the wall clock runtime by a third.
There was this post from cursor https://cursor.com/blog/fast-regex-search today about building an index for agents due to them hitting a limit on ripgrep, but I’m not sure what codebase they are hitting that warrants it. Especially since they would have to be at 100-200 GB to be getting to 15s of runtime. Unless it’s all matches that is.
Yeah, that Cursor blog post is a bit iffy since they just brush over the "ripgrep is slow on large monorepos", move on to techniques they used, and then completely ignore the fact that you have to build and maintain the index.
On a mid-size codebase, I fzf- and rg-ed through the code almost instantly, while watching my coworker's computer slow down to a crawl when Pycharm started reindexing the project.
When I first heard about ripgrep my reaction was laughing. grep had been too established. No way something that isn't 100% compatible with grep could get any traction.
And I was dead wrong. Overnight everyone uses rg (me included).
I don't remember why I didn't switch from ag, but I remember it was a conscious decision. I think it had something to do with configuration, rg using implicit '.ignore' file (a super-generic name instead of a proper tool-specific config) or even .gitignore, or something else very much unwarranted, that made it annoying to use. Cannot remember, really, only remember that I spent too much time trying to make it behave and decided it isn't worth it. Anyway, faster is nice, but somehow I don't ever feel that ag is too slow for anything. The switch from the previous one (what was it? ack?) felt like a drastic improvement, but ag vs. rg wasn't much difference to me in practice.
Totally agree with the author of rg here. Config names should be unambiguous. Anyway, must have been something else, then. As I've said, I cannot remember what was the specific problem, only that it wasn't quite compatible with the workflow I was used to, and now it'd take another full-in attempt to switch to figure out what was so annoying to me back then.
I was just trying to remember why I switched _to_ rg.
It's nice and everything, but I remember being happy with the tools before (I think i moved from grep to ack, then jumped due to perf to ag and for unremembered reasons to pt.)
It took me a while, but I remembered I ran into an issue with pt incorrectly guessing the encoding of some files[0].
I can't remember whether rg suffered from the same issue or not, but I do know after switching to rg everything was plain sailing and I've been happy with it since.
For me it was trying to add a filter to search CMake files to ag and then realizing that the code had some rather stupid design decisions that prevented it. I wrote a pull request that fixed enough things to add the filter, got ignored by the maintainer and later realized that other people had already written the same filter and were ignored too.
Is IRIX experiencing a hobbyist revival or something? This is the second IRIX reference I’ve seen on here in the past two days, and there was a submission a day or two ago (c.f. a Voodoo video card?) as well. I haven’t personally encountered IRIX in the wild since a company I worked at in 2003. I suppose SGI has always had a cool factor but it’s unusual seeing it come up in a cluster of mentions like this.
SGUG tried hard to port newer packages for IRIX for several years but hit a wall with ABI mismatches leading to GOT corruption. This prevented a lot of larger packages from working or even building.
I picked up the effort again after wondering if LLMs would help. I ran into the ABI problems pretty quickly. This time though, I had Claude use Ghidra to RE the IRIX runtime linker daemon, which gave the LLM enough to understand that the memory structures I’d been using in LLVM were all wrong. See https://github.com/unxmaal/mogrix/blob/main/rules/methods/ir... .
After cracking that mystery I was able to quickly get “impossible” packages building, like WebKit, QT5, and even small bits of Go and Rust.
I’m optimistic that we’ll see more useful applications built for this cool old OS.
That is pretty neat. I guess this sort of unlocking and unblocking effort is exactly what’s needed for a revival.
I’m sort of thinking of AmigaOS/Workbench as well although, perhaps because of what I would assume was always a much larger user base than SGI had, it maybe never went away like SGI and IRIX did.
It is great seeing these old platforms get a new lease of life.
Ooh that's super interesting. I assume you shared the recipe with the irix community? I remember keeping Netscape up to date on my Indy was already a struggle in 2002.
I was using ripgrep once and it had a bug that led me downa terrifying rabbit hole - I can't recall what it was but it involved not being able to find text that absolutely should have been there.
Eventually I was considering rebuilding the machine completely but for some reason after a very long time digging deep into the rabbit hole I tried plain old grep and there was the data exactly where it should have been.
So it's such a vague story but it was a while back - I don't remember the specifics but I sure recall the panic.
Sometimes I forget that some of the config files I have for CI in a project are under a dot directory, and therefore ignored by rg by default, so I have to repeat the search giving the path to that config files subdirectory if I want to see the results that are under that one (or use some extra flags for rg to not ignore dot directories other than .git)
Was the file in a .gitignore by any chance? I've got my home folder in git to keep track of dot/config files and that always catches me out. Really dislike it defaulting to that ignoring files that are ignored by git.
I had that happen too recently… Basically rg x would show nothing but grep -r x showed the lines for any x. Tried multiple times with different x, then I kept using grep -r at that time. After a few days, I started using rg again and it worked fine but now I tend to use grep -r occasionally too to make sure.
Next time that happens try looking at the paths, adding a pair of -u, or running with --debug: by default rg will ignore files which are hidden (dotfiles) or excluded by ignore files (.gitignore, .ignore, …).
I use "grep" to search files (it should never skip any unless I tell it to do otherwise) and "git grep" to be a programmer searching a codebase (where it should only look at code files unless I tell it to do otherwise). Two different hats.
I wouldn't want to use tools that straddle the two, unless they had a nice clear way of picking one or the other. ripgrep does have "--no-ignore", though I would prefer -a / --all (one could make their own with alias rga='rg --no-ignore')
And burntsushi is one of us: he's regularly here on HN. Big thanks to him. As soon as rg came out I was building it on Linux. Now it ships stocks with Debian (since Bookworm? Don't remember): thanks, thanks and more thanks.
> > IMO, as long as the time differences remain small, I'm totally okay with ripgrep being slower by default on smaller corpora if it means being a lot faster by default on bigger corpora.
Note that this is the author of ripgrep replying to a third party commenter asking whether rg isn’t already lightweight, and comparing the two under various possible definitions of “lightweight”.
I don’t understand when people typeset some name in verbatim, lowercase, but then have another name for the actual command. That’s confusing to me.
Programmers are too enarmored with lower-case names. Why not Ripgrep? Then I can surmise that there might not be some program ripgrep(1) (there might be a shorter version), since using capital letters is not traditional for CLI programs.
> Stacked Git, StGit for short, is an application for managing Git commits as a stack of patches.
> ... The `stg` command line tool ...
Now, I’ve been puzzled in the past when inputing `stgit` doesn’t work. But here they call it StGit for short and the actual command is typeset in verbatim (stg(1) would have also worked).
Because we are constantly writing variables that are lowercase. Coming up with a name that is both short but immediately understandable is what we live for. Variables are our shrine, we stare at them everyday and are used to their beauty and simplicity.
How would you capitalise it? RipGrep? RIPGrep? You’d need to pick a side and lose the pun. (And of course grep itself would need to be GReP if we took it all the way)
The fun part is it is pretty easy to “rewrite” ripgrep in rust, because burntsushi wrote it as a ton of crates which you can reuse. So you can reuse this to build your own with blackjack and hookers.
A "ton of crates" is IMO the best way to write large Rust programs. Each crate in Rust is a compilation unit, the equivalent of one `.c` file in C. If they don't depend on one another, each crate can be compiled in parallel. It makes building the whole project faster, often significantly so. As with anything one can take it too far, but as long as each crate makes sense as an independent unit it's good.
Gotta add a +1 for this. I wanted to do some ignore files etc for a project.
I thought "well I kinda want to do what rg does". Had a little glance and it was already nicely extracted into a separate crate that was a dream to use.
It seems to me that `rg` is the number one most important part that enables LLMs to be smart agents in a codebase. Who would have thought that a code search tool would enable AGI?
Faster is not always the best thing. I still remember when vs code changed to ripgrep I had to change my habit using it, before then I can just open vs code to any folder and do something with it, even if the folder contains millions of small text files. It worked fine before, but then rg was picked, and it happily used all of my cpu cores scanning files, made me unable to do anything for awhile.
To be honest I hate all the new rust replacement tools, they introduce new behavior just for the sake of it, it's annoying.
One of my favorite moments in HN history was watching the authors of the various search tools decide on a common ".ignore" file as opposed to each having their own: https://news.ycombinator.com/item?id=12568245
I would argue that grep-like tools which read .gitignore violate the Principle of Least Astonishment (POLA). It would be fine if there were a --ignore flag to enable such functionality, but defaulting to it just feels wrong to me. Obviously smarter people than I disagree, but my dumdum head just feels that way.
Back in the day I would have agreed with you, but ever since there is js everywhere you end up with minified js that megabytes big and match everything. I still have muscle memory with `grep -r` and it almost always ends up with some js file, that I didn't know exists ruining the moment.
> Obviously smarter people than I disagree, but my dumdum head just feels that way.
That's absolutely not it. What you're describing is part of the UNIX philosophy: programs should do one thing and do it well, and they should function in a way that makes them very versatile and composable, etc.
And that part of the philosophy works GREAT when everything follows another part of the philosophy: everything should be based on flat text files.
But for a number of reasons, and regardless of whatever we all think of those reasons, we live in a world that has a lot of stuff that is NOT the kind of flat text file grep was made for. Binary formats, minified JS, etc. And so to make the tool more practical on a modern *nix workstation, suddenly more people want defaults that are going to work on their flat text files and transparently ignore things like .git.
It's just that you've showed up to an wildly unprincipled world armed with principles.
I have to agree here. I love ripgrep, but at times I've had to go back to regular grep because I couldn't figure out what it was ignoring and why, and there were far too many settings to figure it out.
FYI, `-uu` turns off both ignoring based on special files (.gitignore, etc) and ignoring hidden files.
And if you want to ignore what you want rg to ignore, not what you want git to ignore? Can you do that?
An `--ignore-file=` flag would be nice I guess:
--ignore-file=.ignore
--ignore-file=.gitignore
--ignore-file=.dockerignore
--ignore-file=.npmignore
etc
but then, assuming all those share the same "ignore file syntax/grammar"...
It's good if they can share syntax. You use the same English words to ask Alice and Bob questions, but when you say "So, tell me, Alice…" you don't want Bob to answer you instead. Using another tool's config by default, making it difficult/impossible to use the dedicated config is the most annoying thing I can imagine. If that's what rg does, I guess that must be the reason I couldn't switch.
Agreed. It's a footgun.
It’s the kind of thing that maybe makes sense today. Less likely to make sense twenty years from now though.
But that’s the kind of problem that only successful things have to worry about.
I’ve read this multiple times over the years and this post is still the most interesting and informative piece describing the problem of making a fast grep-like tool. I love that it doesn’t just describe how ripgrep works but also how all the other tools work and then compares the various techniques. It’s simultaneously a tutorial and an expert deep dive. Just a beautiful piece of writing. In a perfect world, all code would be similarly documented.
Such a good read. I actually went back though it the other day to steal the searching for the least common byte idea out to speed up my search tool https://github.com/boyter/cs which when coupled with the simd upper lower search technique from fzf cut the wall clock runtime by a third.
There was this post from cursor https://cursor.com/blog/fast-regex-search today about building an index for agents due to them hitting a limit on ripgrep, but I’m not sure what codebase they are hitting that warrants it. Especially since they would have to be at 100-200 GB to be getting to 15s of runtime. Unless it’s all matches that is.
Yeah, that Cursor blog post is a bit iffy since they just brush over the "ripgrep is slow on large monorepos", move on to techniques they used, and then completely ignore the fact that you have to build and maintain the index.
On a mid-size codebase, I fzf- and rg-ed through the code almost instantly, while watching my coworker's computer slow down to a crawl when Pycharm started reindexing the project.
When Claude Code uses `grep` it's actually using `rg` underneath
Oh, interesting! I had a user prompt that suggest using rg not grep, but was annoyed that it uses rg.
When I first heard about ripgrep my reaction was laughing. grep had been too established. No way something that isn't 100% compatible with grep could get any traction.
And I was dead wrong. Overnight everyone uses rg (me included).
I don't remember why I didn't switch from ag, but I remember it was a conscious decision. I think it had something to do with configuration, rg using implicit '.ignore' file (a super-generic name instead of a proper tool-specific config) or even .gitignore, or something else very much unwarranted, that made it annoying to use. Cannot remember, really, only remember that I spent too much time trying to make it behave and decided it isn't worth it. Anyway, faster is nice, but somehow I don't ever feel that ag is too slow for anything. The switch from the previous one (what was it? ack?) felt like a drastic improvement, but ag vs. rg wasn't much difference to me in practice.
> I didn't switch from ag, [...] rg using implicit '.ignore' file (a super-generic name instead of a proper tool-specific config)
The ".ignore" name was actually suggested by the author of ag (whereas the author of rg thought it was too generic): https://news.ycombinator.com/item?id=12568245
Totally agree with the author of rg here. Config names should be unambiguous. Anyway, must have been something else, then. As I've said, I cannot remember what was the specific problem, only that it wasn't quite compatible with the workflow I was used to, and now it'd take another full-in attempt to switch to figure out what was so annoying to me back then.
If you ever do, please reply to this with why.
I was just trying to remember why I switched _to_ rg.
It's nice and everything, but I remember being happy with the tools before (I think i moved from grep to ack, then jumped due to perf to ag and for unremembered reasons to pt.)
It took me a while, but I remembered I ran into an issue with pt incorrectly guessing the encoding of some files[0].
I can't remember whether rg suffered from the same issue or not, but I do know after switching to rg everything was plain sailing and I've been happy with it since.
[0] https://github.com/monochromegane/the_platinum_searcher/issu...
For me it was trying to add a filter to search CMake files to ag and then realizing that the code had some rather stupid design decisions that prevented it. I wrote a pull request that fixed enough things to add the filter, got ignored by the maintainer and later realized that other people had already written the same filter and were ignored too.
I just got ripgrep ported to IRIX over the weekend.
It’s fast even on a 300mhz Octane.
Is IRIX experiencing a hobbyist revival or something? This is the second IRIX reference I’ve seen on here in the past two days, and there was a submission a day or two ago (c.f. a Voodoo video card?) as well. I haven’t personally encountered IRIX in the wild since a company I worked at in 2003. I suppose SGI has always had a cool factor but it’s unusual seeing it come up in a cluster of mentions like this.
It ebbs and flows.
SGUG tried hard to port newer packages for IRIX for several years but hit a wall with ABI mismatches leading to GOT corruption. This prevented a lot of larger packages from working or even building.
I picked up the effort again after wondering if LLMs would help. I ran into the ABI problems pretty quickly. This time though, I had Claude use Ghidra to RE the IRIX runtime linker daemon, which gave the LLM enough to understand that the memory structures I’d been using in LLVM were all wrong. See https://github.com/unxmaal/mogrix/blob/main/rules/methods/ir... .
After cracking that mystery I was able to quickly get “impossible” packages building, like WebKit, QT5, and even small bits of Go and Rust.
I’m optimistic that we’ll see more useful applications built for this cool old OS.
That is pretty neat. I guess this sort of unlocking and unblocking effort is exactly what’s needed for a revival.
I’m sort of thinking of AmigaOS/Workbench as well although, perhaps because of what I would assume was always a much larger user base than SGI had, it maybe never went away like SGI and IRIX did.
It is great seeing these old platforms get a new lease of life.
Ooh that's super interesting. I assume you shared the recipe with the irix community? I remember keeping Netscape up to date on my Indy was already a struggle in 2002.
I'd love to read a blog post on porting back to Irix!
I was using ripgrep once and it had a bug that led me downa terrifying rabbit hole - I can't recall what it was but it involved not being able to find text that absolutely should have been there.
Eventually I was considering rebuilding the machine completely but for some reason after a very long time digging deep into the rabbit hole I tried plain old grep and there was the data exactly where it should have been.
So it's such a vague story but it was a while back - I don't remember the specifics but I sure recall the panic.
idk if this was your issue but I’m posting this because it’s not obvious (especially the default behavior):
Was it confirmed to be a bug?
Sometimes I forget that some of the config files I have for CI in a project are under a dot directory, and therefore ignored by rg by default, so I have to repeat the search giving the path to that config files subdirectory if I want to see the results that are under that one (or use some extra flags for rg to not ignore dot directories other than .git)
Sorry I don't recall exactly but I don't think it was anything special like a hidden or binary file.
I still use it but Ive never trusted it fully since then I double check.
Was the file in a .gitignore by any chance? I've got my home folder in git to keep track of dot/config files and that always catches me out. Really dislike it defaulting to that ignoring files that are ignored by git.
I had that happen too recently… Basically rg x would show nothing but grep -r x showed the lines for any x. Tried multiple times with different x, then I kept using grep -r at that time. After a few days, I started using rg again and it worked fine but now I tend to use grep -r occasionally too to make sure.
Next time that happens try looking at the paths, adding a pair of -u, or running with --debug: by default rg will ignore files which are hidden (dotfiles) or excluded by ignore files (.gitignore, .ignore, …).
See https://github.com/BurntSushi/ripgrep/blob/master/GUIDE.md#a... for the details.
I use "grep" to search files (it should never skip any unless I tell it to do otherwise) and "git grep" to be a programmer searching a codebase (where it should only look at code files unless I tell it to do otherwise). Two different hats.
I wouldn't want to use tools that straddle the two, unless they had a nice clear way of picking one or the other. ripgrep does have "--no-ignore", though I would prefer -a / --all (one could make their own with alias rga='rg --no-ignore')
Maybe related to text encodings?
I think riggrep will not search UTF-16 files by default. I had some such issue once at least.
Could have been an incorrectly inferred encoding scheme?
I ran into that with pt, and it definitely made me think I was going mad[0]. I can't fully remember if rg suffered from the same issue or not.
[0] https://github.com/monochromegane/the_platinum_searcher/issu...
and incompatible with grep syntax, which makes it useless to most system admins
(2016)
Maybe the first rust "killer app" in retrospect.
There is also upgrep, which is quite a good project. https://github.com/Genivia/ugrep
I recall that it was also very close in performance. When it was posted here, it beat ripgrep! https://news.ycombinator.com/item?id=38819262
fd:find::rg:grep
Someone please make an awesome new sed and awk.
sd exists, dunno but awk.
https://github.com/chmln/sd
Is it still?
qgrep is faster if you're fine with indexing. worth it
Doesn't ripgrep support indexing too?
And burntsushi is one of us: he's regularly here on HN. Big thanks to him. As soon as rg came out I was building it on Linux. Now it ships stocks with Debian (since Bookworm? Don't remember): thanks, thanks and more thanks.
Big thanks to him indeed (and for other projects in Rust space as well).
// really hoping openai wouldn't now force him to work on some crappy codex stuff if he stays there / in astral.
still a good read
It’s a pure delight to read this docs / pitch.
(2024) gg: A fast, more lightweight ripgrep alternative for daily use cases
https://reddit.com/r/rust/comments/1fvzfnb/gg_a_fast_more_li...
> > IMO, as long as the time differences remain small, I'm totally okay with ripgrep being slower by default on smaller corpora if it means being a lot faster by default on bigger corpora.
Also something-something about dependencies (a Rust staple): https://www.reddit.com/r/rust/comments/1fvzfnb/gg_a_fast_mor...
Note that this is the author of ripgrep replying to a third party commenter asking whether rg isn’t already lightweight, and comparing the two under various possible definitions of “lightweight”.
ugrep is my daily driver. https://ugrep.com
The TUI is great, and approximate matches are insanely useful.
> The binary name for `ripgrep` is `rg`.
I don’t understand when people typeset some name in verbatim, lowercase, but then have another name for the actual command. That’s confusing to me.
Programmers are too enarmored with lower-case names. Why not Ripgrep? Then I can surmise that there might not be some program ripgrep(1) (there might be a shorter version), since using capital letters is not traditional for CLI programs.
Look at Stacked Git:
https://stacked-git.github.io/
> Stacked Git, StGit for short, is an application for managing Git commits as a stack of patches.
> ... The `stg` command line tool ...
Now, I’ve been puzzled in the past when inputing `stgit` doesn’t work. But here they call it StGit for short and the actual command is typeset in verbatim (stg(1) would have also worked).
Because we are constantly writing variables that are lowercase. Coming up with a name that is both short but immediately understandable is what we live for. Variables are our shrine, we stare at them everyday and are used to their beauty and simplicity.
How would you capitalise it? RipGrep? RIPGrep? You’d need to pick a side and lose the pun. (And of course grep itself would need to be GReP if we took it all the way)
I wrote Ripgrep.
And they wrote "... you'd need to pick a side and lose the pun.."
It’s only 2 characters - if you use it all the time it becomes muscle memory.
You can simply add a shell alias with whatever name you like and move on.
True, but easier said than done, because one often need to work in more shells than their local machines..
This is a nonstandard tool. If you can't customize your machine, you already don't have it.
But it could be one day..
Do something like this to fall back to plain grep. You will somehow have to share these configurations across machines though.
You can't in most corporate env machines.
You may be able to download ripgrep, and execute it (!), but god forbid you can create an alias in your shell in a persistant manner.
`[citation needed]`
> You can't in most corporate env machines.
Really? "most" even? What CAN you do if you can't edit files in your own $HOME?
Don't get me started on `nvim` to run neovim...
Hasn’t someone rewritten ripgrep in rust by now? C’mon it’s 2026. Oh wait it was written in Rust (back in 2016).
The fun part is it is pretty easy to “rewrite” ripgrep in rust, because burntsushi wrote it as a ton of crates which you can reuse. So you can reuse this to build your own with blackjack and hookers.
A "ton of crates" is IMO the best way to write large Rust programs. Each crate in Rust is a compilation unit, the equivalent of one `.c` file in C. If they don't depend on one another, each crate can be compiled in parallel. It makes building the whole project faster, often significantly so. As with anything one can take it too far, but as long as each crate makes sense as an independent unit it's good.
Gotta add a +1 for this. I wanted to do some ignore files etc for a project.
I thought "well I kinda want to do what rg does". Had a little glance and it was already nicely extracted into a separate crate that was a dream to use.
Thanks @BurntSushi!
Waiting for the zig port
https://github.com/alexpasmantier/grip-grab
Someone kinda did
looks abandoned. last commit was 2 years ago.
It seems to me that `rg` is the number one most important part that enables LLMs to be smart agents in a codebase. Who would have thought that a code search tool would enable AGI?
Faster is not always the best thing. I still remember when vs code changed to ripgrep I had to change my habit using it, before then I can just open vs code to any folder and do something with it, even if the folder contains millions of small text files. It worked fine before, but then rg was picked, and it happily used all of my cpu cores scanning files, made me unable to do anything for awhile.
To be honest I hate all the new rust replacement tools, they introduce new behavior just for the sake of it, it's annoying.