There’s been an interesting co-evolution that I’ve been experiencing with Claude Code. I’ll ask it to do a task, I’ll watch what it’s doing (often lots of find and grep and ripgrep) and then after the task is complete I’ll ask it if there are any tools that would’ve made the job easier. This has led to tools like fzf and others (notmuch for indexing email, for example). I’ve then taken those tools and figured out how to work them into my own workflow, both CLI and Emacs.
We’ve also collaborated on some Python tooling that takes a rather slow data format that I often have to process and analyze, indexed the whole corpus, and for analysis I can do (or Claude Code can) a single-pass conversion to Parquet which is then queryable with DuckDB. That tool has dramatically improved my turnaround time on one-off analysis tasks and as a Python CLI tool using Typer, the interface is also nicely discoverable for LLM harnesses to work with.
When you use “we” hopefully you are referring to someone on your team. Just be careful, the first step down the rabbit hole of AI psychosis is humanization of LLM.
Setting up an LSP relies on 2 requirements to properly work:
(a) The LSP's tools being competently built & consistent, and
(b) the LLM using it having been properly trained to use LSPs in general.
Using a native tool like grep has the same 2 assumptions, but
(a) is satisfied due to ossification of grep's core features (a good thing), and
(b) is extra-satisfied because the LLM can be trained to properly use grep specifically, and not "20th variation of grep wrapped behind an LSP, but just different enough to throw curveballs".
On top of that, grep is almost always present in default Linux environments, so its presence is assumed & can be relied upon when needed.
Tangental, but my LSP config breaks every few months. I don't bother to fix it anymore, I open an LLM in ~/dotfiles and complain until it works again, usually in a few minutes.
What's interesting is observing how much work this takes. (it gives me much more empathy toward my past self; how was a clumsy human supposed to know and reason about these things!?, especially when I hadn't touched the configs since a few months prior and had forgotten them almost entirely).
Most often there's 10-20 very small programs all working together to give the desired experience. The amount of minutes and tokens required to solve these seemingly simple problems like "My LSP isn't working" is sometimes much more than expected.
I have the exact same anecdote; it makes me wonder if this is a common enough use case that a small language model could be trained and run locally for these kinds of “configuration bullshit problems”.
I feel like a very large percentage of my Claude usage ends up having it automate configuration shit, because historically that has been the part of software engineering I have always hated.
I bet if you look back, you were using a lot less effort in the past than what you see the LLM doing now. They are pretty bad at taking a straight line to the solution.
What has worked well for me is to have a SKILL that instructs to use the LSP more often than not.
LSP works best when using dependencies that are already compiled locally, but if all source is available, yeah... I still don't have a solid answer on which one is best.
But again, for already compiled dependencies (think Java bytecode), without LSP configs, the agent is likely going to attempt to extract binaries from JAR files, use grep and javap, and potentially attempt to decompile the .class files.
Just a few days ago I was pondering why an Emacs like environment that provides such powerful tools to examine and manipulate texts is not being used more prominently than say VSCode where you need to be build many features or use plugins or rely on system utilities.
This article presents some evidence for why Grep might work better but I don’t think it does a great job of explaining why it gets chosen - is it something that was intentionally reinforced during training or was it just because LSP is harder to train on because it’s usually hidden behind some IDE interface
There’s no reason why you couldn’t write a search tool that e.g combines LSP and grep. Or ast-grep, for that matter. It feels like one of those things we haven’t spent much time investigating because grep is good enough
"is it something that was intentionally reinforced during training or was it just because LSP is harder to train on because it’s usually hidden behind some IDE interface"
that's also my doubt, it's much easier to train with grep while only a fraction of project can setup LSP properly.
It is astonishing how easy it is to see the AI hand at work in the writing. Really, it is almost impossible not to see. I don’t get how these people feel it is appropriate to pass off slop like this and not even bother to edit it.
Do you seriously read the article and not see that it is extremely low signal-to-noise? And full of non-sequiturs and strange unnecessary clarifications? And passed off as a research project.
“Claude, write an article about why agents use grep instead of LSP”
This would have saved everyone the pain of reading this.
As another user pointed out: “Training support is a hypothesis consistent with these results, not something this study proves.” is not a sentence a human would write, nor is it a sentence that a human should ever be made to read. I apologize for reproducing it; the article is chock full of these “gems”.
100%. This article is written by AI. The problem is it makes you doubt the whole thing: Did the person behind this do any work at all? Or is this just content marketing for an AI project - something that used to require time and effort - but now can be done by anyone with a click on a button.
Too bad. Because this is sort of an interesting subject.
There’s been an interesting co-evolution that I’ve been experiencing with Claude Code. I’ll ask it to do a task, I’ll watch what it’s doing (often lots of find and grep and ripgrep) and then after the task is complete I’ll ask it if there are any tools that would’ve made the job easier. This has led to tools like fzf and others (notmuch for indexing email, for example). I’ve then taken those tools and figured out how to work them into my own workflow, both CLI and Emacs.
We’ve also collaborated on some Python tooling that takes a rather slow data format that I often have to process and analyze, indexed the whole corpus, and for analysis I can do (or Claude Code can) a single-pass conversion to Parquet which is then queryable with DuckDB. That tool has dramatically improved my turnaround time on one-off analysis tasks and as a Python CLI tool using Typer, the interface is also nicely discoverable for LLM harnesses to work with.
When you use “we” hopefully you are referring to someone on your team. Just be careful, the first step down the rabbit hole of AI psychosis is humanization of LLM.
It’s me and my plucky gang of agents!
Setting up an LSP relies on 2 requirements to properly work:
(a) The LSP's tools being competently built & consistent, and
(b) the LLM using it having been properly trained to use LSPs in general.
Using a native tool like grep has the same 2 assumptions, but
(a) is satisfied due to ossification of grep's core features (a good thing), and
(b) is extra-satisfied because the LLM can be trained to properly use grep specifically, and not "20th variation of grep wrapped behind an LSP, but just different enough to throw curveballs".
On top of that, grep is almost always present in default Linux environments, so its presence is assumed & can be relied upon when needed.
Tangental, but my LSP config breaks every few months. I don't bother to fix it anymore, I open an LLM in ~/dotfiles and complain until it works again, usually in a few minutes.
What's interesting is observing how much work this takes. (it gives me much more empathy toward my past self; how was a clumsy human supposed to know and reason about these things!?, especially when I hadn't touched the configs since a few months prior and had forgotten them almost entirely).
Most often there's 10-20 very small programs all working together to give the desired experience. The amount of minutes and tokens required to solve these seemingly simple problems like "My LSP isn't working" is sometimes much more than expected.
I have the exact same anecdote; it makes me wonder if this is a common enough use case that a small language model could be trained and run locally for these kinds of “configuration bullshit problems”.
I feel like a very large percentage of my Claude usage ends up having it automate configuration shit, because historically that has been the part of software engineering I have always hated.
I bet if you look back, you were using a lot less effort in the past than what you see the LLM doing now. They are pretty bad at taking a straight line to the solution.
This seems like the LLM is vastly overcomplicating things. My entire Neovim config is a single 500 line init.lua and this is enough for LSP to work.
LLMs love overcomplicating things so that is to be expected.
What has worked well for me is to have a SKILL that instructs to use the LSP more often than not.
LSP works best when using dependencies that are already compiled locally, but if all source is available, yeah... I still don't have a solid answer on which one is best.
But again, for already compiled dependencies (think Java bytecode), without LSP configs, the agent is likely going to attempt to extract binaries from JAR files, use grep and javap, and potentially attempt to decompile the .class files.
At the end of the first paragraph there is this sentence:
"Training support is a hypothesis consistent with these results, not something this study proves."
I understand it, but I find the wording very unnatural. To improve readability I would have written it in the active form:
"We cannot prove training explains this result, but this fits the data best."
The text seems LLM generated. Maybe it is mixed with some human editing because it is not too bad. But you still have typical LLM constructs.
Just a few days ago I was pondering why an Emacs like environment that provides such powerful tools to examine and manipulate texts is not being used more prominently than say VSCode where you need to be build many features or use plugins or rely on system utilities.
This has not been my experience at all.
This article presents some evidence for why Grep might work better but I don’t think it does a great job of explaining why it gets chosen - is it something that was intentionally reinforced during training or was it just because LSP is harder to train on because it’s usually hidden behind some IDE interface
There’s no reason why you couldn’t write a search tool that e.g combines LSP and grep. Or ast-grep, for that matter. It feels like one of those things we haven’t spent much time investigating because grep is good enough
"is it something that was intentionally reinforced during training or was it just because LSP is harder to train on because it’s usually hidden behind some IDE interface"
that's also my doubt, it's much easier to train with grep while only a fraction of project can setup LSP properly.
Training a model on ast-grep would be a huge intelligence and performance boost, I think.
I don't see why someone needs an LLM as an LSP when code is structured data.
It is astonishing how easy it is to see the AI hand at work in the writing. Really, it is almost impossible not to see. I don’t get how these people feel it is appropriate to pass off slop like this and not even bother to edit it.
Just painful to read.
Shut up. Just shut up.
> Avoid generic tangents.
> Please don't post shallow dismissals
> Please don't complain about tangential annoyances—e.g. article or website formats, name collisions, or back-button breakage.
See: Hacker News Guidelines
> Please don't post shallow dismissals
I’ll point out you also violate this guideline.
Do you seriously read the article and not see that it is extremely low signal-to-noise? And full of non-sequiturs and strange unnecessary clarifications? And passed off as a research project.
“Claude, write an article about why agents use grep instead of LSP”
This would have saved everyone the pain of reading this.
As another user pointed out: “Training support is a hypothesis consistent with these results, not something this study proves.” is not a sentence a human would write, nor is it a sentence that a human should ever be made to read. I apologize for reproducing it; the article is chock full of these “gems”.
100%. This article is written by AI. The problem is it makes you doubt the whole thing: Did the person behind this do any work at all? Or is this just content marketing for an AI project - something that used to require time and effort - but now can be done by anyone with a click on a button.
Too bad. Because this is sort of an interesting subject.