Well despite my current anti AI sentiment, I have to admit that after reading the article, It was a good use of AI, done by someone with good technical skills. Still I have the feeling that this only works because of the vast accumulated knowledge pre-AI, and if everybody keeps going in this path, it will end up making everyone not advancing their knowledge at the pace they did before. I feel that this AI immersion is really about selling our soul to the devil for short term gains.
> till I have the feeling that this only works because of the vast accumulated knowledge pre-AI
I'm not about to say that there's nothing new under the sun, but parsers are a really well-understood problem where 99.9% of people don't need frontier knowledge and wouldn't be in a position to use it anyway.
And I don't think that people doing research on parsers would ever rely on LLMs for precisely that reason. But we're not parser researchers right?
My point is, we have programming languages like C and C++, we have operating systems like Linux and FreeBSD, we have an empire of software and knowledge accumulated because of the intellectual battles fought by people before AI. With AI, we all are getting our coding easier (and are kind of being forced to), in a way that we will skip these kind of battles. That is, if we all use AI to make our job easier it will have some short term gain but we will end up as a whole ceasing to advance human knowledge with new stuff that has to come from real intellectual work. Like, I don't see people coming up with new outstanding technology if we all sucumb to be AI dependent.
I'm not really sure that I agree. The LLM paradigm basically allows for the same development techniques, for better or worse, but amplified.
So if you were lazily copying the first blog result in Google, getting the first answer from an LLM is equivalent, but the output is actually likely to be better.
If you wanted to do your research on various techniques and evaluate alternatives, LLMs can amplify your capacity to research and to have specific considerations for your specific problem.
LLMs aren't going to solve people's natural inclination towards laziness.
Additionally, while it's true that people may read and learn less about the "lower" levels of software plumbing, it enables enormous possibilities of higher level thinking that before were limited by the amount of manpower you needed.
For example, with LLMs I can try different test sharding strategies or trivially change from factories to fixtures in large test suites. This would have been busywork or drudgery; now I can evaluate several architectural solutions which would not have been possible before.
I think AI is powertool. Period. If you give it to people who are skill, it will create a mess.
I think democratization of intelligence is going to be interesting. You could say the same with same about internet. I think it is part of evolution. May be intelligence or expertise is what does not make us special. May be it is that we are ingenious amd creative with tools and thats how we evolve.
I'm not trying to be pedantic; I think this is an interesting topic and there's a worthwhile distinction to make here. It isn't really being democratized for a couple reasons (at least).
One, access to information isn't truly knowledge in and of itself. People allowing information from LLMs to pass through their brains are not necessarily retaining any of it, and their ability to synthesize and utilize disparate information from LLMs isn't inherently improved by this technology. So the premise of knowledge isn't very sturdy in my mind.
Two, LLMs function across very broad fields of capability, accuracy, content, and so on, and the best models are not accessible to many people. I find people tend to mean the technology is widely available and accessible when they say 'democratization', but that's not necessarily true nor what that word means to begin with.
True democratization would mean something more like "everyone participates in, shapes, regulates, and grows this technology with their own inputs". I don't think that's what happens at all, and in fact, it has been quite the inversion of that so far.
I mention all of this because I agree that it will be interesting to watch what happens, but I don't agree that it will be for the same reasons. I worry about it specifically because there is not an egalitarian distribution of knowledge, and it is not democratically built or shared.
There are some studies that suggest human brain sizes have been shrinking over the last 20,000 years. The theory is that as civilization developed the demand for individual humans to be independently intelligent has weakened because we developed a "collective brain" and also self-domesticated to be more cooperative.
The correlation between brain volume and intelligence is fairly weak. Neanderthals had larger brains than humans, for example. Looking outside the hominids, we have fairly smart corvids with relatively tiny brains.
That means the chain of thought “brains volume decreased, so individuals must have gotten less intelligent. Yet, societies grew smarter, so there must be herd intelligence” breaks at “so individuals must have gotten less intelligent”.
Even if we grant your argument that brain volume between species isn’t good proxy for intelligence, It doesn’t immediately hold for comparisons within a specie.
> May be it is that we are ingenious amd creative with tools and thats how we evolve.
And every time you use the AI to be ingenious or creative, that will be added to the training data. Then someday the AI can be ingenious and creative without you! (It might take a few more breakthroughs. But investors will literally spend trillions chasing those breakthroughs.)
The endgame here is to replace all human intelligence and labor with machines that are smarter and work cheaper. But who controls the machines?
> We didn't write this parser by hand because, at least pre-AI-coding, parsers were extremely difficult to maintain. Writing one without AI would have taken months [...]
> Instead, we use ANTLR, a state-of-the-art, open source parser generator.
I don't agree with this (pre-AI-coding) take. Hand-rolled parsers are much easier to write well and maintain than people think. They also tend to be much faster and produce much better errors than parser generators. I guess if the language you're trying to parse is, say, C++, then you're going to have a miserable time (probably no matter what). But an SQL parser is very doable. (I say this as the author and maintainer of an in-house SQL dialect thingy at work.)
What makes building and maintaining a hand-written parser such a tractable task is:
- The code size can be large, but you can start with a core of a few well-chosen abstractions and then you add lots of parsing code for various language constructs but it's all kind of orthogonal and doesn't add compounding complexity as you go.
- It's just about the most testable kind of code there is. You can cover all the various corner cases with tests and really lock in the behavior so that you can very confidently make changes. One approach I like is to make zillions of tiny test files in the target language accompanied by some golden representation of the AST.
And of course, as the author found out, these properties make writing a parser a really good task for AI coding, too. These tools are very, very good at generating a bunch of new code based on existing abstractions and covering it with lots of test cases.
So I agree with where they ended up, just not where they started :)
That's valid criticism, I kinda hand-waved the "months" part. I read everything I could about parsers while building this (I have a CS background but hadn't thought about parsers in a long time) and came across this blog post https://lakesail.com/blog/sql-parser-in-one-week/ which talked about building a toy parser in a week, so I scaled that up to months for a production one.
Well… tis difficult if one does not understand how grammars work, and therefore parsers. But we’ve seen people use stuff like ContextFreé’s Design Grammar, without even being IT guys, and still figure themselves around.
The whole notion grammars are hard is just wrong. They are not only powerful, but super simple in fact. As is the basic regexp if one cares to spend a focused afternoon to understand it. Probably even less time if working with a decent teacher.
I’ve had very good success in similar setups where you have some sort of “oracle” and can generate enormous corpuses of test data, such that you really, really trust the LLM code must work for the inputs you expect it’ll ever need to handle.
Makes me think of all the algorithms we specify in proof languages and then hand-implement in production languages - this setup could maybe let you just specify the proof of an algorithm and then let LLMs derive efficient implementations with the (slow) proof as an oracle
The key parts of this is how not vibecoded it is. Feels like a model of how you should do software with AI. Now that we can easily set up property testing, fuzzing, etc. there's almost no reason not to.
That's great but I really wish you guys would do something about the llm integration, I tried using it two days ago to create a cohort of users using a sql query, and I was surprised to see that it said that it could not create cohorts for me and i had to resort to exporting data from a sql insight as a cohort cannot use a sql query.
However the worst part was it just writing in the text input slowed down my m4 pro chip to less than 1 fps after 2 prompts and it really left a bad taste in my mouth.
A while ago I had predicted that eventually all coding would eventually become vibe-coding but it would still be a deep engineering discipline (https://news.ycombinator.com/item?id=48040206) -- this is what I meant. Deep technical expertise is still needed, but it shifts from working with the code directly to crafting bespoke comprehensive validation mechanisms around the code. This is a great example of what that could look like.
So it's technically vibe-coding in the sense you don't really look at the code, you just look at the results and "go by the vibes"... except now you're working to rigorously quantify and enforce those vibes. (Philosophical aside: once vibes are rigorously enforced are they "vibes" anymore?)
Recently I was messing around with parquet files in Python and ended up needing to ship the results on Windows, without a Windows machine to test on.
Shipping Python to end users is half mad already, and doing it on Windows is exactly the kind of thing I don't want to spend my life maintaining.
So I figured I'd rewrite it in Go. But that meant embedding a DLL, and how would I test it?
I could spin up a VM, sure. But GitHub Actions already has a Windows environment, and there was my loop: let the agent push to the repo, run tests in GHA, rinse and repeat.
In under an hour it had a full rewrite of my Python, passing every test and producing row-for-row copies of my Parquet output. And it does work on the user machine!
Spotting a loop like that is as satisfying as noticing you can walk your chess opponent into a smothered mate. Truly empowering.
I do consume the parquets with DuckDB but had to read in firebird sql stuff.
I didn't think of checking, but I now learnt there's an extension for DuckDB but it's C++ and also embeds the same DLL [0] https://github.com/flozer/duckdb-firebird
This is super cool, and I am totally going to glean from how you handled testing some of this.
I have a tool I make as a data-plane to a graph engine, and it uses cap'n proto to help (And sqlite as a sort've IPC option). One of the biggest things I have is, I know I am not testing all of it to completion. I am not even really fuzzing, yet.
The thing I would have liked to know is why they don't use an existing fast SQL parser. Was being slightly incompatible with all existing SQL dialects a product requirement?
Our SQL is very similar to ClickHouse SQL, in that we used ClickHouse SQL as a starting point as that's what our underlying DB is. We needed to have our own parser so that we could add additional language features on top.
The previous parser is mostly a declarative grammar file, which is extremely readable. It codegens a C++ parser, which is hard to read. It depends which of those you count as the previous parser's source code!
In the future, we'd make changes by modifying the ANTLR parser first, then using the same approach as in the blog post to get the new parser to parity. We have no plans to get rid of the C++ parser as an oracle!
I think thats exactly what indirectly happened. This guy didnt optimize the parser. Someone else did -- years ago. That work was pulled into the LLM and made it look like magic.
Note that it's not a particularly optimized algorithm: recursive descent + specialized subparser for expressions is simply the standard way to write parsers by hand. It's ANTLR which is super flexible but also dog slow.
Yeah, one of the interesting parts to me while working on this is that the breakpoint for when it's worth writing your own parser vs accepting ANTLR's slowness has shifted massively. Previously it would have been someone's full-time job to maintain. Now with this approach you can get the best of both worlds.
Very good and interesting article, particularly the “loop” that he ended up with.
Amusing anecdotes on LLMs to:
> It did, in fact, make a lot of mistakes, kept doubting whether such a rewrite was even possible, and wanted to call it a day after each round of coding.
> Hilariously one of the most effective was to tell Claude to “think really hard about edge cases" in a background agent.
Very cool use of AI. Shows the power of Autoresearch when you're actually able to build a strict set of tests that clearly delineate successful outcomes.
The old parser has p95 of like 450ms, that seems weirdly slow to me even for a parser generator. Is my intuition wrong? Maybe it's parsing some truly enormous SQL queries?
This must the most compelling look I’ve seen at how software might work with LLMs doing a ton of heavy lifting.
There’s something kind of amazing here in that having read about property based testing I’m pretty confident I could apply it if I had a good use case.
Ha I did consider that! But 70x is plenty fast enough (we still have to query an actual database!) and the parser runs in a shared process on untrusted input, so it wasn't worth the security risk
If we don't care about inspecting the output, why don't we just replace the Rust compiler with something that generates assembly from Rust source, and then cross check the output with a few million random Rust programs?
Surprised anyone gets an interview there. My resume matched the job requirements pretty much exactly with 11yoe and I got an auto rejection mail at 1am.
You have a grammar file in a formal language, and want to generate a faster parser in another formal language.
What's wrong with the source language that it's better to use a sufficiently smart random code generator for the target language, and then fuzz the hell out of the output of it until it behaves the same as the slow translated code, than to create a sufficiently smart compiler from the source to target languages?
I mean this sounds like if we replaced GCC with a really smart random assembly generator and a fuzzer for the output.
> there’s a test for SELECT SELECT FROM FROM WHERE WHERE AND AND which is completely valid SQL
Is this even true? I tried it in SQLite and there's a syntax error after first SELECT.
It would work when "SELECT", "FROM" etc. are quoted, but that's not the same thing.
In what way? This was a geometric mean of the improvements from a small test corpus. In production, where it only parses longer SQL that didn't hit the parser cache, the mean parse time went down by 454x, across millions of parses.
Well despite my current anti AI sentiment, I have to admit that after reading the article, It was a good use of AI, done by someone with good technical skills. Still I have the feeling that this only works because of the vast accumulated knowledge pre-AI, and if everybody keeps going in this path, it will end up making everyone not advancing their knowledge at the pace they did before. I feel that this AI immersion is really about selling our soul to the devil for short term gains.
> till I have the feeling that this only works because of the vast accumulated knowledge pre-AI
I'm not about to say that there's nothing new under the sun, but parsers are a really well-understood problem where 99.9% of people don't need frontier knowledge and wouldn't be in a position to use it anyway.
And I don't think that people doing research on parsers would ever rely on LLMs for precisely that reason. But we're not parser researchers right?
My point is, we have programming languages like C and C++, we have operating systems like Linux and FreeBSD, we have an empire of software and knowledge accumulated because of the intellectual battles fought by people before AI. With AI, we all are getting our coding easier (and are kind of being forced to), in a way that we will skip these kind of battles. That is, if we all use AI to make our job easier it will have some short term gain but we will end up as a whole ceasing to advance human knowledge with new stuff that has to come from real intellectual work. Like, I don't see people coming up with new outstanding technology if we all sucumb to be AI dependent.
I'm not really sure that I agree. The LLM paradigm basically allows for the same development techniques, for better or worse, but amplified.
So if you were lazily copying the first blog result in Google, getting the first answer from an LLM is equivalent, but the output is actually likely to be better.
If you wanted to do your research on various techniques and evaluate alternatives, LLMs can amplify your capacity to research and to have specific considerations for your specific problem.
LLMs aren't going to solve people's natural inclination towards laziness.
Additionally, while it's true that people may read and learn less about the "lower" levels of software plumbing, it enables enormous possibilities of higher level thinking that before were limited by the amount of manpower you needed.
For example, with LLMs I can try different test sharding strategies or trivially change from factories to fixtures in large test suites. This would have been busywork or drudgery; now I can evaluate several architectural solutions which would not have been possible before.
I think AI is powertool. Period. If you give it to people who are skill, it will create a mess.
I think democratization of intelligence is going to be interesting. You could say the same with same about internet. I think it is part of evolution. May be intelligence or expertise is what does not make us special. May be it is that we are ingenious amd creative with tools and thats how we evolve.
> democratization of intelligence
I'm not trying to be pedantic; I think this is an interesting topic and there's a worthwhile distinction to make here. It isn't really being democratized for a couple reasons (at least).
One, access to information isn't truly knowledge in and of itself. People allowing information from LLMs to pass through their brains are not necessarily retaining any of it, and their ability to synthesize and utilize disparate information from LLMs isn't inherently improved by this technology. So the premise of knowledge isn't very sturdy in my mind.
Two, LLMs function across very broad fields of capability, accuracy, content, and so on, and the best models are not accessible to many people. I find people tend to mean the technology is widely available and accessible when they say 'democratization', but that's not necessarily true nor what that word means to begin with.
True democratization would mean something more like "everyone participates in, shapes, regulates, and grows this technology with their own inputs". I don't think that's what happens at all, and in fact, it has been quite the inversion of that so far.
I mention all of this because I agree that it will be interesting to watch what happens, but I don't agree that it will be for the same reasons. I worry about it specifically because there is not an egalitarian distribution of knowledge, and it is not democratically built or shared.
There are some studies that suggest human brain sizes have been shrinking over the last 20,000 years. The theory is that as civilization developed the demand for individual humans to be independently intelligent has weakened because we developed a "collective brain" and also self-domesticated to be more cooperative.
Honestly there might be truth to it, I don't get the downvotes - why?
The correlation between brain volume and intelligence is fairly weak. Neanderthals had larger brains than humans, for example. Looking outside the hominids, we have fairly smart corvids with relatively tiny brains.
That means the chain of thought “brains volume decreased, so individuals must have gotten less intelligent. Yet, societies grew smarter, so there must be herd intelligence” breaks at “so individuals must have gotten less intelligent”.
I think/guess that argument may have merit when replacing brain volume by number of neurons (https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n...)
Even if we grant your argument that brain volume between species isn’t good proxy for intelligence, It doesn’t immediately hold for comparisons within a specie.
No, but correlations there aren’t large, either, if they exist.
Einstein’s brain reportedly was below average size.
That’s a n = 1 example, but there also is a 50/50 example: man vs women. on average, the brains of males are about 10% larger than those of women (https://en.wikipedia.org/wiki/Neuroscience_of_sex_difference...). That doesn’t show up in intelligence differences (https://en.wikipedia.org/wiki/Sex_differences_in_intelligenc...)
Even only looking at males or females, I don’t think larger (fe)males tend to be more intelligent.
> May be it is that we are ingenious amd creative with tools and thats how we evolve.
And every time you use the AI to be ingenious or creative, that will be added to the training data. Then someday the AI can be ingenious and creative without you! (It might take a few more breakthroughs. But investors will literally spend trillions chasing those breakthroughs.)
The endgame here is to replace all human intelligence and labor with machines that are smarter and work cheaper. But who controls the machines?
> And every time you use the AI to be ingenious or creative, that will be added to the training data That's part of evolution.
We as humans have always outsmarted the tools.
It is a tool for some and a crutch for others
That can be said for any technology in history that made work easier.
“Whoa slow down with this ‘writing’ technology. No one will ever remember anything if they can just write it down.”
> We didn't write this parser by hand because, at least pre-AI-coding, parsers were extremely difficult to maintain. Writing one without AI would have taken months [...]
> Instead, we use ANTLR, a state-of-the-art, open source parser generator.
I don't agree with this (pre-AI-coding) take. Hand-rolled parsers are much easier to write well and maintain than people think. They also tend to be much faster and produce much better errors than parser generators. I guess if the language you're trying to parse is, say, C++, then you're going to have a miserable time (probably no matter what). But an SQL parser is very doable. (I say this as the author and maintainer of an in-house SQL dialect thingy at work.)
What makes building and maintaining a hand-written parser such a tractable task is:
- The code size can be large, but you can start with a core of a few well-chosen abstractions and then you add lots of parsing code for various language constructs but it's all kind of orthogonal and doesn't add compounding complexity as you go. - It's just about the most testable kind of code there is. You can cover all the various corner cases with tests and really lock in the behavior so that you can very confidently make changes. One approach I like is to make zillions of tiny test files in the target language accompanied by some golden representation of the AST.
And of course, as the author found out, these properties make writing a parser a really good task for AI coding, too. These tools are very, very good at generating a bunch of new code based on existing abstractions and covering it with lots of test cases.
So I agree with where they ended up, just not where they started :)
That's valid criticism, I kinda hand-waved the "months" part. I read everything I could about parsers while building this (I have a CS background but hadn't thought about parsers in a long time) and came across this blog post https://lakesail.com/blog/sql-parser-in-one-week/ which talked about building a toy parser in a week, so I scaled that up to months for a production one.
Well… tis difficult if one does not understand how grammars work, and therefore parsers. But we’ve seen people use stuff like ContextFreé’s Design Grammar, without even being IT guys, and still figure themselves around.
The whole notion grammars are hard is just wrong. They are not only powerful, but super simple in fact. As is the basic regexp if one cares to spend a focused afternoon to understand it. Probably even less time if working with a decent teacher.
I’ve had very good success in similar setups where you have some sort of “oracle” and can generate enormous corpuses of test data, such that you really, really trust the LLM code must work for the inputs you expect it’ll ever need to handle.
Makes me think of all the algorithms we specify in proof languages and then hand-implement in production languages - this setup could maybe let you just specify the proof of an algorithm and then let LLMs derive efficient implementations with the (slow) proof as an oracle
I cannot believe they're sticking to their guns on this website design. It's awful.
I love that it doesn't feel like every other vibe coded VC backed startup.
They have an excellent branding and have some balls to pull it off, it shows passion, I highly trust it even in company settings.
Try clicking 'switch to website mode' on the left side
thank you!
It's awesome!
Yeah. It locked up my browser. What a pile.
I love it. So different. Slightly BeOS.
This is the type of problem for which LLM generation is great for.
If you have an oracle, and your problem is largely just a pure function, it's pretty good at generating something that both works and is fast.
The key parts of this is how not vibecoded it is. Feels like a model of how you should do software with AI. Now that we can easily set up property testing, fuzzing, etc. there's almost no reason not to.
that is vibecoding these days
That's great but I really wish you guys would do something about the llm integration, I tried using it two days ago to create a cohort of users using a sql query, and I was surprised to see that it said that it could not create cohorts for me and i had to resort to exporting data from a sql insight as a cohort cannot use a sql query. However the worst part was it just writing in the text input slowed down my m4 pro chip to less than 1 fps after 2 prompts and it really left a bad taste in my mouth.
Perhaps the next target for a 100x improvement
A while ago I had predicted that eventually all coding would eventually become vibe-coding but it would still be a deep engineering discipline (https://news.ycombinator.com/item?id=48040206) -- this is what I meant. Deep technical expertise is still needed, but it shifts from working with the code directly to crafting bespoke comprehensive validation mechanisms around the code. This is a great example of what that could look like.
So it's technically vibe-coding in the sense you don't really look at the code, you just look at the results and "go by the vibes"... except now you're working to rigorously quantify and enforce those vibes. (Philosophical aside: once vibes are rigorously enforced are they "vibes" anymore?)
Great loop spotting!
Recently I was messing around with parquet files in Python and ended up needing to ship the results on Windows, without a Windows machine to test on.
Shipping Python to end users is half mad already, and doing it on Windows is exactly the kind of thing I don't want to spend my life maintaining.
So I figured I'd rewrite it in Go. But that meant embedding a DLL, and how would I test it? I could spin up a VM, sure. But GitHub Actions already has a Windows environment, and there was my loop: let the agent push to the repo, run tests in GHA, rinse and repeat.
In under an hour it had a full rewrite of my Python, passing every test and producing row-for-row copies of my Parquet output. And it does work on the user machine!
Spotting a loop like that is as satisfying as noticing you can walk your chess opponent into a smothered mate. Truly empowering.
DuckDB
Also Windows used to have a free VHD with a trial license you could download (and convert to different format with qemu-img)
I do consume the parquets with DuckDB but had to read in firebird sql stuff.
I didn't think of checking, but I now learnt there's an extension for DuckDB but it's C++ and also embeds the same DLL [0] https://github.com/flozer/duckdb-firebird
Really cool! Any chance this is open source?
This is super cool, and I am totally going to glean from how you handled testing some of this.
I have a tool I make as a data-plane to a graph engine, and it uses cap'n proto to help (And sqlite as a sort've IPC option). One of the biggest things I have is, I know I am not testing all of it to completion. I am not even really fuzzing, yet.
Thanks for sharing!
Could the agent traces from this be used to improve sqlglot?
tobymao/sqlglot: Python SQL Parser and Transpiler; with tests and support for 30+ dialects: https://github.com/tobymao/sqlglot
Ibis depends upon sqlglot: https://github.com/tobymao/sqlglot/network/dependents
The thing I would have liked to know is why they don't use an existing fast SQL parser. Was being slightly incompatible with all existing SQL dialects a product requirement?
Our SQL is very similar to ClickHouse SQL, in that we used ClickHouse SQL as a starting point as that's what our underlying DB is. We needed to have our own parser so that we could add additional language features on top.
I think you should clarify that (or whether) while you didn't look at the generated code, you are actually going to adjust it in the future.
How did the two approaches compare in terms of code readability?
The previous parser is mostly a declarative grammar file, which is extremely readable. It codegens a C++ parser, which is hard to read. It depends which of those you count as the previous parser's source code!
In the future, we'd make changes by modifying the ANTLR parser first, then using the same approach as in the blog post to get the new parser to parity. We have no plans to get rid of the C++ parser as an oracle!
Sorry, by two approaches I meant the two parsers generated by the LLM, recursive descent and graph based; not the ANTLR one.
Sort of unexpected that you're keeping the old one as an oracle—but a very good idea for anyone that writes such a complicated parser, indeed!
Yeah curious why they didn't use Presto/Trino, DuckDB, or Clickhouse SQL directly with UDFs and views to augment
Zuora exposes a Trino-based data warehouse which is quite nice and powerful
Besides the parser side, existing dev tools and docs automatically work, too
This is pretty much the case with every SQL dialect
I think thats exactly what indirectly happened. This guy didnt optimize the parser. Someone else did -- years ago. That work was pulled into the LLM and made it look like magic.
Note that it's not a particularly optimized algorithm: recursive descent + specialized subparser for expressions is simply the standard way to write parsers by hand. It's ANTLR which is super flexible but also dog slow.
Yeah, one of the interesting parts to me while working on this is that the breakpoint for when it's worth writing your own parser vs accepting ANTLR's slowness has shifted massively. Previously it would have been someone's full-time job to maintain. Now with this approach you can get the best of both worlds.
Very good and interesting article, particularly the “loop” that he ended up with.
Amusing anecdotes on LLMs to:
> It did, in fact, make a lot of mistakes, kept doubting whether such a rewrite was even possible, and wanted to call it a day after each round of coding.
> Hilariously one of the most effective was to tell Claude to “think really hard about edge cases" in a background agent.
Very cool use of AI. Shows the power of Autoresearch when you're actually able to build a strict set of tests that clearly delineate successful outcomes.
The old parser has p95 of like 450ms, that seems weirdly slow to me even for a parser generator. Is my intuition wrong? Maybe it's parsing some truly enormous SQL queries?
Dunno about the parser, but you broke scrolling on your fancy website without noticing it also ;-)
This must the most compelling look I’ve seen at how software might work with LLMs doing a ton of heavy lifting.
There’s something kind of amazing here in that having read about property based testing I’m pretty confident I could apply it if I had a good use case.
How long did this take?
It took about 2 days to get a proof of concept, and about a week to get something I could ship to production.
I skipped a few features for the PoC (like XML tag support, token positions), so most of the delta was adding those back in!
If you didn’t need to look at the code at all, why not write it in asm instead of Rust, and make it even faster?
Ha I did consider that! But 70x is plenty fast enough (we still have to query an actual database!) and the parser runs in a shared process on untrusted input, so it wasn't worth the security risk
This is a good point.
If we don't care about inspecting the output, why don't we just replace the Rust compiler with something that generates assembly from Rust source, and then cross check the output with a few million random Rust programs?
About 1/1000 of the duration of their interview process where they gloat about wasting your time.
Surprised anyone gets an interview there. My resume matched the job requirements pretty much exactly with 11yoe and I got an auto rejection mail at 1am.
You have a grammar file in a formal language, and want to generate a faster parser in another formal language.
What's wrong with the source language that it's better to use a sufficiently smart random code generator for the target language, and then fuzz the hell out of the output of it until it behaves the same as the slow translated code, than to create a sufficiently smart compiler from the source to target languages?
I mean this sounds like if we replaced GCC with a really smart random assembly generator and a fuzzer for the output.
ha, try to keep going. Run it under samply and Gungraun (need AMD64 for this)
> there’s a test for SELECT SELECT FROM FROM WHERE WHERE AND AND which is completely valid SQL
Is this even true? I tried it in SQLite and there's a syntax error after first SELECT. It would work when "SELECT", "FROM" etc. are quoted, but that's not the same thing.
I interpreted it as referring to a table named "FROM" and fields named "SELECT", "WHERE" and "AND".
[dead]
[dead]
[flagged]
This is true and unkind at the same time.
1. Many cookie banners are required to satisfy the ePrivacy Directive, not just GDPR.
2. Since this is a tracking cookie, not a cookie required for the website to function, it requires consent. So I believe the banner is non compliant.
[flagged]
[dead]
"I didn't rewrite"
Good read, but "70x" is always misleading.
In what way? This was a geometric mean of the improvements from a small test corpus. In production, where it only parses longer SQL that didn't hit the parser cache, the mean parse time went down by 454x, across millions of parses.
Sounds like the real number is 454x faster, not 70x. Checkmate, nerd!
Go read my message again. Every time somebody states it is X time faster, and its the headline, it is pure clickbait.
Yes, you have benchmarks, they will always be synthetic. I'm not dismissing the work, just the selling of it.
You have succeeded though, I did click on it. I even said "Good read" :)
:D
I was hoping to avoid the criticism that I was picking the most favourable number and being misleading!