I started exactly such a project, for org syntax, a while ago. Didn't get very far though because it seems to me that things in org cannot be parsed and understood in one go. For example the TODO keywords can be specified at the top of an org file and then would influence how headings in the document are understood. This cannot be done with merely a PEG.
Another tricky point is nested inline markup. Bold inside italic? Verbatim inside bold, inside italic? And so on. Would be great to support all meaningful combinations via recursive rules. Org has many inline markup things. Even programming language specific inline markup elements.
Another issue is, that Guile's PEG library, when using the non-string form of grammar rules, does not allow mutually recursive grammar rules, due to being buggy (in that way it is not excellent, but in other ways it is!). One has to use the string form or rules, which makes it a lot less nice to use, unfortunately. I hope that will be fixed at some point, because I like the library except for that. And it is in the standard lib, no need to install any dependencies. Like to use it for AoC puzzle inputs for example.
Will be interesting to read how far they got.
EDIT: Also it's going to be great to have an actual grammar based parser for org. This will ease creating one for other tools, like for example parsers of readmes in repos on git hosters.
I had done some work in this direction in my org parser for racket [0]. That work is stalled at the moment (though I still intend to come back to it), but many of the interactions between org and traditional formal grammars have been explored there.
Aha! Interesting to hear you were going in the same direction. There are indeed quite some tricky situations to handle since Org (and Markdown) are complex in many ways.
the approach I am taking is to combine the PEG with a step of AST processing in order to then produce a correct and descriptive tree structure (for later rendering).
I do agree the Guile library has some spots to improve but honestly, I am already enjoying it so much, and it's far better for the Lisper in me than other things I've tried. I also find S-expressions much more comprehensible and self-documenting.
Looking forward to seeing what you do with it :)
I am indeed hoping that this library is useful for more people and that perhaps things like Haunt blog generator can use it (and other interesting uses)
As for nested markup, and nested lists it has been tricky, but so far I've really promising results:
(assert-equal
#:got (inline-org->html (inline-org->ast "*bold with /italic/*"))
#:expect "<strong>bold with <em>italic</em></strong>")
Over the years, I have implemented parsers numerous times, both at work and for side projects, so writing recursive-descent parsers from scratch has become second nature. Once you understand the mechanics, writing a parser by hand is straightforward and offers distinct advantages, particularly much greater flexibility with error handling and reporting. Because of that, I had always viewed PEGs and parser generators as tools primarily for people who couldn’t hand-roll their own because of their circumstances or skill level. (If you look at projects that are neither understaffed nor underskilled you'll find that hand-written recursive-descent parsers are very common: Clang, Go, Rust, TypeScript, Swift and Lua all have hand-written parsers.)
LLMs have changed the equation. It used to take me 2-3 hours to write a parser for a moderately complex grammar. Now, if I hand an LLM a loosely written, BNF-ish grammar and ask for a recursive-descent parser, it finishes the job in five minutes. At this point, writing them by hand is hard to justify. The model does it substantially faster, and lately, often better than I do.
Which makes me wonder: what’s the appeal of PEGs or parser generators now? They used to make sense when hand-writing wasn't practical, but what compelling reasons are left to use them today?
> Over the years, I have implemented parsers numerous times, both at work and for side projects, so writing recursive-descent parsers from scratch has become second nature. Once you understand the mechanics, writing a parser by hand is straightforward
The reason why it became second nature for you is this hand-rolling, manual work. I observed that with math - unless you manually practice problem solving (integrals, differential equations), you might know the mechanics but you'll have hard time solving them. And that knowledge of mechanics will also fade away at some point.
> LLMs have changed the equation. It used to take me 2-3 hours to write a parser for a moderately complex grammar. Now, if I hand an LLM a loosely written, BNF-ish grammar and ask for a recursive-descent parser, it finishes the job in five minutes. At this point, writing them by hand is hard to justify
So, what is the difference between LLMs generated "manual" parsers vs those with parser generators, beside that with LLM that will be done in five minutes + plus some $$ and with parser generators you'll get that for free and under a second?
> Which makes me wonder: what’s the appeal of PEGs or parser generators now? They used to make sense when hand-writing wasn't practical, but what compelling reasons are left to use them today?
The appeal is predictability - use i.e. Bison, feed it with grammar and you'll get always the same output. Not so much with LLMs.
> Which makes me wonder: what’s the appeal of PEGs or parser generators now?
I just had the robots write a PEG parser generator...
Which can do analysis on the grammars which, I suspect, a hand (or LLM) written one can't do so you don't end up chasing infinite recursion, dead rules and whatnot. It also got shoehorned into the regex engine (https://arxiv.org/abs/1210.4992) for my toy Java 1.0 compiler to loop back to TFA.
For my APL interpreter they had to do the 'handwritten' parser (a Pratt parser a sibling comment brings up) as you need to combine parsing and evaluation since there's no way for the parser to tell what it's looking at because APL syntax is just weird.
How small of a model can complete the operation you described above? If writing a parser still requires a software forge with 2T of vram and a petabyte of training data, then I still see value in PEGs.
Maybe the smaller local models, given a structured grammar can use a PEG to generate a parser.
Agreed, and let me add that Pratt parsers are both powerful, and efficient, and the code is fairly easy to understand and write by hand (while being slightly more powerful than recursive descent).
No LLMs needed, and you can jump straight to coding, and you don't have to go through precedence shenanigans.
If I understand correctly, one of the recent cool generative AI-assisted results is actually supposed to have closed this problem, complete with a Lean proof!
I think language designers in general have realized that there's nothing particularly virtuous about a programming language with a syntax that's hard to parse.
Go and Rust (and Pascal) have cleaned up C's awful legacy enough that they're parseable with LL(X) parsers that don't have scary corner cases or polynomial behavior.
Indeed. Regexes are _regular_, which is non hierarchical by definition.
The two are useful for different layers of abstraction: regex is for lexing and PEG is for parsing.
Speaking of the Chomsky hierarchy, last I checked, it is still unproven whether or not PEGs can parse all context free languages. Intuitively, they're _probably_ weaker than CFGs, but no one has yet provided a counter example.
PEGs aren't contained within the context-free languages, so it's not really intuitive that they're weaker, particularly as nobody has yet come up with a context-free language that a PEG cannot parse. They're capable of parsing things context-free grammars cannot.
Indeed you make a good point. My naive self thought it could work out, but actually the more you zoom out, the more tree-structures you see in many formats. And in my opinion, then it's time to sheathe Valyrian Steel of Lisp, IMHO the language to rule them all when it comes to working with tree based structures (Org, HTML, SXML, many more)
Due to personal failings, my main takeaway from this piece is that Regex is under attack -- I will not stand by as CS theory blorbo is maligned so! Of course there's an undeniable elegance to Lisp in all cases, and seeing Lisp in use today feels like seeing someone unsheathe Valerian steel. But still, the fact remains: regex is vastly underappreciated by the standard engineer, largely because a few of the most common implementations drop some of the most important features. Namely, composition.
I have no easy way to express this other than to plug my recently-released OSS, namely a library I wrote to implement "RegexStores" in Python[1] to compile complex, composed regexes upfront from a custom DSL. I'll link a representative usecase below[2], which I think drives home two things:
1. Regex deserves to be composed using named groups, at the very least! Most people aren't even aware that you can define named groups upfront and then reference them by name throughout, even 'capturing' them multiple times in one match.[3] Even when you do write patterns that use subroutines, the ergonomics of actually accessing, say, the three matched substrings for the `username` group in your single match is finnicky at best (without some magic[4]...).
2. Regex needs high quality syntax highlighting. The linked page is hopefully skimmable in a python sense, but trying to decipher the individual patterns in GitLab would be tough even for me, and I wrote the darn things. It's kinda awkwardly sized, but this screenshot drives home the basics of the point -- note the bright yellow named groups, which are basically subroutine invocations as discussed above: https://i.imgur.com/NllqM6S.png
Sorry to quasihijack the thread. Hopefully the fact that all this stuff has never been announced or published anywhere is proof enough of my good intentions -- that is, to defend my one n' only :)
Most of this functionality was conceived and implemented in a rage after I found out that Rust's main (only?) regex library avoids the possibility of infinite loops by just dropping half the features of the language, so excuse the jank. Even the halting problem seems solvable when you're willing to make moves like that!
[3]: I wrote up an overview of advanced regexes in Python a while back, but was hit by the agential engineering bus before I had the time to polish and release it. Some may find it interesting, if overly long -- the aforementioned subroutines are covered under `3.1`, and repeated captures under `6.2`: https://gitlab.com/doering-ai/libs/basis/-/blob/main/docs/re...
I think you make a great point in highlighting the value of Regex. I don't want to undermine it at all, although some use-cases are just not a good fit for it (even if you can horseshoe things). I think a higher-level abstraction for regex is super welcome, and I quite admire what you are doing there, thanks a lot for sharing. I do think regex's indeed quite a lot more difficult to grasp when complexity rises compared to grammars or other approaches.
Worth looking into nested words aka visibly pushdown languages.
They're a proper superset of regular languages and a proper subset of deterministic context-free languages, but they retain many of the nice properties of regular languages that DCFLs don't - they're closed under intersection, union, concatenation, Kleene Star and reversal.
They can parse more languages that Regular Expressions (non PCRE), but fewer than deterministic CFG subsets like LL/LR. They're expressive enough to parse languages which have a regular tree structure like S-expressions, JSON, XHTML.
Ha, that's interesting.
I started exactly such a project, for org syntax, a while ago. Didn't get very far though because it seems to me that things in org cannot be parsed and understood in one go. For example the TODO keywords can be specified at the top of an org file and then would influence how headings in the document are understood. This cannot be done with merely a PEG.
Another tricky point is nested inline markup. Bold inside italic? Verbatim inside bold, inside italic? And so on. Would be great to support all meaningful combinations via recursive rules. Org has many inline markup things. Even programming language specific inline markup elements.
Another issue is, that Guile's PEG library, when using the non-string form of grammar rules, does not allow mutually recursive grammar rules, due to being buggy (in that way it is not excellent, but in other ways it is!). One has to use the string form or rules, which makes it a lot less nice to use, unfortunately. I hope that will be fixed at some point, because I like the library except for that. And it is in the standard lib, no need to install any dependencies. Like to use it for AoC puzzle inputs for example.
Will be interesting to read how far they got.
EDIT: Also it's going to be great to have an actual grammar based parser for org. This will ease creating one for other tools, like for example parsers of readmes in repos on git hosters.
I had done some work in this direction in my org parser for racket [0]. That work is stalled at the moment (though I still intend to come back to it), but many of the interactions between org and traditional formal grammars have been explored there.
0. https://github.com/tgbugs/laundry
Some really impressive work there! thanks for sharing!
Aha! Interesting to hear you were going in the same direction. There are indeed quite some tricky situations to handle since Org (and Markdown) are complex in many ways.
the approach I am taking is to combine the PEG with a step of AST processing in order to then produce a correct and descriptive tree structure (for later rendering).
I do agree the Guile library has some spots to improve but honestly, I am already enjoying it so much, and it's far better for the Lisper in me than other things I've tried. I also find S-expressions much more comprehensible and self-documenting.
Looking forward to seeing what you do with it :)
I am indeed hoping that this library is useful for more people and that perhaps things like Haunt blog generator can use it (and other interesting uses)
As for nested markup, and nested lists it has been tricky, but so far I've really promising results:
(assert-equal #:got (inline-org->html (inline-org->ast "*bold with /italic/*")) #:expect "<strong>bold with <em>italic</em></strong>")
[flagged]
Over the years, I have implemented parsers numerous times, both at work and for side projects, so writing recursive-descent parsers from scratch has become second nature. Once you understand the mechanics, writing a parser by hand is straightforward and offers distinct advantages, particularly much greater flexibility with error handling and reporting. Because of that, I had always viewed PEGs and parser generators as tools primarily for people who couldn’t hand-roll their own because of their circumstances or skill level. (If you look at projects that are neither understaffed nor underskilled you'll find that hand-written recursive-descent parsers are very common: Clang, Go, Rust, TypeScript, Swift and Lua all have hand-written parsers.)
LLMs have changed the equation. It used to take me 2-3 hours to write a parser for a moderately complex grammar. Now, if I hand an LLM a loosely written, BNF-ish grammar and ask for a recursive-descent parser, it finishes the job in five minutes. At this point, writing them by hand is hard to justify. The model does it substantially faster, and lately, often better than I do.
Which makes me wonder: what’s the appeal of PEGs or parser generators now? They used to make sense when hand-writing wasn't practical, but what compelling reasons are left to use them today?
> Over the years, I have implemented parsers numerous times, both at work and for side projects, so writing recursive-descent parsers from scratch has become second nature. Once you understand the mechanics, writing a parser by hand is straightforward
The reason why it became second nature for you is this hand-rolling, manual work. I observed that with math - unless you manually practice problem solving (integrals, differential equations), you might know the mechanics but you'll have hard time solving them. And that knowledge of mechanics will also fade away at some point.
> LLMs have changed the equation. It used to take me 2-3 hours to write a parser for a moderately complex grammar. Now, if I hand an LLM a loosely written, BNF-ish grammar and ask for a recursive-descent parser, it finishes the job in five minutes. At this point, writing them by hand is hard to justify
So, what is the difference between LLMs generated "manual" parsers vs those with parser generators, beside that with LLM that will be done in five minutes + plus some $$ and with parser generators you'll get that for free and under a second?
> Which makes me wonder: what’s the appeal of PEGs or parser generators now? They used to make sense when hand-writing wasn't practical, but what compelling reasons are left to use them today?
The appeal is predictability - use i.e. Bison, feed it with grammar and you'll get always the same output. Not so much with LLMs.
> Which makes me wonder: what’s the appeal of PEGs or parser generators now?
I just had the robots write a PEG parser generator...
Which can do analysis on the grammars which, I suspect, a hand (or LLM) written one can't do so you don't end up chasing infinite recursion, dead rules and whatnot. It also got shoehorned into the regex engine (https://arxiv.org/abs/1210.4992) for my toy Java 1.0 compiler to loop back to TFA.
For my APL interpreter they had to do the 'handwritten' parser (a Pratt parser a sibling comment brings up) as you need to combine parsing and evaluation since there's no way for the parser to tell what it's looking at because APL syntax is just weird.
Horses for courses, as they say.
Maintenance when you don't have access to an LLM?
How small of a model can complete the operation you described above? If writing a parser still requires a software forge with 2T of vram and a petabyte of training data, then I still see value in PEGs.
Maybe the smaller local models, given a structured grammar can use a PEG to generate a parser.
Agreed, and let me add that Pratt parsers are both powerful, and efficient, and the code is fairly easy to understand and write by hand (while being slightly more powerful than recursive descent).
No LLMs needed, and you can jump straight to coding, and you don't have to go through precedence shenanigans.
One cool factoid about PEGs is that it is an open problem if they can parse all context free languages.
If I understand correctly, one of the recent cool generative AI-assisted results is actually supposed to have closed this problem, complete with a Lean proof!
https://arxiv.org/abs/2608.29592
I think language designers in general have realized that there's nothing particularly virtuous about a programming language with a syntax that's hard to parse.
Go and Rust (and Pascal) have cleaned up C's awful legacy enough that they're parseable with LL(X) parsers that don't have scary corner cases or polynomial behavior.
“Org” in this context is emacs org mode. Seems odd in this day and age that anyone would see hierarchical text format and think “regex”!
Indeed. Regexes are _regular_, which is non hierarchical by definition.
The two are useful for different layers of abstraction: regex is for lexing and PEG is for parsing.
Speaking of the Chomsky hierarchy, last I checked, it is still unproven whether or not PEGs can parse all context free languages. Intuitively, they're _probably_ weaker than CFGs, but no one has yet provided a counter example.
PEGs aren't contained within the context-free languages, so it's not really intuitive that they're weaker, particularly as nobody has yet come up with a context-free language that a PEG cannot parse. They're capable of parsing things context-free grammars cannot.
Ordered choice may cause a PEG parser to fail where a CFG parser would recognize the string.
Indeed you make a good point. My naive self thought it could work out, but actually the more you zoom out, the more tree-structures you see in many formats. And in my opinion, then it's time to sheathe Valyrian Steel of Lisp, IMHO the language to rule them all when it comes to working with tree based structures (Org, HTML, SXML, many more)
Due to personal failings, my main takeaway from this piece is that Regex is under attack -- I will not stand by as CS theory blorbo is maligned so! Of course there's an undeniable elegance to Lisp in all cases, and seeing Lisp in use today feels like seeing someone unsheathe Valerian steel. But still, the fact remains: regex is vastly underappreciated by the standard engineer, largely because a few of the most common implementations drop some of the most important features. Namely, composition.
I have no easy way to express this other than to plug my recently-released OSS, namely a library I wrote to implement "RegexStores" in Python[1] to compile complex, composed regexes upfront from a custom DSL. I'll link a representative usecase below[2], which I think drives home two things:
1. Regex deserves to be composed using named groups, at the very least! Most people aren't even aware that you can define named groups upfront and then reference them by name throughout, even 'capturing' them multiple times in one match.[3] Even when you do write patterns that use subroutines, the ergonomics of actually accessing, say, the three matched substrings for the `username` group in your single match is finnicky at best (without some magic[4]...).
2. Regex needs high quality syntax highlighting. The linked page is hopefully skimmable in a python sense, but trying to decipher the individual patterns in GitLab would be tough even for me, and I wrote the darn things. It's kinda awkwardly sized, but this screenshot drives home the basics of the point -- note the bright yellow named groups, which are basically subroutine invocations as discussed above: https://i.imgur.com/NllqM6S.png
Sorry to quasihijack the thread. Hopefully the fact that all this stuff has never been announced or published anywhere is proof enough of my good intentions -- that is, to defend my one n' only :)
Most of this functionality was conceived and implemented in a rage after I found out that Rust's main (only?) regex library avoids the possibility of infinite loops by just dropping half the features of the language, so excuse the jank. Even the halting problem seems solvable when you're willing to make moves like that!
[1]: https://gitlab.com/doering-ai/libs/basis/#regular-expression...
[2]: https://gitlab.com/doering-ai/apps/wiki-parse/-/blob/main/wi...
[3]: I wrote up an overview of advanced regexes in Python a while back, but was hit by the agential engineering bus before I had the time to polish and release it. Some may find it interesting, if overly long -- the aforementioned subroutines are covered under `3.1`, and repeated captures under `6.2`: https://gitlab.com/doering-ai/libs/basis/-/blob/main/docs/re...
[4]: https://gitlab.com/doering-ai/libs/basis/-/blob/main/my/rege...
I think you make a great point in highlighting the value of Regex. I don't want to undermine it at all, although some use-cases are just not a good fit for it (even if you can horseshoe things). I think a higher-level abstraction for regex is super welcome, and I quite admire what you are doing there, thanks a lot for sharing. I do think regex's indeed quite a lot more difficult to grasp when complexity rises compared to grammars or other approaches.
Worth looking into nested words aka visibly pushdown languages.
They're a proper superset of regular languages and a proper subset of deterministic context-free languages, but they retain many of the nice properties of regular languages that DCFLs don't - they're closed under intersection, union, concatenation, Kleene Star and reversal.
They can parse more languages that Regular Expressions (non PCRE), but fewer than deterministic CFG subsets like LL/LR. They're expressive enough to parse languages which have a regular tree structure like S-expressions, JSON, XHTML.