"None of the challenges described above are new; I've been pondering them for many years now, and yet biting the bullet and rewriting the parser didn't feel like something I'd like to get into. By my private estimates it'd take at least a week of deep heads-down work to port the gritty 2000 lines of YACC grammar rules to a recursive descent parser [4]."
"In this post, I'll describe how I collaborated with an LLM coding agent (Codex) to help me rewrite pycparser to use a hand-written recursive-descent parser and remove the dependency on PLY. This has been an interesting experience and the post contains lots of information and is therefore quite long; if you're just interested in the final result, check out the latest code of pycparser - the main branch already has the new implementation."
"I've definitely noticed the improvement in capabilities of LLM coding agents in the past few months, and many reputable people online rave about using them for increasingly larger projects. That said, would an LLM agent really be able to accomplish such a complex project on its own? This isn't just a toy, it's thousands of lines of dense parsing code. What gave me hope is the concept of conformance suites mentioned by Simon Willison. Agents seem to do well when there's a very clear and rigid goal function - s"
pycparser is a widely used pure-Python C parser producing ASTs inspired by Python's. The project historically relied on PLY for lexing and parsing. The parser was rewritten to a hand-written recursive-descent implementation with assistance from an LLM coding agent (Codex), eliminating the PLY dependency. Several longstanding implementation issues and the perceived large effort created a mental barrier to manual rewriting. Improved LLM capabilities and the use of tests and conformance suites provided a concrete goal and confidence to attempt automated assistance. The rewrite focused on maintainability, preserving parsing functionality while reducing external dependencies.
Read at Thegreenplace
Unable to calculate read time
Collection
[
|
...
]