Woolf explained, "If code can indeed be improved simply through iterative prompting such as asking the LLM to 'make the code better' - even though it's very silly - it would be a massive productivity increase. And if that's the case, what happens if you iterate on the code too much?"
The initial solution, which Woolf characterized as something a novice programmer might write, took an average 657 milliseconds to run on an Apple M3 Pro Macbook Pro. When asked to 'write better code,' Claude responded with optimized code that performed 2.7x faster.
Iterations three and four produced further gains, resulting in speedups of 4.1x and 99.7x. Woolf then repeated the experiment using "prompt engineering," which simply means providing the LLM with more detail about what's expected and how to proceed.
Woolf concluded, "Although it's both counterintuitive and unfun, a small amount of guidance asking the LLM specifically what to improve can lead to significant performance enhancements, proving that refinement through prompting is essential for maximizing productivity."
Collection
[
|
...
]