As AI keeps improving, mathematicians struggle to foretell their own future

"We were quite impressed with how the AI models did, says Lauren Williams, a Harvard University mathematician and First Proof team member. The problems that we proposed really are on the forefront of what AI models—perhaps together with experts—can solve."

"First Proof grew out of its 11-person team's own eye-opening—if sometimes frustrating—experiences with AI. No preexisting benchmarks seemed sufficient for testing LLMs as a mathematician's assistant. In principle, an LLM could save time by proving smaller lemmas—intermediate propositions along a mathematician's path to developing larger theorems of greater interest."

"In just the past few months, the best publicly available models have begun generating valid proofs for minor theorems of actual use for working mathematicians. To some experts, the opening round of First Proof was a pivotal moment in this ongoing story."

First Proof, an 11-person team effort, is conducting the second round of benchmarking large language models' capabilities in research-level mathematics. Recent months have shown publicly available AI models generating valid proofs for minor theorems useful to working mathematicians, marking a significant shift in the field. The initiative emerged from the team's experiences with AI limitations in mathematical assistance, as existing benchmarks proved insufficient for testing LLMs as mathematician assistants. While AI could theoretically save time by proving intermediate propositions, practical applications have often failed. The first round tested 10 lemmas from unpublished papers with a one-week deadline, impressing experts with model performance on frontier-level problems.

Read at www.scientificamerican.com