#ai-evaluation

#ai-evaluation

Cornell Tech faculty made a strong showing at the 2025 Conference on Neural Information Processing Systems (NeurIPS), held Dec. 2-7 in San Diego, presenting 23 research papers at one of the world's premier gatherings for artificial intelligence and machine learning. NeurIPS draws thousands of scholars and industry leaders each year and is widely recognized as a leading forum for breakthroughs in AI, computational neuroscience, statistics, and large-scale modeling.

On Thursday, the Laude Institute announced its first batch of Slingshots grants, aimed at "advancing the science and practice of artificial intelligence." Designed as an accelerator for researchers, the Slingshots program is meant to provide resources that would be unavailable in most academic settings, whether it's funding, compute power, or product and engineering support. In exchange, the recipients pledge to produce some final work product, whether it's a startup, an open-source codebase, or another type of artifact.

Some lawyers have learnt that the hard way, and have been fined for filing AI-generated court briefs that misrepresented principles of law and cited non-existent cases. The same is true in other fields. For example, AI models can pass the gold-standard test in finance - the Chartered Financial Analyst exam - yet score poorly on simple tasks required of entry-level financial analysts (see go.nature.com/42tbrgb).

What if the secret to unlocking AI's full potential wasn't in the technology itself but in how we use it? After spending over 200 hours teaching AI to write, Nate B Jones discovered that the biggest mistakes aren't about algorithms or software limitations, they're about human misunderstanding. Too often, we assume AI can read between the lines of vague instructions or magically produce brilliance without guidance. The result? Generic, uninspired content that misses the mark.

ChatGPT maker OpenAI has released a new evaluation, dubbed GDPval, to measure how well its AIs perform on "economically valuable, real-world tasks across 44 occupations." "People often speculate about AI's broader impact on society, but the clearest way to understand its potential is by looking at what models are already capable of doing," the company wrote in an accompanying blog post. "Evaluations like GDPval help ground conversations about future AI improvements in evidence rather than guesswork, and can help us track model improvement over time," OpenAI added.

Result ambiguity can come in different forms. The lack of agreement among raters is the most common one, known as Inter Rater Reliability (IRR).