LAB is an open-source evaluation framework for assessing how well AI agents perform extended, real-world legal work rather than short reasoning tasks. The first version includes more than 1,200 tasks across 24 legal practice areas. Tasks are graded against more than 75,000 expert-written rubric criteria. The framework is designed to show where agents can do all, some, or none of a task, supporting law firms in measuring the ROI of AI investments and identifying where AI can augment teams. LAB is launched without a leaderboard, with plans to work with research partners to produce baseline results and publish standards for normalizing submissions before any rankings appear.
"“The goal of LAB is to provide a clear picture of how agents can be deployed to support legal work in the real world,” the researchers write. “By articulating where agents can do all, some, or none of a task, LAB helps law firms measure the ROI of AI investments and where such investments can augment their teams' work.”"
"“We're intentionally launching LAB without a leaderboard because we expect the dataset to evolve over time and we want to work with the community to ensure results are clear and intuitive in how they convey agent performance,” Harvey says."
"The first version of LAB contains more than 1,200 tasks spanning 24 legal practice areas, graded against more than 75,000 expert-written rubric criteria. The code and a portion of the dataset are available on GitHub."
"In creating LAB, Harvey says that existing legal AI benchmarks - including LegalBench, CUAD, LEXam, and Harvey's own earlier BigLaw Bench - measure short-horizon reasoning, such as ability to read a contract, answer a question, compare cases, or analyze an argument. LAB is meant to measure something closer to the unit of work that actually gets delegated inside a law firm."
Read at LawSites
Unable to calculate read time
Collection
[
|
...
]