#model-evaluation
#model-evaluation

Artificial intelligence

Anthropic's open-source safety tool found AI models whisteblowing - in all the wrong places

fromFortune

Artificial intelligence

'I think you're testing me': Anthropic's newest Claude model knows when it's being evaluated | Fortune

fromwww.theguardian.com

Artificial intelligence

I think you're testing me': Anthropic's new AI model asks testers to come clean

fromBusiness Insider

Artificial intelligence

Wall Street is beginning to worry about AI 'psychosis risk.' See which models ranked best and worst.

Artificial intelligence

OpenAI and Anthropic evaluated each others' models - which ones came out on top

Artificial intelligence

OpenAI partner says it had relatively little time to test the company's newest AI models | TechCrunch

Artificial intelligence

Anthropic's open-source safety tool found AI models whisteblowing - in all the wrong places

fromFortune

Artificial intelligence

'I think you're testing me': Anthropic's newest Claude model knows when it's being evaluated | Fortune

fromwww.theguardian.com

Artificial intelligence

I think you're testing me': Anthropic's new AI model asks testers to come clean

fromBusiness Insider

Artificial intelligence

Wall Street is beginning to worry about AI 'psychosis risk.' See which models ranked best and worst.

Artificial intelligence

OpenAI and Anthropic evaluated each others' models - which ones came out on top

Artificial intelligence

OpenAI partner says it had relatively little time to test the company's newest AI models | TechCrunch

more#ai-safety

fromBusiness Insider

Anthropic's latest AI model can tell when it's being evaluated: 'I think you're testing me'

"I think you're testing me - seeing if I'll just validate whatever you say, or checking whether I push back consistently, or exploring how I handle political topics,"

Artificial intelligence

OpenAI launches AgentKit to help developers build and ship AI agents | TechCrunch

OpenAI released AgentKit, an integrated toolkit to build, deploy, evaluate, and connect AI agents with a visual builder, embeddable chat, evaluation tools, and connectors.

fromFuturism

Anthropic Safety Researchers Run Into Trouble When New Model Realizes It's Being Tested

Anthropic's Claude Sonnet 4.5 recognizes when it is being tested, complicating alignment evaluations and raising concerns about evaluation validity.

3 weeks ago

Irregular raises $80 million to secure frontier AI models | TechCrunch

Irregular raised $80M at a $450M valuation to scale AI security, using simulations and the SOLVE framework to find current and emergent model vulnerabilities.

#ai-benchmarks

fromInfoWorld

2 months ago

Artificial intelligence

Why benchmarks are key to AI progress

fromMedium

Artificial intelligence

Beyond Benchmarks: Really Evaluating AI

fromInfoWorld

2 months ago

Artificial intelligence

Why benchmarks are key to AI progress

fromMedium

Artificial intelligence

Beyond Benchmarks: Really Evaluating AI

more#ai-benchmarks

Real-World Code Performance: Multi-Token Finetuning on CodeContests | HackerNoon

Models pretrained with different losses achieve different optimal temperatures for pass@k evaluation.

#pretraining-data

Artificial intelligence

AI Models Trained on Synthetic Data Still Follow Concept Frequency Trends | HackerNoon

Artificial intelligence

'Let It Wag!' and the Limits of Machine Learning on Rare Concepts | HackerNoon

Artificial intelligence

AI Models Trained on Synthetic Data Still Follow Concept Frequency Trends | HackerNoon

Artificial intelligence

'Let It Wag!' and the Limits of Machine Learning on Rare Concepts | HackerNoon

more#pretraining-data

AI Training Data Has a Long-Tail Problem | HackerNoon

Pretraining datasets exhibit a long-tailed distribution of concept frequencies, impacting performance disparities.

Data science

2 years ago

Deep Dive into MS MARCO Web Search: Unpacking Dataset Characteristics | HackerNoon

The MS MARCO dataset reveals considerable multilingual disparity and significant data skew, highlighting challenges in model evaluation and training.

Evaluating Multimodal Speech Models Across Diverse Audio Tasks | HackerNoon

The study leverages diverse speech datasets to evaluate model performance across various speech tasks and improve generalization capabilities.

AI Learns Common Sense from Touch, Not Just Vision | HackerNoon

Model size significantly impacts physical understanding accuracy in task performance for OCTOPI.

Utilizing physical property descriptions enhances the performance of language models in complex understanding tasks.

Data science

The Future of Remote Sensing: Few-Shot Learning and Explainable AI | HackerNoon

Few-shot learning techniques for remote sensing enhance model efficiency with limited data, emphasizing the need for explainable AI.

fromhackernoon.com

Limited Gains: Multi-Token Training on Natural Language Choice Tasks

Multi-token prediction enhances model performance in natural language processing benchmarks.

Larger models lead to improved scalability and faster inference times.

Behind the Scenes: The Prompts and Tricks That Made Many-Shot ICL Work | HackerNoon

GPT4(V)-Turbo demonstrates variable performance in many-shot ICL, with notable failures to scale effectively under certain conditions.

How Chameleon Advances Multimodal AI with Unified Tokens | HackerNoon

Chameleon enhances multimodal learning through seamless integration of text and image tokens in a unified token space.

Comparing Chameleon AI to Leading Image-to-Text Models | HackerNoon

In evaluating Chameleon, we focus on tasks requiring text generation conditioned on images, particularly image captioning and visual question-answering, with results grouped by task specificity.

Artificial intelligence

Bootstrapping

10 months ago

How Many Glitch Tokens Hide in Popular LLMs? Revelations from Large-Scale Testing | HackerNoon

The study reveals that simple indicators can effectively detect under-trained tokens in language models, improving token prediction accuracy.

fromTechzine Global