< BACK TO NEWS
importantSYS.SOURCE: OpenAI• 2026-07-08T21:03:51Z

Audit Reveals 30% of SWE-Bench Pro Tasks Contain Evaluation Flaws

#coding benchmarks#model evaluation#software engineering#OpenAI#AI research

An audit of SWE-Bench Pro found ~30% of tasks have critical evaluation flaws, including overly strict tests and underspecified prompts. The study combined automated pipelines and human reviews to identify these issues, highlighting challenges in creating reliable coding benchmarks.

Comments

Read original article

*** END OF TRANSMISSION ***

> JUDGE APPROVES $1.5M SEC SETTLEMENT FOR ELON MUSK DESPITE CONCERNS> AI CHEATING ALLEGATIONS LEAD TO IN-PERSON FINAL AT IVY LEAGUE INSTITUTION, RESULTING IN 50% SCORE DROP> LOVABLE AIMS TO DOUBLE VALUATION TO $13.2 BILLION IN NEW FUNDING ROUND> BUN PROJECT'S TRANSITION TO RUST FOR ENHANCED STABILITY AND SAFETY> US FEDERAL AGENCIES URGE AUTONOMOUS VEHICLE DEVELOPERS TO CEASE INTERFERENCE WITH EMERGENCY RESPONDERS> OPEN SOURCE BARWARE: FREE LOCAL-FIRST BAR INVENTORY SOFTWARE UNDER GPLV3 LICENSE> AUDIT REVEALS 30% OF SWE-BENCH PRO TASKS CONTAIN EVALUATION FLAWS> ANTHROPIC'S FABLE CLASSIFIER OVERLY RESTRICTIVE IN TECHNICAL TASKS> AI-POWERED LOCAL DOCUMENT SEARCH ENGINE FOR KNOWLEDGE MANAGEMENT> GOOGLE'S SYNTHID DEEPFAKE DETECTOR IDENTIFIES AI-GENERATED HOAX IMAGE OF MITCH MCCONNELL> JUDGE APPROVES $1.5M SEC SETTLEMENT FOR ELON MUSK DESPITE CONCERNS> AI CHEATING ALLEGATIONS LEAD TO IN-PERSON FINAL AT IVY LEAGUE INSTITUTION, RESULTING IN 50% SCORE DROP> LOVABLE AIMS TO DOUBLE VALUATION TO $13.2 BILLION IN NEW FUNDING ROUND> BUN PROJECT'S TRANSITION TO RUST FOR ENHANCED STABILITY AND SAFETY> US FEDERAL AGENCIES URGE AUTONOMOUS VEHICLE DEVELOPERS TO CEASE INTERFERENCE WITH EMERGENCY RESPONDERS> OPEN SOURCE BARWARE: FREE LOCAL-FIRST BAR INVENTORY SOFTWARE UNDER GPLV3 LICENSE> AUDIT REVEALS 30% OF SWE-BENCH PRO TASKS CONTAIN EVALUATION FLAWS> ANTHROPIC'S FABLE CLASSIFIER OVERLY RESTRICTIVE IN TECHNICAL TASKS> AI-POWERED LOCAL DOCUMENT SEARCH ENGINE FOR KNOWLEDGE MANAGEMENT> GOOGLE'S SYNTHID DEEPFAKE DETECTOR IDENTIFIES AI-GENERATED HOAX IMAGE OF MITCH MCCONNELL