Artificial Analysis Revamps Intelligence Index After GPT-6 Astra Benchmarking Dispute

Artificial Analysis has released version 4.2 of its Intelligence Index, revising its methodology after earlier benchmarks drew sharp criticism for underestimating OpenAI’s GPT-6 Astra. The update, published on September 5, 2026, follows weeks of debate among AI researchers and developers who noted a significant gap between Artificial Analysis’ scores and those from other evaluation platforms.

The revised index now shows GPT-6 Astra scoring four points higher than its predecessor, GPT-6. However, Anthropic’s Claude Fable 5.1 retains the top position on the leaderboard, with OpenAI’s model in second place and Meta’s latest offering in third.

What changed in version 4.2

Artificial Analysis addressed several structural issues in its benchmarking approach. The platform removed GPQA-Diamond from its test suite, noting that models have effectively solved that benchmark, making it no longer useful for differentiation. In its place, two new benchmarks have been added: AA-Briefcase, designed to evaluate real-world knowledge work capabilities, and GDP.pdf from Surge AI, which tests PDF document analysis.

A significant methodological change involves the weighting of private test data, which now accounts for 40 percent of the overall scoring. This adjustment is intended to make it harder for model developers to optimise specifically for the benchmarks. The platform also corrected scoring errors across several existing benchmarks and refined its grading systems for more stable results.

The company explained that it had delayed updates to maintain score stability during a period of rapid major model launches. However, the pace of advancement at the top of the leaderboard forced an interim refresh. Version 5 of the Intelligence Index has been under development for eight months and will be rolled out in stages.

The context of the scoring dispute

The controversy arose when Artificial Analysis’ earlier scoring placed GPT-6 Astra roughly on par with its predecessor, contradicting other evaluation results. OpenAI’s own assessments, along with Epoch AI’s ranking, had placed Astra well ahead of the field. Epoch AI ranked it first out of 267 models with a score of 169 points across more than 50 benchmarks. Additionally, ARC-AGI-3 results showed a large to very large improvement depending on the testing harness used.

This discrepancy led to scepticism among researchers who rely on benchmark scores for model selection and procurement decisions. For Indian enterprises and AI labs evaluating frontier models, accurate benchmarking is critical for deployment choices, cost planning, and capability assessments.

Efficiency and cost metrics

Beyond raw intelligence scores, Artificial Analysis’ data highlights GPT-6 Astra’s efficiency advantages. According to the platform, Astra uses fewer tokens per task than any other frontier model. On the cost-to-performance ratio, Anthropic, OpenAI, Meta, and China’s Zhipu AI share the lead position.

This efficiency metric is particularly relevant for Indian startups and enterprises operating under cost constraints. Lower token consumption translates directly to reduced inference costs, which can significantly affect the total cost of ownership for AI deployments at scale.

Datos clave

Metric	Value
Intelligence Index version	2
GPT-6 Astra score change	+4 points over predecessor
Top-ranked model	Claude Fable 5.1 (Anthropic)
Private test data weighting	40 percent of total score

What this means for Indian AI buyers

For technology teams evaluating AI models in India, the Artificial Analysis revision underscores the importance of cross-referencing benchmark scores rather than relying on a single evaluation platform. The discrepancy between different ranking systems highlights how methodology choices can produce divergent conclusions about model capabilities.

The addition of real-world knowledge work and document analysis benchmarks aligns with common enterprise use cases in India, including contract review, compliance document processing, and knowledge management. The removal of solved benchmarks like GPQA-Diamond reflects the evolving nature of AI evaluation as models continue to improve.

Artificial Analysis has indicated that version 5 will bring more substantial changes. Until then, version 4.2 serves as an interim correction to address the most immediate criticism. Developers and procurement teams should monitor these updates closely, as benchmark methodology changes can shift relative model rankings and affect deployment decisions.

Source: The Decoder – https://the-decoder.com/artificial-analysis-overhauls-its-intelligence-index-after-gpt-6-astra-scoring-drew-skepticism/