Debates over AI benchmarking have reached Pokémon
Even Pokémon is not safe for AI -benchmarking controversy.
Last week, one Post on X Viral went and claimed that the newest Gemini model from Google the flagship Claude model from Anthropic surpassed in the original Pokémon video game trilogy. Allegedly Gemini Lavendar Town had reached the Twitch current of a developer; Claude was stuck on Mount Moon at the end of February.
Gemini literally leads to Claude ATM in Pokemon after reaching Lavender Town
119 Live views Only by the way, incredibly underestimated stream pic.twitter.com/8avsovai4x
– Jush (@jush21e8) April 10, 2025
But what the mail did not mention is that Gemini had an advantage.
As Users on Reddit Noted, the developer who maintains the Gemini stream has built an adapted minimap that helps the model to identify “tiles” in the game, such as a chainable trees. This reduces the need for Gemini to analyze screenshots before the gameplay decisions makes.
Now Pokémon is at best a semi-serious AI-benchmark-we would claim that it is a very informative test of the possibilities of a model. But the is An instructive example of how different implementations of a benchmark can influence the results.
For example anthropic reported Two scores for his recent Anthropic 3.7-Sonnet model on the Benchmark SWE-Bench verified, which was designed to evaluate the coding options of a model. Claude 3.7 Sonnet achieved 62.3% accuracy on SWE-Bench verified, but 70.3% with a “adapted scaffolding” that developed anthropically.
More recently, Meta has a version of one of its newer models, Llama 4 Maverick, tailored to perform well at a certain benchmark, LM Arena. The vanilla version of the model scores considerably worse with the same evaluation.
In view of the fact that AI-Benchmarks inclusive Pokémon-in the start are imperfect measures, used and non-standard implementations The waters are even more muddy. That is, it seems unlikely that it will be easier to compare models while they are released.
