Almost all Android devices cheat at benchmarks, report says
The only Android manufacturers who aren't cheating at benchmarks are Google and its subsidiary Motorola, according to a new report.
The only Android manufacturers who aren't cheating at benchmarks are Google and its subsidiary Motorola, according to a new report from hardcore testing site AnandTech.
The Samsung Galaxy Note 3 is the latest high-end Android phone found to be gaming the tests tech sites use to rank processor performance by artificially optimising for them, with Samsung yesterday denying that that was its intention.
AnandTech has made a table of 13 of the most prominent Android devices released this year, including the Galaxy Note 3, Galaxy S4, HTC One, Moto X and new Nexus 7. It shows which it has found to be cheating at benchmarks (wonderfully, it's entitled 'I Can't Believe I Have To Make This Table').
All but the Moto and Google Nexus devices (and the Nvidia Shield gaming tablet, which isn't out in the UK) have been caught cheating at least one of the seven major benchmarks. Samsung seems to be the most prolific gamer of tests, with its Tab 3 tweaking two, the S4 three and the Note 3 a whopping six out of seven.
LG, HTC and Asus all seem to be gaming tests too, to some extent. The Nexus devices don't because the software required isn't part of the basic, clean version of Android they run.
Apple, which of course doesn't have any direct competition selling iOS devices, does not use optimisation.
Do benchmarks matter?
What happens when cheating devices take these tests? They have code, written by the manufacturer and added to their version of Android, that detects a prominent benchmark is being performed. It ramps up all the cores of their processors to full speed to race through the data processing required.
But is there anything wrong with that? It's not like they're somehow faking data. It's not an accurate depiction of real-life performance, though -- the phone or tablet will normally restrict the speeds at which various cores run, to save battery and prevent overheating. When you compare those scores with that of another device, it's not giving an accurate comparison of how they might cope with editing your photos or running a cool new game.
It's like a school getting advance warning of an OFSTED inspection. The teachers might drill the kids to behave better, or produce lots of colourful art to go on the walls, and if they're very good they get an extra mufti day next week. It's not what the school is normally like, so prospective parents, who rely on the results of that inspection, will get the wrong idea.
Samsung yesterday said in a statement to CNET UK, "The Galaxy Note 3 maximises its CPU/GPU frequencies when running features that demand substantial performance. This was not an attempt to exaggerate particular benchmarking results."
How do you decide which new phone or tablet to get? Do you rely on benchmarks as part of your decision? Let me know in the comments below, or on our completely above-board Facebook page.
