X

Debate sparked about benchmark for Intel, ARM chips

An analyst raises questions about a benchmark for the Intel and ARM chips that go into smartphones.

Brooke Crothers Former CNET contributor
Brooke Crothers writes about mobile computer systems, including laptops, tablets, smartphones: how they define the computing experience and the hardware that makes them tick. He has served as an editor at large at CNET News and a contributing reporter to The New York Times' Bits and Technology sections. His interest in things small began when living in Tokyo in a very small apartment for a very long time.
Brooke Crothers
2 min read
The Galaxy S4 packs a Samsung Exynos 5 Octa processor.  Questions were raised about benchmarks that pit the Lenovo K900, with an Intel chip, versus the Galaxy S4, with an ARM chip.
The Galaxy S4 packs a Samsung Exynos 5 Octa processor. Questions were raised about benchmarks that pit the Lenovo K900, with an Intel chip, versus the Galaxy S4, with an ARM chip. Samsung

Nothing like a debate about processor benchmarks to get enthusiasts blood up.

EE Times posted a story titled Has Intel really beaten ARM? on Wednesday that calls into question the use of the widely-used AnTuTu Benchmark.

"There has been a considerable amount of press around recent AnTuTu benchmark results claiming, Intel...processor outperforms Nvidia, Qualcomm, and Samsung," wrote the EE Times story's author analyst Jim McGregor of Tirias Research.

The original article states.

"The benchmarks were impressive but the real surprise was the current consumption recorded during the benchmarks; the new [Intel] processor not only outperformed the competition in performance but it did so with up to half the current drain," said Jim Mielke, vice president of engineering at ABI research, in a statement.

Not so fast, McGregor says. After making the not uncommon assertion that chip companies and device vendors have attempted in the past to "manipulate" benchmarks, he says that the results (linked to above) are "odd," as it's the only benchmark that shows an Intel smartphone chip -- the Atom Z2580 -- greatly outperforming ARM.

Samsung Galaxy S4 (with Samsung ARM chip) vs. Lenovo K900 (with Intel silicon).  'The results demonstrate a significant advantage for the Intel processor relative to the Samsung processor...only in the AnTuTu benchmark,' McGregor said.
Samsung Galaxy S4 (with Samsung ARM chip) vs. Lenovo K900 (with Intel silicon). 'The results demonstrate a significant advantage for the Intel processor relative to the Samsung processor...only in the AnTuTu benchmark,' McGregor said. Tirias Research / EE Times

To illustrate his concerns about the AnTuTu benchmark as applied in this case, McGregor "compiled a variety of benchmark information from tech reviewers, benchmarking organizations, and other industry resources."

The compiled results show the Intel chip not faring as well in other benchmarks (see chart).

McGregor also makes the point that a new version of the AnTuTu benchmark seems to favor the Intel chip more than ARM processors.

Finally, a separate critique posted on Berkeley Design Technology's (BDTI) Web site found that "the ARM-based [Samsung] Exynos processor performs all the operations specified in the benchmark source code, while the Intel Z2580 processor skips some steps."