August 21, 2013, 4:17 PM — The performance and battery life of smartphones and tablets may not match the numbers provided by device makers, but development is under way on a tool that could bring consistency to the measurement of those metrics.
A "user experience" benchmark being developed by Berkeley Design Technology (BDTI) analyzes system-level efficiency to predict performance and battery life in mobile devices. The ratings will be based on device configurations and usage modes such as Web browsing, video and phone calls.
"When we looked at the kinds of benchmarks people were using, we were horrified at how bad some of them are. We saw an opportunity to improve the situation and provide the industry with better metrics," said Jeff Bier, president and founder of BDTI.
The overall mobile device experience is what matters and honest measurements are needed for apples-to-apples comparisons with other devices, Bier said. He likened the benchmark to measuring the performance of a car, in which fuel efficiency and overall performance are measured while taking all components into account.
As the mobile industry matures, chip makers and analysts agreed there is a need for more accurate benchmarks.
Mobile-device benchmarking has been in "a horrible state for a decade" and needs to be addressed, said Patrick Moorhead, founder and president of Moor Insights and Strategy.
"Consumers are being misled by some of the benchmarks, resulting in a bad purchase. Many benchmarks are synthetic in nature and most are unable to correlate to real-world usage models. Others are easy to manipulate because they use the honor system to keep code clean from someone gaming the benchmark," Moorhead said.
Moorhead for more than a decade has been vocal about the need to establish a honest benchmarks in PCs, and is now pushing for metrics that represent real-world mobile usage scenarios.
BDTI is trying to differentiate its benchmark by not measuring the performance of individual components such as CPUs and GPUs, which is common in other tools. One component may not necessarily reflect the performance of a device and the sum of all parts needs to be taken into account when determining power efficiency and user experience, Bier said.
A graphics processor may perform well, but a screen may consume a lot of power to render 3D polygons, Bier said. Or a device may simply lack the bandwidth to deliver the graphics.
"This is one of the perils in isolation, it may look good, but when lapped into a system, it may look different. It you're a component provider.... you want to test in a whole system with a whole set of applications," Bier said.