Benchmarking the mobile experience

How do people use a mobile phone today? Turns out, the question is rather complex.

By Jon Peddie

In trying to benchmark a mobile phone, Jon Peddie Research (JPR) recently looked at how a mobile phone is used. It became quickly apparent that game playing wasn’t the only, or even the most impor­tant, thing a mobile phone was used for. Yet most of the benchmarks we found (and we found over 65) offered a poly­gons per second test. The modern smart­phone is capable of considerably more than drawing polygons, and the uses the phone will be put to vary from person to person.

The MyTouchQ from Huawei is a recent entry into the smartphone wars. (Source: T-Mobile)

It makes sense to first look at what smartphones can do today even before we talk about what they are being used for.

Key smartphone components

Today’s modern smartphone has evolved to embrace a large variety of features and functions, as well as an amazing number of components. It’s safe to say the mobile device revolution began when multimedia el­ements first appeared in the mobile phone beginning with an FM tuner, camera, and stereophonic sound. From those early, somewhat exploratory steps, consumers and application de­velopers have responded, enabling the growth of the functions and capabilities we have today.

The uses enabled by the features and the environment for free development is encouraged by the evolution of app stores and is fostering a rapid evolution­ary development for both the platforms and the applications.

Key user experiences

The mobile phone is used for so many things that it becomes a challenge to list them all. A few uses include augmented reality, photography, pictures and video, gaming, GPS/navigation, mobile bank­ing, near field communication (NFC), stereovision, music, eReaders, and more. And of course, some people use them as telephones.

The mobile has become so ubiqui­tous and so powerful that it’s almost impossible to catalog all of its uses and applications. Email and text are among the most common ways that people use a mobile phone besides phone calls, and various surveys have been conduct­ed to find out what users do with the devices. No conclu­sive results could be found; the sur­veys gave differ­ent results depend­ing on who did the survey and, most of all, who the sur­veyors could get to take the survey. For example, the respected Pew Inter­net research center found that 76% of those surveyed used their phone to take pictures and 34% used them to play games. Well-known surveyor Nielsen found that 61% of users played games, and picture taking wasn’t even men­tioned. Overall what is interesting in looking at the surveys is the list of ac­tivities considered.

Actually, for all of the emphasis on gaming and polygon count, the most common use for smartphones is surf­ing the web and reading emails—pret­ty much what most people do with their computers. Social media runs a very close third. Several studies, as well as common sense, can be found to back up this contention. But, people do play games and especially free games. Nielsen has studies that report that games are the most popular applications in app stores.

The Galaxy Nexus, designed by Google, sold out quickly after its launch earlier this year. (Source: Verizon)

How do you test it?

Given that people use their phones in different ways, how do you compare performance? There are almost as many benchmarks for a smartphone as there are functions. It is only recent­ly that the smart­phone has had an open and accessible operating system, so benchmark and testing programs in the public domain have been limited.

Testing a smart­phone is easy to prescribe, difficult to implement. Test for everything? That’s not practical. The system-on-a-chip (SoC) suppliers have their own tests to back up their performance claims. In addition, those SoC companies who also supply the radios and modems have to test for even more situations, plus submit the reference designs to various government agen­cies throughout the world that regulate the spectrum. But users—and websites—aren’t going to invest that kind of time in testing.

Test suites

We propose an industry-wide agree­ment on a suite of tests for five or six of the most common usage scenarios. For example, we need a test that shows how fast a device can receive a full-resolution picture and how many times it can do it before the battery is depleted or the memory is full. A test that reports on the device’s ability to download a full-length HD movie, store it, and then play it at HD on the device’s screen and an HDTV. A test that reports on the perfor­mance of the device running a full-reso­lution game with all graphics functions at maximum—and for how long the de­vice can run that test. Other usage sce­narios would include transcoding, audio recording, and accuracy of the GPS, ac­celerometer, and gyro are also needed. We need a robust suite of tests that can be applied to various user modalities.

Given the complexity of smartphones and the proliferation of benchmarks, we have to acknowledge we don’t have an ideal world, nor do we have a complete solution, and yet we have to move on. Therefore, at JPR we have selected a few benchmarks to use in a collective manner to try to arrive at a reasonable evaluation of the smartphone for a generalized use case.

To find out more, download the new Jon Peddie Research report, “The Mobile Experience” here:

Jon Peddie is President of Jon Peddie Research, publisher of GraphicSpeak and the JPR TechWatch newsletter.