The new Basemark has a new benchmark

OpenGL ES 3.1 and Metal can now be tested and easily compared across OSs

After having gone through a management buyout just three months ago, the newly formed Basemark has introduced a suite of mobile benchmarks for iOS, OpenGL ES 3.1, and Metal. That’s impressive in of and by itself, but even more so because now for the first time a comparative test suite can be run across OSs with the same workloads and profile. This is a big deal in testing land.

Basemark ES 3.1 is the only graphics benchmark in the market that offers a toggle feature for easy performance bottleneck detection. The benchmark comes in three different versions: Free, Professional, and Corporate. The following table describes the main differences between the versions:

The Basemark Benchmark comes in three version.

Basemark ES 3.1 contains the following features:

Deferred Lighting with Alpha Blending
Four render targets are attached to shaders in geometry stage, two are used for storing positions, normals and material properties while the other two are used for decal textures and semi-transparent objects, respectively
There are 80 Lights in the scene
A Stencil Test is used to optimize the lighting, ensuring that only areas affected by the lighting range of each light object will be shaded
Water physics simulation using compute shaders
Based on Fast Fourier Transform for wave propagation.
Screen space Ambient occlusion using fragment shaders
Indirect Rendering
Particle Effects
Normal Mapping
Multi-pass water reflection
Cube map reflections
Blur
Depth of field
Bloom shading
Screenspace anti-aliasing

The benchmark’s results and scoring are based on the following rules and premises. The score computation is derived from FPS, which is calculated as an arithmetic mean with the following rules:

Total number of frames rendered is constant for all target devices running the test with the same configuration.
For on-screen and off-screen test runs, the number of frames consists of the whole scene and for each feature test the number of frames is approximately 1/3 of the whole scene.

To get official test results, make sure you have enabled Official Test Settings from the Configuration menu. The official Main Score is based on the off-screen test run, because it eliminates the differences in screen pixel sizes and the v-sync limit of 60 FPS, by rendering the test frames 36 times to an auxiliary buffer before refreshing the screen.

The Main Score is also normalized according to a constant (C) generated by baseline devices and then scaled such that the score range for the current top performing GPUs is near 1000. The formula for calculating the official MainScore is:

Screenshot displaying the scene with (right) and without (left) Bloom Shading + Depth of Field. (Source: Basemark)

Effect of Screenspace anti-aliasing (enabled only on the right). (Source: Basemark)

Screenshots showing contents of Deferred Shader’s G-buffer: normal, depth, color, and semi-transparent. (Source: Basemark).

Ambient Occlusion in action in the screenshot on the right. (Source: Basemark)

The results show up in a dashboard dialog box. Additionally, the official test result page provides four sub-categories: Lighting, Compute, Instancing and Post-Processing. These tests measure the delta between the baseline measurement of running the scene without any effects on and the performance hit you get by enabling a particular effect. The score of these is expressed as a percentage where bigger is better. The theoretical maximum score is 100, which would imply that there were zero penalties for enabling the particular effect.

Basemark ES 3.1 Professional version allows user to customize test configurations via the Configuration menu. Please note that if you change any of the settings, then the score you obtain will not be the official score. Only test runs with Official Test Settings on will display the official results screen. For all other test runs, Basemark ES 3.1 shows a custom score page as depicted in the screenshot below. Only official scores are sent to Power Board.

You can download the benchmark at Google Play.

What do we think?

At last. Evaluating the processors in a mobile device is extremely difficult because the device builders have such leeway to adjust things such as clock, memory type and speed, and thermal throttles. Given that the GPU plays such a dominate role in a mobile device, its critical that we have a tool to make comparisons possible, and adjustments if necessary. Basemark ES 3.1 gives us all that and more.