Snapdragon 888 benchmarks are out - Here's the problem with them

The Snapdragon 888 is Qualcomm's flagship chipset for Android smartphones in 2021, and the stakes have arguably never been higher. On the one hand, Apple is flexing its chip-design chops, and while Qualcomm modems may be in its current hardware, that's not going to be the case forever. Meanwhile, Qualcomm's Android SoC rivals aren't standing still.

Qualcomm has pulled out all the stops for the Snapdragon 888, then, even skipping ahead in its nomenclature to borrow some traditional Chinese good luck in the process. Were 2020 a normal year, we'd have had a chance to play with the company's Snapdragon reference design, Qualcomm's demo hardware for developers and engineers to tinker with the specs.

This, of course, is no ordinary year. Since we can't run our own benchmarks, Qualcomm has done that itself, running the reference design through popular tests like AnTuTu, Geekbench, GFX Bench, and AITuTu.

"Each benchmark was run on a Snapdragon 888 reference design and the subsequent scores are the average result across three iterations," Qualcomm says of its figures. "All benchmarks were run using default settings on the Snapdragon 888 reference design, which is recommended to deliver the best balance between performance and power consumption for user's typical day-to-day use."

The Snapdragon 888 reference design may not look like much – since it's not a commercial device, the aesthetics aren't really important – but it's a powerhouse inside. The Snapdragon 888 itself has a 2.84 GHz Arm Cortex-X1 core for maximum performance, three 2.42 GHz "big" cores, and four 1.8 GHz "little" cores for frugality.

It's paired with 12GB of LPDDR5 memory and 512GB of storage. There's a 6.65-inch 1080 x 2340 120Hz display, USB Type-C 3.1 Gen2, and a 3,780 mAh battery. Of course, device-makers will choose exactly how they'll configure their Android phones.

Clearly – and coming as no great surprise – the Snapdragon 888 is more potent than its predecessor. The mobile chip world is cutthroat: Qualcomm couldn't do anything less than crank things up to their maximum. While that makes for a speedy SoC on paper, it's also tricky to get too excited about the numbers.

Just as we've seen with 5G, the capabilities of your device are only one component in the overall experience, and probably a relatively minor component at that. In the case of 5G, you're dependent on your carrier's network and coverage, arguably more so than the cellular hardware inside your phone or tablet. Yes, mmWave 5G can be blisteringly fast, but most people in the US don't live anywhere near a patch of mmWave coverage.

Much in the same way, the raw performance of a chip is only as useful as what applications do with those capabilities. Twelve months ago, with the Snapdragon 865 reference design in hand, I wrote that you probably shouldn't care about its benchmarking numbers. The same holds true for the Snapdragon 888's results today.

In theory, having a method to compare devices, or chipsets, is handy. In reality, though, that's where the benchmarks' value ends. Extrapolating user experience from a table of numbers is almost impossible.

If you're a photographer, AI-powered image processing may be the most important metric to your contentedness with a device. Gamers' experience will likely hinge on the GPU, and how well graphics are rendered with Vulkan. That's before you get to everyday tasks which increasingly call upon some combination of CPU, GPU, AI co-processors, and various specialized ISPs, or user-experience metrics like wireless management that synthetic benchmarks don't even attempt to quantify.

Software plays a huge part, too. Your Snapdragon 888 powered device may end up delivering a lesser experience if it's stuck on an older version of Android; it'll depend on things like drivers, and how much effort app-developers have put into taking advantage of the hardware they have to play with. That's before you get to the cruft carriers love to install, and the customizations smartphone-makers put in. Just how much part Qualcomm and its chipset actually plays in all this starts to seem surprisingly small.

That's why, I suspect, we've seen the chipmaker collaborate more closely with Google on things like Project Treble. There, the two companies recently announced they'd be working together to ensure new Android phones – based on the Snapdragon 888 and other upcoming Qualcomm chipsets – could get four years of Android OS updates, and four years of security updates. It's a move that will hopefully mean more Android phones running the latest version of Google's OS, and getting to install that quicker too.

In the end, the Snapdragon 888 is a mighty fast chipset. It'll be in the heart of numerous flagship Android phones next year, and they'll undoubtedly boast incredible displays, eminently capable camera arrays, longer battery life, and broader connectivity options. While it's easy to be swayed by headline numbers and big SoC boasts, though, it's equally important to demand commitment from the OEMs using Qualcomm's latest and greatest. After all, having the headroom to run cutting-edge software is no use at all if the software isn't there to install.