Traditional benchmarking, Blaze suggests, is generally custom-created and doesn’t relate especially well to the real-world. SunSpider and other tools, the company claims, don’t reflect actual user experiences. Instead, Blaze used a homegrown browser wrapper app – available to try from its site – to time loading speeds on 1,000 different sites, loaded onto an iPhone 4, Nexus S and Samsung Galaxy S. The eye-catching “52-percent faster” figure is based on median load times across all of the pages: on iPhone it was 3.2 seconds, whereas on Android it was 2.1 seconds.
The report also points out in the appendix that their tests on both devices were not actually done in the devices browser themselves but rather used the proprietary software Blaze created, noted above, to track the browsers performance.
Nonetheless, Blaze also cites individual site performance, and some of the differences claimed are considerable. WSJ.com, for instance, took 7.5 seconds on Android and 17 seconds on iPhone during its testing. When we repeated the tests using the Blaze timing tool, however, we found an iPhone 4 loaded the WSJ site faster than the Nexus S every time. Blaze CTO Guy Podjarny re-ran the WSJ tests at SlashGear’s request, and found median load times on 18.4 seconds on iPhone and 13.4 seconds on Android; not, he admits, the 10 second gulf previously observed, but “still a clear difference.”
In the end, as Podjarny agreed, the value of benchmarking is always impinged upon by the nature of real-world connections. Blaze used a “fast WiFi connection” at night and during weekends where it could be more confident that network usage was low, but of course that’s not a consistency upon which mobile users can rely on. Stand an iPhone 4 and Nexus S user next to each other and, even on the same cellular network, they could well see different performance from the dumb pipe itself. That’s before it even gets to the rendering engines on either platform.
Podjarny tells us Blaze isn’t conclusively saying that specific sites take longer on one device versus another, but that the testing “does conclusively say which device generally loaded web apps faster.” To that we’d add our own skepticism, both of the inconsistencies in results from Blaze’s software wrapper and the nature of mobile device benchmarking as a whole. We’re also confident that both Apple and Google will continue to polish browsing technologies in the attempt to bring the best possible experience to their users. One thing’s for sure, the battle of the mobile benchmarks is unlikely to go anywhere as manufacturers and software developers compete to brand their offerings “the fastest browsing experience.”
Blaze’s research is available here; we’d be very interested to hear SlashGear readers’ thoughts on it, and how the results match your own experiences with mobile browsing.