"You expect better of us, I expect better of us." So RIM co-CEO Mike Lazaridis opened the company's apologetic BlackBerry press conference this morning, before swiftly pointing out that services have now been "fully restored". As for compensation, however, fellow co-CEO Jim Balsillie said RIM has no idea, at present, what or even if that could be "Our focus has been 100-percent on getting services restored," he told investors and journalists, "that's something we're going to focus on now, but it hasn't been our priority overnight."
According to Lazaridis, a dual-redundancy high capacity switch failed in Europe; that should have been handled by a backup system, but that system also collapsed. The failure in Europe resulted in overloading elsewhere. When the system was restarted, processing the backlog took longer than RIM had expected. RIM is still unclear on why the initial switch failed, but says it was "most likely caused by hardware [failure]."
In the past 18 months, Lazaridis highlighted, RIM has managed 99.97-percent uptime, something which will come as little comfort to those at the tail-end of a more than three day outage. The company will now undertake "aggressive immediate steps" to reduce the likelihood of it happening again, and is working with vendors to analyse the hardware failure, is performing a root core analysis to figure out why the system took longer than expected to come back online, and - perhaps most critically in the same week the iPhone 4S and iMessage hits shelves - attempt to restore the trust of users.
However, RIM also denied that the recent layoffs had any impact on the outage. "This was unfortunately the largest [outage] that we've experienced" Lazaridis concluded, "it was unfortunate."