Extended Hotmail and Outlook Outage Highlight Dark Side of Cloud
Some users of Microsoft's recently minted Outlook.com and its ancient Hotmail services were having a bear of a time accessing their email messages the last couple of days due to an outage. The service disruption also affected SkyDrive, Microsoft's online storage scheme, though a fix was doled out in a matter of hours.
Hotmail and Outlook.com users started complaining about not being able to access their emails on Tuesday, March 12th, building in up frustration after several hours rolled by without a fix in sight. That's not to say Microsoft's engineers were sitting on their thumbs or asleep at the wheel like Homer Simpson in the control room of a nuclear reactor. Whatever the problem was (Edit -- overheated datacenter, see update below), however, an easy remedy just wasn't in the cards.
"Fixing the problem is taking longer than we hoped. We apologize for the lengthy interruption in service," Microsoft posted on its Service Status page.
The above message was posted on three separate occasions, the first one on Wednesday, March 13th at 1:13 AM and the last one later that morning at 9:12 AM. Service was fully restored several hours later.
Hotmail is home to 360 million email accounts, though it's being phased out in favor of Outlook.com, which officially launched last month. Prior to that, it had been available in preview form, amassing 60 million accounts in six months. It's Microsoft's goal to have all Hotmail users migrated over to Outlook.com by summer.
"On the afternoon of the 12th, in one physical region of one of our datacenters, we performed our regular process of updating the firmware on a core part of our physical plant. This is an update that had been done successfully previously, but failed in this specific instance in an unexpected way," Microsoft explained in a blog post. "This failure resulted in a rapid and substantial temperature spike in the datacenter. This spike was significant enough before it was mitigated that it caused our safeguards to come in to place for a large number of servers in this part of the datacenter.
"These safeguards prevented access to mailboxes housed on these servers and also prevented any other pieces of our infrastructure to automatically failover and allow continued access. This area of the datacenter houses parts of the Hotmail.com, Outlook.com, and SkyDrive infrastructure, and so some people trying to access those services were impacted."
Hotmail and Outlook.com users started complaining about not being able to access their emails on Tuesday, March 12th, building in up frustration after several hours rolled by without a fix in sight. That's not to say Microsoft's engineers were sitting on their thumbs or asleep at the wheel like Homer Simpson in the control room of a nuclear reactor. Whatever the problem was (Edit -- overheated datacenter, see update below), however, an easy remedy just wasn't in the cards.
"Fixing the problem is taking longer than we hoped. We apologize for the lengthy interruption in service," Microsoft posted on its Service Status page.
The above message was posted on three separate occasions, the first one on Wednesday, March 13th at 1:13 AM and the last one later that morning at 9:12 AM. Service was fully restored several hours later.
Hotmail is home to 360 million email accounts, though it's being phased out in favor of Outlook.com, which officially launched last month. Prior to that, it had been available in preview form, amassing 60 million accounts in six months. It's Microsoft's goal to have all Hotmail users migrated over to Outlook.com by summer.
Update
Microsoft provided more details on the outage today, blaming the service disruption on a botched firmware upgrade that caused its datacenter to overheat."On the afternoon of the 12th, in one physical region of one of our datacenters, we performed our regular process of updating the firmware on a core part of our physical plant. This is an update that had been done successfully previously, but failed in this specific instance in an unexpected way," Microsoft explained in a blog post. "This failure resulted in a rapid and substantial temperature spike in the datacenter. This spike was significant enough before it was mitigated that it caused our safeguards to come in to place for a large number of servers in this part of the datacenter.
"These safeguards prevented access to mailboxes housed on these servers and also prevented any other pieces of our infrastructure to automatically failover and allow continued access. This area of the datacenter houses parts of the Hotmail.com, Outlook.com, and SkyDrive infrastructure, and so some people trying to access those services were impacted."
