Sections:Fails

At 1:49 AM EDT on 6/30/09, a brief power interruption in our Data Center appears to have severely damaged one of the four UPS's in one of our racks. (This UPS had not exhibited any symptoms of issues going into the power interruption.)  The damaged UPS resulted in half of the rack's AC power supply being removed.The infrastructure in that rack was designed to continue to function in this type of partial power outage, but several limitations in this design were exposed yesterday,  resulting in the queuing of all inbound email of organizations using the Ultimate Anti-Spam Protector option, an outage of BlackBerry service, and issues with one of our two infrastructure monitoring systems.  At 8:42 AM, we re-routed inbound email from the queues to the Ultimate Anti-Spam Protector service, and the inbound email resumed processing. Due to a configuration issue with the re-routing, some organizations' inbound email was 'bounced' back to the email sender, instead of being successfully delivered.  We were able to restore most email service by 9:30 AM.  Some isolated issues with email and BlackBerry service remained until everything was fully resolved at 12:40 PM. 

To prevent this type of issue in the future, we have taken corrective actions so that the Ultimate Anti-Spam Protector processing and BlackBerry services will continue functioning in the event of this type of  issue in the future.  We are in the process of updating the affected infrastructure monitoring system so that it too will operate properly during this type of issue. And we are replacing the affected UPS with a model that will provide our monitoring system with more diagnostic information, to help reduce the probability of a UPS-caused AC power outage occurring again. 

We apologize for the service interruption, and we will build on the corrective actions we have already taken, as we continue to strive to provide the highest possible service level on a proactive basis.  If you have any questions or concerns please do not hesitate to contact us at This email address is hidden from email harvesters via JavaScript or 1-800-882-8701.

 

- The SureTech.com Solutions Team