Post Mortem of Outage Thursday 12/26/2019

Description of Outage:

At 8:00 AM ET on 12/26/2019 we identified through our morning testing that SureDesk users were unable to login when trying to login from the website so5.  Anyone attempting login would receive a message that login failed.  Users using the direct Citrix Receiver connection were able to log in.  

Outage Review:

We determined that the Netscaler was having a load balancing issue that affected the ability to receive logins through the website portal.  This was restored by 9:00 PM ET 12/26/2019.   Most users were unaffected by this issue as they login directly from the Citrix Receiver, however any Mac users or users who could not set up this method to login were affected.    

Remediation:

From the review of the incident we identified an improved process to load balance the netscaler and are implementing those improvements with Citrix engineers now to avoid a similar issue in the future.

We strive for better than 99.9% uptime and are glad to report that over the past 6 months we continue to exceed this service estimate.

If you have any questions, please contact solutions@suretech.com or call 1-800-882-8701.

Post Mortem of Outage 12/23/2019

Description of Outage:

At 12 PM ET on 12/23/2019 we identified that users on one of our Standard server pool servers had an unexpected disconnect from the file server.  Anyone on that server were unable to connect to their files and all work in progress was closed as well.  This issue was isolated only to users on that specific server in the pool as we run small redundant pool servers for optimal performance. 

Outage Review:

The outage affected users for approximately half an hour.  After that affected users were immediately able to log back in and access all files.  However, any open work in progress had been closed.  We determined that the cause of the shutdown was caused by an automatic reboot of that server.  

Remediation:

We are continuing to work with Amazon engineers to determine if the sudden reboot was a fluke or if there is something specific that cuased this to ensure it does not happen again.  We will update this post after that additional research.

Post Mortem of Outage 09/06/2018

Description of Outage:

At 5 PM ET on 9/6/2018 we identified that one of our Standard server pool servers had an unexpected shutdown.  Anyone on that server immediately had a session closed and all work in progress was closed as well.  This issue was isolated only to users on that specific server in the pool.  The shutdown only affected 10 users in our pool environment as we run small redundant pool servers for optimal performance. 

Outage Review:

Affected users were immediately able to log back in.  However, any open work in progress had been closed.  We determined that the cause of the shutdown was a critical software failure which was isloated to that server. 

Remediation:

During our scheduled maintenance the same night we updated the software that had failed.  After testing and reviewing the system we believe the issue is now fully resolved.

Post Mortem of Outage Monday 07/02/2018

Description of Outage:

At 6:30 AM ET on 7/2/2018 we identified through our morning testing that SureDesk users were unable to login.  Anyone attempting login would receive a message that apps and desktops were unavailable. 

Outage Review:

We determined that the Delivery Controller was not responding and thus preventing users from connecting to their desktop environments.  Our normal restore efforts for this kind of report did not immediately resolve the issue as some services still remained locked up.  We began a restore process of all the SQL Databases running on the login server.  We determined the database that had the corruption and restored.  All services were restored at 9:30 AM and most customers notified.  We then did additional testing for customers with custom servers to ensure all logins all restored.  All customers were notified by 10:15 AM.   Total system outage was 3 hours. 

Remediation:

From the review of the incident we identified an improved process to compact the databases every 3 months to enable restore to be much faster if a similar database corruption were to occur again in the Delivery Controller.   We implememted and tested this process and the restore should be less than 10 minutes to address a similar issue in the future should it occur.

Post Mortem of Outage Friday 08/18/2017

Description of Outage:

Standard SureDesks and some Custom Server SureDesks rebooted without warning. Overall outage was 30 minutes. Customers were working post reboot without any major problems.

Outage Review:

From the review of Logs and talking with Citrix the issue seems to be that the Delivery Controllers failed to reboot some of the Server this weekend during regular server maintenance on Sunday. It seems that the Delivery Controllers could finally complete the reboot that was scheduled weekly to allow for consistent performance. Nothing indicates that this was anything other than that function running at the wrong time. The message and the timing all point to the process that should take place at 3am on Sunday Morning.

Remediation:

This clearly was a bug in the Citrix Version. The Delivery Controllers were upgraded to the latest version to help prevent this from happening in the future. We will continue to streamline the process so that these can happen at a regular pace to prevent these types of outages from occurring.

Dear SureDesk Users,

At 11:15 am ET on June 13, 2017 we identified an overload on our File Server.  We immediately messaged users requesting an emergency log off from the server at 11:30 am ET.  All users were fully logged off by 11:45 am ET.  We then performed an emergency reboot of the File Server to resolve the issue.  When rebooting the server a small but critical service on the system that handles user profile data, such as personal documents and settings like shortcuts, mapped drives and internet settings, etc., did not come back up upon server reboot.  We immediately escalated to our Amazon Engineer team to resolve the issue.  

Our engineers restored the server and all services were restored at 1:45 pm ET with no data loss.  Additionally we learned how to recover 4x faster from such an outage in the future.  

We strive for better than 99.9% uptime and this falls within that SLA.  This was the first all user SureDesk outage we have experienced.  

In a stranger than fiction irony in the middle of this exerpience we were doing emgency resuce of the large majority of users from another cloud service provider that suffered a catastrophic server loss in their data center.  All of this reinforced our greatest strengths in our focus on building the most resilient cloud IT network possible - but also overstretched our HelpDesk and our team was not able to message users as quickly as we would like during this outage.  We have already put updates in place to improve this messaging and updates to customers as well.  

If you have any questions, please email solutions@suretech.com or call 1-800-882-8701 x1.

Dear SureDesk Users,

At 1:45 pm ET this morning we identified a processing issue on the server causing perfomance issues which required a server reboot.  Adobe changed their autoupdate process for Adobe Reader, which required a server reboot to affect update and this may be the root cuase of the processing issue on the server.

Our engineers restarted server and performance issues were all resolved by 2:30 pm ET.  During scheduled server maintenance this weekend we will be researching further to determine the root cause of the hung services and how to handle the new update processes by Adobe.

We strive for better than 99.9% uptime and are glad to report that over past 6 month we continue to exceed this service estimate.

If you have any questions, please email solutions@suretech.com or call 1-800-882-8701 x1.

Thank You,

- Your SureTech Solutions Team

Dear SureMail Users,

January 12, 2016 starting at approximately 3:25pm we identified an issue on our SureMail Exchange servers that is causing intermittent disconnections from Outlook.   Email access from smart phones and webmail is performing as usual. 

While the issue is being addressed, we recommend you use https://webmail.suretech.com.

A server fix over the evening allowed most users to launch Outlook successfully as of first thing morning of January 13. 

A small handful of users required some direct assistance to get their local computer to properly accept SureMail password ...

Dear SureDesk Users,

At 7:45 am ET this morning we identified an issue affected QuickBooks usage.  We investigated the issue and resolved the application processing issue by 11:15 am ET.  We determined the underlying cause was due to an internal Quickbooks applciation service that was in a hung state.  We will be improving our monitoring system alrets to incldue monitoring on internal application services to ensure we can more quickly identify hung application services and resolve the issue more expediently.

We strive for better than 99.9% uptime and are glad to report that over past 6 month we continue to exceed this service estimate.

If you have any questions, please email solutions@suretech.com or call 1-800-882-8701 x1.

Thank You,

- Your SureTech Solutions Team

Dear SureDesk Users,

At 8:45 am EST this morning we identified an issue interfering with SureDesk usage for some users. A pop up message from our systems reported a Server Certificate out of date. Connection dropped error.

Our engineers researched and were able to identify an issue with one of the certificates and resolved the issue by 9:30. The resolution took approximately 45 minutes to take effect across all servers. At 10:22 am EST we were able to confirm that the update had resolved the issues and usage had been restored for all effected users. We are putting additional monitoring systems in place to avoid any downtime from a certificate issue again.

We strive for better than 99.9% uptime and are glad to report that over past 6 month we continue to exceed this service estimate.

If you have any questions, please email solutions@suretech.com or call 1-800-882-8701 x1.

Thank You,

- Your SureTech Solutions Team

Dear SureMail Users,

On February 10th, 2015 we had connectivity slowness from 2 pm EST to 3:30 pm EST.  

The slowness on the system was caused by network hacking attempts which slowed down the server farm.  Security is our utmost priority and we are pleased we were able to mitigate a network attack thanks to our protection systems and the rapid intervention of our network and security specialists. We also implemented countermeasures to prevent similar signature attacks in the future. We apologize for any inconvenience caused by this incident. 

Please note that although we strive and hope for 100% uptime, with our current infrastructure we realistically expect, and continue to deliver 99.9+% uptime and this outage continues to support our expected level of service. No mail was lost and users should have experienced at most a short delay in sending or receiving mail.

If you have any questions, please email solutions@suretech.com or call 1-800-882-8701 x1.

Thank You,

- Your SureTech Solutions Team

Dear SureMail Users,

On November 5th, 2014 latencies were observed on eight Exchange 2013 servers around 8:00 am EDT.

These latencies reduced the speed of the mail flow and affected the Outlook Web Application (OWA).

We diagnosed the issue by working with our equipment vendor and Microsoft. A maintenance operation started at 7 pm EDT and lasted until 11 pm EDT. This operation involved adding an SSD to each server to handle the Transport layer database traffic. The operation successfully resolved the latencies.

This is a temporary measure until the root cause is dealt with, at which point the proper course of action will be taken. We are confident that this temporary measure will fully eliminate re-occurrences until the final fix is applied.

Please note that although we strive and hope for 100% uptime, with our current infrastructure we realistically expect, and continue to deliver 99.9+% uptime and this outage continues to support our expected level of service. No mail was lost and users should have experienced at most a short delay in sending or receiving mail.

If you have any questions, please email solutions@suretech.com or call 1-800-882-8701 x1.

Thank You,

- Your SureTech Solutions Team

Dear SureMail Users,

On October 30th, 2014 we partially lost connectivity to our site 2 datacenter at 3:39 pm EDT. Four minutes later, at 3:43 pm EDT, connectivity was completely lost. Network connectivity went back to normal at 3:52 pm EDT but services took longer to recover.

The network recovered itself after thirteen minutes but our sysadmin team had to bring all the servers and databases back online manually which took longer. By 4:25 pm EDT, 98% of the services were back online and by 5:04 pm EDT, everything was back online.

After an investigation, we found that the loss of network connectivity was due to a crash in both our primary and backup core switches. The crash occurred within a four minute interval and unfortunately, those switches take more than four minutes to complete their boot up sequence. The Exchange servers did an automatic shutdown to prevent any further issues.

Please note that although we strive and hope for 100% uptime, with our current infrastructure we realistically expect, and continue to deliver 99.9+% uptime and this outage continues to support our expected level of service. No mail was lost and users should have experienced at most a short delay in sending or receiving mail.

If you have any questions, please email solutions@suretech.com or call 1-800-882-8701 x1.

Thank You,

- Your SureTech Solutions Team

Dear SureDesk Users,

This morning at 1:32am the web server which hosts the portal for our citrix login access to the servers had an overload on the CPU. This caused a failure to launch the so4.suretech.com portal for citrix and rdc services. Additionally we had a systems alert failure of this issue on the server at the time it occurred. What should have been a 10 minute outage was not fully resolved until 9 am due to the monitoring systems not sending proper alerts when the outage first occurred at 1:32 am EST.  We have resolved the cause of the overload issue and our alerting software to ensure prompt resolution on any such issues in future.

If you have any questions, please email solutions@suretech.com or call 1-800-882-8701 x1.

Thank You,

- Your SureTech Solutions Team

Dear SureDesk Users,

This morning at 9:33 am we identified an issue for some users logging in to the SureDesk environment due to an issue with Citrix license deployment.  We rebooted the servers at 10:15 am to resolve the issue and confirmed all logins were renabled by 11 am.  We are determining full fix on the root cause of teh Citrix license issue and will post additional details to this page when the root cause if resolved. 

If you have any questions, please email solutions@suretech.com or call 1-800-882-8701 x1.

Thank You,

- Your SureTech Solutions Team

Dear SureMail Users,

On May 28, 2014 we experienced downtime of our MS Exchange 2010 servers starting at approximately 10:50 am.

The issue started at around 10:50 am EST and the issue was resolved by 11:30 am EST.

There was a problem with a piece of the firewall equipment which affected the end user.  We have put changes in place to avoid this issue again in the future.   

Please note that although we strive and hope for 100% uptime, with our current infrastructure we realistically expect, and continue to deliver 99.9+% uptime and this outage continues to support our expected level of service. No mail was lost and users should have experienced at most a short delay in sending or receiving mail.

If you have any questions, please email solutions@suretech.com or call 1-800-882-8701 x1.

Thank You,

- Your SureTech Solutions Team

Dear SureMail Users,

On April 7, 2014 we experienced outbound and inbound delay for some users on our MS Exchange 2010 servers starting at approximately 1:30 pm.

The issue started at around 1:30 pm EST and we identified the issue by 2:45 pm EST.

There was an issue with our SMTP relay servers that was causing the issue.  We have put changes in place to avoid this issue again in the future.   

Please note that although we strive and hope for 100% uptime, with our current infrastructure we realistically expect, and continue to deliver 99.9+% uptime and this outage continues to support our expected level of service. No mail was lost and users should have experienced at most a short delay in sending or receiving mail.

If you have any questions, please email solutions@suretech.com or call 1-800-882-8701 x1.

Thank You,

- Your SureTech Solutions Team

From March 1 to March 5 some of our Hosted Exchange 2013 users experienced difficulty in accessing their mailboxes through MAPI (Outlook) or ActiveSync protocol.  The issue was caused by a limitation of Exchange 2013 Enterprise software for RPC protocol connections on Exchange Mailbox Servers.  Our monitoring logs showed that between 10:30 am and 4:00 PM EST up to 10% of connections to some our Mailbox servers were dropped and users had difficulty reconnecting, due to a problem with the RPC service on the Mailbox servers. Our Exchange engineers coordinated directly with MS Engineers to finally diagnose the root cause of the issue and fix for the servers. 

The fix was successfully applied on the night of March 5.  This corrected the improper Exchange behavior ensuring that the RPC connections would no longer be dropped or blocked.  There continued to be isolated issues due to the emergency work-arounds we had to put in place during this troubleshooting on March 6 and full fixes for all issues were resolved as of that evening. 

This is the longest troubleshooting of a single issue we have experienced in almost 10 years working with these hosted Exchange services.  We deployed all available resources to diagnose and work with Microsoft to implement the repair and understand the headaches and expense the connectivity issues cost.  OWA access was available during the connection issues to local Outlook clients. 

We are glad to report this full resolution and our continued commitment to ensure improvements to our architecture so we can address issues even better for the future ...

Dear SureMail Users, On February 4, 2014 we experienced MS Outlook connectivity issues for some users on our SureMail system.

The issue started at about 12:30 PM EST and services were fully restored for all users by 1:40 PM EST.

The issue came from one of the SureMail Front end servers, the service responsible for establishing and managing the Outlook connections, known as MSExchangeRPC, entered a state where it was not able to process the connections properly. The system monitoring the service tried to restart it and failed.

We had to manually and forcibly end the service to make sure it was restarted properly. We are putting improvement in place to ensure services restart automatically. 

 If you have any questions, please email solutions@suretech.com or call 1-800-882-8701 x1.

Thank You,

- Your SureTech Solutions Team

Dear SureMail Users, On November 13, 2013 we experienced intermittent connectivity issues for some users on our SureMail system.

The issue started at about 12:00 PM EST and services were fully restored for all users by 12:10 PM EST.

There was hardware failure on one of our Exchange servers which resulted in connectivity loss for some users and has since been fully resolved. We apologize for any inconvenience that this may have caused.

Please note that although we strive and hope for 100% uptime, with our current infrastructure we realistically expect, and continue to deliver 99.9+% uptime and this outage continues to support our expected level of service.
No mail was lost and users should have experienced at most a short delay in sending or receiving mail.

 If you have any questions, please email solutions@suretech.com or call 1-800-882-8701 x1.

Thank You,

- Your SureTech Solutions Team

Dear SureMail Users,

On August 5, 2013 we experienced intermittent connectivity issues for some users on our SureMail system.

The issue started at 9:45 AM EST and became more consistent at 10:02 AM EST. Services were fully restored for all users by 11:10 AM EST.

There was an issue with one of our application firewalls which resulted in Exchange connectivity loss for some users. We are investigating the issue to prevent it from causing downtime in the future. We apologize for any inconvenience that this may have caused.

Please note that although we strive and hope for 100% uptime, with our current infrastructure we realistically expect, and continue to deliver 99.9+% uptime and this outage continues to support our expected level of service. No mail was lost and users should have experienced at most a short delay in sending or receiving mail.

If you have any questions, please email solutions@suretech.com or call 1-800-882-8701 x1.

Thank You,

- Your SureTech Solutions Team

Dear SureDesk Users,

On April 22, 2013 we experienced a service outage for some users on our SureDesk environment.

At approximately 8:15 AM EST we began experiencing log-in issues for some users due to a Citrix networking issue with the Netscaler device, which enables log-ins to the hosted desktop and all published applications. The settings for the Netscaler device were resolved and tested to confirm that issues were fully resolved by 10:15 AM EST. We are continuing an investigation regarding this matter to prevent it from causing downtime in the future. We apologize for any inconvenience that this may have caused.

Please note that although we strive and hope for 100% uptime, with our current infrastructure we realistically expect, and continue to deliver 99.9+% uptime and this outage continues to support our expected level of service.

If you have any questions, please email solutions@suretech.com or call 1-800-882-8701 x1.

Thank You,

- Your SureTech Solutions Team

Wouldnt it be great if the glory days lasted forever? But reality is trophies get dusty, records are broken, and what it took to be the best ten years ago, just isnt enough for todays standards. Things get better, faster. And eventually, its time to move from good enough to something much better.

Windows XP had an amazing run and millions of PC users are grateful for it. But its time to move on. Two reasons: 1) Extended support for Windows XP is running out in less than 1000 days, and 2) theres an OS out there thats much better than Windows XP.

On April 8, 2014, security patches and ...

As amazing as software power and features have become, we are sometimes amazed in the wrong way.

We recently spent more than 20 hours troubleshooting OneNote on our SureDesk in an attempt to address crashes and notebook corruptions. We used several different repositories for the OneNote files including, SkyDrive free, SkyDrive business, our own SharePoint, and saving files locally; and this is what I have concluded;

• OneNote is buggy and prone to corrupting notebooks that have images in them.
• SkyDrive has terrible outbound throughout and is both miserably slow when loading OneNote and also causes OneNote.exe to crash. The only way I found to use SkyDrive reliably was to use OneNote online on SkyDrive.

In the end (hit the cymbals) we gave up and moved our notes to Evernote for anywhere access capability and no crashes.

Dear SureDesk Users,

On February 25, 2013 we experienced a service outage for some users on our SureDesk environment.

At approximately 12:45 PM EST we began experiencing a service slowness for some users on our SureDesk environment. Issues were found with the network hubs in Houston, Texas where the was ISP causing some bandwidth connection issues into our data center for some users. The data center went to a failover ISP to resolve the immediate issue. Issues were fully resolved by 1:10 PM EST. We are continuing an investigation regarding this matter to prevent it from causing downtime in the future. We apologize for any inconvenience that this may have caused.

Please note that although we strive and hope for 100% uptime, with our current infrastructure we realistically expect, and continue to deliver 99.9+% uptime and this outage continues to support our expected level of service.p>

If you have any questions, please email solutions@suretech.com or call 1-800-882-8701 x1.

Thank You,

- Your SureTech Solutions Team

Dear SureMail Users,

On February 14, 2013 we experienced a service outage for some users on our SureMail system.

The issue started at 8:45 AM EST and services were resumed by 8:55 AM EST. There was an issue with one of our application firewalls which resulted in Exchange connectivity loss for some users. We are investigating the issue to prevent it from causing downtime in the future. We apologize for any inconvenience that this may have caused.

Please note that although we strive and hope for 100% uptime, with our current infrastructure we realistically expect, and continue to deliver 99.9+% uptime and this outage continues to support our expected level of service. No mail was lost and users should have experienced at most a short delay in sending or receiving mail.

If you have any questions, please email solutions@suretech.com or call 1-800-882-8701 x1.

Thank You,

- Your SureTech Solutions Team

Dear SureMail Users,

On February 1, 2013 we experienced a service outage for all users on our SureMail system.

The issue started at 7:22 AM EST and services were largely resumed by 7:59 AM EST. There was a residual issue with 1 server that caused some service connectivity problems to remain until 8:50 AM EST.  The issue was caused by a power loss on one of our racks and a failure in the UPS backup to perform as it should have to maintain service during the power outage.  We are working now to improve power redundancy configuration in the rack that encountered the issue to prevent the same issue in the future.

Please note that although we strive and hope for 100% uptime, with our current infrastructure we realistically expect, and continue to deliver 99.9+% uptime and this outage continues to support our expected level of service. No mail was lost and users should have experienced at most a short delay in sending or receiving mail.

Thank You,

- Your SureTech Solutions Team

From October 5th through October 6th we had a confluence of unrelated issues causing downtime for some users on our SureDesk environment.

We had a File Server Corruption which caused File Access issues for some users. In addition, several of our streaming Apps including LaCerte, QuickBooks and PeachTree experienced user delays. Finally, the Citrix Single Sign On plugin was not consistent and sometimes required users to re-enter their SureDesk login upon launching their SureDesk in order to get all streaming apps.

We addressed many of these issues this weekend by doing maintenance on the File Server, adding a redundant QuickBooks server for better performance and failover, reinstalling PeachTree and LaCerte, and testing the Single Sign On plugin for work arounds to this known bug in the latest Citrix release.

We continue to make regular updates and improvements to our infrastructure. This past week did not meet our expected SureDesk uptime and it is our commitment to our customers to continue to improve our systems to meet our 99.9% effectiveness and reliability.

If you have any questions, please email solutions@suretech.com or call 1-800-882-8701 x1.

Thank You,

- Your SureTech Solutions Team

On October 4th, 2012 we experienced slowness for all users on our SureMail system.

The issue started at 11:12 AM EST and subsided at approximately 5:16 PM EDT. We are continuing to troubleshoot and have opened an incident with Microsoft to determine the root cause of the issue so that we can implement corrective action and prevent the same issue in the future.

This is the kind of failure, where there continue to be unknown bugaboos, which is the hardest for us and our customers. We look forward to a world where tech is rid of bugaboos.

If you have any questions, please email solutions@suretech.com or call 1-800-882-8701 x1.

Thank You,

- Your SureTech Solutions Team

Some users may be experiencing periodic slowness in their email connection this morning.

We wanted to let all users know that we are aware of the issue and are in the process of troubleshooting and will be determining a full fix asap. We will have a status update on the resolution within the next 2 hours.

If you have any questions do not hesitate to contact us at 1-800-882-8701 x1 or solutions@suretech.com.

-Your SureTech™ Solutions Team

We are happy to report the SureMail slowness issues some users experienced earlier today has been resolved.

The problem was caused by an update to 1 of our 5 anti-virus engines triggering a condition on 2 of our Exchange 2010 mailbox servers that resulted in a service outage for mailboxes located on those servers. The condition was first reported at 10:45 AM EST and due to the complex nature of the issue, it took until 11:23 AM EST to resume normal mailbox access on one of the servers and until 11:38 AM EST to normal mailbox access on the second server. There may have been some lag time for some users to feel the full effects of the fix.

For the future, we have fixed the way these updates are triggered. Furthermore, we are adding several system monitors so that we can detect and resolve this type of issue right away should it come up again in the future.

If you are still experiencing any delays or issues with your email, please contact our HelpDesk support so we can help you right away ...]]> Corey Meyers SureTech.com Thu, 20 Sep 2012 17:28:05 -0400 http://w99.suretech.com/17622/Outage_Details_for_MS_Exchange_September_20_2012 http://w99.suretech.com/17567/Go_Daddy_Attack_Affects_Login_to_SureDesk

We are taking corrective actions to develop a monitor that can detect the problem proactively in the future.

Should you have any questions about the above, please let us know at solutions@suretech.com

- Your SureTech.com Solutions Team

We work hard everyday planning and managing risk for our services not to fail.  But they do sometimes, and when they do it is emotionally and physically painful.  It hurts our pride, and we know it hurts our clients and perhaps worse, costs our clients money.

At the same time we have worked hard in our business, with our clients and our staff to be realistic and to pay attention to the math.

Unfortunately machines break, and more importantly, complex machines with multiple types of resources, tools and programs fail with certainty.

Part of our work is predicting that certainty despite all our best efforts to avoid fails.  Surprisingly, our historical data and observance of industry best practices leads to fairly reliable prediction of failure rates, despite our very best efforts to avoid failures all together.

For our current $50-$100 monthly SureDesk price we have a target of 99.9% reliability and trailing 12 month uptime is 99.75% which is about 8 hours more annual downtime than we target, but within our expectations given the size and age of our SureDesk 3.0 service.  Our SureMail service had similar uptime the first years of our service and is currently reliably fully functional over 99.9% of the time over the last 18 months.

Our target uptime is a function of the amount of redundancy built into the system, as well as our expertise managing the system.  We currently have a great deal of redundancy and failover and the testing necessary to go ...

We have found that most of our service outages are caused by unexpected behavior in our business continuity systems and architecture which we continually adapt and improve.

Please note that although we strive and hope for 100% uptime, with our current infrastructure we realistically expect, and continue to deliver 99.9+% uptime and this outage continues to support our expected level of service.  No mail was lost and users should have experienced at most a short delay in sending or recieving mail.

Should you have any questions about the above, please let us know at solutions@suretech.com

 - Your SureTech.com Solutions Team

Monday morning (5/9) some SureDesk users were experiencing sluggishness and intermitted connection instability which was determined to be caused by the new switch.

Configuration troubleshooting and failover systems were not responsive to a fix and the entire internet connection at the Data Center was reset at noon to allow the network problem to be circumvented.  The network was reactivated at 12:40 and normal SureDesk service was restored at 12:48PM.

We continue to provide scheduled routine maintenance to our Data Center during non-peak and weekend hours to ensure network availability and stability.  SureTech.com will always notify clients ahead of time if particular maintenance will cause a server interruption.  We did not anticipate a service interruption while performing the above maintenance.

Should you have any questions about the above, please let us know at solutions@suretech.com

- Your SureTech.com Solutions Team

As part of our continued efforts to provide a secure SureMail environment we updated the configuration of a firewall on the Exchange Server today at approximately 4:20 AM EDT.  This update went smoothly and our monitoring indicated no issues with the new configuration. 

However, some SureMail clients did begin to experience connectivity issues at that time; this was due to a load-balancing problem caused by our updates.  This issue was resolved at 8:20 AM EDT; no mail was lost only connectivity to local clients was affected.

We are currently updating our monitoring to detect this type of load-balancing issue in the future to prevent further connection issues of this kind.

Should you have any questions about the above, please let us know at solutions@suretech.com

 - Your SureTech.com Solutions Team

Troubleshooting Problems with Microsoft Outlook Add-Ins

Third-party add-ins (such as desktop antivirus scanners) cause a disproportionately high number of connection-related issues in Outlook.

The easiest way to test this is to start Outlook in safe mode, which disables all add-ins.  To do this, shut Outlook down completely, by choosing File Exit, and then hit Ctrl-Alt-Delete to enter the Task Manger.  From there, click the Processes tab, and terminate any processes called OUTLOOK.EXE.  Next, go to Start Run, and type in outlook.exe /safe. (Note that there MUST be a space character between outlook.exe and /safe.)

If running in Safe Mode fixes whatever connection issue you were having, the next step is to restart Outlook normally and start disabling or removing Add-Ins.  There are two areas youll need to check COM+ Addins, and Exchange Client Extensions.  These are accessed differently on Outlook 2007 and Outlook 2003.

For Outlook 2010:  Go to File Options Add-ins.  Click the Go button thats located towards the bottom of the window.

For Outlook 2007:  From within Outlook, navigate Tools Trust Center Add-ins.  Towards the bottom, choose COM Add-ins and click Go.

For Outlook 2003:  Within Outlook, choose Tools Options, and then click the Other tab, and then the Advanced Options button.  Towards the bottom of the screen youll see a button labeled COM Add-Ins.  Click that.

Youll get to a window with a number of Add-ins listed:

Friday afternoon, June 17th, our server administrators noticed that the hard disk on one of our web servers was showing signs of potential failure.

We experienced a new failure mode today with one of our application firewalls. It started returning errors to some customer requests from the Internet at approximately 11:46 AM EST. The issue, while seriously affecting some customer organizations, was not detected via our multiple monitoring systems.  However based on some issues we were able to see, we re-started the affected application firewall at 12:09 PM EST. The resulting re-convergence of load balancing that occurred affected the other application firewall starting at approximately 12:12 PM EST and ending by 12:19 PM EST. All services returned to normal production availability via the originally affected application firewall by 12:22 PM EST.  The aggregate time during which any customer organizations were affected by the issue was 36 minutes.

We are taking corrective actions to detect the memory fragmentation issue that caused inbound requests to fail on the affected application firewall. We will update our monitoring systems to alert us of this issue prior to inbound requests being rejected, so that we can remediate the issue without customer organizations being affected.

Should you have any questions about the above, please let ...]]> Alberto Molina SureTech.com Wed, 16 Feb 2011 14:14:10 -0400 http://w99.suretech.com/15552/Outage_Details_for_MS_Exchange_Dec_16_2010 http://w99.suretech.com/13552 SureMail™ MS Exchange Outage

Your MS Exchange is fully recovered after Tuesday's service outage.  All data has been fully recovered to your mailbox with no loss of data.

The issue we experienced was precipitated by a problem with an HP Storage Area Network that caused an enclosure with 12-drives to fail.  Our work to recover from this was followed by an additional drive failure during the rebuild of the degraded array that hosted your mailbox, causing a catastrophic failure in the array.  

Please note we regularly deal with upgrades, maintenance and occasional hardware failures transparently and without affecting your service.  In this case, however, all data and data redundancy in the production environment was lost causing us to ...]]> Alberto Molina SureTech.com Wed, 03 Feb 2010 13:56:30 -0400 http://w99.suretech.com/13552 http://w99.suretech.com/13347 We strive to ensure that all our products are reliable and consistent.  Whenever services are interrupted we work hard to get to the bottom of the cause and solutions so that such an event does not happen again.

On Monday, December 28 we experienced an outage for selected clients from 8:58 a.m. to 9:57 a.m. EST. 

The underlying cause:
An error in the Storage Area Network (SAN) supporting mailboxes hosted on the MAIL34 server removed client access to the mailboxes.  We troubleshooted the issue and were able to bring the SAN and server back online within an hour.

Steps taken to prevent reoccurence:

We have implemented additional monitoring of the SAN in order to be informed quickly of this specific condition, so that if this issue ever occurs again, the downtime associated will be much shorter.

We are researching this issue further in an effort to eliminiate the possibility of it occuring again. 

Network Stability:

Overall, the entire SureMail™ environment has enjoyed a 99.902% availabilty rate over the past 365 days, along with very few scheduled maintentance periods.  The ...]]> Alberto Molina SureTech.com Tue, 29 Dec 2009 17:13:44 -0400 http://w99.suretech.com/13347 http://w99.suretech.com/12794 We strive to ensure that all our products are reliable and consistent.  Whenever services are interrupted we work hard to get to the bottom of the cause and solutions so that such an event does not happen again.

On Tuesday, July 22 we experienced an outage for selected clients from approximately 5 a.m. to 11 a.m. EST. 

The underlying cause:
An error in the behavior of clustering services led to the offlining of a number of mailbox stores which prevented access to those mailboxes. The same event also introduced inconsistencies into the log files that are generated for these mailbox stores which made bringing them back online a lengthy process with some element of risk. Once we had taken steps to ensure that incoming mail would continue to be accepted by our incoming mail servers we made copies of all affected mailbox stores to ensure that existing data was secure before beginning the process of rebuilding the mailbox stores. The rebuild process is resource intensive and to minimise the downtime for our customers we allocated additional hardware resources to the recovery process. Recovery ...]]> Alberto Molina SureTech.com Sun, 26 Jul 2009 21:27:38 -0400 http://w99.suretech.com/12794 http://w99.suretech.com/12743  

 

 

 

At 1:49 AM EDT on 6/30/09, a brief power interruption in our Data Center appears to have severely damaged one of the four UPS's in one of our racks. (This UPS had not exhibited any symptoms of issues going into the power interruption.)  The damaged UPS resulted in half of the rack's AC power supply being removed.The infrastructure in that rack was designed to continue to function in this type of partial power outage, but several limitations in this design were exposed yesterday,  resulting in the queuing of all inbound email ...]]> Alberto Molina SureTech.com Wed, 01 Jul 2009 10:24:43 -0400 http://w99.suretech.com/12743 http://w99.suretech.com/12712 On June 22, 2009 we experienced a serious outage on our SureDesk™ systems.  While attempting a minor stability upgrade, our systems admins encountered an unfortunate irreversible bug that crashed the connection service to our SureDesk™ Gold environment

As it happens we also had a parallel upgrade standing by for release this weekend that we were able to move up to be in effect today and include when we restored service.

Service was down from 7:30am to 3:07pm and we sincerely regret the inconvenience to all affected SureDesk™ Gold users.   Going forward we have adjusted our upgrade policy for bugs to take less risks while system upgrades are also being rolled out. Please note SureDesk™ Platinum users were not affected.  Our Gold services don’t have a fully redundant failover standby which contributes to the difficulties in restoring service we saw today.

Also please note in addition to policy changes, we are streamlining work arounds if this were to happen again (which we do NOT expect) including old-school “Terminal Services” access and streamlined local synchronization of SureFiles™.   Feel free to contact us for more information.

On the ...]]> Alberto Molina SureTech.com Tue, 23 Jun 2009 16:21:59 -0400 http://w99.suretech.com/12712 http://w99.suretech.com/14358 We strive to ensure that all our products are reliable and consistent.  Whenever services are interrupted we work hard to get to the bottom of the cause and solutions so that such an event does not happen again.

On Monday, August 2 we experienced an outage for selected clients from 3:27 p.m. to 5:20 p.m. EST. 

The underlying cause:
The partial outage today was caused by a problem with one of the application firewalls.  It failed in such a way as to 'lock' those sessions that had been using it, and it required an on-site intervention to correct the issue. 

Steps taken to prevent reoccurence:

We have taken action to prevent this failure mode from happening again, and also to enable remotely correcting this issue so that if this issue ever occurs again, the downtime associated will be much shorter.

Network Stability:

Overall, the entire SureMail™ environment has enjoyed a 99.902% availabilty rate over the past 365 days, along with very few scheduled maintentance periods.  The MAIL34 mailbox server has experienced an availability rate of 99.906% since it was ...]]> Alberto Molina SureTech.com Tue, 03 Aug 2010 10:31:34 -0400 http://w99.suretech.com/14358 http://w99.suretech.com/7737/SurePhones_Service_Outage_4_05_pm_Sept_23_2008 4:05 pm

The Virtual PBX we use, RingCentral.com completely went down nation wide and all Phone service was completely interrupted.

================= 

4:15 pm

The delicious annectdote in all this is we had just five minutes before the outage fixed a number of configuration and programming issues on the local phones for one of our clients.

 

Virtual PBX used to not work at all and per our experiments over the summer it seems to have come of age.  Getting the correct firmware and configurations for the phone with the RingCentral service, however was a 3 day troubleshooting project, which we were proud to have finished and to report to the client with our success, at 4pm - and then the whole system went kaput!  Sweet irony, right?

 

Calls to Ring Central engineering don't work correctly - we can hear them but they can't hear us.  Some of the same problems we experienced in setting up configurations on our local network.

 

Eventually we get through and ---- they tell us service is down.  Estimate is One Hour Minimum for repair, so I advise our clients it's out for the rest of the day.

We expend so much effort finding the right solutions and working to integrate them into Solutions for clients, the worst thing to then have to tell a client the system just doesn't work and there's nothing we can do.

...]]> Alberto Molina SureTech.com Tue, 23 Sep 2008 16:54:50 -0400 http://w99.suretech.com/7737/SurePhones_Service_Outage_4_05_pm_Sept_23_2008 http://w99.suretech.com/14817 We strive to ensure that all our products are reliable and consistent.  Whenever services are interrupted we work hard to get to the bottom of the cause and solutions so that such an event does not happen again.

On Friday, Nov 12 we experienced an issue with our SureFiles Active Directory system for selected clients and became aware that one of our shared SureFiles servers was non responsive at 7 a.m. EST and clients were unable to access their Files.

The underlying cause:

Our systems engineers determined that one of our shared SureFiles Active Dirtory file servers was non responsive due to a Operating System virus: Win32.Parite.A/B/C that was revealed by our system updates from the previous evening and and in the process damaged the server Operating System.  

All traces of the Virus were removed, we rebuilt the server to ensure stability and main services were restored by 4pm EST and all services were restored by 7pm. 

Steps taken to prevent reoccurance:

We have implemented additional defences of the Active Directory structure to prevent any down time of our servers and also are currently putting in place added fail-over fault tolerance so that we can guarantee continued uptime.

This added fault tolerance will be released in a major new upgrade to both SureFiles and SureDesk systems in December and will ensure increased guaranteed uptime for our SureFiles and SureDesk services. 

Should you have any questions about the above, please email us at solutions@suretech.com or call 609-688-1111 ext 1.

Best regards,

- The Technical Support Team at SureTech.com

]]> Dana Molina SureTech.com Mon, 15 Nov 2010 00:10:49 -0400 http://w99.suretech.com/14817 http://w99.suretech.com/12331/Avoiding_Great_Service_but_Bad_Value 13 Hours and $1,400.00 To upgrade my Hard Drive?!?

 

Weve always said that Managed Services for IT is usually a flawed business model.  Pretty much the better job you do the less you make.  Kinda like lawyers, I guess, except at least we talk about it!

 

13 hours to reformat and upgrade a client laptop is the perfect example of providing GREAT service at a TERRIBLE value.  I worked over a long day and night replacing the hard drive of this particular laptop.  Theres no question the client got a lot of benefit.   3.5x more storage space, 3x improvement in speed.  And almost no worries he handed me the laptop in the morning and came back for it the next day fully upgraded.  I couldnt save him from a couple headaches, though he had to hunt down passwords that needed to be reached in his browser .... and he had to see my 13 hour bill!  Which points to the value problem the work took 13 hours, but no one wants to pay labor of 13 hours for 3.5x space and  3x speed.  Think of it this way the 320GB drive cost less than $100 and the 3x speed was just getting the computer back to the speed it was the day he bought it.  Who wants to pay almost the cost of the computer to simply set (or in this case reset) it up?

 

Ill tell you: nobody.  But whats the alternative?  It takes hours and hours to set these things up so we as a ...

]]> Alberto Molina SureTech.com Thu, 19 Mar 2009 08:56:22 -0400 http://w99.suretech.com/12331/Avoiding_Great_Service_but_Bad_Value http://w99.suretech.com/8135/Xobni_for_Outlook_is_awesome_except_it_doesn_t_work Xobni which is inbox spelled backwards is an absolutely terrific plug in for Microsoft Outlook except for the small fact that it doesn't work... - view comments

 

]]> Alberto Molina SureTech.com Fri, 03 Oct 2008 07:25:23 -0400 http://w99.suretech.com/8135/Xobni_for_Outlook_is_awesome_except_it_doesn_t_work http://w99.suretech.com/11003/Apple_Scores_on_Service American based, for american customers - and email.

That's pretty much the price of excellent service these days.  If you outsource your service to a place that doesn't care about your customers ...  - view comments

]]> Alberto Molina SureTech.com Tue, 20 Jan 2009 01:22:28 -0400 http://w99.suretech.com/11003/Apple_Scores_on_Service