Mail Server Outage
Early on May 30, our mail server briefly went down, and our team quickly ascertained that the issue was a storage problem. We were able to get the system running again in about 5 minutes, but new that a proper investigation and fix was necessary once the business day was over for most of our clients. Unfortunately, it was discovered that we had suffered a catastrophic file system error, and our partition tables have been irrecoverably damaged.
After working with our team and some outside vendors, it was decided that recovery was the only option, and this is currently underway. Due to the massive amount of data that needs to be recovered, this process will take a very long time, at least 24 hours or longer. As we have updates, we will post them at https://twitter.com/nerdshosting.
Our recovery is currently underway, and we expect to be able to retrieve greater than 99% of all mail data.
During this recovery no mail services will be available. You will not be able to access webmail, POP, IMAP or SMTP. However, all new, incoming email is being queued by our incoming gateway, and will be delivered just as soon as the recovery is completed and we are back up and running. Any client using an outside mail service (Google Apps, Office 365, etc.) is completely unaffected by this.
We sincerely apologize to our clients for failing you in this situation, and are working as hard and as fast as we can to get all systems back up and running as soon as possible.