Become a Nerd Own A Region

  Blog main page

Explanation of Brief Outage

Today our team had a small issue with DNS that caused a brief outage for about a dozen of our hosted clients. We want to publish a brief explanation of this unfortunate event, which is now fully resolved.
Earlier today our team detected a determined brute force attack against two of our clients. While these type of attacks are common, these two attacks were special in that they were unrelated to each other and markedly more powerful than most attacks. (It may be interesting to the reader to note that both attacks were against un-patched copies of WordPress, which are particularly tasty targets for attackers.) Thus, we essentially were being attacked by two simultaneous attacks.
As part of the strategy in mitigating these attacks, our team was forced to restart some critical web hosting services, and unwittingly one of these restarts interrupted a DNS synchronization that happened to be underway at that moment.  Unfortunately this interruption caused the DNS records of about 1 dozen of our hosted sites to be lost and deleted from all our DNS servers and in the heat of the work against the malicious attacks our team never noticed the damage to the DNS system.  The loss of these DNS records occurred at approximately 6PM GMT.  Thankfully, safety measures built into our DNS software prevented this issue from spreading and becoming a larger issue beyond the affected dozen sites.
About 3 hours after the loss of these DNS records, our team was notified by a client that their site was non-operational, and upon investigation we discovered the problem and immediately set about repairing it.  Thankfully, we had full backups of the DNS entries and were able to rebuild everything very quickly.  At this time, both malicious attacks are fully mitigated and stopped, and all DNS issues have been resolved.  We estimate the total downtime to the affected dozen clients was approximately 4 hours.
Our team sincerely apologies for the issue today, and thank our clients for their business!