Nerds On Site Hosting: Summary of July 9 Hosting Incident
Article Contents
On Saturday, July 9, 2022, our Managed WP hosting platform suffered an outage, which lasted until Monday, July 11, 2022.
This outage was caused, in part, by the actions of cyber criminals that had begun to compromise client WordPress credentials for a large number of sites on our platform, estimated at about 1/3 of our client’s sites.
On the afternoon of July 9, our team determined that a full-scale roll-back of our hosting platform was necessary to clean out the intrusion properly, and it was this roll-back that caused the outage. This post summarizes what happened and what our next steps will be to better protect and recover from such an incident in the future.
Incident summary:
Starting in early June, attacks on WordPress were significantly stepped up worldwide, with several commentators on social media pointing the finger at Russia as a source for these probing attacks.
Our team first noticed a successful intrusion of a client site on our platform on Tuesday, July 5, and these intrusions took the form of account compromises. At the time, these intrusions appeared very limited, as they appeared to be isolated to two clients with a collection of sites hosted with us, with shared usernames across all sites. At this point, our team reacted by correcting the affected sites, notifying and working with the affected clients, and checking the sites for any additional intrusions.
By Friday, July 8, it was clear that the attacker had started to focus on clients on our platform, with the account compromises beginning to affect a larger number of clients than the original two clients.
There were common threads and connections between the affected sites, but it appeared clear that the criminal actors had realized they had found a WordPress-heavy target to focus on for a while.
At this time, we also started seeing the criminals deploying payloads on the affected sites. Based on our team’s experience with previous wide-scale attacks, we deployed rapid credential roll-back tools. Still, we quickly realized this was ineffective, as these recovery operations did not clean out the malicious payloads.
Our team went to work cleaning out these malicious payloads as we found them but quickly realized that the attacker, with their combined surface of compromised accounts and existing inserted backdoors, was able to move quicker than our cleaning efforts.
Because of this realization, our team took the platform offline and rolled back the entire platform to a time before the first payloads appeared. Most services were restored by the morning of July 11, with all services restored by the evening of July 11,
Since then, our team has painstakingly reviewed and examined each site on our platform, using automated tools and human interventions, no less than three times.
We noticed first that the cyber criminals attempted to compromise the sites after the rollback.
Still, as a result of some of the measures put in place as part of our recovery, the malicious actors were foiled and eventually frustrated to the point that they ultimately gave up and moved on. At the time of this posting, our team believes that all sites are clean and safe and no longer under attack by the specific group involved in the attack.
How was compromise achieved?
Our team is unclear how the compromise was achieved, though we believe a zero-day or other recently discovered WordPress vulnerability may have played a role.
We could confirm that some of the compromised sites had no plugins, and all compromised sites were fully patched or had been fully patched within three days of the compromise.
We found that all of the compromised accounts were either abandoned administrator accounts (admin accounts the site owners had not cleaned up once those people had left staff) or developer accounts that were re-used across many sites.
To protect some of our client’s privacy and spare anyone undue embarrassment, we will not elaborate more on that side of things.
Next steps:
Our team has implemented the following changes as a result of this compromise:
- Starting July 31, all ‘administrator’ level accounts must have 2FA set up. 2FA has been offered from day one on our Managed WP platform but never enforced. Moving forward, this will be required, and we believe it would have prevented this attack. You can read more about this here: Nerds On Site Hosting Announcement: New MFA Requirement for Managed WP Hosting
- Our team was unhappy with the speed of the roll-back and has identified several areas that could be significantly improved. We are working on this now to ensure that any future restorations or roll-backs will be sped up considerably.
- In conclusion…
In conclusion:
Everyone at Nerds On Site sincerely apologizes to our clients for this incident. We promise to learn from it to prevent such attacks and execute a faster recovery if it ever happens again.
The world of WordPress hosting is a particularly vulnerable field, with nary a week going by that a hosting company suffers an embarrassing outage. WordPress powers 43% of the internet and thus is a desirable target for criminals. Despite this incident, our team loves this challenge, and thanks for your continued trust in us.
