Security Disclosure: Cerb Forums/Wiki (via WordPress vulnerability) - tips for limiting your exposure
As outlined in our privacy policy, we’re committed to transparently disclosing any security breaches, whether verified or potential.
As a preface, this incident did not in any way affect Cerb On-Demand customers. The On-Demand machines only run Cerb and do not expose any other software to the web. This incident was strictly limited to one virtual machine which was isolated to serve public web content related to Cerb (the forums, wiki, legacy blog, and main project website).
On Tuesday, September 25th 2012, around 2:39PM PST, a few public websites on that virtual machine were defaced through an exploit in a two year old (3.1) version of WordPress. The vulnerable site was an archive of the old Cerb4 blog, which hasn’t been in regular use for many years. It should have been upgraded, “staticized”, or deleted.
We saw the monitoring notifications within about 15 minutes of the content changing, and we brought the entire machine offline for forensic analysis.
The affected neighboring sites on that virtual machine had an <IFRAME> prepended to any index.php files, or WordPress themes, to serve arbitrary ad content to website visitors. This is the only change we’ve found in the filesystem analysis on that machine, although the exploit would have allowed the attackers to run arbitrary PHP code as the Apache user. This was possible because several .php files were writeable by the web server when all they needed was read access.
The attack vector was a known vulnerability in WordPress 3.x, and our analysis of the server logs and filesystem contents, combined with the public experience of others who report this same exploit on a daily basis, leads us to conclude that it was most likely automated and had the sole objective of delivering fraudulent ad revenue to the attacker with redirected web traffic.
We have seen nothing in the logs or filesystem to indicate that any database content was viewed or copied. However, it would have been possible for a determined attacker to do so by reading the database connection info from the configuration files of WordPress, vBulletin, and Mediawiki. They could have then crafted custom queries in PHP to read user information from those databases. Again, we have no evidence that this took place, but in a situation where we can’t say with 100% certainty that it didn’t happen, we feel that it’s prudent to react according to the worst case scenario.
If you had an account on our forums, wiki, or blog, your email address may have been disclosed. The passwords were all hashed, and many were also salted, but if your password was fairly simplistic (short, and mostly letters and numbers) then it would be possible to recover using tools like MD5 reverse lookups, brute force dictionary matching, or rainbow tables.
We highly recommend using a unique password per website using tools like 1Password. If you use the same password on multiple sites, especially for your email address, we strongly recommend that you change it now. We’ve taken the blog and forums offline, and the passwords on the wiki have been force randomized. You will need to use the ‘forgot password’ feature before logging in again.
The virtual machine hosted the following sites:
- cerb4.com/blog
An archive of the Cerb4 blog running WordPress 3.1. This is where the exploit occurred. The sensitive information involved would be blog user passwords (hashed) and email addresses. We’ve had our project news on Facebook for several years, so this only affected a few people who had posted comments on Cerb4 blog posts.
Actions taken: We’ve taken the Cerb4 blog archive offline permanently and notified past blog commenters. - wiki.cerb5.com
The community wiki for recent versions of Cerb, running Mediawiki. The wiki allows anonymous updates so there’s very little sensitive information contained in its database. Users who created accounts provided an email address. That email address and the password hash (generally salted) would have been accessible.
Actions taken: We’ve reinstalled Mediawiki from a fresh copy of the files and randomized all user passwords. The wiki is running on an isolated virtual machine. Affected users have been notified. - forums.cerberusweb.com
The forums ran an old version of vBulletin and receive very little attention from us. For those reasons, we’ve brought them offline until we can upgrade them to the latest version and isolate them on a new virtual machine. Ideally we’d handle public support interactions in a different way. The possible disclosure would include email addresses and salted password hashes.
Actions taken: We’ve brought the forums offline for the time being. When restored they will be isolated on a dedicated virtual machine. Affected users have been notified. - cerberusweb.com
Our website is just a community portal plugin in our Cerb installation. As such, there was only a single index.php file on the virtual machine that served as a proxy to our On-Demand instance in another datacenter. The On-Demand network was not affected. This defacement was limited to the injection of the <IFRAME> to the top of that single file.
As further evidence that the attack was blindly automated, the <IFRAME> broke all the images, stylesheets, scripts, and links on our project website because those links are all served through the proxy from elsewhere. Our project website receives the most traffic of any of these affected sites and would have been the most practical target for traffic redirection. The site was likely defaced for about 15 minutes.
This site has no database and stores no potentially sensitive information. However, if you visited the website on Tuesday, Sept 25 2012 between 2-3pm PST then you may have been tagged with a tracking cookie from an external website. This applies to a very, very small number of people (visitors in a 15 min timespan). We have their IPs, but it isn’t feasible to reverse those to identities that we can email. The website would have appeared completely broken during that small window and hopefully it would have dissuaded any visitors from continuing to browse.
Actions taken: The community portal file was re-installed from a fresh copy.
In addition to taking deprecated software applications offline, we’ve also removed the write privileges from the virtual machine web servers on content files where it is not explicitly required. In cases where the web server writes content to directories (like uploaded photos), we’ve disabled the ability for PHP to execute in those paths. We’ve also isolated untrusted third-party applications to independent virtual machines to limit the scope of any future vulnerabilities.
The affected virtual machine is no longer serving public content. We’re continuing to examine its filesystem and logs for any evidence of additional tampering, but there is no evidence that there is more to the attack than described above. The incident was on par with a shared hosting environment where multiple clients can read each other’s files through a shared web server user (e.g. www-data). That’s not an ideal situation, and permission separation is obviously something we should be concerned with even for our own supporting websites like the forums, blog, and wiki. A base_dir restriction, or distinct web server users per website, would have isolated this issue to the old Cerb4 blog and potentially only affected only a couple dozen people. Comparatively, there are about 7,000 accounts on the forums.
In conclusion, we’re very confident that no user data was disclosed. However, had we pretended that this never happened, and then information from our machines was later publicly posted to the web (or shared more nefariously), we’d have potentially jeopardized thousands of accounts that used the same email address or password.
The vast majority of these passwords were salted, which drastically increases the complexity of cracking them. Regardless, please don’t use the same password on multiple websites. Use software like 1Password to generate and store a random password per website. Use a mixture of upper case, lower case, numbers, and symbols. Better yet, do that and write out an entire sentence. Once you authenticate with 1Password you won’t have to type these incomprehensible passwords by hand.
With distinct passwords, you’ll never have to worry about what other online doors that password would unlock. For example, if someone gets into your email address they can probably reset your passwords everywhere else. Where possible, set up two-factor authentication (i.e. “something you know and something you have”). Gmail supports this. In addition to typing your password, you also have to enter a rolling authorization code from an app running on your mobile phone. Even if someone knows your password they can’t log in without access to your mobile, which would be unlikely in an event like this.
This incident appears to have been harmless, but it will further improve security on the periphery of our network. We’re committed to being vigilant custodians of the information you entrust us with.
- Jeff Standen, Chief of R&D
WebGroup Media, LLC.
