CPU jumping to 100% from PHP
I run several WordPress sites, and everything's fully up to date, and nothing looks hacked. I don't get a lot of traffic either. I have also scoured my logs for details as to what is hitting the sites at that time (of course, I have no idea which one it might be). The only oddity is beging bombarded by requests from something call BUbiNG (no relation to Bing crawler), which gives a source URL of
But my main questions are: why would that be causing a problem? and has anyone else had this problem? And what is php(random)_ippme? My Google-Fu is not turning up anything.
12 Replies
On a Wordpress site, the cpu would soar to 100% and wireshark showed weird non-web traffic. After some investigation, I found the following results. One of the Wordpress plugins had a vulnerability, a remote attacker used that to upload and execute a php script. This particular php script, once executed, it would delete itself and just remain resident.
Once the server was rebooted, the attackers script would be gone, but he would monitor this and remotely re-upload and re-execute the same script once the server was back online. This method, allowed the attacker to by-pass any file integrity checks, anti-virus scans and anything else that might cause the admin to think that something was wrong.
The attackers php script was basically doing mining for one of the less popular crypto currencies. Once the vulnerability was closed and the server restored from backup, the attacks stopped.
IfThenElse, can you tell me more about this? Or where I can get more info? Wordpress and all plugins for all word press sites are up-to-date, and have always been kept up to date, so that might not be the issue, but I would like to investigate it more.
Any other suggestions would be greatly appreciated! I'm currently at a total loss.
Since its wordpress, I'd start with disabling ALL plugins and removing them from the plugins directory (so they are not accessible at all). Then reboot for the memory resident process to go away, then download wordpress-latest.zip from the wordpress.org site and install it on top, just in case any files have been modified. Also change all passwords. If things improve then its obvious that wordpress has been compromised.
You can try various tricks, like use the "find" command to detect all modified files from the past X number days/weeks, CentOS comes with SELinux which will warn you about AVC errors. Run rkhunter or other popular detection engines. The rest is your typical hardening tricks, like modevasive, modsecurity for apache, disable all exec (shell_exec, exec, etc) and remote access (curl, wget, etc) related function in php.
If you can't manage it yourself, then ask a professional, I believe Linode offers paid service that may help.
But somehow the attackers had modified numerous Wordpress PHP files to have includes to files that actually didn't even exist on the system (I checked, of course). Those modified files I restore to the latest official WP version. There were also lots of randomly named .php files, which I deleted. There was also an admin user installed called wp.service.controller.(some random stuff), which I of course deleted immediately and proceeded to change all legit user passwords.
The only extra plugin the site had was Contact Form 7, and that plugin is not on any of my other sites, which I verfied to be fine. So maybe that was the attack vector? I'm not sure. My uncertainty causes me some consternation. If I knew how they did this for certain, I could protect my sites in the future.
In my experience, it is quite hard to completely clean a hacked site, usually there is always something left behind that you won't catch and eventually the attacker will be back. My suggestion in these cases, is to restore from backup and at the same time delete the offending Contact Form 7 plugin.
Based on your description, the modified files should have given you a clear and advanced warning that something went wrong, it is your responsibility to use file integrity tools like AIDE or Tripwire. In addition, the effects of the hack can be limited with process accounting and SELinux.
If you keep apache logs old enough, then you may get an idea how the attack started, google for Contact Form 7 exploits and see if they apply to your version of the plugin.
I've wiped and re-installed the site (had to keep the DB though, as that has all the content), and I've also installed WordFence, as that appears to be highly recommend for monitoring these sorts of things. I've installed that to all my sites now.
Thanks for your help on putting me on the right path to finding the problem.
Just for future reference, here is a good idea how to monitor Wordpress integrity.
* Install AIDE
yum install aide
.
Configure the /etc/aide.conf file, remove all the default paths (we don't care about root level hacks here). So just add the base Wordpress files, for example:
/home/whatever/public_html/index.php NORMAL /home/whatever/public_html/wp-admin/ NORMAL /home/whatever/public_html/wp-includes/ NORMAL.
Generate the initial database
aide --init.
Set the "new" database as the current one
mv /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz.
Check for modifications with:
aide --check
You may do the check within a cron job, once a day or even several times a day. If anything changes Wordpress core files, you'll be notified of the change. The only downside, is that you will get notified when Wordpress updates itself, then you need to re-init the database, but that is not a big deal. You can even arrange for things to happen with Ansible.
phpoRfrE2_ippme looks like a temporary name for an uploaded file. Someone found a POST exploit and uploads files to a location from which they can then execute code.
Here is a firewall I wrote when this happened to me.