- server load was really high and ram was maxed out triggering the alarm
- sshed into the server and ran htop to verify the load and to see if there was anything standing out
- seemed like it was coming from the webserver side so I checked the logs
- noticed a shitton of traffic in the logs coming through trying to hit admin and database urls, so it seemed like my webserver was under attack
- started trying to narrow it down why I was seeing a spike in traffic
- was not sure what i was looking for in the logs so i changed directions and checked the frontend
- i went through a couple of the sites i was hosting and noticed this come up instead of the site
- ran this image past a friend who was a bit more versed on the security side of things and the let me know this was a reverse shell
- FML i know what that means
- i took a snapshot of the server at that moment beccause I knew i was gonna nuke the server
- i took a clone of the VM and killed the network and started it back up to start digging
- knowing i was reversed shelled, i looked for the typical stuff these fucks do, and BAM SPAM
- noticed a lot of traffic on the vm knocking on the email ports
25 110 - checked the processes and sure enough i saw postfix (an email server) installed
- looked into postfix and found out it was trying to send out emails, those emails were hitting my firewall, the firewall would bounce them back causing the memory issue that triggered my alarm on the monitoring server
- fuck email, that is the first thing i disabled and locked down to prevent this type of shit
- here is one of the many emails it was trying to send out unsccessfully.
- it was sending out spam emails for a Russian Ladies of the Night thingamajig
- ended up tracking down the site to these fucks
- site was: j 0 b g i r 1 2 4 DOT r u (spelled wrong but general idea)
- fuck these people
lookin for the entry point
- started with the reverse shell found how it was running / installed
- reverse shell was being started by a cron job using an older version that somehow managed to get installed on the machine
- tracked the orgin of the job to a wordpress install i was hosting for my cousin
- the sites custom theme that the dev had stopped maintain that had some code in it that allowed them access to install that old cron version with a job that would install the reverse shell
postmortem
- WordPress sucks when you do not maintain it. I should know better
- custom coded themes suck, stick to the ones that have a long support cycle
- keep shit up to date, especially running WordPress
- monitoring and alerts saved my ass. this could have gone unnoticed for a while before i had seen this or thought to check
- current hosting method is not the best idea. 1 VM with multiple WordPress installs behind Apache and Nginx
- 1 website took down all my other websites
- gonna look into containerizing my stack with docker or some other container manager
- should be more secure each website is contained to its own instance separate from the host machine usually
- good thing i had backups too, made it easy to restore the sites to the new temp server
- made sure to go through each site and sanitize it as best as i could. Nuked everything but the DB wp-config.php and the wp-content folders
