How to manage a security breach on your server

Modern servers frequently run a multitude of software applications.

In our case, we use Docker on the server to run multiple services.

One of these services, Gitlab, had a security issue (gitlab workhorse more specifically).

Which led to a trojan infection. (In our case, the binaries were called “wate” in case this matters to someone – although this could have been purely random names as well).

We became aware of this due to our hoster, Hetzner, shutting down our IP (and sending several warning messages). The trojan had been used as part of a DDoS attack on a Chinese IP and was generating suspicious amounts of packets.

We took action, and some useful knowledge emerged due to this situation.

Luckily the server was reachable through a different IP (and by the way – Hetzner will also allow you to whitelist an IP for 2 hours so you can even work with a server with just one IP).

Here are some key tips to diagnose such a security breach

• Do not panic. Secure docker logs first (and possibly create a backup) before shutting the server down – you might need these logs to diagnose the issue.
• check which users were last logged in / whether new accounts have been created
• Use a virus scanner on your server (on the host operating system for Docker). We used clamav to find the trojans.
• Note: if you are running a mail server as well (as we do), the virus scanner will probably pick up on a lot of suspicious mails, too.
• Use a rootkit scanner – there are two we used, rkhunter and chkrootkit
• Use htop to find out which processes are loading the server
• Use a network traffic monitoring utility
• look through the output of ps –aux to see any suspicious processes

My recommendation is that before shutting down the container you will create a dump of ps –aux | grep wate (in our case) to see the running trojan processes, and then after shutting it down again.

This way, you will be able to confirm to yourself that the security breach has been contained, and that the processes did not spread outside the Docker container.

WordPress performance issues

We also experienced some WordPress performance issues during this situation – while the main IP of the server was blocked.

After Hetzner unblocked our IP the performance issues went away. (Our initial suspicion was that the breach had been in a WordPress container)

Actions to avoid the situation in the future

1. We are going to set up automatic container updates. The boyfriend of my sister recommended the Docker “Watchtower” container to me.
2. There is a tool called Lynis which is a security audit tool which we will look into
3. Use public / private key authentication on the server for SSH. Limit specifically which accounts are allowed to log in
4. Use a SaaS auditing service / scanner – we’re planning to run this on a regular base
5. Start creating rolling backups if you’re not doing them already – so you can go back to a past date

There is no shame in a mistake – modern IT is a complex endeavour. As long as you keep learning from the situations, and using different approaches to IT problems (and life in general), there’s growth and movement forward. Life.

Luckily, most situations can be managed and turned around. We hope you do have backups in place, and can recover from your situation.

Best of luck and thank you for reading this.