Detecting Kernel Rootkits

Sandra E. Ring and Eric Cole

Rootkits come in two basic forms: application (or file) level rootkits, which are based on a series of trojaned programs, and the more complex kernel level rootkits. Kernel level rootkits subvert the actual kernel of an operating system, usually through loadable kernel modules (LKMs). Each kernel rootkit has its own distinct signature, method of detection, and remediation techniques. Examples of application level rootkits include t0rn and lrk5. Examples of kernel level rootkits include slkm, knark, and adore.

Application level rootkits are the more common variant. Most systems administrators are familiar with at least some of the available tools for searching out application level rootkits (see sidebar "Detecting Application Level Rootkits"). Finding application- or file-based rootkits can be as simple as searching for the trojaned binaries. Kernel level rootkits, on the other hand, are more powerful and less detectable than traditional application level rootkits, and good documentation on finding kernel level rootkits is more difficult to find.

Kernel level rootkits compromise the kernel and are capable of subverting any application level program without having to physically trojan it. To comprehend this technique, it is important to first understand the interaction between user space applications and the kernel (see Figure 1).

How Rootkits Work

When the user types "ls", the ls program is loaded into user memory and is executed. User and kernel instruction execution is separated into two distinct "modes". User applications such as "ls" execute in user mode, whereas kernel instructions execute in supervisor mode on the processor. This division prevents user applications from directly accessing kernel instructions. The only way that a user application can access a kernel instruction (other than directly modifying /dev/kmem) is to use the system call table interface. The system call table consists of an indexed array of system calls that contains addresses to their corresponding functions. When a system call is made, the indexed number of the system call is placed in a register and an interrupt is made. This interrupt transfers execution from user mode to supervisor mode and the kernel instructions are executed. A kernel rootkit subverts this by either replacing the pointers in the system call table or by replacing the kernel instructions directly (see Figure 2).

Because of the simplicity and portability, most kernel level rootkits operate by replacing the system call table instruction pointers. This is commonly referred to as "syscall patching". Although this technique easily defeats application integrity checkers like Tripwire and RPM, a simple integrity check of the system call table can typically expose it.

The more complicated method of direct kernel instruction patching requires the developer of the rootkit to be well versed in assembly language. The rootkit must "search" for a particular signature in memory to identify the correct patch location. This method defeats system call table integrity checks, but is far less portable because it depends on a strict set of signatures to implant itself. Kernel patches and operating system upgrades quickly alter the signature and leave the rootkit exposed. The basic method of protection for this type of attack comes from verifying the integrity of the entire running kernel. Because monitoring of this magnitude can be complex to implement and processor intensive to execute, it is not very feasible. Other than prevention techniques, there has been very little work done in this area to identify memory patching kernel modules.

SuckIT is one of the most advanced kernel rootkits. It combines the portability of syscall patching with the stealth of direct memory patching to develop an extremely powerful (and dangerous) kernel rootkit. SuckIT is capable of patching directly into kernel memory through /dev/kmem. This clever method of insertion was first introduced by Silvio Cesare many years ago, and provides a means of loading without relying on loadable kernel module support or access to /boot/System.map. Patch addresses are selected with lseek(), reading is accomplished though read(), and write() is used to install the implant.

SuckIT uses an elegant method to obtain a pointer to the system call table. It executes the sidt instruction [asm("sid %0" : "=m" (idtr));] in conjunction with readkmem and offset calculations to obtain a pointer to the intercept descriptor table. Using a technique first introduced by Spacewalker, it targets the system call table in a different way. Instead of redirecting addresses in the original table, it creates a copy of the table and simply directs the int $0x80 calls to the new copy. Patching is then done to the new table as needed to hide files, processes, and connections. This redirection of the "int $0x80 call" address causes userspace programs to rely on the trojaned copy of the system call table, without touching the original table. Therefore, this bypasses most standard integrity checks of the system call table.

One of the most advanced process hiding techniques was implemented by Dark Angel in Phantasmagoria. Phantasmagoria is an advanced Linux process hider. It operates by completely removing running processes from the task list using the REMOVE_LINKS macro instead of filtering results of kernel calls. This implementation essentially prevents detection of running processes from anyone analyzing the task structure list. It has been implemented in the most recent version of Adore, and we expect that it will be used in other future rootkits as well.

Adore also introduces the notion of "PID 0 hiding" on Linux. By default, processes with an id of 0 are not displayed by any of the systems administration utilities; therefore simply renumbering the PID can be used to easily hide a process. The downside is that the standard "exit()" function does not operate properly with a PID of zero. The attacker must trap all exits made by the hidden process and replace the PID to a valid number prior to exiting. (See the sidebar "Chkrootkit").

Detecting Kernel Level Rootkits

You can detect kernel level rootkits by:

• Their startup mechanism
• How the module implants itself into the kernel
• How processes are hidden

If possible, the best method of detection is to boot from a trusted floppy or CD-ROM that contains a kernel and standard system administration utilities. For Linux, I am fond of the Linux Care Bootable Toolbox CD, but vendor operating disks work just fine. Booting independent of the compromised kernel prevents a kernel level rootkit from hiding, and trusted applications on the boot media provide dependable mechanisms to expose the attacker's files. From this, you can choose to do either manual or automated inspections.

On Linux the detection of most kernel modules is as simple as checking the integrity of the system call table. Each time the Linux kernel is compiled, a file containing the map of kernel symbols and addresses is created (/boot/System.map). This can be checked manually or automatically verified. Listing 1 shows an example that prints the addresses of some of the more commonly attacked system calls.

This generates a list of results in dmesg that can be compared to the /boot/System.map using a simple Perl script (see Listing 2). Keep in mind that the attacker may have modified your /boot/System.map, so use a copy that you trust. The result of running dmesg -c | ./compare.pl demonstrates the differences between a clean and infected system (see Figure 3).

You will notice that the current addresses listed for the infected system above are in a different range of memory. Instead of the address space used at boot up, c01xxxxx, the new addresses are in the d087bxxx address space. This address range corresponds to where the attacker's module is loaded into memory. Similar integrity checks can be implemented on Solaris. Although there is no /boot/System.map for comparison, you will still see a dramatic jump in memory addresses for patched system calls on Solaris as well.

Further forensics to identify the suspect module can be completed by issuing the following commands:

# cat /proc/ksysms | grep d087b
d087b000 __insmod_adore_O/root/research/rootkits/adore/adore.o_M3E80DAAB_V132114 [adore]
d087b060 __insmod_adore_S.text_L5488    [adore]

Kernel Rootkit Process Hiding Detection

You can use a similar technique to search for hidden processes. First, the module in Listing 3 is installed into memory. This will print each process that the kernel maintains on its task queue. Second, a simple "ps" is run in user space and the two results are compared. Any process that shows up in kernel space, but not user space, is likely to be a "hidden" process (see Listing 3).

Note that this technique will not detect processes running using Phantasmagoria-style hiding techniques. At this time, there is not a concrete method to detect them. Additionally, most publicly available rootkits fail to account for the "lsof" utility. Running it periodically can also expose many hidden processes.

Remediation

Remediation involves dealing with rootkits and minimizing their impact to your organization. It occurs after you determine that your system was compromised and you have identified the presences of a rootkit. You have two options, targeted replacement or complete system install. If you opt for targeted replacement, you need to identify:

• How the attacker gained access to your system
• How the attacker starts the rootkit on boot up
• What the attacker uses to communicate with the rootkit (command and control backdoor)
• What the attacker has done to hide processes
• What the attacker has done to hide files
• What the attacker has done to hide network connections

Using the same methods that you used to expose the attacker's utilities, replace any trojans with trusted copies (note that backup does not necessarily mean trusted; you do not know how long the attacker has been on your system). Additionally, remove all startup scripts and files that the attacker has added to your system.

You can assume that the attacker has gathered passwords for all users on the system. All passwords should be immediately changed. Next, you must determine how the attacker compromised your system in the first place (or he'll do it again as soon as you remove all of his tools). Verify that only necessary services are running and ensure that all current patches have been applied.

Resources and References

Adore -- Linux-based Loadable Kernel Module. Features smart PROMISC flag hiding, persistent file and directory hiding, and netstat hiding:

http://wwww.team-teso.net/releases/adore-0.42.tgz

http://www.chrootkit.org

http://www.packetstormsecurity.com/UNIX/penetration/rootkits/ddb-ste.tar.gz

http://www.codito.de/prog/icpm-shell.tar.gz

http://www.packetstormsecurity.com/UNIX/penetration/rootkits/knark-2.4.3.tgz

http://lbt.linuxcare.com

http://www.packetstormsecurity.org/UNIX/penetration/rootkits/lrk5.src.tar.gz

http://www.sans.org/y2k/lion.htm

http://www.redhat.com/swr/i386/nc-1.10-16.i386.html

http://sourceforge.net/projects/nmap

http://darkangel.antifork.org/codes

http://www.rpm.org

http://www.sun.com/blueprints/0501/Fingerprint.pdf

http://www.big.net.au/~silvio

http://www.packetstormsecurity.com/groups/thc/slkm-1.0.tar.gz

Spacewalker:

http://projet7.tuxfamily.org/docs/security/lkm.txt

http://sd.q-art.nl/sk

http://www.packetstormsecurity.org/UNIX/penetration/rootkits/tk.tgz

http://www.tripwire.com

http://www.packetstormsecurity.com/UNIX/penetration/rootkits/udp_backdoor.tar.gz

http://www.packetstormsecurity.org/UNIX/penetration/rootkits/ulogin.c

Sandy Ring is a lead scientist at the Advanced Technology Research Center, TSGI. She has extensive experiences providing guidance and designing and building systems for government and commercial clients for over 10 years. One of her areas of expertise is kernel development and network programming for Unix systems.