tl;dr

I’ve been fascinated by offensive applications of eBPF for a while, and so I’ve both given a talk and released a few tools demonstrating a range of offensive, defensive, and anti-anti-RE techniques.

Trace Away

At this year’s DEF CON 29, I presented a talk discussing a number of offensive and defences uses for eBPF (Video of presentation on YouTube). One aspect of rootkits and eBPF that has fascinated me the most is the ability to present different versions of the ‘truth’ to different programs and users, effectivly ‘warping reality’ around usermode proceses. While not strictly an eBPF-only feature, eBPF makes it easier to warp reality across many different machines, and therefore the opportunity for offensive and defensive teams are greater than perhaps it’s ever been.

This blog accompanies my talk, sharing a bit more background and detail into what I discussed at DEF CON. I’ll also point out I wasn’t the only one thinking about offensive eBPF: At this year’s BlackHat USA, Guillaume Fournier, Sylvain Afchain, and Sylvain Baubeau presented a talk about offensive eBPF that I also recommend checking out.

eBPF Primer

This article explains what eBPF is much better than I can, but in essence it allows an administrator to attach programmable trace points to the Linux Kernel for one of 2 broad purposes:

  • Efficiently routing network packets
  • Observing function calls for debugging or security monitoring

The second point is the one that I will focus on in this blog, although both have interesting offensive and defensive uses.

What makes eBPF more interesting that traditional Kernel modules is that it comes with very strong memory safety and efficiency guarantees, thanks to a program verifier that checks code statically and dynamically to ensure eBPF programs can’t crash a system, or otherwise break something important.

For this reason, it is increasingly possible to run eBPF programs in places where you can’t run Kernel Modules, such as in Managed kubernetes environments like Amazon EKS.

eBPF Programs are limited in what they can see and do, but this limit still allows for a level of shenanigans beyond what you can do in userspace, and (as I’ll discuss later) there’s not a lot of security tooling looking into malicious eBPF usage.

Malicious Tracing

A common use-case for eBPF is to attach programs to two parts of the Linux Syscalls, which are the main way user space programs communicate with the kernel:

  1. At the entry to the syscall, to observe the parameters sent to the kernel
  2. At the exit to the syscall, to observe the response from the kernel.

For example, when a userspace program wants to read a file, it will commonly use two syscalls: openat and read:

Picture highlighting syscall flow from openat to read

The first call to openat returns an fd file descriptor, which is then passed into read to get the data from the file.

If we wanted to track what programs are reading what files (and what the data in the file was), we could attach a number of eBPF programs to the entry and exit of those two syscalls:

Picture highlighting syscall flow from openat to read, with eBPF Programs attached to entry and exit of each syscall

The programs attached to openat record what filename is associated with what fd number, and the programs attached to read record what was read from what file.

If you were to run these programs as-is, you’d probably quickly be inundated with events, as it will log every file opened by every program across the entire system.

Thankfully eBPF provides a number of ‘helper’ functions to only run based upon executable, the user running the program, or really anything in the parameters or return value. For example, to only log if the program executable name is test, we can use the bpf_get_current_comm helper:

// trace the entry to the openat syscall
SEC("tp/syscalls/sys_enter_openat")
int handle_read_enter(struct trace_event_raw_sys_enter *ctx)
{
    // eBPF has restrictions on strings and loops
    // so staticaly declare size and string upfront
    const int test_len = 5;
    const char test[test_len] = "test";
    char comm[test_len];

    // Use helper to get executable name:
    bpf_get_current_comm(&comm, sizeof(comm));

    // check to see if name matches
    if (int i = 0; i < test_len; i++) {
        if (test[i] != comm[i]) {
            // Name missmatch,
            // return and don't do anything
            return 0;
        }
    }

    // executable name is 'test', log event,
    // perform additional checks, to do anything else
    ...

    return 0;
}

This is one of the advantages of using eBPF for tracing - The ability to intelligently select when to run your trace programs. For malicious programs, this ability to behave differently in different circumstances can have fascinating results.

Additionally, eBPF programs are restricted from altering the parameters or return values it observes. However, if the parameter or return value is a pointer to userspace memory, then the program is allowed to read and write to wherever it points to, thanks to the two eBPF helper functions bpf_probe_read_user and bpf_probe_write_user.

Combined, the ability to selectively alter data in between userspace and the kernel is a powerful offensive primitive, that can have a wide range of possible uses.

Sudo lie to me

As an example, let’s say an attacker has compromised a web server, and temporarily has root access. They know they can always get back on the machine by leaving behind a webshell, but on this machine the web server runs as the low-privileged apache user, which is not in the list of users allowed to sudo to become root. The attacker needs a way to allow it’s webshell apache user to become root, in a way that can’t be easily detected by an administrator or security tool.

When a user runs sudo, the program will look in the /etc/sudoers file for details on what that user can or cannot do. If a user isn’t allowed to use sudo, their name won’t be in the file.

Based upon our earlier syscall diagram, a malicious eBPF program can the call to read /etc/sudoers call from sudo, overwriting the data returned from the kernel to inject a line saying apache is allowed to become root:

Picture highlighting how eBPF can intercept paramaters and return codes from syscalls

The eBPF program can overwrite the first line in the data returned from read to include apache ALL=(ALL:ALL) NOPASSWD:ALL, giving the apache user the ability to become sudo. But the real file on disk is never altered, which means to programs other that sudo (including kernel programs such as auditd) the file reamins untouched and without the apache user. Only to sudo will the file look different:

# Initially, user apache cannot use sudo:
root@ubuntu$ sudo -l -U 'apache'
User apache is not allowed to run sudo on ubuntu.

# Now load the eBPF Programs to warp reality around sudo:
root@ubuntu$ ./bad-bpf-sudoadd --username 'apache'

# To 'cat', the file looks normal:
root@ubuntu$ cat /etc/sudoers | grep 'apache' | wc -l
0

# But to 'sudo', it's a different story:
root@ubuntu$ sudo -l -U 'apache'
User apache may run the following commands on ubuntu:
    (ALL : ALL) NOPASSWD: ALL

# Now when we are in an apache webshell
# we can run sudo to become root again
apache@ubuntu$ sudo whoami
root

This technique would also work if sudo is using a custom PAM Module to enforce multi-factor, as PAM configuration is stored as text inside /etc/pam.d/. The eBPF program can just lie to sudo about the PAM configuration to say no multi-factor or even password is needed.

Hiding Processes, Hijacking execve

Lying about the contents of files isn’t the only thing malicious eBPF programs can do. Using bpf_probe_write_user anything that gets sent or received as a userspace buffer can be altered. Some other possibilities include:

  • Hiding process from ps by lying about the contents of the /proc/ pseudo-folder
  • Changing the Executable filename when a program calls the execve, hijacking execution
  • Hiding an “ignore all” AuditD rule by intercepting data being sent from the AuditD’s netlink socket to a userspace controller

By combining the ability to alter userspace data, with the ability to precisely target the alterations at specific processes and users, eBPF becomes a powerful tampering tool that is hard to detect and defeat without the right knowledge.

Faking Syscalls

Another recent feature of eBPF is the ability to bypass the syscall entirely, and instead return an arbitry return code to make it look like the function was called.

For example, if you wanted to block a process from writing to a file, you can intercept and fake the return to the write syscall, returing the expected value that makes it look like the data was written to disk, without the real function atually being called:

// Attach to the 'write' syscall
SEC("fmod_ret/__x64_sys_write")
int BPF_PROG(fake_write, struct pt_regs *regs)
{
    // Get expected write amount
    u32 count = PT_REGS_PARM3(regs);

    // Overwrite return
    return count;
}

Detection Challenges and Opportunities

If you didn’t know eBPF existed or could be malicious, detecting such attacks would be challenging. Unlike traditional hooking or KProbes, eBPF programs using tracepoints don’t alter the function’s address or code, so it’s much less obvious there is something ‘in between’ userspace and the kernel.

However, with knowledge of eBPF, you can use the bpf syscall and the bpftool cli program to list out all running eBPF programs and any related usermode processes. Of course a malicious eBPF program can intercept this syscall, so your best bet may be one of:

  • raw memory forensics
  • Kernel Module Sensor

For memory forensics, Volatility was recently updated to include a new plugin to extract bpf information from a memory capture. Valatility can read memory snapshots taken either from inside a running machine using a tools like AVML, or can even be taked from ‘underneeth’ the machine at the hypervisor level.

Otherwise for a live system, the good news is eBPF cannot write to kernel memory, which makes it much harder to hide from kernel-based security monitoring. So while eBPF can hide processes from ps, the actual process information still resides in the kernel, and security callbacks to systems like Linux Security Modules or auditD occur untampered.

Additionally, eBPF programs are unloaded after a reboot. By starting monitoring for uses of eBPF early in the machines boot cycle, you may be able to detect when an unexpected eBPF program is loaded later on. You can even use eBPF to monitor itself by monitoring the bpf syscall.

Anti-Anti Debugging

My DEF CON talk will covered a number of malicious eBPF examples, but these technqiues are not only useful to attackers.

Most malware authors don’t like having their code reverse engineered or analysed, so they often implement checks to determine if they are running on a ‘real’ machine, or inside an analysis sandbox. One way they might do this is to look at the MAC Address of the network card, to determine if the address is linked to a real card, or one attached to a virtual machine.

On linux the MAC Address can be read by reading the address file in /sys/class/net/, so if we wanted to fool some malware into thinking it’s running on a real machine, we could intercept the reading of this file and replace the Address with benign data. There are plenty of other tricks we could play on the malware, lying about IP Addresses, kernel modules (to hide analysis tools), and we could even create a full fake filesystem, lying about the entire contents of the disk and allowing the malware to read and write without it ever actually seeing or affecting the real host.

[Bad BPF]

I’ve created a collection of offensive eBPF Programs named Bad BPF. These tools demonstrate a number of offensive eBPF techniques:

  • Intercepting sudo read calls to enable a low-privileged user to elevate to root
  • Hijacking calls the execve to change the program being launched
  • Hiding process from ps by intercepting directory listings to /proc
  • Replacing arbitrary text in files to demonstrate kernel module hiding and MAC Address faking
  • Deny any calls other uses of the bpf syscall by sending a SIGKILL to kill the process
  • Lie to a program about writing to a file by faking the return to sys_write

The code should be well commented to help understand how they are able to work.

Conclusion

This blog only scratches the surface of what’s possible when eBPF is used for offensive purposes. To learn more, check out my DEF CON talk, and also check out:

For future research, I’m very interesting in exploring how eBPF can affect things when attached to not just syscalls, but instead further inside the kernel where there is usermode buffers to alter. I’ve also been experimenting with using eBPF to detect malicious hooking and rootkits, which I’ll write more about as I learn more.