tl;dr

I ported my Commandline Cloaking tests to Nim, and used it to explore Tetragon, the latest eBPF security observability tool.

Tetragon

Recently Isovalent, the company behind the eBPF-based network monitoring and security program Cilium released Tetragon, an open source “eBPF-based security observability and runtime enforcement platform”.

Isovalent has many very talented eBPF experts, so I thought it would be neat to compare Tetragon to the same Commandline Cloaking techniques I tested Sysmon for Linux with, another eBPF-based security tool.

Tetragon can do a lot more than just observe commandlines, but for this blog I will mostly stick to just commandline detections.

Nim

As well as looking into Tetragon, I also checked out Nim, a statically typed programming language that has been growing in popularity over the last few years. As a compiled system language, it appears to be striking a balance between lower-level systems access and C-interoperability with higher-level conveniences like iterators and a rich standard library.

To me, it fits in a similar place as Go, without the large library ecosystem (due to its popularity Vs google-backed Go), but also without some of the baggage that comes with Go, such as the super-thick static binaries.

As I don’t work as a low-level software engineer I’m not the right person to provide a full evaluation of Nim, but I thought I’d share my thoughts on porting my Commandline Cloaking toy programs from C/Go.

Lab setup

Install the Nim compiler was as simple as following this guide:

curl https://nim-lang.org/choosenim/init.sh -sSf | sh

Tetragon is mostly designed to be run as part of a Kubernetes cluster. I just wanted to run Tetragon by itself (and didn’t have a Kubernetes cluster already setup for testing), thankfully, Djalal Harouni provided this method to run Tetragon as a standalone docker container:

# 1. Run Tetragon in privliged Docker, mounting required folders
docker run --name tetragon \
--rm -it --pid=host --cgroupns=host \
--privileged --detach \
-v /sys/kernel/btf/vmlinux:/var/lib/tetragon/btf \
quay.io/cilium/tetragon:v0.8.0 \
bash -c "/usr/bin/tetragon"

#2. Get events in pretty format:
docker exec -t tetragon bash -c "/usr/bin/tetra getevents -o compact"
#   OR get full JSON of process ecex events:
docker exec -t tetragon bash -c "/usr/bin/tetra getevents" | jq 'select(.process_exec)'

# When finished:
docker rm -f tetragon

Tests

For more information on each of the test programs I created and ran, see my previous blog where I go into more detail.

Test #0 - No cloaking

This program doesn’t do any shenanigans, it just prints some basic commandline information to the screen.

Nim

This was understandably very straightforward to port to Nim:

import os
import strformat
import posix

proc main() =
    echo(fmt"  PID     {getpid()}")
    echo(fmt"  PPID    {getppid()}")
    echo(fmt"  argc    {paramCount() + 1}")  # Nim doesn't count argv[0] in paramCount

    for i in 0..paramCount():
        echo(fmt"  argv[{i}] {paramStr(i)}")

    echo("  Sleeping for 60 seconds so you can lookup the PID")
    setControlCHook(proc() {.noconv.} = discard)
    sleep(60 * 1000)

main()

It is odd that the Nim built-in paramCount() isn’t exactly like C’s argc in that it doesn’t count argv[0] (which is usually the program name), but accessing arguments with paramStr() does include the 0th arg.

Tetragon

Running ./basic_nim AAAA produces a predictable output from Tetragon:

{
  "process_exec": {
    "process": {
      "exec_id": "Ojk1OTg2MDUwNTAyNzU3OjcwMDE=",
      "pid": 7001,
      "uid": 1000,
      "cwd": "/home/path/code/commandline_cloaking/bin/",
      "binary": "/home/path/code/commandline_cloaking/bin/basic_nim",
      "arguments": "AAAA",
      "flags": "execve clone",
      "start_time": "2022-06-16T03:33:18.153Z",
      "auid": 1000,
      "parent_exec_id": "Ojk0OTI0ODgwMDAwMDAwOjU4NzM=",
      "refcnt": 1
    }
  },
  "time": "2022-06-16T03:33:18.153Z"
}

Tetragon helpfully captures the full commandline, as well as other useful information such as UserID, start time, parent process etc. Tetragon maintains its own unique ID system, so you can link and track child and parent processes, even in the event of PID reuse.

Test #1 - Post-execution cloaking

This test runs a few tricks to modify it’s commandline arguments after it executes, to hide from tools such as ps. This includes:

  • Editing our /proc/pid/cmdline by editing our argv array
  • Using prctl and PR_SET_NAME to change the process name in /proc/pid/comm
  • Forking twice and killing parents, which sets the process’ Parent PID to PID 1

Nim

Writing this in Nim was also quite straightforward. To use syscalls such as prctl, it was as simple as:

# Declare the c definitions in Nim 
proc syscall(number: clong): clong {.importc, varargs, header: "sys/syscall.h".}
var NR_PRCTL {.importc: "__NR_prctl", header: "unistd.h".}: int
var PR_SET_NAME {.importc: "PR_SET_NAME", header: "sys/prctl.h".}: int

# Call syscall function to invoke prctl to set our /proc/self/comm
var err = syscall(NR_PRCTL, PR_SET_NAME, cstring("faked"))

This was the same for memset, which doesn’t have a Nim equivalent in the standard lib, but was super easy to ‘import’ and call the C version.

To edit the raw argv array, I did have to ‘turn off’ some of Nim’s magic, as the built-in argument array paramStr() isn’t the actual memory address of /proc/self/cmdline, but the converted-to-Nim string version, and therefore changing that doesn’t actually cloak the commandline from ps.

Working around this was a simple as compiling the program with the --nomain flag, then implementing the Nim startup process ourselves:

proc main(argc: int, argv: cstringArray, envp: cstringArray): int {.cdecl, exportc.} =
    # Need to call NimMain ourselves first to avoid explosions
    NimMain()

    # Do stuff with argv, etc. ...
    discard memset(argv[0], ord('F'), csize_t(len(argv[0])))

For a reason I’ll explain below, once this dodgy_nim program did it’s double-fork-and-kill trick to make its parent PID 1, I then made it call execve to run the basic_nim program.

Tetragon

To test, I ran dodgy_nim like this:

$> ./dodgy_nim AAAA
-------- REAL --------
  PID     8883
  PPID    5873
  argc    2
  argv[0] ./dodgy_nim
  argv[1] AAAA
---- FORK & FAKE -----
  PID     8885
  PPID    1
  argc    2
  argv[0] FFFFFFFFFFF
  argv[1] BBBB
---- EXECVE BASIC ----
  PID     8885
  PPID    1
  argc    2
  argv[0] from_dodgy
  argv[1] CCCC
----------------------

# Check ps output
$> ps aux | grep 7613
path        7613  0.0  0.0   3468   196 pts/1    S    14:02   0:00 FFFFFFFFFFF BBBB

Like Sysmon for Linux, Tetragon logs the process information before it runs, which means the original commandline that the process was started with.

But as you can see below, things do get a little confusion, as it marks the original process as exited, and when basic starts it states its parent ID PID 1:

{
  "process_exec": {
    "process": {
      "exec_id": "Ojk4NDUxNDQ0MTgyMTYxOjg4ODM=",
      "pid": 8883,
      "uid": 1000,
      "cwd": "/home/path/code/commandline_cloaking/bin/",
      "binary": "/home/path/code/commandline_cloaking/bin/dodgy_nim",
      "arguments": "AAAA",
      "flags": "execve clone",
      "start_time": "2022-06-16T04:14:23.547Z",
      "auid": 1000,
      "parent_exec_id": "Ojk0OTI0ODgwMDAwMDAwOjU4NzM=",
      "refcnt": 1
    },
    "parent": {
      "exec_id": "Ojk0OTI0ODgwMDAwMDAwOjU4NzM=",
      "pid": 5873,
      "uid": 0,
      "cwd": "/home/path/code/commandline_cloaking/bin",
      "binary": "/usr/bin/bash",
      "flags": "procFS auid",
      "start_time": "2022-06-16T03:15:36.983Z",
      "auid": 0,
      "parent_exec_id": "Ojk0OTI0NzcwMDAwMDAwOjU4NzI=",
      "refcnt": 4294967281
    }
  },
  "time": "2022-06-16T04:14:23.547Z"
}
{
  "process_exit": {
    "process": {
      "exec_id": "Ojk4NDUxNDQ0MTgyMTYxOjg4ODM=",
      "pid": 8883,
      "uid": 1000,
      "cwd": "/home/path/code/commandline_cloaking/bin/",
      "binary": "/home/path/code/commandline_cloaking/bin/dodgy_nim",
      "arguments": "AAAA",
      "flags": "execve clone",
      "start_time": "2022-06-16T04:14:23.547Z",
      "auid": 1000,
      "parent_exec_id": "Ojk0OTI0ODgwMDAwMDAwOjU4NzM="
    }
  },
  "time": "2022-06-16T04:14:23.548Z"
}
{
  "process_exec": {
    "process": {
      "exec_id": "Ojk4NDUyNDQ1NDI1MTg2Ojg4ODU=",
      "pid": 8885,
      "uid": 1000,
      "cwd": "/home/path/code/commandline_cloaking/bin/",
      "binary": "/home/path/code/commandline_cloaking/bin/basic_nim",
      "arguments": "CCCC",
      "flags": "execve clone",
      "start_time": "2022-06-16T04:14:24.548Z",
      "auid": 1000,
      "parent_exec_id": "OjQzMDAwMDAwMDox",
      "refcnt": 1
    },
    "parent": {
      "exec_id": "OjQzMDAwMDAwMDox",
      "pid": 1,
      "uid": 0,
      "cwd": "/",
      "binary": "/usr/lib/systemd/systemd",
      "arguments": "auto noprompt splash",
      "flags": "procFS auid rootcwd",
      "start_time": "2022-06-15T00:53:32.533Z",
      "auid": 0,
      "refcnt": 1
    }
  },
  "time": "2022-06-16T04:14:24.548Z"
}

This is all technically correct from the kernel’s point of view, but it does show that the double-fork technique is enough to break the commandline chain for analysts. I hadn’t checked this with Sysmon for Linux in my previous blog, so I went back and tried and it also failed to properly link dodgy to basic.

Tests #2 and #3 - In-memory loader and Piping input

This test uses memfd_create to create an in-memory file, writing an ELF program it gets from stdin, before running it with execve.

Nim

Again, thanks to being super easy to call libC functions, this was also straightforward in Nim:

# Define from C
proc memfd_create(name: cstring, flags: cuint): cint {.importc, header: "sys/mman.h".}
proc execve(pathname: cstring, argv: cstringArray, envp: cstringArray): cint {.importc: "execve", header: "stdlib.h".}

# Create in-memory file
let fd = memfd_create("", 0)

# Read from stdin
let data = readAll(stdin)

# Write data to in-memory file
var fwrite = open(fp, fmWrite)
fwrite.write(data)
fwrite.close()

# Create new argv
var args: seq[string]
args.add("from_loader")
for i in 2..paramCount():
    args.add(paramStr(i))
let argv = alloccstringArray(args)

# Exec program
discard execve(cstring(fmt"/proc/self/fd/{fd}"), argv, nil)

Tetragon

I ran the loader like this, to read and launch basic_nim:

$> cat ./bin/basic_nim | ./bin/loader_nim -

Tetragon records both programs executing, and links them together using exec_id:

{
  "process_exec": {
    "process": {
      "exec_id": "Ojk5OTE4Nzc1NzkwNDY1OjEwMzkw",
      "pid": 10390,
      "uid": 1000,
      "cwd": "/home/path/code/commandline_cloaking/bin/",
      "binary": "/home/path/code/commandline_cloaking/bin/loader_nim",
      "arguments": "-",
      "flags": "execve clone",
      "start_time": "2022-06-16T04:38:50.879Z",
      "auid": 1000,
      "parent_exec_id": "Ojk0OTI0ODgwMDAwMDAwOjU4NzM=",
      "refcnt": 1
    },
    "parent": {
      "exec_id": "Ojk0OTI0ODgwMDAwMDAwOjU4NzM=",
      "pid": 5873,
      "uid": 0,
      "cwd": "/home/path/code/commandline_cloaking/bin",
      "binary": "/usr/bin/bash",
      "flags": "procFS auid",
      "start_time": "2022-06-16T03:15:36.983Z",
      "auid": 0,
      "parent_exec_id": "Ojk0OTI0NzcwMDAwMDAwOjU4NzI=",
      "refcnt": 4294967293
    }
  },
  "time": "2022-06-16T04:38:50.879Z"
}
{
  "process_exec": {
    "process": {
      "exec_id": "Ojk5OTE4Nzc2Nzc1MjMzOjEwMzkw",
      "pid": 10390,
      "uid": 1000,
      "cwd": "/home/path/code/commandline_cloaking/bin/",
      "binary": "/proc/self/fd/3",
      "flags": "execve",
      "start_time": "2022-06-16T04:38:50.880Z",
      "auid": 1000,
      "parent_exec_id": "Ojk5OTE4Nzc1NzkwNDY1OjEwMzkw",
      "refcnt": 1
    },
    "parent": {
      "exec_id": "Ojk5OTE4Nzc1NzkwNDY1OjEwMzkw",
      "pid": 10390,
      "uid": 1000,
      "cwd": "/home/path/code/commandline_cloaking/bin/",
      "binary": "/home/path/code/commandline_cloaking/bin/loader_nim",
      "arguments": "-",
      "flags": "execve clone",
      "start_time": "2022-06-16T04:38:50.879Z",
      "auid": 1000,
      "parent_exec_id": "Ojk0OTI0ODgwMDAwMDAwOjU4NzM=",
      "refcnt": 1
    }
  },
  "time": "2022-06-16T04:38:50.880Z"
}

While Tetragon doesn’t show the 2nd process as being an in-memory file, the binary path /proc/self/fd/3 is probably the same level of suspiciousness. We don’t know the in-memory file as the basic program, and while we do see the cat executing, with in-memory file technique the file could have been read from a veriety of sources that don’t leave a commandline, such as remote webserver or hardcoded within the same binary.

Test #4 - LD_PRELOAD

One common technique used by malware to hide is to make use of LD_PRELOAD and similar environment variables.

This test uses LD_PRELOAD to hook the libc function in between the programs initial execution and it’s main() function, altering the input commandline arguments.

Nim

Like in C, this was more straightforward than Go.

# Declare function type matching hooked function signature
type
  LibCMain = proc (
    main: pointer, argc: cint, argv: cstringArray,
    init: pointer, fini: pointer, rtld_fini: pointer
    ): int {.cdecl}

# This hooks __libc_start_main
proc hookedLibcMain(
    main: pointer, argc: cint, argv: cstringArray,
    init: pointer, fini: pointer, rtld_fini: pointer
): int {.cdecl, exportc:"__libc_start_main", dynlib} =
    # Find original function
    let lib = loadLib("libc.so.6")
    assert lib != nil, "Error loading library"
    let origFunc = cast[LibCMain](lib.symAddr("__libc_start_main"))
    assert origFunc != nil, "Error loading function from library"

    # Create new argv
    let newArgc = cint(2)
    var newArgv = alloccstringArray(["from_preload", "BBBB"])
    
    # Call into real function
    return origFunc(main, newArgc, newArgv, init, fini, rtld_fini)

I had to use the .exportc declaration as Nim doesn’t allow functions to start with an _. To use LD_PRELOAD on non-underscore functions you simply have the name the function in Nim the same as the function to be hooked.

Tetragon

I ran the program like this, hooking and replacing the commandline args to basic:

$> LD_PRELOAD=./preload_nim.so ./basic_nim AAAA
  PID     11292
  PPID    5873
  argc    2
  argv[0] from_preload
  argv[1] BBBB
  Sleeping for 60 seconds, so you can look up the PID

Just like Sysmon, it records the arguments the program was launched with, which is technically correct again, but not effectivley happened:

{
  "process_exec": {
    "process": {
      "exec_id": "OjEwMTAwMzk4Mzc5ODg1NToxMTI5Mg==",
      "pid": 11292,
      "uid": 1000,
      "cwd": "/home/path/code/commandline_cloaking/bin/",
      "binary": "/home/path/code/commandline_cloaking/bin/basic_nim",
      "arguments": "AAAA",
      "flags": "execve clone",
      "start_time": "2022-06-16T04:56:56.087Z",
      "auid": 1000,
      "parent_exec_id": "Ojk0OTI0ODgwMDAwMDAwOjU4NzM=",
      "refcnt": 1
    }
  },
  "time": "2022-06-16T04:56:56.087Z"
}

Unfortunately, Tetragon doesn’t log the environment variables a program was launched with, but if it did so, you should be able to see the LD_PRELOAD, which is not commonly used in production environments, and will be pointing to the malicious library.

Test #5 - In-memory patching

Using Silvio’s Parasite technique, we can patch a binary to edit its arguments without LD_PRELOAD. For more information, see this section in the previous blog.

Nim

I didn’t implement this in Nim due to time and motivational constraints.

Tetragon

When patching and loading a binary in-memory, the output is similar to the LD_PRELOAD test.

A twist on the Parasite technique is that the program could write the patch binary back to disk (overwriting the original file), run the on-disk binary, then replace it with the original binary once the program has run.

This would mean the binary is no longer run from in-memory. Tetragon can monitor file access (and more) by supplying custom TracingPolicies, so it could watch of any process writing to e.g. /bin/ to detect the tampering. I ran Tetragon with this policy to look for writes to files under /bin/:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "sys-write-follow-fd-bin"
spec:
  kprobes:
  - call: "fd_install"
    syscall: false
    args:
    - index: 0
      type: int
    - index: 1
      type: "file"
    selectors:
    - matchArgs:
      - index: 1
        operator: "Prefix"
        values:
        - "bin"
      matchActions:
      - action: FollowFD
        argFd: 0
        argName: 1
  - call: "__x64_sys_write"
    syscall: true
    args:
    - index: 0
      type: "fd"
    - index: 1
      type: "char_buf"
      sizeArgIndex: 3
    - index: 2
      type: "size_t"

To run Tetragon with a custom policy policy_file.yml in the current directory, run:

# 1. Start Tetragon with a custom policy file:
docker run --name tetragon \
--rm -it --pid=host --cgroupns=host \
--privileged \
-v /sys/kernel/btf/vmlinux:/var/lib/tetragon/btf \
-v `pwd`:/policies \
quay.io/cilium/tetragon:v0.8.0 \
bash -c "/usr/bin/tetragon --config-file /policies/policy_file.yml"

# 2. Get events in pretty format:
docker exec -t tetragon bash -c "/usr/bin/tetra getevents -o compact"
#   OR get full JSON of process kprobe events:
docker exec -t tetragon bash -c "/usr/bin/tetra getevents" | jq 'select(.process_kprobe)'

# Cleanup when finished:
docker rm -f tetragon

Which when I overwrote /bin/bash using Python I got this event:

{
  "process_kprobe": {
    "process": {
      "exec_id": "OjEwNDg1OTU2ODY0NjQ2MDoxNTQzMQ==",
      "pid": 15431,
      "uid": 1000,
      "cwd": "/home/path/code/commandline_cloaking/",
      "binary": "/usr/bin/python",
      "arguments": "./overwrite_bash.py",
      "flags": "execve clone",
      "start_time": "2022-06-16T06:01:11.671Z",
      "auid": 1000,
      "parent_exec_id": "Ojk0OTI0ODgwMDAwMDAwOjU4NzM=",
      "refcnt": 1
    },
    "function_name": "__x64_sys_write",
    "args": [
      {
        "file_arg": {
          "path": "/bin/bash"
        }
      },
      {
        "truncated_bytes_arg": {
          "bytes_arg": "f0VMRgIBAQAAAAAAAAAAAAMAPgABAAAAgBIAAAAAAABAAAAAAAAAAAhqAgAAAAAAAAAAAEAAOAANAEAAIAAfAAYAAAAEAAAAQAAAAAAAAABAAAAAAAAAAEAAAAAAAAAA2AIAAAAAAADYAgAAAAAAAAgAAAAAAAAAAwAAAAQAAAAYAwAAAAAAABgDAAAAAAAAGAMAAAAAAAAcAAAAAAAAABwAAAAAAAAAAQAAAAAAAAABAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMgKAAAAAAAAyAoAAAAAAAAAEAAAAAAAAAEAAAAFAAAAABAAAAAAAAAAEAAAAAAAAAAQAAAAAAAA+Z0BAAAAAAD5nQEAAAAAAAAQAAAAAAAAAQAAAAQAAAAAsAEAAAAAAACwAQAAAAAAALABAAAAAABUVAAAAAAAAFRUAAAAAAAAABAAAAAAAAABAAAABgAAACgNAgAAAAAAKB0CAAAAAAAoHQIAAAAAADgDAAAAAAAAIBwBAAAAAAAAEAAAAAAAAAIAAAAGAAAAQA0CAAAAAABAHQIAAAAAAEAdAgAAAAAA8AEAAAAAAADwAQAAAAAAAAgAAAAAAAAABAAAAAQAAAA4AwAAAAAAADgDAAAAAAAAOAMAAAAAAAAwAAAAAAAAADAAAAAAAAAACAAAAAAAAAAEAAAABAAAAA==",
          "orig_size": "160264"
        }
      },
      {
        "size_arg": "160264"
      }
    ],
    "action": "KPROBE_ACTION_POST"
  },
  "time": "2022-06-16T06:01:11.677Z"
}

The bytes_arg is indeed the first 520 bytes of the ELF I overrode bash with.

I found creating these custom TracingPolicies to require experienced knowledge of eBPF, Tetragon, and the Linux kernel. But it is possible to detect this technique. Sysmon for Linux also has a similar ability to track file writes as one of its default event types, although I didn’t test it out.

Test #6 - Chroot dorking

While reading Liz Rice’s Excellent book on container security, I started thinking about another commandline cloaking method using chroot jails.

chroot is a system that allows you to change what a process sees as the ‘root’ (/) folder. This allows a level of isolation and abstraction from the ‘real’ files on the host, and is a key concept in how containers work. For example, say a machine has this filesystem:

/
├── bin
│   ├── sh
│   ├── python
│   └── bash
├── home
│   └── user
│       └── fake_root
│           ├── bin
|           |    └── bash
|           └── fake_tmp
├── proc
└── tmp

If we run a program that calls chroot to set the root to /home/user/fake_root, it will see the filesystem like this:

/
├── bin
│   └── bash
└── fake_tmp

When the program attempts to run /bin/bash, it will actually launch /home/user/fake_root/bin/bash. This is how containers work (well part of how they work, I can’t recommend Liz Rice’s book highly enough for a more in-depth explanation).

When you run /bin/bash inside a docker container, on the host machine, it is actually running something like /var/lib/docker/overlay2/<very_long_hash>/merged/bin/bash.

As a result, using chroot also has implications for Tetragon and commandline monitors.

To test how chroot can affect commandline monitors like Tetragon, I created this Nim program:

proc main() =
    if getuid() != 0:
        echo("Need to run as root (technically just CAP_SYS_CHROOT but I'm lazy)")
        quit(1)

    # Create fake root and /bin folder with 
    createDir("fake_root")
    createDir("fake_root/bin")
    
    # Create a link from a dodgy program to fake_root/bin/bash
    link("/path/to/dodgy_nim", "fake_root/bin/bash")

    # Move into fake root and call chroot
    setCurrentDir("fake_root")
    chroot(".")

    # Run fake bin bash, which is now at /bin/bash
    execve("/bin/bash", nil, nil)

There are a few tricks to getting this program to work:

  • The program needs the CAP_SYS_CHROOT capability, which usually means running it at as root
  • The faked bash program either needs to be statically compiled, or you have to also copy in all the libraries into the fake_root

But given those constraints, Tetragon’s logging records the activity like this:

{
  "process_exec": {
    "process": {
      "exec_id": "OjI5Mzc2MTgxOTk3NTcxMzoyMTgzNg==",
      "pid": 21836,
      "uid": 0,
      "cwd": "/path/to/fake_root/",
      "binary": "/bin/bash",
      "arguments": "",
      "flags": "execve",
      "start_time": "2022-07-05T03:13:04.511Z",
      "auid": 1000,
      "parent_exec_id": "OjI5Mzc2MTgxOTI1MDIwNzoyMTgzNg==",
      "refcnt": 1
    },
    "parent": {
      "exec_id": "OjI5Mzc2MTgxOTI1MDIwNzoyMTgzNg==",
      "pid": 21836,
      "uid": 0,
      "cwd": "/path/to/fake_root/",
      "binary": "/path/to/chroot_parent",
      "arguments": "",
      "flags": "execve clone",
      "start_time": "2022-07-05T03:13:04.510Z",
      "auid": 1000,
      "parent_exec_id": "OjI5MzczOTUxNDI1NTM0MjoyMTc4Mw==",
      "refcnt": 1
    }
  },
  "time": "2022-07-05T03:13:04.511Z"
}

According to Tetragon, chroot_parent launched /bin/bash, which sounds like a completely benign activity. But this makes sense - Tetragon is designed to monitor containers as well as host applications, so recording the path as the ‘container’ sees it would help with observing and writing detections for containers.

Tetragon could also monitor and detect this type of suspicious chroot usage by hooking the chmod syscall. After some reading, I created this very basic TracingPolicy:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "sys-chroot"
spec:
  kprobes:
  # __x64_sys_chroot(const char *path)
  - call: "__x64_sys_chroot"
    syscall: true
    args:
      - index: 0
        type: "string"

Which produces this Tetragon event, highlighting the activity:

{
  "process_kprobe": {
    "process": {
      "exec_id": "OjE4MTExMjk2ODkwMTE6NTkxMA==",
      "pid": 5910,
      "uid": 0,
      "cwd": "/path/to/",
      "binary": "/path/to/chroot_parent",
      "arguments": "",
      "flags": "execve clone",
      "start_time": "2022-08-02T03:40:48.950Z",
      "auid": 1000,
      "parent_exec_id": "OjMwNTI3MDAwMDAwMDoyNDQx",
      "refcnt": 1
    },
    "parent": {
      "exec_id": "OjMwNTI3MDAwMDAwMDoyNDQx",
      "pid": 2441,
      "uid": 0,
      "cwd": "/path/to",
      "binary": "/usr/bin/bash",
      "flags": "procFS auid",
      "start_time": "2022-08-02T03:15:43.090Z",
      "auid": 0,
      "parent_exec_id": "OjMwNTI2MDAwMDAwMDoyNDQw",
      "refcnt": 1
    },
    "function_name": "__x64_sys_chroot",
    "args": [
      {
        "string_arg": "."
      }
    ],
    "action": "KPROBE_ACTION_POST"
  },
  "time": "2022-08-02T03:40:48.951Z"
}

Like the file monitor, this policy isn’t ‘on by default’ and requires someone to understand enough about Tetragon and the Linux kernel to write and share it. In the real world, you would want to edit the policy to also filter out known-good binaries or chroot folders, to reduce the amount of false positives.

Conclusions and final thoughts

Writing Nim

I thoroughly enjoyed writing code in Nim, and really liked that it was ‘low enough’ to make it easy to do things like syscalls and memory operations, while being ‘high enough’ to have nice things baked in like argument parsing, slices, and string manipulation.

I’m unsure how much I’ll keep writing in Nim as my main goal of programming nowadays is to share information, and that means sharing code in a language more people use.

Using Tetragon

Tetragon from a programmer’s perspective Tetragon is really cool, easy to run and attach Kprobes to observe all kinds of activity. I could see how you could create some really crafty TracingPolicies to catch all sort of unusual behaviour.

Like Sysmon for Windows and other observability tools, it doesn’t come with its own detections, which means it’s on the users and community to create and share new policies to detect and prevent suspicious behaviour. But writing new policies for Tetragon requires someone with close-to-expert knowledge and experience in Linux and Tetragon.

It reminded me a bit of AuditD rules, where there people who get the most out of AuditD are those whole deeply understand the Linux kernel, or who are lucky enough that someone else’s rules work in their environment without being flooded by false positives. But those without the luck or skills end up feeling frustrated, or simply just don’t use it, losing out on what should be a great source of secutity data.

I hope the Tetragon team can create a community of detection writers, as I feel without that many people won’t be able to get the most out of it.

A good addition to the work would be to be able to automatically translate Sigma rules to and from Tetragon TracePolicies. This would enable people to share detections in a common format beyond Tetragon, and convert rules they may be using with other tools and systems into Tetragon-specific policies.

Example Code

I’ve updated my Commandline Cloaking repository with all the Nim examples, the new Chroot example, as well as the Tetragon TracePolicies.