Commandline Cloaking and Sysmon for Linux

tl;dr

I investigated running Sysmon for Linux against a number of techniques to fake a program’s commandline. Code and Sysmon config available on GitHub.

• tl;dr
• Sysmon for Linux
• Attack Scenario
• Lab setup
• Tests
  • Test #0 - No cloaking
  • Test #1 - Post-execution cloaking
  • Test #2 - Piping input
  • Test #3 - In-memory loader
  • Test #4 - LD_PRELOAD
  • Test #5 - In-memory patching
• Conclusions and final thoughts
  • Sysmon for Linux
  • Writing low-level Go
• Example Code

Sysmon for Linux

Earlier this year, Microsoft released Sysmon for Linux, which is an awesome new addition to free Linux monitoring tools. Typically, sysadmins have used AuditD (or tools that use AuditD under the hood) to monitor Linux systems. However, using AuditD can be tricky to configure, and can require a level of knowledge of Linux internals that might put off administrators that just want basic process monitoring.

Sysmon for Linux conversely uses the same event schema as Sysmon on Windows, which abstracts away the internals in how it generates events, instead groping them into “ProcessCreate”, “NetworkConnect “, “FileDelete”, etc. Using the same schema as Windows is quite useful for people already familiar with the Windows version of Sysmon, although documentation could be improved to help those who aren’t familiar, or only know AuditD.

I was interested in exploring what it’s like to set up and run Sysmon for Linux, and how it might fare against some common techniques malware uses to hide from system administrators.

Attack Scenario

A lot of common malware detections on Linux are based upon on pattern matching commandlines and program filenames. I wanted to explore the various ways a program can alter or hide commandlines from Audit tools, and see what is recorded by Sysmon for Linux.

It was also an excuse to learn more about writing low-level programs in Go, which I’ve always wanted to learn more about, instead of defaulting to C all the time.

Lab setup

To set things up, I used both Roberto Rodriguez’s dope Sysmon Overview and the official instructions.

As I was only interested in process events, I used this simple config to just get process creations:

<Sysmon schemaversion="4.70">
  <EventFiltering>
    <!-- Event ID 1 == ProcessCreate. Log all newly created processes -->
    <RuleGroup name="" groupRelation="or">
      <ProcessCreate onmatch="exclude"/>
    </RuleGroup>
  </EventFiltering>
</Sysmon>

Once Sysmon was installed and running, to make things simple I didn’t forward the events to a SIEM, I just used the sysmonLogView helper tool to print events out to the console in a human-readable format:

sudo bash -c '
    tail -f /var/log/syslog |
    /opt/sysmon/sysmonLogView -e 1
'

Tests

Test #0 - No cloaking

To start, I made a simple program named basic that just prints out the arguments passed into it:

func main() {
  pid := os.Getpid()
  ppid := os.Getppid()

  argc := len(os.Args)
  fmt.Printf("  PID     %d\n", pid)
  fmt.Printf("  PPID    %d\n", ppid)
  fmt.Printf("  argc    %d\n", argc)
  for i := 0; i < argc; i++ {
    fmt.Printf("  argv[%d] %s\n", i, os.Args[i])
  }

  fmt.Print("  Sleeping for 60 seconds so you can lookup the PID")
  time.Sleep(60 * time.Second)
}

When run with a single argument, basic prints out:

$> /path/to/basic AAAA
  PID     984173
  PPID    1054354
  argc    2
  argv[0] /path/to/basic
  argv[1] AAAA
  Sleeping for 60 seconds so you can lookup the PID

This generates the following Sysmon Event as expected:

Event SYSMONEVENT_CREATE_PROCESS
  RuleName: -
  UtcTime: 2021-11-30 23:51:08.592
  ProcessGuid: {35dd0383-b8ec-61a6-b8d0-480000000000}
  ProcessId: 984173
  Image: /path/to/basic
  FileVersion: -
  Description: -
  Product: -
  Company: -
  OriginalFileName: -
  CommandLine: /path/to/basic AAAA
  CurrentDirectory: /path/to
  User: path
  LogonGuid: {35dd0383-9d97-61a4-e803-000002000000}
  LogonId: 1000
  TerminalSessionId: 347
  IntegrityLevel: no level
  Hashes: -
  ParentProcessGuid: {35dd0383-9a98-61a4-fddb-e43082550000}
  ParentProcessId: 1054354
  ParentImage: /usr/bin/bash
  ParentCommandLine: -bash
  ParentUser: path

We can see basic was launched by bash, the process IDs match up, and the commandline is the same. All as expected.

A really neat aspect of Sysmon for Linux is how it extracts the commandline arguments from a process. Other tools use the arguments passed into the execve syscall, but these arguments can be vulnerable to a race condition, such as the one detailed by Rex Guo and Junyuan Zeng at DEF CON 29, due to the fact they exist in the parent process’ userspace memory. Sysmon for Linux however extracts the arguments from the task struct within the kernel’s memory, which (at the time Sysmon reads it) is inaccessible to the parent process, and will always be the real arguments used to start the process.

We can also compare Sysmon’s output to other sysadmin tools, proving the commandline, process ID, and filename match up:

# Use ps to look for 'basic' process
$> ps u -C "basic"
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
path     1066830  0.0  0.1 702896  8968 pts/2    Sl+  11:23   0:00 /path/to/basic AAAA

Under the hood, ps parses data from the /proc/ pseudo-filesystem, so we can also do that:

# Look up the commandline using the PID and the proc psudo filesystem
$> xxd -g 1 -c 6 /proc/1066830/cmdline
00000000: 2e 2f 62 69 6e 2f  /path/to/
00000006: 62 61 73 69 63 00  basic.
0000000c: 41 41 41 41 00     AAAA.

By reading /proc/<pid>/cmdline we can see the strings in argv, each ending in a 00 NULL character. This data exists within basic’s memory, which means that a dodgy program could alter this data.

Test #1 - Post-execution cloaking

Next, I wrote a program named dodgy to alter the contents of it’s argv after it has started to run. While this is easy to do in either C or Go, the Go code is harder to read, so I created dodgy in C, which looks like this:

int main(int argc, char* argv[]) {
  // Overwrite address of argv to fool 'ps'
  memset(argv[0], 'F', strlen(argv[0]));
  if (argc > 1) {
      memset(argv[1], 'B', strlen(argv[1]));
  }

  // Print arguments just like 'basic' did
  printf("  PID     %d\n", getpid());
  printf("  PPID    %d\n", getppid());
  printf("  argc    %d\n", argc);
  for (int i = 0; i < argc; i++) {
    printf("  argv[%d] %s\n", i, argv[i]);
  }
  printf("Sleeping for 60 seconds so you can lookup the PID\n");
  sleep(60);
  return 0;
}

Now when we run the program we get:

$> /path/to/dodgy AAAA
  PID     1067696
  PPID    1054354
  argc    2
  argv[0] FFFFFFFFFFF
  argv[1] BBBB
Sleeping for 60 seconds so you can lookup the PID

# Then in another tab look for all 'dodgy' processes
$> ps u -C "dodgy"
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
path     1067696  0.0  0.0   2304   556 pts/2    S+   12:06   0:00 FFFFFFFFFFF BBBB

# Look up the commandline using the PID and the proc psudo filesystem
$> xxd -g 1 -c 6 /proc/1067696/cmdline
00000000: 46 46 46 46 46 46  FFFFFF
00000006: 46 46 46 46 46 00  FFFFF.
0000000c: 42 42 42 42 00     BBBB.

There are similar techniques programs can use to fake information once it has executed, including:

• Using prctl and PR_SET_NAME to change the process name in /proc/pid/comm
• Fork twice and kill parents, which sets the process’ Parent PID to PID 1

But to Sysmon all these efforts are in vain - As it logs the process information before it runs, it sees the original commandline that the process was started with:

Event SYSMONEVENT_CREATE_PROCESS
  RuleName: -
  UtcTime: 2021-11-30 23:51:08.592
  ProcessGuid: {35dd0383-b8ec-61a6-b8d0-480000000000}
  ProcessId: 1067696
  Image: /path/to/dodgy
  FileVersion: -
  Description: -
  Product: -
  Company: -
  OriginalFileName: -
  CommandLine: /path/to/dodgy AAAA
  CurrentDirectory: /path/to
  User: path
  LogonGuid: {35dd0383-9d97-61a4-e803-000002000000}
  LogonId: 1000
  TerminalSessionId: 347
  IntegrityLevel: no level
  Hashes: -
  ParentProcessGuid: {35dd0383-9a98-61a4-fddb-e43082550000}
  ParentProcessId: 1054354
  ParentImage: /usr/bin/bash
  ParentCommandLine: -bash
  ParentUser: path

Test #2 - Piping input

Some programs support reading data in from standard input, which means it can either be | piped in or read in using < file redirection. One example is Python, where the following examples are all equivalent:

# Run the file
python3 quietquokka.py
python3 $(bash -c "echo cXVpZXRxdW9ra2EucHkK | base64 -d")
# Pipe file in
cat quietquokka.py | python3
# File redirection
python3 < quietquokka.py

The first two generate Syslog events matching what you expect (the $(bash -c ... part gets evaluated prior to the syscall):

Event SYSMONEVENT_CREATE_PROCESS
  RuleName: -
  UtcTime: 2021-12-01 02:57:11.135
  ProcessGuid: {35dd0383-e487-61a6-fdbb-b3691d560000}
  ProcessId: 1068450
  Image: /usr/bin/bash
  FileVersion: -
  Description: -
  Product: -
  Company: -
  OriginalFileName: -
  CommandLine: bash -c echo cXVpZXRxdW9ra2EucHkK | base64 -d
  CurrentDirectory: /path/to
  User: path
  LogonGuid: {35dd0383-9d97-61a4-e803-000002000000}
  LogonId: 1000
  TerminalSessionId: 347
  IntegrityLevel: no level
  Hashes: -
  ParentProcessGuid: {35dd0383-9a98-61a4-fddb-e43082550000}
  ParentProcessId: 1054354
  ParentImage: /usr/bin/bash
  ParentCommandLine: -bash
  ParentUser: path
Event SYSMONEVENT_CREATE_PROCESS
  RuleName: -
  UtcTime: 2021-12-01 02:57:11.137
  ProcessGuid: {35dd0383-e487-61a6-69ee-60001a560000}
  ProcessId: 1068452
  Image: /usr/bin/base64
  FileVersion: -
  Description: -
  Product: -
  Company: -
  OriginalFileName: -
  CommandLine: base64 -d
  CurrentDirectory: /path/to
  User: path
  LogonGuid: {35dd0383-9d97-61a4-e803-000002000000}
  LogonId: 1000
  TerminalSessionId: 347
  IntegrityLevel: no level
  Hashes: -
  ParentProcessGuid: {35dd0383-e487-61a6-fdbb-b3691d560000}
  ParentProcessId: 1068450
  ParentImage: /usr/bin/bash
  ParentCommandLine: bash
  ParentUser: path
Event SYSMONEVENT_CREATE_PROCESS
  RuleName: -
  UtcTime: 2021-12-01 02:57:11.137
  ProcessGuid: {35dd0383-e487-61a6-ed63-6a0000000000}
  ProcessId: 1068453
  Image: /usr/bin/python3.9
  FileVersion: -
  Description: -
  Product: -
  Company: -
  OriginalFileName: -
  CommandLine: python3 quietquokka.py
  CurrentDirectory: /path/to
  User: path
  LogonGuid: {35dd0383-9d97-61a4-e803-000002000000}
  LogonId: 1000
  TerminalSessionId: 347
  IntegrityLevel: no level
  Hashes: -
  ParentProcessGuid: {35dd0383-9a98-61a4-fddb-e43082550000}
  ParentProcessId: 1054354
  ParentImage: /usr/bin/bash
  ParentCommandLine: -bash
  ParentUser: path

However, in the other two examples (piping in data and file redirection), the Python commandline is empty:

Event SYSMONEVENT_CREATE_PROCESS
  RuleName: -
  UtcTime: 2021-12-01 02:58:58.656
  ProcessGuid: {35dd0383-e4f2-61a6-ed63-6a0000000000}
  ProcessId: 1068470
  Image: /usr/bin/python3.9
  FileVersion: -
  Description: -
  Product: -
  Company: -
  OriginalFileName: -
  CommandLine: python3
  CurrentDirectory: /path/to
  User: path
  LogonGuid: {35dd0383-9d97-61a4-e803-000002000000}
  LogonId: 1000
  TerminalSessionId: 347
  IntegrityLevel: no level
  Hashes: -
  ParentProcessGuid: {35dd0383-9a98-61a4-fddb-e43082550000}
  ParentProcessId: 1054354
  ParentImage: /usr/bin/bash
  ParentCommandLine: -bash
  ParentUser: path

I’ve covered this technique before, including thoughts on how to make use of eBPF to detect it. And to be clear Sysmon isn’t wrong - the Python was started with just the commandline “python3”.

You can’t pipe in all arguments to a program, however, only in specific cases where the program supports it. But this does include a number of tools like python, curl, and bash, and is important to think about when looking to make commandline-focussed detection.

Test #3 - In-memory loader

Another common technique used by malware is to execute a program from memory. This can be easily done by using memfd_create:

func main() {
  // Print PID of loader program
  fmt.Printf("  Ldr PID %d\n", os.Getpid())

  // Use memfd to create in-memory file
  fd, _ := unix.MemfdCreate("", 0)

  // open created in-memory file
  fp := fmt.Sprintf("/proc/self/fd/%d", fd)
  memfd_file := os.NewFile(uintptr(fd), fp)
  defer memfd_file.Close()

  // Read ELF from disk and write into in-memory file
  // ELF could also be read from a C2 server, encrypted
  // on disk, hardcoded, piped in from stdin, etc.
  data, _ := ioutil.ReadFile("/path/to/basic")
  memfd_file.Write(data)

  // Execute in-memory binary, fake argv[0] filename
  argv := []string{"from_loader", "AAAA"}
  unix.Exec(fp, argv, os.Environ())
}

This loader program can be then be run to launch basic from in-memory:

$> /path/to/loader
  Ldr PID 17436
  PID     17436
  PPID    15924
  argc    2
  argv[0] from_loader
  argv[1] AAAA
  Sleeping for 60 seconds so you can lookup the PID

# Look up pid
$> ps u | grep 17436
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
path       17436  0.0  0.0 703064  2872 pts/0    Sl+  15:13   0:00 from_loader AAAA

Due to how the execve syscall works, basic’s PID is the same as loader’s. This is because execve doesn’t really create a new process per sey, but instead replaces the program running from loader to basic. But to Sysmon, both programs are recorded:

Event SYSMONEVENT_CREATE_PROCESS
  RuleName: -
  UtcTime: 2022-01-02 04:19:04.901
  ProcessGuid: {35dd0383-27b8-61d1-8cdf-480000000000}
  ProcessId: 17436
  Image: /path/to//bin/loader
  FileVersion: -
  Description: -
  Product: -
  Company: -
  OriginalFileName: -
  CommandLine: /path/to/loader /path/to/basic AAAA
  CurrentDirectory: /path/to/
  User: path
  LogonGuid: {35dd0383-21ff-61d1-e803-000000000000}
  LogonId: 1000
  TerminalSessionId: 177
  IntegrityLevel: no level
  Hashes: -
  ParentProcessGuid: {00000000-0000-0000-0000-000000000000}
  ParentProcessId: 15924
  ParentImage: -
  ParentCommandLine: -
  ParentUser: -
Event SYSMONEVENT_CREATE_PROCESS
  RuleName: -
  UtcTime: 2022-01-02 04:19:04.901
  ProcessGuid: {35dd0383-27b8-61d1-1000-480000000000}
  ProcessId: 17436
  Image: /memfd:
  FileVersion: -
  Description: -
  Product: -
  Company: -
  OriginalFileName: -
  CommandLine: from_loader AAAA
  CurrentDirectory: /path/to/
  User: path
  LogonGuid: {35dd0383-21ff-61d1-e803-000000000000}
  LogonId: 1000
  TerminalSessionId: 177
  IntegrityLevel: no level
  Hashes: -
  ParentProcessGuid: {00000000-0000-0000-0000-000000000000}
  ParentProcessId: 15924
  ParentImage: -
  ParentCommandLine: -
  ParentUser: -

Sysmon can only tell the second ELF executed was from in-memory (marking it as /memfd), and it records the ‘fake’ from_loader commandline. It cannot tell that the binary run was actually /path/to/basic.

But depending on the machine, user, or parent process, it is probably highly suspicious to see a /memfd in-memory ELF being run. So while you cannot tell what binary it was, this alone might be enough to investigate.

Test #4 - LD_PRELOAD

One common technique used by malware to hide is to make use of LD_PRELOAD and similar environment variables. By setting LD_PRELOAD to the path to a library on disk, the Linux dynamic linker will inject this library into all new programs, and will look in that library first for any exported functions that the program may have instead wanted to use from other libraries. This means a malicious library and can hook and alter the program, and is commonly used as a backdoor to alter functions that relate to file reading or network connections, so the malware can hide data on disk, or allow a normally blocked connection through.

But another thing a malicious LD_PRELOAD library can do is hook the program just as it starts, and alter the argv arguments before they are seen and used. For programs written in C using libc, this means hooking the __libc_start_main function.

In C, this type of hooking is simple:

int __libc_start_main(
  int (*main)(int, char **, char **),
  int argc,
  char **argv,
  int (*init)(int, char **, char **),
  void (*fini)(void),
  void (*rtld_fini)(void),
  void *stack_end)
{
  // Overwrite args like in 'dodgy'
  memset(argv[0], 'F', strlen(argv[0]));
  if (argc > 1) {
    memset(argv[1], 'B', strlen(argv[1]));
  }

  // Lookup the real libc function
  typeof(&__libc_start_main) orig_start_main = dlsym(RTLD_NEXT, "__libc_start_main");

  // Call real __libc_start_main
  return orig_start_main(main, argc, argv, init, fini, rtld_fini, stack_end);
}

After writing a C version of the basic program to be hooked, I ran with and set the LD_PRELOAD variable to be my malicious library:

# Set the LD_PRELOAD variable to path to preload
# NOTE: this won't work on Go programs, so use the C version of Basic
LD_PRELOAD=/path/to/preload.so /path/to/basic_c AAAA
  PID     26687
  PPID    16961
  argc    2
  argv[0] FFFFFFFFFFFFF
  argv[1] BBBB
  Sleeping for 60 seconds so you can lookup the PID

# Check ps output for pid
$> ps aux | grep 26687
path       26687  0.0  0.0   2360   532 pts/1    S+   15:47   0:00 FFFFFFFFFFFFF BBBB

# Extract LD_PRELOAD env variable from ps output
$> ps ue | grep 26687 | grep -o "LD_PRELOAD=.*" | awk '{print $1}'
LD_PRELOAD=/path/to/preload.so

As preload’s shenanigans run after the process starts, Sysmon only see’s the original arguments, and not what the effective arguments to basic_c are:

Event SYSMONEVENT_CREATE_PROCESS
  RuleName: -
  UtcTime: 2021-12-02 03:29:01.320
  ProcessGuid: {35dd0383-3d7d-61a8-9d12-65fb78550000}
  ProcessId: 26687
  Image: /path/to/basic_c
  FileVersion: -
  Description: -
  Product: -
  Company: -
  OriginalFileName: -
  CommandLine: /path/to/basic_c AAAA
  CurrentDirectory: /path/to
  User: path
  LogonGuid: {35dd0383-4d8c-61a4-e803-000000000000}
  LogonId: 1000
  TerminalSessionId: 334
  IntegrityLevel: no level
  Hashes: -
  ParentProcessGuid: {35dd0383-4d8c-61a4-fd3b-d4af46560000}
  ParentProcessId: 1052430
  ParentImage: /usr/bin/bash
  ParentCommandLine: -bash
  ParentUser: path

This is accurate, technically basic_c was launched with AAAA, and preload runs after the process has launched. But the basic_c program will act as if the fake BBBB argument was passed into it.

To add to the confusion, instead of overwriting argv, the preload library could instead point __libc_start_main to a new argv array, which will fool ps as well:

// Create our own argv and argc
static char* new_argv[] = {
    "from_preload",
    "BBBB"
};
static int new_argc = 2;

int __libc_start_main(
  int (*main)(int, char **, char **),
  int argc,
  char **argv,
  int (*init)(int, char **, char **),
  void (*fini)(void),
  void (*rtld_fini)(void),
  void *stack_end)
{
  // Lookup the real libc function
  typeof(&__libc_start_main) orig_start_main = dlsym(RTLD_NEXT, "__libc_start_main");

  // Call real __libc_start_main, note we instead pass it our new argc and argv
  // instead of the original ones
  return orig_start_main(main, new_argc, new_argv, init, fini, rtld_fini, stack_end);
}

When run this looks like:

# Set the LD_PRELOAD variable to path to preload
# NOTE: this won't work on Go programs, so use the C version of Basic
LD_PRELOAD=/path/to/preload.so /path/to/basic_c AAAA
  PID     28152
  PPID    16961
  argc    2
  argv[0] from_preload
  argv[1] BBBB
  Sleeping for 60 seconds so you can lookup the PID

# Check ps output for pid, note the original arguments and not what basic effectivly ran with
$> ps aux | grep 28152
path       28152  0.0  0.0   2360   536 pts/1    S+   15:57   0:00 /path/to/basic_c AAAA

This blog was also meant to be about how to do things in Go and not C, so thanks to Awgh on the Bloodhound Slack’s Go channel, I was able to create a similar library in Go:

//export __libc_start_main
func __libc_start_main(main *C.void, argc C.int, argv **C.char,    init *C.void,    fini *C.void,    rtld_fini *C.void,) C.int {
  // Create and use new argv[0] and argv[1]
  offset := unsafe.Sizeof(uintptr(0))
  argv_copy := argv
  new_argv_0 := C.CString("from_preload")
  *argv_copy = new_argv_0
  if argc > 1 {
    new_argv_1 := C.CString("BBBB")
    argv_copy = (**C.char)(unsafe.Pointer(uintptr(unsafe.Pointer(argv_copy)) + offset))
    *argv_copy = new_argv_1
  }

  // Find and call real __libc_start_main function
  fp := C.dlopen(C.CString("libc.so.6"), C.RTLD_LAZY)
  orig_func := C.dlsym(fp, C.CString("__libc_start_main"))
  val, err := cdecl.Call(
    uintptr(orig_func),
    uintptr(unsafe.Pointer(main)),
    uintptr(argc),
    uintptr(unsafe.Pointer(argv)),
    uintptr(unsafe.Pointer(init)),
    uintptr(unsafe.Pointer(fini)),
    uintptr(unsafe.Pointer(rtld_fini)),
  )

  return (C.int)(val)
}

(see the full code here)

Unfortunately, Sysmon doesn’t log the environment variables a program was launched with, most likely due to size constraints, but if it did so you should be able to see the LD_PRELOAD, which is not commonly used in production environments, and will be pointing to the malicious library.

Test #5 - In-memory patching

The final technique to discuss is making use of in-memory patching, to achieve the same goal as LD_PRELOAD, but without using the environment variable.

Back in 1998, Silvio Cesare wrote a paper on Unix ELF parasites and viruses. In the paper he wrote about how to patch ELF binaries to inject code at the program’s entrypoint, running custom shellcode in between when the program has been execve‘d, and the program’s “main” function.

The awesome Binjection library has implemented Silvio’s technique in Go, so I was able to use their library and my in-memory loader code to:

1. Read a binary to run from disk
2. Patch the entrypoint with custom shellcode that alters argv before returning to the real entrypoint
3. Run the patched binary from memory using memfd.

The full code I called injector, and is available here. The shellcode to alter the arguments at the ELF entrypoint is actually very simple, as at the start of the ELF execution the kernel will place the memory addresses to the arguments on the top of the stack, so as long as you don’t care about memory corruption, to overwrite the first argument from AAAA to BBBB the shellcode simply looks like:

; Get address of to argv[1] string
mov  rax, [rsp+0x10]

; overwrite argv[1] string to "BBBB\0"
; which in hex is: 42 42 42 42 00
mov [rax],   dword 0x42424242
mov [rax+4], byte 0x00

; Reset rax to 0
xor rax, rax

The end result looks like this:

$> /path/to/injector /path/to/shellcode.bin /path/to/basic AAAA
  PID     28638
  PPID    16961
  argc    2
  argv[0] from_injector
  argv[1] BBBB
  Sleeping for 60 seconds so you can lookup the PID

# Check ps output for pid, note the original arguments and not what `basic` saw
$> ps aux | grep 28152
path       28638  0.0  0.0 703064  2876 pts/1    Sl+  16:34   0:00 from_injector BBBB

Sysmon for Linux will log the loader fully, but again will have some trouble with the loaded binary, only recording /memfd:

Event SYSMONEVENT_CREATE_PROCESS
 RuleName: -
 UtcTime: 2022-01-02 05:36:48.387
 ProcessGuid: {35dd0383-39f0-61d1-a01b-4b0000000000}
 ProcessId: 28638
 Image: /path/to/injector
 FileVersion: -
 Description: -
 Product: -
 Company: -
 OriginalFileName: -
 CommandLine: /path/to/injector /path/to/shellcode.bin /path/to/basic AAAA
 CurrentDirectory: /path/to/
 User: path
 LogonGuid: {35dd0383-25d1-61d1-e803-000001000000}
 LogonId: 1000
 TerminalSessionId: 178
 IntegrityLevel: no level
 Hashes: -
 ParentProcessGuid: {00000000-0000-0000-0000-000000000000}
 ParentProcessId: 16961
 ParentImage: -
 ParentCommandLine: -
 ParentUser: -
Event SYSMONEVENT_CREATE_PROCESS
 RuleName: -
 UtcTime: 2022-01-02 05:36:48.387
 ProcessGuid: {35dd0383-39f0-61d1-2700-480000000000}
 ProcessId: 28638
 Image: /memfd:
 FileVersion: -
 Description: -
 Product: -
 Company: -
 OriginalFileName: -
 CommandLine: from_injector AAAA
 CurrentDirectory: /path/to/
 User: path
 LogonGuid: {35dd0383-25d1-61d1-e803-000001000000}
 LogonId: 1000
 TerminalSessionId: 178
 IntegrityLevel: no level
 Hashes: -
 ParentProcessGuid: {00000000-0000-0000-0000-000000000000}
 ParentProcessId: 16961
 ParentImage: -
 ParentCommandLine: -
 ParentUser: -

Sysmon for Linux doesn’t currently log the hash of the image being executed. If it did, the Sha256 hash of the binary launched would be the hash for the original binary+shellcode:

$> sha256sum /path/to/basic
9cadc2121a28ffe4c9c30afe093794206fdc429b8d56b4b0ddae70ac9dc993cd  /path/to/basic
# the patched binary can be read from the proc filesystem:
$> sha256sum /proc/28152/exe
0a21c7e1bb346db045e2d182ec2678f13d431689ff2d0a8ab83e641bd94a5b4f  /proc/28152/exe

The patched binary hash would be extremely uncommon, possibly unique, and doesn’t match the original binary on disk. Depending on the environment, the uncommonness of the hash may be enough to investigate.

Conclusions and final thoughts

Sysmon for Linux

Sysmon for Linux is looking like a great free monitoring tool for Linux. For people familiar with the Windows version of Sysmon, it provides a common configuration language that will definitely be useful for admin to maintain, and it’s process creation events would provide excellent insight into a lot of standard Linux malware attacks.

It is missing a few features that might catch more advanced attacks, such a file hashing and environment variable logging, and I didn’t explore its performance on a production-sized machine, but it’s a solid monitor to rival using AuditD, or other free monitors such as Tracee.

One thing not covered in this blog is the ability for an attacker to tamper with Sysmon’s configuration. Sysmon uses eBPF under the hood, and uses eBPF Maps to store the configuration (What event types to record, processes to ignore, etc.), which as I’ve written about in the past can be susceptible to tampering by an attacker with admin/root level privileges. While it is difficult for a program to protect itself for such a highly privileged attacker, it’s worth noting that the attack I described in my previous blog does work on Sysmon without generating any tamper-detection events. This is something the Sysmon team might consider detecting in the future, but it’s not an easy thing to prevent, and they may rightfully consider it out of scope for a free monitoring tool.

Writing low-level Go

Thanks to the Awgh, the Bloodhound Slack, and the Binjection project, I was able to pretty easily get up to speed and write Go code to do raw syscalls, LD_PRELOADing, and other low-level tricks pretty easily. Raw pointer manipulation looks pretty ugly in Go compared to C, but doing things like argument parsing and safe string manipulation was much nicer in Go.

Example Code

I’ve uploaded the source for all the different programs mentioned in this blog here: https://github.com/pathtofile/commandline_cloaking. With examples in both C and Go, it has these programs:

• Basic’s argument printing
• Dodgy’s argv manipulation
• LD_PRELOAD libraries
• In-memory loading to alter argv after execution
• ELF Patching to alter argv after execution
• Sample Sysmon for Linux configuration and setup