pat_h/to/file

Investigating realtime detections on iOS using Unified Logging

2024-10-24T12:00:00+00:00

In an attempt to get back into blogging, I’m publishing some notes on a current research project I’m working on, rather than waiting until I have fully chased down every research lead like the other projects I’ve been working on-and-off for over the years.

While looking for a new research project, I recently watched an excellent talk by Matthias Frielingsdorf on the Current State Of IOS Malware Detection. While my career thus far has focussed on desktop and server security, I had heard from friends who work in the mobile antimalware space that iOS presents a unique challenge to defenders.

To summarise my friends’ opinions, Apple’s strong focus on security and privacy has created devices that are difficult to exploit (especially with advanced features such as Lockdown Mode), but that once an attacker has compromised a device it was difficult for the user or external security team to detect an attack, due to the difficulties in getting actionable ‘live’ data from the device at runtime. Matthias’ talk convinced me to learn about iOS security, and how I could apply my expertise in protecting other Operating Systems to this problem.

Current Detection Efforts
Enter Lockdown
Working Remotely
Unified Logging
Configuration Profiles
Collecting Data
Messaging
- iMessage
- SMS
Other Activity
Conclusions and Future work
Epilogue - Offensive Uses

Current Detection Efforts

The most common technique defenders have in uncovering intrusions on iOS devices is to create a forensic snapshot of the device using the Sysdiagnose feature. This has been used by Amnesty International, The Citizen Lab, and others to great success in the past. But they suffer from two major flaws:

1. Snapshots give time for attackers to hide

As snapshots have to be manually triggered, can take up to 30 minutes to collect, and only offer a view into the device at that specific point in time, they give the attacker time to ‘clean up’ their footprint on the device, removing forensic artefacts their initial exploitation and execution may have created.

2. You need to know when to make a snapshot

It is unlikely a user will know exactly when they are compromised. So unless the user employs a routine of proactively and constantly taking snapshots, they might never know the ‘right’ time to take one.

In Kaspersky’s reporting on the TriangleDB Malware, they discussed initially being tipped off to the infection via network monitoring. But network monitoring is far from a reliable detection mechanism, especially in the age of Global Content Delivery Networks, TLS encryption, and the increasing use of encrypted DNS via DNS-Over-TLS.

I wanted to investigate if there was a way to get realtime information from an iOS device, that would either reveal an attack, or at least let a user know that something suspicious has occurred, and that they should consider making a forensic snapshot to investigate.

Enter Lockdown

Remote machines can communicate to a special daemon running on iOS devices called lockdownd. This communication requires that the remote machine is first trusted by a human physically agreeing to ‘pair’ the iOS and remote machine together, as well as the devices being able to communicate over either physical USB, or local WiFi. For WiFi to work, the devices must be on the same subnet (more on that later).

To explain both how the pairing and communication works, here is 2 excellent writeups that will explain it far better than I can:

Understanding usbmux and the iOS lockdown service - Jon Gabilondo
Understanding iDevice protocol layers - PyMobileDevice3 dev team

Once the remote machine is paired with the iOS device, a surprising amount of both snapshot and realtime data becomes available to query via the creation one of a number of ‘Lockdown Services’. These services became the focus of my research.

Screenshot of an iOS device, with a prompt asking if the human trusts the connected device

Working Remotely

The USB/Local WiFi limitation initially seemed like a big blow the practicality of using Lockdown Services for detections ‘in the real world’. Initially I had assumed this limitation was due to the fact the initial device handshake was setup via the Bonjour/Multicast DNS protocol. This protocol by design can’t traverse across network subnets, which means the only option for an iOS device outside of local Wi-Fi range is to connect is to carry around something like a portable Linux Wi-Fi router that has the ability create a low-level L2 VPN tunnel to the remote machine, then connect the iOS to the router via Wi-Fi. Kinda lame.

However, it turns out the initial handshake isn’t strictly necessary every connection. An excellent tool for communicating with lockdownd and Lockdown Services is pyMobileDevice3, a pure-python implementation of the older libmobiledevice library. This tools provides the ability to communicate with a any remote device that is accessible over TCP, provided it has first been connected ‘locally’. In my situation I first connected my iOS device to my remote machine via USB and ran:

# First, get device identifier from:
pymobiledevice3 usbmux list

# Ensure devices are paired:
pymobiledevice3 lockdown pair

# Turn connections via 'Wi-Fi' on
pymobiledevice3 lockdown wifi-connections on

# Save pair record for future access:
# This contains sensitive data so don't post to Twitter or something
pymobiledevice3 lockdown save-pair-record pair_record.plist

Once that was done, I was able to connect to my device while it was fully remote, as long as it was accessible over TCP, for example using a ‘normal’ VPN like Tailscale to link the 2 devices together. Using Python on my remote machine, I did this by:

import plistlib
from pymobiledevice3.lockdown import create_using_tcp

# Open and parse saved pair record
with open("pair_record.plist", "rb") as f:
  plist = plistlib.load(f)

lockdown = create_using_tcp(
  # IP Address or hostname of iOS device
  hostname="10.11.22.33",
  # Identifier from 'pymobiledevice3 usbmux list'
  identifier="11111111-2222222222222222",
  # plist from disk
  pair_record=plist,
)

# Can now use lockdown to connect to services, see next paragraphs
trace_service = OsTraceService(lockdown)

There is a massive caveat with this technique, however - It still requires the iOS device to be connected to some type of Wi-Fi network. It doesn’t have to be the same Wi-Fi network as the remote machine, but unless the device is connected to something over Wi-Fi, the lockdownd daemon will refuse any connection. So you could walk around tethered to something like a small Android Phone, and it’d work I guess?

That still isn’t ideal, and I feel there is more research to understand the exact limitation here. I decided to call this “good enough” for now, and started to look into what you can actually do with access the Lockdown services.

Unified Logging

Like most UNIXes, iOS has a centralised system-wide logging system. Unlike other UNIXs that primarily use syslog, modern iOS uses a system called Unified Logging, which enables applications and programs to produce structured log entries in that can be performantly written and read in real-time. Managed by the logd daemon on the device, there is no on-device UI to read these logs, mostly because they are more useful to developers and apple rather than the end-user.

Applications can’t access the logs of other applications or system components, but a paired remote device can read logs from the whole system, by using lockdownd to connect to the os_trace_relay Lockdown Service. This is helpfully implemented in pyMobileDevice3, where we can simply run this to start reading and parsing log messages in real-time:

import plistlib
from pymobiledevice3.lockdown import create_using_tcp
from pymobiledevice3.services.os_trace import OsTraceService

# Connect to os_trace_relay service and start reading entries:
# lockdown = create_using_tcp(...)
trace_service = OsTraceService(lockdown)
for entry in trace_service.syslog():
  # pyMobileDevice converts this data to a Python object.
  # To write to disk, the below code converts this into JSON:
  entry_json = {
        "pid": entry.pid,
        "timestamp": datetime.isoformat(entry.timestamp),
        "level": entry.level.name,
        "image_name": entry.image_name,
        "filename": entry.filename,
        "message": entry.message,
        "label": {
            "category": "",
            "subsystem": "",
        },
    }
    if entry.label is not None:
        entry_json["label"]["category"] = entry.label.category
        entry_json["label"]["subsystem"] = entry.label.subsystem
  
  # You can now print or filter the JSON
  print(json.dumps(entry_json))

When using this script, we can capture events that look like this:

{
  "pid": 65424,
  "timestamp": "2024-10-04T14:52:28.100478",
  "level": "INFO",
  "image_name": "/System/Library/PrivateFrameworks/TCC.framework/TCC",
  "filename": "/Applications/MobileSlideShow.app/MobileSlideShow",
  "message": "SEND: 0/7 synchronous to com.apple.tccd: request: msgID=65424.1, function=TCCAccessRequest, service=kTCCServicePhotos,",
  "label": { "category": "access", "subsystem": "com.apple.TCC" },
}

pid and filename are the Process ID and program responsible the event, in this case the default Photos app (called MobileSlideShow.app to iOS). image_name Will be where the log message was actually generated from, which can be the main process binary, or like in this case an external Library the process has loaded.

The label can help link logs messages from the same Application/Library together, however they are ‘arbitrarily’ named by the developers, so the same category name across different Libraries might not mean the same thing. And the message field can also be any sort of data, including milti-line stack traces, encoded JSON, hex- or base64-encoded data, or really anything else the developer found worth logging.

There are plenty of other Lockdown services that are worth investigating (some of which are mentioned below), but for the most part my research focussed on parsing events from this log, and understanding what events get created when I undertake various suspicious and malicious actions on my devices.

Configuration Profiles

Before detailing what data I was able to collect from my device’s logs, it’s useful to talk about Configuration Profiles. Designed to either be installed ‘manually’ on a device or via a Mobile Device Management (MDM) solution, these (usually signed) profiles are designed to ensure a device is setup consistently across an organisation. For example, they can ensure certain applications are installed or specific settings are turned on or off. When deployed via an MDM no user interaction is required, but they can also be installed manually by the user physically clicking through a number of security prompts:

They have been used for malicious purposes, which requires the user to be tricked/phished/coerced into installing them. The reason Configuration Profiles are relevant to this research is that Apple publishes a number of Special Configuration Profiles, designed specifically for developers to help diagnose difficult problems with their applications. Most of the profiles supplied by Apple accomplish this by enabling extra verbose logging across the system, and oftentimes also disabling special checks that redact sensitive information from the logs. For example the CFNetwork Diagnostics profile states:

The CFNetwork Diagnostics profile generates files that allow Apple to troubleshoot issues with your device and help Apple to improve its products and services.

The generated files will contain all data transferred over the Internet to or from your device, including personal information such as the contents of the emails and iMessage texts you send and receive, the websites you visit, your Maps searches, and your requests to Siri. The generated files will contain account passwords that websites and your apps transfer over the Internet to access secure resources.

The profile will expire after 7 days.

In my testing, the settings these profiles alter can only be changed by Configuration Profiles specifically signed by Apple, and as the profiles are signed they cannot be altered, although they can be parsed to see exactly what settings they change. I’ve uploaded the parsed plist data to this repo for reference.

One other interesting factor is the note about the expiry. While the CFNetwork description states the profile will expire in 7 days, in reality the actual expiry appears to be much longer. For example, below is the XML plist data from the CFNetwork profile currently downloadable from Apple. As you can see it has a ‘Removal Date’ of July 30th 2025, 10 months after I initially downloaded it:

 version="1.0">

  ConsentText
  
    default
    The CFNetwork Diagnostics profile generates files that allow Apple to troubleshoot issues with your device and help Apple to improve its products and services. The generated files will contain all data transferred over the Internet to or from your device, including personal information such as the contents of the emails and iMessage texts you send and receive, the websites you visit, your Maps searches, and your requests to Siri. The generated files will contain account passwords that websites and your apps transfer over the Internet to access secure resources. The profile will expire after 7 days.

You will be able to turn on and off logging at any time while the Profile is installed, and will be able to review the log files on your computer prior to sending them to Apple.  To turn off logging, open Settings, tap General, tap "Profiles," tap "CFNetwork Diagnostics," and tap "Delete Profile."

By enabling this diagnostic tool and sending a copy of the generated files to Apple, you are consenting to Apple's use of the content of such files in accordance with its privacy policy (http://www.apple.com/legal/privacy).

Apple Confidential Profile. Do not distribute. Not to be used or disclosed without permission from Apple. Copyright © 2024, Apple Inc. All rights reserved.
  
  DurationUntilRemoval
  604800.0
  PayloadContent
  
    
      PayloadContent
      
        
          DefaultsData
          
            AppleCFNetworkDiagnosticLogging
            3
          
          DefaultsDomainName
          .GlobalPreferences
        
      
      PayloadDescription
      Enables CFNetwork diagnostic logging
      PayloadDisplayName
      CFNetwork Extended Logging
      PayloadIdentifier
      com.apple.defaults.managed.cfnetworklogging
      PayloadOrganization
      Apple, Inc
      PayloadType
      com.apple.defaults.managed
      PayloadUUID
      3A614BF2-2651-4A2A-9FA1-7556444401E9
      PayloadVersion
      1
    
  
  PayloadDescription
  Enables CFNetwork logging. Reboot required.
  PayloadDisplayName
  CFNetwork Diagnostics
  PayloadIdentifier
  com.apple.cfnetworklogging
  PayloadOrganization
  Apple Inc.
  PayloadRebootSuggested
  
  PayloadType
  Configuration
  PayloadUUID
  E524E135-447D-439D-BBDD-F7E49FCE36E5
  PayloadVersion
  1
  RemovalDate
  2025-06-30T07:00:00Z

This is similar for most of the profiles I investigated.

Another important note is Configuration Profiles cannot be installed while a device is in Lockdown Mode. While a smart decision by Apple as part of Lockdown Mode’s efforts to reduce possible attack surface, it means that to load a new Configuration profile, you need to:

Go into settings to turn off Lockdown Mode
Reboot the device
Install the new Configuration Profile (which may require another reboot)
Turn Lockdown Mode back on
Reboot the device again

While fine for research purposes, this isn’t ideal to have to do ‘in the real world’, especially if it needs to be done semi-reguarly to ‘reload’ expired Profiles. So during my research I checked what data was generated with and without any profiles, to see if they were actually needed.

Collecting Data

My research was conducted using 2 iOS devices that I happen to have on hand:

A Jailbroken iPhoneX running iOS 16
A non-Jailbroken iPhone 14 running iOS 18

On both devices I tested various activities such as starting Applications, browsing to websites, receiving messages, etc., with and without Configuration Profiles. On the Jailbroken device I also used SSH to launch a few ‘native’ executables outside any Application sandbox.

I started looking at the ‘normal’ types of events used by security platforms on other Operating Systems, i.e. events related to Process, Networking, and File activity.

Next, I looked at more iOS-specific events, such as events related to common exploitation vectors, inter-process communications, and various post-exploitation activities undertaken by known iOS malware such as NSO’s Pegasus and TriangleDB.

Process Creation

When launching an Application, I observed the following event being logged, that contained the Application name and some other details (Safari in this case):

'application' Constructed job description (context ):
 { count = 18, transaction: 0, voucher = 0x0, contents =
  "ProcessType" =>  { length = 9, contents = "SystemApp" }
  "EnableTransactions" => : false
  "_ManagedBy" =>  { length = 22, contents = "com.apple.runningboard" }
  "CFBundleIdentifier" =>  { length = 22, contents = "com.apple.mobilesafari" }
  "_ResourceCoalition" =>  { length = 35, contents = "application" }
  "ThrottleInterval" => : 2147483647
  "PersonaEnterprise" => : 1000
  "MachServices" =>  { count = 0, transaction: 0, voucher = 0x0, contents =
  }
  "EnablePressuredExit" => : false
  "InitialTaskRole" => : 1
  "UserName" =>  { length = 6, contents = "mobile" }
  "EnvironmentV<…>

Unfortunately, I did not observe these or similar events when launching ‘native’ executables. As an alternative to relying on just the logs, the os_trace_relay Lockdown Service does allow you to query the current list of running processes. This is done with pyMobileDevice3 by doing:

# lockdown = create_using_tcp(...)
trace_service = OsTraceService(lockdown)
pids = trace_service.get_pid_list().get("Payload")
print(json.dumps(pids, indent=2))

The results contains just the PID and Executable:

{
  "172": {
    "ProcessName": "itunescloudd"
  },
  "142": {
    "ProcessName": "analyticsd"
  },
  "34": {
    "ProcessName": "mediaserverd"
  },
  "112": {
    "ProcessName": "nehelper"
  },
  ...
}

While the outputted data is missing information typically seen in a process listing on other Operating Systems, such as commandline arguments or username, this technique could enable you to periodically query the list to try to discover new or unusual processes. This isn’t a great solution, as it will miss anything that runs in between queries, and it may be hard to tell legitimate from illegitimate processes.

It could also be possible to simply keep track of all processes that generate any type of log entry. While malware authors are unlikely to directly write log entries, the dynamic libraries the malware imports, or the processes they inject their malware into, just might. This could lead to the process “leaking” events that give away the malware’s presence or intentions. This would require keeping a list of all processes previously seen in the logs, which might get quite long the longer a phone is on. Neither of these are great solutions.

Networking

Logging Network activity is where things get a bit more interesting. By default, without any extra Configuration Profile I oberved a number of DNS-related events. However the specific details about what host was looked up was hashed for privacy:

// Request
{
  "pid": 68558,
  "timestamp": "2024-10-07T16:24:12.388888",
  "level": "NOTICE",
  "image_name": "/usr/sbin/mDNSResponder",
  "filename": "/usr/sbin/mDNSResponder",
  "message": "[R337] getaddrinfo start -- flags: 0xC000D000, ifindex: 0, protocols: 0, hostname: , options: 0xC {in-app-browser, use-failover}, client pid: 288 (com.apple.WebKi), delegator pid: 282 (MobileSafari)",
  "label": { "category": "dnssd_server", "subsystem": "com.apple.mdns" }
}

// Response
{
  "pid": 68558,
  "timestamp": "2024-10-07T16:23:47.588887",
  "level": "NOTICE",
  "image_name": "/usr/sbin/mDNSResponder",
  "filename": "/usr/sbin/mDNSResponder",
  "message": "[R335->Q40326] getaddrinfo result -- event: add, ifindex: 0, name: , type: A, rdata: , expired: no",
  "label": { "category": "dnssd_server", "subsystem": "com.apple.mdns" },
}

These events show that MobileSafari with PID 282 looked up a domain using the device’s central DNS service, but not what it looked up. However, on the iOS16 an event did contain the domain requested:

{
  "pid": 175,
  "timestamp": "2024-10-07T17:09:17.579674",
  "level": "DEBUG",
  "image_name": "/usr/lib/libnetworkextension.dylib",
  "filename": "/usr/sbin/mDNSResponder",
  "message": "NEHelperTrackerGetDisposition: lookup for  length 47 (app info ref C2910FF0 pid 282 for Web)",
  "label": { "category": "", "subsystem": "com.apple.networkextension" }
}

This event (or anything similar) was missing on the iOS18 device. If you install the CFNetwork Diagnostics Configuration Profile, the DNS lookups are logged unredacted:

// Request
{
  "pid": 144,
  "timestamp": "2024-10-02T16:55:35.665548",
  "level": "NOTICE",
  "image_name": "/usr/sbin/mDNSResponder",
  "filename": "/usr/sbin/mDNSResponder",
  "message": "[R26617] getaddrinfo start -- flags: 0xC000D000, ifindex: 0, protocols: 0, hostname: ec2-54-215-0-19.us-west-1.compute.amazonaws.com, options: 0xC {in-app-browser, use-failover}, client pid: 47601 (com.apple.WebKi), delegator pid: 47598 (MobileSafari)",
  "label": { "category": "dnssd_server", "subsystem": "com.apple.mdns" }
}

// Response
{
    "pid": 144,
    "timestamp": "2024-10-02T16:55:35.666772",
    "level": "NOTICE",
    "image_name": "/usr/sbin/mDNSResponder",
    "filename": "/usr/sbin/mDNSResponder",
    "message": "[R26617->Q25753] getaddrinfo result -- event: add, ifindex: 0, name: ec2-54-215-0-19.us-west-1.compute.amazonaws.com., type: A, rdata: 13.57.37.227, expired: no",
    "label": {
      "category": "dnssd_server",
      "subsystem": "com.apple.mdns"
    }
}

In terms of actual network connection information, without needing a Configuration Profile these events were periodically sent detailing some network information, but not actual IP addresses etc:

{
  "pid": 68550,
  "timestamp": "2024-10-07T16:51:34.222655",
  "level": "NOTICE",
  "image_name": "/System/Library/PrivateFrameworks/Symptoms.framework/Frameworks/SymptomEvaluator.framework/SymptomEvaluator",
  "filename": "/usr/libexec/symptomsd",
  "message": "Data Usage for wget on flow 7176 - WiFi in/out: 1080/757, WiFi delta_in/delta_out: 270/171, Cell in/out: 572451/1563, Cell delta_in/delta_out: 0/0, RNF: 0, subscriber tag: 0",
  "label": { "category": "analytics", "subsystem": "com.apple.symptomsd" }
}

While not useful for telling exactly what hosts the device is talking to, these events may be useful to spot unusual processes transferring an unusual amount of data over the network.

I also observed these events while using commandline tools on the jailbroken iOS16 device, e.g. when using wget from “Procursus Team” on Sileo Store:

{
  "pid": 69302,
  "timestamp": "2024-10-07T16:44:45.586970",
  "level": "DEBUG",
  "image_name": "/usr/lib/system/libsystem_info.dylib",
  "filename": "/private/preboot/DEE19D683D13EA5F16C71B676AF1D2C3C0F84C8F33659BE2BEE207865AA501FC9B38E5051439961162D5A61A2AC4415B/jb-qL8pPRKx/procursus/usr/bin/wget",
  "message": "nat64_v4_requires_synthesis(54.215.0.19) == false",
  "label": {
    "category": "getaddrinfo",
    "subsystem": "com.apple.network.libinfo"
  }
}

None of this contains port information, but there is an easier way to collect network information from a device…

Networking - Using PCAPs

Starting with iOS 5, apple added a remote virtual interface (RVI) facility that allows mirroring networks traffic from an iOS device. This is done via another Lockdown Service com.apple.pcapd. Helpfully, unlike like a typical external network tap, the events received from pcapd contain both the raw traffic and information on what Process is responsible for the traffic.

As a demonstration, this code combines pyMobileDevice3 with Scapy to parse unencrypted DNS traffic:

from pymobiledevice3.services.pcapd import PcapdService
from scapy.all import sr1,IP,ICMP, Ether, DNS

# lockdown = create_using_tcp(...)
service = PcapdService(lockdown)
    for packet in service.watch():
        e = Ether(packet.data)
        if e.haslayer(DNS):
            print(f"epid: {packet.epid} | pid: {packet.pid} | comm: {packet.comm}")
            print(e.show())

An example of this code’s output is:

epid: 49370 | pid: 144 | comm: mDNSResponder
###[ Ethernet ]###
  dst       = 64:5a:ed:12:5a:4f
  src       = 74:ac:b9:37:87:04
  type      = IPv4
###[ IP ]###
     version   = 4
     ihl       = 5
     tos       = 0x0
     len       = 110
     id        = 29395
     flags     = DF
     frag      = 0
     ttl       = 64
     proto     = udp
     chksum    = 0x4377
     src       = 192.168.1.1
     dst       = 192.168.1.227
     \options   \
###[ UDP ]###
        sport     = domain
        dport     = 57369
        len       = 90
        chksum    = 0x3703
###[ DNS ]###
           id        = 39303
           qr        = 1
           opcode    = QUERY
           aa        = 0
           tc        = 0
           rd        = 1
           ra        = 1
           z         = 0
           ad        = 0
           cd        = 0
           rcode     = ok
           qdcount   = 1
           ancount   = 1
           nscount   = 0
           arcount   = 0
           \qd        \
            |###[ DNS Question Record ]###
            |  qname     = b'ec2-13-57-37-224.us-west-1.compute.amazonaws.com.'
            |  qtype     = A
            |  unicastresponse= 0
            |  qclass    = IN
           \an        \
            |###[ DNS Resource Record ]###
            |  rrname    = b'ec2-13-57-37-224.us-west-1.compute.amazonaws.com.'
            |  type      = A
            |  cacheflush= 0
            |  rclass    = IN
            |  ttl       = 86400
            |  rdlen     = None
            |  rdata     = 13.57.37.224
           \ns        \
           \ar        \

The pid:144 and comm:mDNSResponder point to the service doing the actual lookup, which in this case is the device’s centralised DNS service But the epid:49370 is the Process ID that initiated the DNS lookup, which in this example was curl. By parsing more packets, we would see both the initial request, and the actual network connection to 13.57.37.224.

The caveat to this method is most network data would still be encrypted under TLS, including DNS traffic if using DNS-Over-HTTPS/TLS), but the addition of process information does make this more useful than regular external network monitoring, as we can can spot situations where an unusual process is talking to a ‘normal’ remote server.

File Access

The last of the ‘usual’ events to capture on a system, getting information on interesting files being written to disk also proved difficult.

One of the more interesting files to observe being written is Crash Logs, which might be an indication of a process being exploited and left in an unstable state, for example the BLASTPASS attack. Helpfully we get events whenever a new crash report is generated, for example when I accidentally made Safari crash:

{
  "pid": 213,
  "timestamp": "2024-10-20T09:21:51.268620",
  "level": "NOTICE",
  "image_name": "/System/Library/PrivateFrameworks/OSAnalytics.framework/OSAnalytics",
  "filename": "/System/Library/CoreServices/osanalyticshelper",
  "message": "Saved type '309()' report (4 of max 25) at /private/var/containers/Shared/SystemGroup/systemgroup.com.apple.osanalytics/DiagnosticReports/MobileSafari-2024-10-20-092151.ips",
  "label": { "category": "", "subsystem": "" }
}
{
  "pid": 7551,
  "timestamp": "2024-10-20T09:21:52.048922",
  "level": "NOTICE",
  "image_name": "/System/Library/PrivateFrameworks/OSAnalytics.framework/OSAnalytics",
  "filename": "/System/Library/CoreServices/OTACrashCopier",
  "message": "Retiring (309) submitted '/private/var/mobile/Library/Logs/CrashReporter/Retired/MobileSafari-2024-10-20-092151.ips': success",
  "label": { "category": "", "subsystem": "" }
}

As the filename of the crashdump (MobileSafari-2024-10-20-092151.ips in this case) relates to the name of the process that crashed, and depending on what crashed this might be an excellent reason to trigger a forensic snapshot. This crashdump can be collected remotely from the device, either as part of a sysdiagnose snapshot, or directly from the com.apple.crashreportcopymobile Lockdown Service/ with pyMobileDevice’s CrashReportsManager class:

from pymobiledevice3.services.crash_reports import CrashReportsManager

# lockdown = create_using_tcp(...)
crash_service = CrashReportsManager(lockdown)
crash_service.pull("output.ips", "/Retired/MobileSafari-2024-10-20-092151.ips")

To crack the creation or access of other files, on iOS the fseventsd daemon keeps track of filesystem changes, storing the logs on the device in the /private/var/db/fseventsd directory. These logs can collected from Jailbroken devices as the root user, which can then be parsed by tools such as this one from David Cowen. I believe collecting these logs from a non-jailbroken device can only be done as part of the sysdiagnose snapshot.

Whenever fseventsd went to create a new logfile on the jailbroken iOS16 device I saw this event, however this appears to be a jailbreak artefact and not something seen normally:

{
  "pid": 65285,
  "timestamp": "2024-10-04T14:49:36.798972",
  "level": "ERROR",
  "image_name": "/usr/libexec/fseventsd",
  "filename": "/usr/libexec/fseventsd",
  "message": "scan_old: bailing out because device mounted @  has dls 0x0 and dls->fci 0x0",
  "label": { "category": "daemon", "subsystem": "com.apple.fsevents" }
}

Messaging

The next part of my research focussed on more specific signs of initial exploitation or malicious data collection.

Based on the Research into iOS malware such as Pegasus and TriangleDB, a common way to exploit an iOS device appears to be sending a malicious message to the device using formats like iMessage or SMS, oftentimes exploiting iOS’ parsing of files attached to the messages. I wanted to look for events related to incoming messages, as well as any attachments that may have been sent, so I tested sending a different types of messages to the devices, including SMS, iMessage, and MMS. For iMessage I sent messages with and without file attachments.

iMessage

On iOS16, without a Configuration Profile, no logs contained the sender information or message text unredacted. When sending various types of file attachments, I would occasionally see the following error log that does contain the filename, although no other useful information:

{
  "pid": 0,
  "timestamp": "2024-10-07T17:23:56.047715",
  "level": "ERROR",
  "image_name": "/System/Library/Extensions/Sandbox.kext/Sandbox",
  "filename": "/kernel",
  "message": "System Policy: suggestd(200) deny(1) file-read-metadata /private/var/mobile/Library/SMS/Attachments/74/04/D9C5BF2C-1C0A-4260-BA3D-6459FFA6D961/Form-10-Inventory-of-assets-and-liabilities-1.docx",
  "label": { "category": "", "subsystem": "" }
}

On iOS18, without Profile I observed an event that contained the sender info (+61411111111 in this example) :

{
  "pid": 78921,
  "timestamp": "2024-10-07T23:03:32.005659",
  "level": "INFO",
  "image_name": "/System/Library/PrivateFrameworks/IMCore.framework/IMCore",
  "filename": "/Applications/MobileSMS.app/MobileSMS",
  "message": "--> Chat iMessage;-;+61411111111 has 0 remaining holds: ",
  "label": { "category": "IMChat", "subsystem": "com.apple.Messages" }
}

When sending an attachment, this event was also created, but no other information:

{
  "pid": 220,
  "timestamp": "2024-10-07T23:03:31.900702",
  "level": "INFO",
  "image_name": "/System/Library/PrivateFrameworks/IMCore.framework/IMCore",
  "filename": "/System/Library/PrivateFrameworks/SOS.framework/sosd",
  "message": "Posting transfer, guid: ",
  "label": {
    "category": "IMFileTransferCenter",
    "subsystem": "com.apple.Messages"
  }
}

Adding the FaceTime and Call Activity Logging Configuration Profile is where things got interesting. This profile unredacts a lot of information in the logs, for example this event that contains the sender and receiver’s details:

{
    "pid": 69,
    "timestamp": "2024-10-07T17:48:37.770384",
    "level": "INFO",
    "image_name": "/System/Library/Messages/PlugIns/iMessage.imservice/iMessage",
    "filename": "/System/Library/PrivateFrameworks/IMCore.framework/imagent.app/imagent",
    "message": "Starting processing post blastdoor for messageItem: IMMessageItem[outgoing: NO sender=+61411111111; service=iMessage; handle=(null); destinationCallerID= +61422222222, unformatted=(null); country=(null); roomName='(null)'; flags=1; subject='(null)' text='(null)' messageID: 0 GUID:'3DFC4B49-2D9A-4C45-8D1C-8624D4339D3F' sortID: 0 date:'749976517.015652' date-delivered:'0.000000' date-read:'0.000000' date-played:'0.000000' transfer guids: '(\n    \"BFF68274-7B55-46E6-AC74-3A0B70F88504\"\n)' empty: NO finished: YES sent: NO read: NO delivered: NO audio: NO played: NO from-me: NO DD results: NO DD Scanned: NO Downgraded: NO emote: NO expirable: NO expire-state: 0 balloon-bundle-id: (null) expressive-send-style-id: (null) time-expressive-send-played: (null) bizIntent: (null) locale: (null) error: 0 sync-state 0 corrupt: NO shouldSendMeCard: NO isSpam: NO hasUnseenMention: NO threadIdentifier: (null), threadOriginator: (null), replyCountsByPart: (null), isChoros: NO, chorosConversationID: -1, syndicationRanges: (null), syncedSyndicationRanges: (null), dateEdited",
    "label": {
      "category": "MessageService",
      "subsystem": "com.apple.Messages"
    }
  }

While this event doesn’t contain much information about the attachment, if you lookup the ‘transfer guid’ in other events (BFF68274-7B55-46E6-AC74-3A0B70F88504 in this example), you would find information on other events about both the filename and the iCloud URL used to facilitate the file transfer:

{
    "pid": 272,
    "timestamp": "2024-10-07T17:48:37.831417",
    "level": "INFO",
    "image_name": "/System/Library/PrivateFrameworks/IMCore.framework/IMCore",
    "filename": "/System/Library/PrivateFrameworks/SOS.framework/sosd",
    "message": "Posting transfer created: [IMFileTransfer: 0x66eb4a410  state: Waiting for Accept  sync state: Not synced  local path: (null)  transferred name: (null)  guid: BFF68274-7B55-46E6-AC74-3A0B70F88504  error: -1  total bytes: 2427577  created: 2024-10-07 06:48:37 +0000 commSafety: 0]  transferName: apple-platform-security-guide.pdf",
    "label": {
      "category": "IMFileTransferCenter",
      "subsystem": "com.apple.Messages"
    }
}
{
    "pid": 69,
    "timestamp": "2024-10-07T17:48:37.834690",
    "level": "INFO",
    "image_name": "/System/Library/Messages/PlugIns/iMessage.imservice/iMessage",
    "filename": "/System/Library/PrivateFrameworks/IMCore.framework/imagent.app/imagent",
    "message": "    user info: {\n    \"decryption-key\" = AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA;\n    \"file-size\" = 2427577;\n    \"mime-type\" = \"application/pdf\";\n    \"mmcs-owner\" = \"CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC.DDDDDDDDDDDDDDDD.EEEEEEEE\";\n    \"mmcs-signature-hex\" = BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB;\n    \"mmcs-url\" = \"https://p40-content.icloud.com/CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC.DDDDDDDDDDDDDDDD.EEEEEEEE\";\n    name = \"apple-platform-security-guide.pdf\";\n    \"uti-type\" = \"com.adobe.pdf\";\n}",
    "label": {
      "category": "Attachments",
      "subsystem": "com.apple.Messages"
    }
}
{
    "pid": 69,
    "timestamp": "2024-10-07T17:48:38.076867",
    "level": "INFO",
    "image_name": "/System/Library/PrivateFrameworks/IMDaemonCore.framework/IMDaemonCore",
    "filename": "/System/Library/PrivateFrameworks/IMCore.framework/imagent.app/imagent",
    "message": "Local path: (null), filename: apple-platform-security-guide.pdf",
    "label": {
      "category": "Attachments",
      "subsystem": "com.apple.Messages"
    }
}

The initial event does make mention as occurring after running the file is run through Blastdoor, a security service designed to deal with untrusted data in a secure way. This means exploitation could occur prior to this event being created, in which case even with a Configuration Profile we only have this message to indicate blastdoor is about the be activated:

{
  "pid": 61,
  "timestamp": "2024-10-07T13:23:08.237016",
  "level": "NOTICE",
  "image_name": "/System/Library/PrivateFrameworks/IDS.framework/identityservicesd.app/identityservicesd",
  "filename": "/System/Library/PrivateFrameworks/IDS.framework/identityservicesd.app/identityservicesd",
  "message": "IDSBlastDoor: Incoming message  has constraint type 0 and BlastDoor context ",
  "label": { "category": "IDSDaemon", "subsystem": "com.apple.IDS" }
}

In theory, either a program crash, or otherwise lack of follow-up event after this initial blast door event, could mean something suspicious happened during blast door processing.

Finally, with the Configuration Profile, the message content was sometimes available in the following message under CKBBContentKeyAttachmentSummary (I sent the message “AAAA” in this example):

{
  "pid": 185,
  "timestamp": "2024-10-07T17:48:38.783849",
  "level": "INFO",
  "image_name": "/System/Library/PrivateFrameworks/IMDPersistence.framework/IMDPersistence",
  "filename": "/System/Library/PrivateFrameworks/IMDPersistence.framework/XPCServices/IMDPersistenceAgent.xpc/IMDPersistenceAgent",
  "message": "Set userInfo for dictionaries {\n    CKBBContentKeyAttachmentCount = 1;\n    CKBBContentKeyAttachmentSummary = AAAA;\n    CKBBContentKeyCountByAttachmentType =     {\n        \"%lu Files\" = 1;\n    };\n    CKBBContextKeyChatGUIDs =     (\n        \"iMessage;-;+6142222222\"\n    );\n    CKBBContextKeyMessageGUID = \"3DFC4B49-2D9A-4C45-8D1C-8624D4339D3F\";\n    CKBBContextKeySenderName = Host;\n    CKBBContextKeySenderPersonCentricID = \"9EFC13F1-34E0-44EA-A3E8-D63A574CA9B3:ABPerson\";\n    CKBBUserInfoKeyChatIdentifier = \"+614222222222\";\n}",
  "label": {
    "category": "IMDNotificationsController",
    "subsystem": "com.apple.Messages"
  }
}

And in this event (I sent “Apple pie”):

{
  "pid": 185,
  "timestamp": "2024-10-07T18:04:41.387819",
  "level": "DEBUG",
  "image_name": "/System/Library/PrivateFrameworks/IMFoundation.framework/IMFoundation",
  "filename": "/System/Library/PrivateFrameworks/IMDPersistence.framework/XPCServices/IMDPersistenceAgent.xpc/IMDPersistenceAgent",
  "message": " => Out attributed string: Apple pie{\n    \"__kIMMessagePartAttributeName\" = 0;\n}",
  "label": {
    "category": "DataDetector",
    "subsystem": "com.apple.IDS"
  }
}

SMS

Without any Configuration Profile, on iOS18 I observed the following event that contained the sender’s phone number:

{
  "pid": 88956,
  "timestamp": "2024-10-22T16:16:31.305446",
  "level": "INFO",
  "image_name": "/System/Library/PrivateFrameworks/IMCore.framework/IMCore",
  "filename": "/Applications/MobileSMS.app/MobileSMS",
  "message": "--> Chat SMS;-;+61411111111 has 0 remaining holds: ",
  "label": { "category": "IMChat", "subsystem": "com.apple.Messages" }
}

Again without any profile, when sending an image with MMS I would observe the following event that contains both the the destination filename and sender information:

{
    "pid": 32,
    "timestamp": "2024-10-07T17:32:24.931615",
    "level": "NOTICE",
    "image_name": "/System/Library/PrivateFrameworks/UserNotificationsServer.framework/UserNotificationsServer",
    "filename": "/System/Library/CoreServices/SpringBoard.app/SpringBoard",
    "message": "Successfully resolved request: resolvedRequest=, subtitle: (null), body: , summaryArgument: , summaryArgumentCount: 0, categoryIdentifier: MessageExtension-SMS, launchImageName: , threadIdentifier: +61411111111, attachments: (\n    \"\\n}>>\"\n), badge: (null), sound: , realert: 1, interruptionLevel: 1, relevanceScore: 0.00, filterCriteria: (null), trigger: (null)>, resolutionSuccess=YES, continueOnFailure=NO",
    "label":
      {
        "category": "AttachmentsService",
        "subsystem": "com.apple.UserNotifications",
      },
  }

Other Activity

Lastly, I wanted to look for any other smaller indicators of post-exploitation activity. While many of these didn’t result in hard ‘proof’ of an attack, like detection engineering for other Operating Systems when small events are coupled together with other pieces of information they might just form enough of a pattern to prompt for a more in-depth look or forensic snapshot.

Enabling Ad Tracking

Kaspersky observed that the TriangleDB malware would enable personalized ad tracking. While I am unsure of exactly how the malware achieved this, when I toggled the option in the Setting UI it created this event:

{
  "pid": 128,
  "timestamp": "2024-10-04T13:31:35.345429",
  "level": "NOTICE",
  "image_name": "/System/Library/PrivateFrameworks/ManagedConfiguration.framework/ManagedConfiguration",
  "filename": "/usr/libexec/adprivacyd",
  "message": "Set Bool value YES for settings: allowApplePersonalizedAdvertising",
  "label": {
    "category": "ProfileConnection",
    "subsystem": "com.apple.ManagedConfiguration"
  }
}

Location

Without a Profile, when I used an App that requested the device’s location I observed this event:

{
  "pid": 70,
  "timestamp": "2024-10-04T13:59:24.527649",
  "level": "NOTICE",
  "image_name": "/usr/libexec/locationd",
  "filename": "/usr/libexec/locationd",
  "message": "client '[mobile]com.apple.Maps' authorized for location; starting now, desiredAccuracy, -1.0, distanceFilter, -1.0, operatingMode 0, dynamicAccuracyReductionEnabled 0, allowsAlteredAccessoryLocations 1, activityType 0",
  "label": { "category": "Client", "subsystem": "com.apple.locationd.Core" }
}

With the Location and Motion Configuration profile, I observed the actual devices location in the logs, which is expected given the Profile’s description. I am unsure if other methods of getting the device’s location produce similar events.

Entitlements

Without a profile I observed these events when a App was requesting certain Entitlements from the Transparency, Consent and Control (TCC) framework:

{
  "pid": 65424,
  "timestamp": "2024-10-04T14:52:28.100478",
  "level": "INFO",
  "image_name": "/System/Library/PrivateFrameworks/TCC.framework/TCC",
  "filename": "/Applications/MobileSlideShow.app/MobileSlideShow",
  "message": "SEND: 0/7 synchronous to com.apple.tccd: request: msgID=65424.1, function=TCCAccessRequest, service=kTCCServicePhotos,",
  "label": { "category": "access", "subsystem": "com.apple.TCC" },
}
{
  "pid": 44,
  "timestamp": "2024-10-04T14:49:49.519418",
  "level": "NOTICE",
  "image_name": "/System/Library/PrivateFrameworks/TCC.framework/Support/tccd",
  "filename": "/System/Library/PrivateFrameworks/TCC.framework/Support/tccd",
  "message": "Granting TCCDProcess: identifier=com.apple.mobileslideshow, pid=65396, auid=501, euid=501, binary_path=/Applications/MobileSlideShow.app/MobileSlideShow access to kTCCServicePhotos via entitlement 'com.apple.private.tcc.allow'",
  "label": { "category": "access", "subsystem": "com.apple.TCC" }
}

In this example the photos app MobileSlideshow was requesting access to the devices photo library. Similar to getting the device’s location, I am unsure if other methods of interacting with TCC produce similar events.

KeyChain access

When I used Safari to request my saved passwords from the iOS Keychain, I observed this event:

{
  "pid": 66574,
  "timestamp": "2024-10-04T16:46:49.079914",
  "level": "NOTICE",
  "image_name": "/System/Library/PrivateFrameworks/SafariCore.framework/SafariCore",
  "filename": "/Applications/SafariViewService.app/SafariViewService",
  "message": "-[WBSSavedAccountStore _savedAccounts]: Returning 2 saved accounts",
  "label": { "category": "Keychain", "subsystem": "com.apple.SafariShared" },
}

While this event didn’t require any Configuration profile, malware commonly collects this data from disk or other lower-level APIs, in which case this event would probably not be created.

Cookies

Similarly to keychain access, malware typically collects cookies are by reading the database stored on the devices disk, requiring file access information to detect. However in 2021 Google Observed Russia’s SVR deploy a javascript cookie stealer, that coupled with an exploit to the underlying WebKit engine was used to send sensitive cookies to an SVR-controlled website using ‘regular’ javascript.

Using the CFNetwork Configuration Profile, I observed this event when I visited some websites using Safari that read my saved cookies, that contained information on the website visited:

{
  "pid": 42891,
  "timestamp": "2024-10-04T13:52:24.335812",
  "level": "NOTICE",
  "image_name": "/System/Library/Frameworks/CFNetwork.framework/CFNetwork",
  "filename": "/System/Library/PrivateFrameworks/CoreParsec.framework/parsecd",
  "message": "CFNetwork Diagnostics [3:758] 13:52:24.335 {\nHTTPCookieStorage::copyCookiesForURL: \n                         Request URL: https://api-glb-aapse2c.smoot.apple.com/\n                    MainDocument URL: null\n} [3:758]",
  "label": { "category": "Diagnostics", "subsystem": "com.apple.CFNetwork" },
}

Whether the SVR technique was ‘regular’ enough to generate these events is unknown.

XPC

XPC is the main inter-process communication format on iOS. This format was explicitly used by NSO’s FORCEDENTRY exploit to escape the Application Sandbox. Without any Configuration Profile I observed a number of XPC related messages being regularly created by my devices. These events contain information on what processes are being communicated with and by whom:

{
  "pid": 223,
  "timestamp": "2024-10-04T13:43:24.510463",
  "level": "NOTICE",
  "image_name": "/System/Library/PrivateFrameworks/SPShared.framework/SPShared",
  "filename": "/usr/libexec/searchpartyd",
  "message": "XPCSessionManager(BeaconManagerService): New XPC connection:  connection from pid 42697 on mach service named com.apple.icloud.searchpartyd.pairingmanager",
  "label":
    {
      "category": "xpcSessionManager",
      "subsystem": "com.apple.icloud.spshared",
    },
}
{
  "pid": 68634,
  "timestamp": "2024-10-07T14:08:06.935270",
  "level": "INFO",
  "image_name": "/System/Library/Frameworks/LocalAuthentication.framework/Support/coreauthd",
  "filename": "/System/Library/Frameworks/LocalAuthentication.framework/Support/coreauthd",
  "message": "MainServiceListener has accepted a new connection:  connection from pid 68598 on mach service named com.apple.CoreAuthentication.daemon hash:10d047d0",
  "label": {
    "category": "Server,LAContext",
    "subsystem": "com.apple.LocalAuthentication"
  }
}

I am uncertain exactly what messages would have been created during FORCEDENTRY’s exploitation.

Conclusions and Future work

While this research is still very much an ongoing project, I feel there is already enough threads to pull on to warrant more investigation. A lot of ‘normal’ events used by endpoint security systems on other Operating Systems are lacking, which does make a system build solely on Unified Logging data unlikely to be that useful as a security solution. Especially given that a proper security system would be expected to do something about a detected threat or anomalous activity, beyond just a notification.

But, there are enough events, that with a proper understanding of what is ‘normal’ for that device (e.g. who that phone usually receives messages from, or what Apps ask for what data), it might be possible to at least inform a user that they should take and analyse a forensics snapshot, which might then might contain more concrete indicators of compromise. An especially interesting detection strategy might be to look for unusual process crashes, then pulling and analysing the Crash Logs to determine if a full forensic snapshot is needed.

I do plan to continue more research into this topic, starting with understanding the Wi-Fi limitation. The next will probably be around choosing specific events to collect, parse, and send to a SIEM, to understand the volume and usefulness of data collected from an actual in-use device. The third will be looking for more interesting events that might contain atypical-but-useful data that could show signs of exploitation or post-exploitation activity. This includes looking at other Lockdown Services for other interesting sources of data.

Epilogue - Offensive Uses

I want to touch on a few final thoughts on the topic of Offensive uses of this technique.

Firstly, sending log data to a remote machine does does come with some security risk. Due to their less locked-down architectures, other Operating Systems such as MacOS, Windows, or Linux, may be more open to attack by malicious actors. If the remote machine is compromised, the attackers would be able to collect the logs sent from the iOS device, which if certain Configuration Profiles are used could contain sensitive personal information.

The positive take, however, is that these other Operating Systems also allow advanced security products to detect sophisticated attacks, in ways not possible on iOS. In fact in some scenarios, with the increased visibility enabled by these security products, ‘forcing’ the attacker onto the other operating systems might even be preferable.

Secondly, there is a question about whether a malicious actor could use this technique in replacement of deploying their own malware on the device. There are documented attacks where a user is phished or coerced into installing malicious Configuration Profiles, which require a similar amount of physical interacting with the device to setup, and would therefore require a similar amount of deception/coercion to setup malicious Unified Logging collection.

As a more sophisticated hypothetical, an attacker who has exploited a device to gain root privileges would have the ability to both perform log collection on the device, as well as unredact any sensitive information before sending the logs to their own (untrusted) Command-And-Control servers. Any malware that employs this may be able to reduce its on-device footprint, as it would not need to touch or hook certain parts of the device in order to collect information available in the logs.

Lastly, if this technique becomes ubiquitous enough for defenders to regularly use, it may push malicious actors to develop techniques to tamper with the Unified Logging system, to either remove their activity from the logs, or intercept the reading of the logs and ‘filter out’ or obfuscate their activity. The advantage of reading and reacting to the logs in real-time could mean that enough ‘suspicious’ events might be generated during the attackers initial attack stages, before the attacker has enough control over the device to scrub the logs. If this is the case, the user may be notified to take a forensic snap before the malware can react. And as evidenced by similar log-scrubbing attacks on other Operating Systems, removing log entries in a non-suspicious way can be more difficult than it appears, as gaps in logs may be just as suspicious as the log entries themselves.

Gaining Threat-Intelligence the REALLY dodgy way

2022-11-30T12:00:00+00:00

tl;dr

I forked and used hfiref0x’s awesome Kernel Driver Utility project to subscribe to the Microsft-Threat-Intelligence feed without using a signed binary or test machine.

Back to ETW-TI

Despite spending most of my time away from Windows these days, I was recently motivated to check back in with ETW, especially after the exploit used by SealighterTI was patched.

To refresh your memory on the systems and acronyms covered in this blog series so far (or skip to the new content):

Event Tracing for Windows (ETW)

ETW is a system on Microsoft Windows that provides tracing and event logs for a variety of kernel- and user-mode systems, both from Microsoft and other developers. Grouped into ‘Providers’, these events can provide a wealth of information for anyone looking to understand what is happening on a system, from Security products looking to detect malicious activity to bug hunters looking for vulnerabilities.

KrabsETW and Sealighter

KrabsETW is a C++ and .NET library that makes it very easy to subscribe to and receive ETW events.

Sealighter is a tool I built on top of Krabs to provide a basic commandline interface into ETW, making it easy to explore ETW events and data.

Protected Processes, PPL, and Early Launch Anti-Malware

Starting with Windows 8, Microsoft started developing new ways to provide extra protection to critical and sensitive information, such as user credentials or Anti-Malware processes.

One of these mechanisms is called Protected Proceses, and its sibling ‘Protected Process - Light’, or PPL. These are user-mode programs that have extra protections and security to prevent tampering but are also allowed access to data that typical programs cannot.

They require (among other things) that the program binary and libraries are checked and signed by Microsoft, which will then mark the binaries with a special entitlement indicating they can be run as a protected process.

One of these entitlements is called ‘Early Launch Anti-Malware’ or ELAM. Designed for security products, it allows the programs to both protect themselves from attackers, and gain special access to the system, such as the ability to scan Kernel activity early in the machine’s boot process. ELAM user-mode programs have to be loaded alongside a similarly checked-and-signed Kernel Driver, and there is a vetting process in place before a person or company can create ELAM programs.

Protected processes with the ELAM entitlement are sometimes called ‘PPL-AntiMalware’.

Microsft-Security-Threat-Intelligence

Microsft-Security-Threat-Intelligence is the name of a special ETW Provider, that can only be read by PPL-AntiMalware processes.

It contains lots of information that is very interesting to security software, such as suspicious memory allocations that could be evidence of process injection or Kernel exploitation.

Previous attempts

Test-Signing Mode

As mentioned above, to run a PPL process and subscribe to Microsft-Security-Threat-Intelligence the legitimate way on a normal Windows machine, you need a driver checked and signed by Microsoft with the ELAM entitlement.

If you put the Machine into ‘Test Signing’ or ‘Debug’ mode, while the program is still required to be signed and accompanied by an ELAM driver, it does not need to be signed by Microsoft.

I covered the steps to collect data from the Threat-Intelligence Provider on debug machine in my blog Experimenting with Protected Processes and Threat-Intelligence .

Exploitation

Sometimes an exploit is discovered allowing you to inject code into an existing PPL process.

Although strong security mechanisms typically prevent this, back in 2018 Alex Ionescu and James Forshaw presented a series of talks, and blogs, covering many ways you could trick Windows into illegitimately running arbitrary code at the PPL level.

One of their techniques was worked up by Clément Labro into PPLDump, which would trick certain PPL Programs into loading an unsigned driver.

In my blog Gaining Threat-Intelligence the dodgy way, I built off Labro’s code to create SealighterTI, which uses the same exploit to subscribe to the Threat-Intelligence feed and send the results to the Event Log to be read and analysed.

Earlier this year, Microsoft patched the exploit used by PPLDump and SealighterTI, so these no longer work on up-to-date systems.

Even Dodgier Method - Vulnerable Drivers

Recently there has been growing public interest and research into utilising old drivers with known vulnerabilities for attack purposes.

As these old drivers are legitimetly signed by Microsoft, they can be used to exploit the system and gain Kernel-level privileges on up-to-date systems, and can circumvent a lot of Windows security mechanisms such as Secure Boot.

Microsoft can revoke the certificate of the drivers to prevent them from loading, and both Microsoft and 3rd-party security vendors provide ways to block these drivers based on file hash or certificate, but this is often a difficult cat-and-mouse game, as driver developers might not be forthcoming with details about exactly what versions of their driver are vulnerable, and what file hashes and signatures to block. And there is a lot of drivers out there to monitor and write signatures for.

For defensive purposes, the best protection is to know what drivers are normally installed on a system (to spot anomalies), and to closely monitor who, when, and how new drivers are installed, as installing new drivers does require Administrator privileges and should not be a common administrator task.

But for research purposes, vulnerable drivers provide an excellent way to explore Windows from one of the highest privilege levels, without altering system behaviour by putting it into a non-production debug state.

Enter the Kernel Driver Utility

The Kernel Driver Utility (KDU) from hfiref0x is an excellent project to explore the possibilities of vulnerable drivers.

Hfiref0x has collected a range of known vulnerable drivers (30 at the time of this blog), and abstracted exploiting each driver into a common interface to be able to read and write arbitrary kernel memory from user-land.

It also comes with several shellcode-based Kernel Driver loaders, making it trivial to write your own unsigned driver (perhaps first testing it normally on a debug system), and deploy it to a fully patched and protected system.

KDU and Sealighter

For interacting with PPL processes, KDU originally could only strip protections from processes, a not-uncommon tactic to be able to dump PPL memory or kill the process.

For Sealighter and The Threat-Intelligence Provider, I added a new feature to KDU to enable it to launch new processes as PPL-AntiMalware.

To achieve this, KDU first creates the process suspended usinh CREATE_SUSPENDED, so that while the process object in the kernel is created, the program hasn’t started to run. Then KDU uses a vulnerable driver to edit kernel memory, altering the process object in memory to change its protection type and signer to be PsProtectedTypeProtectedLight and PsProtectedSignerAntimalware, i.e. PPL-AntiMalware.

With this new feature, it was easy to then use KDU to run Sealighter and get Threat-Intelligence events. I simply created this Sealighter config:

{

  "session_properties": {
    "session_name": "Sealighter-Trace",
    "output_format": "stdout"
  },

  "user_traces": [
    {
        "trace_name": "TI-Trace",
        "provider_name": "Microsoft-Windows-Threat-Intelligence"
    }
  ]
}

Then I built and ran KDU like this:

kdu.exe -pse "C:\path\to\Sealighter.exe C:\path\to\config.json"

And just like that, we get Threat-Intelligence events from a fully patched, non-debug Windows environment:

As you can see from the picture, the machine is running the latest Windows 22H2, Secure Boot is enables, and Driver signing using WHQL is enforced. Despite all that, KDU was able to load an old version of the legitimate MSI Afterburner RTCore driver, which is vulnerable to the exploit CVE-2019-16098 . It used this driver and the exploit to edit the EPROCESS->PS_PROTECTION object in the kernel’s memory to set the protection level of the process to be PPL-AntiMalware.

Once KDU does it’s job, the Sealighter Process is un-suspended, and it can now subscribe to and log events from Microsoft-Windows-Threat-Intelligence.

Conclusion

This blog doesn’t cover anything particularly new but does demonstrate yet another way to explore PPL Processes and the Threat-Intelligence ETW Provider. Vulnerable drivers will always be a security problem, but for both defensive and offensive research they also provide an excellent resource to learn more about Windows.

My additions to KDU are now merged into the main branch, so you can explore creating PPL Processes yourself by cloning the latest version from GitHub

Thanks go to hfiref0x for KDU, to lean and explore some more, I recommend you check out:

Linux cloud memory forensics tutorial

2022-08-22T12:00:00+00:00

tl;dr

This is a quick and simple guide to getting and analysing volatile memory dumps from Linux VMs on Azure, AWS, and other cloud providers.

tl;dr
Volatility and Linux
A tale of two halves
1. A Memory capture
2. An Intermediate Symbol File (ISF)
- Auto-Generate ISF
- Manually Generating
Using ISF
Conclusion

Volatility and Linux

Volatility is the main open source tool to forensicically analyse volatile memory captures. But using Volatility to analyse Linux images can be confusing, especially as the process changed massivly from Volatility 2 to 3, so I thought I’d write up a quick guide explaining how to go about getting and analysing memory captures of Linux Virtual Machines (VMs) when they are hosted on a cloud provider (Azure, AWS, etc.)

A tale of two halves

There are two components needed to analyse a Linux memory Image, A volatile memory capture and an Intermediate Symbol File, or ISF.

1. A Memory capture

The memory capture will be taken from the machine you wish to analyse. The easiest way to do this is using AVML from Microsoft. This script will download the latest version of avml, and write a compressed memory capture to disk as capture.lime.compressed:

wget https://github.com/microsoft/avml/releases/latest/download/avml
# OR if wget is not installed:
curl -LO https://github.com/microsoft/avml/releases/latest/download/avml

chmod ugo+x ./avml
sudo ./avml --compress capture.lime.compressed

I don’t do the analysis on the same machine, but instead transfer the image to my workstation for further analysis. On some version of Linux and Volatility I encounter an issue analysing compressed images, where the program hangs indefinetly. I work around this by using avml-convert to decompress the image on my workstation first:

wget https://github.com/microsoft/avml/releases/latest/download/avml-convert
chmod u+x ./avml-convert
./avml-convert capture.lime.compressed capture.lime

2. An Intermediate Symbol File (ISF)

This is the part that has changed the most from Volatility 2, and is the part that initially confuses most people (including me).

You need to generate a special JSON file that contains the symbols from a matching Linux kernel. This file is called an Intermediate Symbol File or ISF. If they don’t match, Volatility won’t let you analyse the image, as it need to be able to precisly say where to look in the memory capture. Even very minor version differences can move the location of kernel objects needed for the analysis, so it’s important to generate a matching ISF for each memory capture.

There are lists of pre-generated ISF files, such as KevTheHermits’s excelent site. But as you kernel has to match exactly with the ISF file, it’s possible that you may need to generate your own.

Auto-Generate ISF

Thankfully, KevTheHermit has also created a super useful tool to generate new ISFs. It can create ISF files for:

Ubuntu
Debian
Fedora
AWS’ Amazon Linux 2
Azure’s CBL-Mariner 2

To use KevTheHermit’s tool, first run this command on the target image, to get the exact kernel version:

uname -r
# output should be something like:
# "5.11.0-43-generic" or
# "4.14.281-212.502.amzn2.x86_64"

Then run the tool, selecting the Linux distribution and passing in the output from uname. e.g. to get a CBL-Mariner ISF:

python symbol_maker.py --distro cbl-mariner --kernel "5.15.48.1-2.cm2"

To generate an ISF for Debian or Ubuntu, use --branch to tell the tool what cloud service the image is on, either:

linux If a non-cloud image
linux-aws If on AWS
linux-azure If on Azure
linux-gcp If on Google Cloud Compute

e.g. to generate an ISF for an Ubuntu machine running on AWS:

python symbol_maker.py --distro ubuntu --branch linux-aws --kernel "5.13.0-1031-aws"

If generating an Amazon Linux ISF, use --branch 2 for Amazon Linux 2 (the only version supported).

The tool will take a while as it downloads about 1GB of data, before outputting the ISF json file to disk in the symbol_files// folder.

Manually Generating

Unfortunetly, sometimes even KevTheHermit’s tool won’t be able to get exactly the right kernel. Sometimes if the ‘banner’ (the string that identifies the exact kernel version) is extremely similar (e.g. only the build timestamp is different), you can edit the Banner in the ISF JSON file to match your image. But typically, if KevTheHermit’s tool can’t get the same kernel, it means you need to manually generate your own ISF.

The advantage of the cloud is that VMs should be running from a known and re-usable base image. I reccomend creating a new-but-matching VM to the one you caputred memory from, so you don’t taint the machine with uneeded data.

In AWS, the ID of the image is the AMI, and you can list the AMI ID of a running VM using either the Web UI, or the aws commandline:

# Look for "ImageId"
aws ec2 describe-instances --instance-ids i-1234567890abcdef0

You can then use this exact AMI ID to create a new machine that should have a matching kernel. Other cloud providers have different ways of getting the base Image ID. You don’t need a particularly powerful VM (my tests were on a 1CPU 2GB RAM), but you will need ~15GB of disk space.

To generate an ISF, you need two files:

System.map
A debug version of the current Linux kernel

Most linux distributions should have the System.Map file under the /boot/ directory with a name that matches the version of the kenrel, i.e.:

/boot/System.map-$(uname -r)

To download a debug kernel, the process is different depending on the Linux distribution. I’ve captured the most common ones below:

Ubuntu:

(Tested on AWS, Azure, and DigitalOcean)

sudo apt update
sudo apt install --yes ubuntu-dbgsym-keyring
sudo tee /etc/apt/sources.list.d/debug.list << EOF
deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse
EOF

sudo apt update
sudo apt install --yes linux-image-$(uname -r)-dbgsym
# Debug kernel is at: /usr/lib/debug/boot/vmlinux-$(uname -r)

Debian:

(Tested on AWS)

sudo tee /etc/apt/sources.list.d/debug.list << EOF
deb http://deb.debian.org/debian-debug/ $(lsb_release -cs)-debug main
deb http://deb.debian.org/debian-debug/ $(lsb_release -cs)-proposed-updates-debug main
EOF

sudo apt update
sudo apt install --yes linux-image-$(uname -r)-dbg
# Debug kernel is at: /usr/lib/debug/boot/vmlinux-$(uname -r)

Centos8, Fedora, Amazon Linux:

(Tested on AWS, should work for other RHEL-like distros)

sudo yum --enablerepo='*-debuginfo' install kernel-debuginfo-$(uname -r)
# Debug kernel is at: /usr/lib/debug/lib/modules/$(uname -r)/vmlinux

Azure CBL-Mariner:

sudo yum install mariner-repos-debug
sudo yum install kernel-debuginfo-$(uname -r)
# Debug kernel is at: /usr/lib/debug/lib/modules/$(uname -r)/vmlinux

Generating an ISF

Once I have a debug kernel, I transfer it from the VM to my workstation. You can in theory do this next step on the VM, but it requires a machine with 4GB+ of memory and a decent CPU, and I typically just run tiny VMs in the cloud and do bulky things on my workstation.

Build a copy of Dwarf2JSON, a tool from the Volatility team. If you don’t already have Go installed, you can grab a pre-built version by me on GiHub here. Then run dwarf2JSON to generate the ISF, pointing it at the debug kernel you downloaded from the VM:

wget https://github.com/pathtofile/dwarf2json/releases/latest/download/dwarf2json-linux-amd64 -O dwarf2json
chmod u+x dwarf2json
./dwarf2json linux --system-map  --elf  > dbgkernel_aws_ubuntu.json

Using ISF

Once you’ve generated the ISF JSON file, Clone the Volatility3 repo and install the Python3 dependecies:

git clone https://github.com/volatilityfoundation/volatility3.git
cd volatility3
pip install -r requirements.txt

Next copy the ISF JSON file to volatility3/volatility3/framework/symbols/linux/. Run this command to check that Volatility found your ISF file and was able to parse the banner text:

$> python vol.py isfinfo.IsfInfo
Volatility 3 Framework 2.3.0
URI     Valid   Number of base_types    Number of types Number of symbols       Number of enums Windows info    Linux banner    Mac banner
file:///home/path/code/volatility/volatility3/volatility3/framework/symbols/linux/dbgkernel_aws_ubuntu.json     Unknown 19      11925   199625  2110    -       Linux version 5.15.0-1015-aws (buildd@lcy02-amd64-063) (gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #19~20.04.1-Ubuntu SMP Wed Jun 22 19:07:51 UTC 2022 (Ubuntu 5.15.0-1015.19~20.04.1-aws 5.15.39)
....

You can also check the banner in your memory capture by running:

$> python vol.py -f  banners.Banners
Volatility 3 Framework 2.3.0
Progress:  100.00               Stacking attempts finished
Offset  Banner
0x5da8f38       Linux version 5.15.0-1015-aws (buildd@lcy02-amd64-063) (gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #19~20.04.1-Ubuntu SMP Wed Jun 22 19:07:51 UTC 2022 (Ubuntu 5.15.0-1015.19~20.04.1-aws 5.15.39)
...

If they match up, then you can start using Volatility to analyse the image. For example, to list the processes running at the time of catprue:

$> python vol.py -f  linux.pslist
Volatility 3 Framework 2.3.0
Progress:  100.00               Stacking attempts finished
OFFSET (V)      PID     TID     PPID    COMM

0x9ea501284b00  1       1       0       systemd
0x9ea501281900  2       2       0       kthreadd
0x9ea501280000  3       3       2       rcu_gp
...

Conclusion

Capturing and analysing memory from Linux machines does take a few steps, however hopefully this blog clears up some of the confusion about Volatility 3 and Linux, and provides a useful guide to getting and analysing cloud images.

Tool - Use TouchID and the Secure Enclave from the commandline

2022-08-21T12:00:00+00:00

I just released Toucli, a simple tool to use TouchID and the Secure Enclave to encrypt data from the commandline.

See the details in the README for more information about how it works and how to use it.

Commandline Cloaking 2 - Tetragon and Nim

2022-08-04T12:00:00+00:00

tl;dr

I ported my Commandline Cloaking tests to Nim, and used it to explore Tetragon, the latest eBPF security observability tool.

tl;dr
Tetragon
Nim
Lab setup
Tests
Conclusions and final thoughts
- Writing Nim
- Using Tetragon
Example Code

Tetragon

Recently Isovalent, the company behind the eBPF-based network monitoring and security program Cilium released Tetragon, an open source “eBPF-based security observability and runtime enforcement platform”.

Isovalent has many very talented eBPF experts, so I thought it would be neat to compare Tetragon to the same Commandline Cloaking techniques I tested Sysmon for Linux with, another eBPF-based security tool.

Tetragon can do a lot more than just observe commandlines, but for this blog I will mostly stick to just commandline detections.

Nim

As well as looking into Tetragon, I also checked out Nim, a statically typed programming language that has been growing in popularity over the last few years. As a compiled system language, it appears to be striking a balance between lower-level systems access and C-interoperability with higher-level conveniences like iterators and a rich standard library.

To me, it fits in a similar place as Go, without the large library ecosystem (due to its popularity Vs google-backed Go), but also without some of the baggage that comes with Go, such as the super-thick static binaries.

As I don’t work as a low-level software engineer I’m not the right person to provide a full evaluation of Nim, but I thought I’d share my thoughts on porting my Commandline Cloaking toy programs from C/Go.

Lab setup

Install the Nim compiler was as simple as following this guide:

curl https://nim-lang.org/choosenim/init.sh -sSf | sh

Tetragon is mostly designed to be run as part of a Kubernetes cluster. I just wanted to run Tetragon by itself (and didn’t have a Kubernetes cluster already setup for testing), thankfully, Djalal Harouni provided this method to run Tetragon as a standalone docker container:

# 1. Run Tetragon in privliged Docker, mounting required folders
docker run --name tetragon \
--rm -it --pid=host --cgroupns=host \
--privileged --detach \
-v /sys/kernel/btf/vmlinux:/var/lib/tetragon/btf \
quay.io/cilium/tetragon:v0.8.0 \
bash -c "/usr/bin/tetragon"

#2. Get events in pretty format:
docker exec -t tetragon bash -c "/usr/bin/tetra getevents -o compact"
#   OR get full JSON of process ecex events:
docker exec -t tetragon bash -c "/usr/bin/tetra getevents" | jq 'select(.process_exec)'

# When finished:
docker rm -f tetragon

Tests

For more information on each of the test programs I created and ran, see my previous blog where I go into more detail.

Test #0 - No cloaking

This program doesn’t do any shenanigans, it just prints some basic commandline information to the screen.

Nim

This was understandably very straightforward to port to Nim:

import os
import strformat
import posix

proc main() =
    echo(fmt"  PID     {getpid()}")
    echo(fmt"  PPID    {getppid()}")
    echo(fmt"  argc    {paramCount() + 1}")  # Nim doesn't count argv[0] in paramCount

    for i in 0..paramCount():
        echo(fmt"  argv[{i}] {paramStr(i)}")

    echo("  Sleeping for 60 seconds so you can lookup the PID")
    setControlCHook(proc() {.noconv.} = discard)
    sleep(60 * 1000)

main()

It is odd that the Nim built-in paramCount() isn’t exactly like C’s argc in that it doesn’t count argv[0] (which is usually the program name), but accessing arguments with paramStr() does include the 0th arg.

Tetragon

Running ./basic_nim AAAA produces a predictable output from Tetragon:

{
  "process_exec": {
    "process": {
      "exec_id": "Ojk1OTg2MDUwNTAyNzU3OjcwMDE=",
      "pid": 7001,
      "uid": 1000,
      "cwd": "/home/path/code/commandline_cloaking/bin/",
      "binary": "/home/path/code/commandline_cloaking/bin/basic_nim",
      "arguments": "AAAA",
      "flags": "execve clone",
      "start_time": "2022-06-16T03:33:18.153Z",
      "auid": 1000,
      "parent_exec_id": "Ojk0OTI0ODgwMDAwMDAwOjU4NzM=",
      "refcnt": 1
    }
  },
  "time": "2022-06-16T03:33:18.153Z"
}

Tetragon helpfully captures the full commandline, as well as other useful information such as UserID, start time, parent process etc. Tetragon maintains its own unique ID system, so you can link and track child and parent processes, even in the event of PID reuse.

Test #1 - Post-execution cloaking

This test runs a few tricks to modify it’s commandline arguments after it executes, to hide from tools such as ps. This includes:

Editing our /proc/pid/cmdline by editing our argv array
Using prctl and PR_SET_NAME to change the process name in /proc/pid/comm
Forking twice and killing parents, which sets the process’ Parent PID to PID 1

Nim

Writing this in Nim was also quite straightforward. To use syscalls such as prctl, it was as simple as:

# Declare the c definitions in Nim 
proc syscall(number: clong): clong {.importc, varargs, header: "sys/syscall.h".}
var NR_PRCTL {.importc: "__NR_prctl", header: "unistd.h".}: int
var PR_SET_NAME {.importc: "PR_SET_NAME", header: "sys/prctl.h".}: int

# Call syscall function to invoke prctl to set our /proc/self/comm
var err = syscall(NR_PRCTL, PR_SET_NAME, cstring("faked"))

This was the same for memset, which doesn’t have a Nim equivalent in the standard lib, but was super easy to ‘import’ and call the C version.

To edit the raw argv array, I did have to ‘turn off’ some of Nim’s magic, as the built-in argument array paramStr() isn’t the actual memory address of /proc/self/cmdline, but the converted-to-Nim string version, and therefore changing that doesn’t actually cloak the commandline from ps.

Working around this was a simple as compiling the program with the --nomain flag, then implementing the Nim startup process ourselves:

proc main(argc: int, argv: cstringArray, envp: cstringArray): int {.cdecl, exportc.} =
    # Need to call NimMain ourselves first to avoid explosions
    NimMain()

    # Do stuff with argv, etc. ...
    discard memset(argv[0], ord('F'), csize_t(len(argv[0])))

For a reason I’ll explain below, once this dodgy_nim program did it’s double-fork-and-kill trick to make its parent PID 1, I then made it call execve to run the basic_nim program.

Tetragon

To test, I ran dodgy_nim like this:

$> ./dodgy_nim AAAA
-------- REAL --------
  PID     8883
  PPID    5873
  argc    2
  argv[0] ./dodgy_nim
  argv[1] AAAA
---- FORK & FAKE -----
  PID     8885
  PPID    1
  argc    2
  argv[0] FFFFFFFFFFF
  argv[1] BBBB
---- EXECVE BASIC ----
  PID     8885
  PPID    1
  argc    2
  argv[0] from_dodgy
  argv[1] CCCC
----------------------

# Check ps output
$> ps aux | grep 7613
path        7613  0.0  0.0   3468   196 pts/1    S    14:02   0:00 FFFFFFFFFFF BBBB

Like Sysmon for Linux, Tetragon logs the process information before it runs, which means the original commandline that the process was started with.

But as you can see below, things do get a little confusion, as it marks the original process as exited, and when basic starts it states its parent ID PID 1:

{
  "process_exec": {
    "process": {
      "exec_id": "Ojk4NDUxNDQ0MTgyMTYxOjg4ODM=",
      "pid": 8883,
      "uid": 1000,
      "cwd": "/home/path/code/commandline_cloaking/bin/",
      "binary": "/home/path/code/commandline_cloaking/bin/dodgy_nim",
      "arguments": "AAAA",
      "flags": "execve clone",
      "start_time": "2022-06-16T04:14:23.547Z",
      "auid": 1000,
      "parent_exec_id": "Ojk0OTI0ODgwMDAwMDAwOjU4NzM=",
      "refcnt": 1
    },
    "parent": {
      "exec_id": "Ojk0OTI0ODgwMDAwMDAwOjU4NzM=",
      "pid": 5873,
      "uid": 0,
      "cwd": "/home/path/code/commandline_cloaking/bin",
      "binary": "/usr/bin/bash",
      "flags": "procFS auid",
      "start_time": "2022-06-16T03:15:36.983Z",
      "auid": 0,
      "parent_exec_id": "Ojk0OTI0NzcwMDAwMDAwOjU4NzI=",
      "refcnt": 4294967281
    }
  },
  "time": "2022-06-16T04:14:23.547Z"
}
{
  "process_exit": {
    "process": {
      "exec_id": "Ojk4NDUxNDQ0MTgyMTYxOjg4ODM=",
      "pid": 8883,
      "uid": 1000,
      "cwd": "/home/path/code/commandline_cloaking/bin/",
      "binary": "/home/path/code/commandline_cloaking/bin/dodgy_nim",
      "arguments": "AAAA",
      "flags": "execve clone",
      "start_time": "2022-06-16T04:14:23.547Z",
      "auid": 1000,
      "parent_exec_id": "Ojk0OTI0ODgwMDAwMDAwOjU4NzM="
    }
  },
  "time": "2022-06-16T04:14:23.548Z"
}
{
  "process_exec": {
    "process": {
      "exec_id": "Ojk4NDUyNDQ1NDI1MTg2Ojg4ODU=",
      "pid": 8885,
      "uid": 1000,
      "cwd": "/home/path/code/commandline_cloaking/bin/",
      "binary": "/home/path/code/commandline_cloaking/bin/basic_nim",
      "arguments": "CCCC",
      "flags": "execve clone",
      "start_time": "2022-06-16T04:14:24.548Z",
      "auid": 1000,
      "parent_exec_id": "OjQzMDAwMDAwMDox",
      "refcnt": 1
    },
    "parent": {
      "exec_id": "OjQzMDAwMDAwMDox",
      "pid": 1,
      "uid": 0,
      "cwd": "/",
      "binary": "/usr/lib/systemd/systemd",
      "arguments": "auto noprompt splash",
      "flags": "procFS auid rootcwd",
      "start_time": "2022-06-15T00:53:32.533Z",
      "auid": 0,
      "refcnt": 1
    }
  },
  "time": "2022-06-16T04:14:24.548Z"
}

This is all technically correct from the kernel’s point of view, but it does show that the double-fork technique is enough to break the commandline chain for analysts. I hadn’t checked this with Sysmon for Linux in my previous blog, so I went back and tried and it also failed to properly link dodgy to basic.

Tests #2 and #3 - In-memory loader and Piping input

This test uses memfd_create to create an in-memory file, writing an ELF program it gets from stdin, before running it with execve.

Nim

Again, thanks to being super easy to call libC functions, this was also straightforward in Nim:

# Define from C
proc memfd_create(name: cstring, flags: cuint): cint {.importc, header: "sys/mman.h".}
proc execve(pathname: cstring, argv: cstringArray, envp: cstringArray): cint {.importc: "execve", header: "stdlib.h".}

# Create in-memory file
let fd = memfd_create("", 0)

# Read from stdin
let data = readAll(stdin)

# Write data to in-memory file
var fwrite = open(fp, fmWrite)
fwrite.write(data)
fwrite.close()

# Create new argv
var args: seq[string]
args.add("from_loader")
for i in 2..paramCount():
    args.add(paramStr(i))
let argv = alloccstringArray(args)

# Exec program
discard execve(cstring(fmt"/proc/self/fd/{fd}"), argv, nil)

Tetragon

I ran the loader like this, to read and launch basic_nim:

$> cat ./bin/basic_nim | ./bin/loader_nim -

Tetragon records both programs executing, and links them together using exec_id:

{
  "process_exec": {
    "process": {
      "exec_id": "Ojk5OTE4Nzc1NzkwNDY1OjEwMzkw",
      "pid": 10390,
      "uid": 1000,
      "cwd": "/home/path/code/commandline_cloaking/bin/",
      "binary": "/home/path/code/commandline_cloaking/bin/loader_nim",
      "arguments": "-",
      "flags": "execve clone",
      "start_time": "2022-06-16T04:38:50.879Z",
      "auid": 1000,
      "parent_exec_id": "Ojk0OTI0ODgwMDAwMDAwOjU4NzM=",
      "refcnt": 1
    },
    "parent": {
      "exec_id": "Ojk0OTI0ODgwMDAwMDAwOjU4NzM=",
      "pid": 5873,
      "uid": 0,
      "cwd": "/home/path/code/commandline_cloaking/bin",
      "binary": "/usr/bin/bash",
      "flags": "procFS auid",
      "start_time": "2022-06-16T03:15:36.983Z",
      "auid": 0,
      "parent_exec_id": "Ojk0OTI0NzcwMDAwMDAwOjU4NzI=",
      "refcnt": 4294967293
    }
  },
  "time": "2022-06-16T04:38:50.879Z"
}
{
  "process_exec": {
    "process": {
      "exec_id": "Ojk5OTE4Nzc2Nzc1MjMzOjEwMzkw",
      "pid": 10390,
      "uid": 1000,
      "cwd": "/home/path/code/commandline_cloaking/bin/",
      "binary": "/proc/self/fd/3",
      "flags": "execve",
      "start_time": "2022-06-16T04:38:50.880Z",
      "auid": 1000,
      "parent_exec_id": "Ojk5OTE4Nzc1NzkwNDY1OjEwMzkw",
      "refcnt": 1
    },
    "parent": {
      "exec_id": "Ojk5OTE4Nzc1NzkwNDY1OjEwMzkw",
      "pid": 10390,
      "uid": 1000,
      "cwd": "/home/path/code/commandline_cloaking/bin/",
      "binary": "/home/path/code/commandline_cloaking/bin/loader_nim",
      "arguments": "-",
      "flags": "execve clone",
      "start_time": "2022-06-16T04:38:50.879Z",
      "auid": 1000,
      "parent_exec_id": "Ojk0OTI0ODgwMDAwMDAwOjU4NzM=",
      "refcnt": 1
    }
  },
  "time": "2022-06-16T04:38:50.880Z"
}

While Tetragon doesn’t show the 2nd process as being an in-memory file, the binary path /proc/self/fd/3 is probably the same level of suspiciousness. We don’t know the in-memory file as the basic program, and while we do see the cat executing, with in-memory file technique the file could have been read from a veriety of sources that don’t leave a commandline, such as remote webserver or hardcoded within the same binary.

Test #4 - LD_PRELOAD

One common technique used by malware to hide is to make use of LD_PRELOAD and similar environment variables.

This test uses LD_PRELOAD to hook the libc function in between the programs initial execution and it’s main() function, altering the input commandline arguments.

Nim

Like in C, this was more straightforward than Go.

# Declare function type matching hooked function signature
type
  LibCMain = proc (
    main: pointer, argc: cint, argv: cstringArray,
    init: pointer, fini: pointer, rtld_fini: pointer
    ): int {.cdecl}

# This hooks __libc_start_main
proc hookedLibcMain(
    main: pointer, argc: cint, argv: cstringArray,
    init: pointer, fini: pointer, rtld_fini: pointer
): int {.cdecl, exportc:"__libc_start_main", dynlib} =
    # Find original function
    let lib = loadLib("libc.so.6")
    assert lib != nil, "Error loading library"
    let origFunc = cast[LibCMain](lib.symAddr("__libc_start_main"))
    assert origFunc != nil, "Error loading function from library"

    # Create new argv
    let newArgc = cint(2)
    var newArgv = alloccstringArray(["from_preload", "BBBB"])
    
    # Call into real function
    return origFunc(main, newArgc, newArgv, init, fini, rtld_fini)

I had to use the .exportc declaration as Nim doesn’t allow functions to start with an _. To use LD_PRELOAD on non-underscore functions you simply have the name the function in Nim the same as the function to be hooked.

Tetragon

I ran the program like this, hooking and replacing the commandline args to basic:

$> LD_PRELOAD=./preload_nim.so ./basic_nim AAAA
  PID     11292
  PPID    5873
  argc    2
  argv[0] from_preload
  argv[1] BBBB
  Sleeping for 60 seconds, so you can look up the PID

Just like Sysmon, it records the arguments the program was launched with, which is technically correct again, but not effectivley happened:

{
  "process_exec": {
    "process": {
      "exec_id": "OjEwMTAwMzk4Mzc5ODg1NToxMTI5Mg==",
      "pid": 11292,
      "uid": 1000,
      "cwd": "/home/path/code/commandline_cloaking/bin/",
      "binary": "/home/path/code/commandline_cloaking/bin/basic_nim",
      "arguments": "AAAA",
      "flags": "execve clone",
      "start_time": "2022-06-16T04:56:56.087Z",
      "auid": 1000,
      "parent_exec_id": "Ojk0OTI0ODgwMDAwMDAwOjU4NzM=",
      "refcnt": 1
    }
  },
  "time": "2022-06-16T04:56:56.087Z"
}

Unfortunately, Tetragon doesn’t log the environment variables a program was launched with, but if it did so, you should be able to see the LD_PRELOAD, which is not commonly used in production environments, and will be pointing to the malicious library.

Test #5 - In-memory patching

Using Silvio’s Parasite technique, we can patch a binary to edit its arguments without LD_PRELOAD. For more information, see this section in the previous blog.

Nim

I didn’t implement this in Nim due to time and motivational constraints.

Tetragon

When patching and loading a binary in-memory, the output is similar to the LD_PRELOAD test.

A twist on the Parasite technique is that the program could write the patch binary back to disk (overwriting the original file), run the on-disk binary, then replace it with the original binary once the program has run.

This would mean the binary is no longer run from in-memory. Tetragon can monitor file access (and more) by supplying custom TracingPolicies, so it could watch of any process writing to e.g. /bin/ to detect the tampering. I ran Tetragon with this policy to look for writes to files under /bin/:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "sys-write-follow-fd-bin"
spec:
  kprobes:
  - call: "fd_install"
    syscall: false
    args:
    - index: 0
      type: int
    - index: 1
      type: "file"
    selectors:
    - matchArgs:
      - index: 1
        operator: "Prefix"
        values:
        - "bin"
      matchActions:
      - action: FollowFD
        argFd: 0
        argName: 1
  - call: "__x64_sys_write"
    syscall: true
    args:
    - index: 0
      type: "fd"
    - index: 1
      type: "char_buf"
      sizeArgIndex: 3
    - index: 2
      type: "size_t"

To run Tetragon with a custom policy policy_file.yml in the current directory, run:

# 1. Start Tetragon with a custom policy file:
docker run --name tetragon \
--rm -it --pid=host --cgroupns=host \
--privileged \
-v /sys/kernel/btf/vmlinux:/var/lib/tetragon/btf \
-v `pwd`:/policies \
quay.io/cilium/tetragon:v0.8.0 \
bash -c "/usr/bin/tetragon --config-file /policies/policy_file.yml"

# 2. Get events in pretty format:
docker exec -t tetragon bash -c "/usr/bin/tetra getevents -o compact"
#   OR get full JSON of process kprobe events:
docker exec -t tetragon bash -c "/usr/bin/tetra getevents" | jq 'select(.process_kprobe)'

# Cleanup when finished:
docker rm -f tetragon

Which when I overwrote /bin/bash using Python I got this event:

{
  "process_kprobe": {
    "process": {
      "exec_id": "OjEwNDg1OTU2ODY0NjQ2MDoxNTQzMQ==",
      "pid": 15431,
      "uid": 1000,
      "cwd": "/home/path/code/commandline_cloaking/",
      "binary": "/usr/bin/python",
      "arguments": "./overwrite_bash.py",
      "flags": "execve clone",
      "start_time": "2022-06-16T06:01:11.671Z",
      "auid": 1000,
      "parent_exec_id": "Ojk0OTI0ODgwMDAwMDAwOjU4NzM=",
      "refcnt": 1
    },
    "function_name": "__x64_sys_write",
    "args": [
      {
        "file_arg": {
          "path": "/bin/bash"
        }
      },
      {
        "truncated_bytes_arg": {
          "bytes_arg": "f0VMRgIBAQAAAAAAAAAAAAMAPgABAAAAgBIAAAAAAABAAAAAAAAAAAhqAgAAAAAAAAAAAEAAOAANAEAAIAAfAAYAAAAEAAAAQAAAAAAAAABAAAAAAAAAAEAAAAAAAAAA2AIAAAAAAADYAgAAAAAAAAgAAAAAAAAAAwAAAAQAAAAYAwAAAAAAABgDAAAAAAAAGAMAAAAAAAAcAAAAAAAAABwAAAAAAAAAAQAAAAAAAAABAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMgKAAAAAAAAyAoAAAAAAAAAEAAAAAAAAAEAAAAFAAAAABAAAAAAAAAAEAAAAAAAAAAQAAAAAAAA+Z0BAAAAAAD5nQEAAAAAAAAQAAAAAAAAAQAAAAQAAAAAsAEAAAAAAACwAQAAAAAAALABAAAAAABUVAAAAAAAAFRUAAAAAAAAABAAAAAAAAABAAAABgAAACgNAgAAAAAAKB0CAAAAAAAoHQIAAAAAADgDAAAAAAAAIBwBAAAAAAAAEAAAAAAAAAIAAAAGAAAAQA0CAAAAAABAHQIAAAAAAEAdAgAAAAAA8AEAAAAAAADwAQAAAAAAAAgAAAAAAAAABAAAAAQAAAA4AwAAAAAAADgDAAAAAAAAOAMAAAAAAAAwAAAAAAAAADAAAAAAAAAACAAAAAAAAAAEAAAABAAAAA==",
          "orig_size": "160264"
        }
      },
      {
        "size_arg": "160264"
      }
    ],
    "action": "KPROBE_ACTION_POST"
  },
  "time": "2022-06-16T06:01:11.677Z"
}

The bytes_arg is indeed the first 520 bytes of the ELF I overrode bash with.

I found creating these custom TracingPolicies to require experienced knowledge of eBPF, Tetragon, and the Linux kernel. But it is possible to detect this technique. Sysmon for Linux also has a similar ability to track file writes as one of its default event types, although I didn’t test it out.

Test #6 - Chroot dorking

While reading Liz Rice’s Excellent book on container security, I started thinking about another commandline cloaking method using chroot jails.

chroot is a system that allows you to change what a process sees as the ‘root’ (/) folder. This allows a level of isolation and abstraction from the ‘real’ files on the host, and is a key concept in how containers work. For example, say a machine has this filesystem:

/
├── bin
│   ├── sh
│   ├── python
│   └── bash
├── home
│   └── user
│       └── fake_root
│           ├── bin
|           |    └── bash
|           └── fake_tmp
├── proc
└── tmp

If we run a program that calls chroot to set the root to /home/user/fake_root, it will see the filesystem like this:

/
├── bin
│   └── bash
└── fake_tmp

When the program attempts to run /bin/bash, it will actually launch /home/user/fake_root/bin/bash. This is how containers work (well part of how they work, I can’t recommend Liz Rice’s book highly enough for a more in-depth explanation).

When you run /bin/bash inside a docker container, on the host machine, it is actually running something like /var/lib/docker/overlay2//merged/bin/bash.

As a result, using chroot also has implications for Tetragon and commandline monitors.

To test how chroot can affect commandline monitors like Tetragon, I created this Nim program:

proc main() =
    if getuid() != 0:
        echo("Need to run as root (technically just CAP_SYS_CHROOT but I'm lazy)")
        quit(1)

    # Create fake root and /bin folder with 
    createDir("fake_root")
    createDir("fake_root/bin")
    
    # Create a link from a dodgy program to fake_root/bin/bash
    link("/path/to/dodgy_nim", "fake_root/bin/bash")

    # Move into fake root and call chroot
    setCurrentDir("fake_root")
    chroot(".")

    # Run fake bin bash, which is now at /bin/bash
    execve("/bin/bash", nil, nil)

There are a few tricks to getting this program to work:

The program needs the CAP_SYS_CHROOT capability, which usually means running it at as root
The faked bash program either needs to be statically compiled, or you have to also copy in all the libraries into the fake_root

But given those constraints, Tetragon’s logging records the activity like this:

{
  "process_exec": {
    "process": {
      "exec_id": "OjI5Mzc2MTgxOTk3NTcxMzoyMTgzNg==",
      "pid": 21836,
      "uid": 0,
      "cwd": "/path/to/fake_root/",
      "binary": "/bin/bash",
      "arguments": "",
      "flags": "execve",
      "start_time": "2022-07-05T03:13:04.511Z",
      "auid": 1000,
      "parent_exec_id": "OjI5Mzc2MTgxOTI1MDIwNzoyMTgzNg==",
      "refcnt": 1
    },
    "parent": {
      "exec_id": "OjI5Mzc2MTgxOTI1MDIwNzoyMTgzNg==",
      "pid": 21836,
      "uid": 0,
      "cwd": "/path/to/fake_root/",
      "binary": "/path/to/chroot_parent",
      "arguments": "",
      "flags": "execve clone",
      "start_time": "2022-07-05T03:13:04.510Z",
      "auid": 1000,
      "parent_exec_id": "OjI5MzczOTUxNDI1NTM0MjoyMTc4Mw==",
      "refcnt": 1
    }
  },
  "time": "2022-07-05T03:13:04.511Z"
}

According to Tetragon, chroot_parent launched /bin/bash, which sounds like a completely benign activity. But this makes sense - Tetragon is designed to monitor containers as well as host applications, so recording the path as the ‘container’ sees it would help with observing and writing detections for containers.

Tetragon could also monitor and detect this type of suspicious chroot usage by hooking the chmod syscall. After some reading, I created this very basic TracingPolicy:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "sys-chroot"
spec:
  kprobes:
  # __x64_sys_chroot(const char *path)
  - call: "__x64_sys_chroot"
    syscall: true
    args:
      - index: 0
        type: "string"

Which produces this Tetragon event, highlighting the activity:

{
  "process_kprobe": {
    "process": {
      "exec_id": "OjE4MTExMjk2ODkwMTE6NTkxMA==",
      "pid": 5910,
      "uid": 0,
      "cwd": "/path/to/",
      "binary": "/path/to/chroot_parent",
      "arguments": "",
      "flags": "execve clone",
      "start_time": "2022-08-02T03:40:48.950Z",
      "auid": 1000,
      "parent_exec_id": "OjMwNTI3MDAwMDAwMDoyNDQx",
      "refcnt": 1
    },
    "parent": {
      "exec_id": "OjMwNTI3MDAwMDAwMDoyNDQx",
      "pid": 2441,
      "uid": 0,
      "cwd": "/path/to",
      "binary": "/usr/bin/bash",
      "flags": "procFS auid",
      "start_time": "2022-08-02T03:15:43.090Z",
      "auid": 0,
      "parent_exec_id": "OjMwNTI2MDAwMDAwMDoyNDQw",
      "refcnt": 1
    },
    "function_name": "__x64_sys_chroot",
    "args": [
      {
        "string_arg": "."
      }
    ],
    "action": "KPROBE_ACTION_POST"
  },
  "time": "2022-08-02T03:40:48.951Z"
}

Like the file monitor, this policy isn’t ‘on by default’ and requires someone to understand enough about Tetragon and the Linux kernel to write and share it. In the real world, you would want to edit the policy to also filter out known-good binaries or chroot folders, to reduce the amount of false positives.

Conclusions and final thoughts

Writing Nim

I thoroughly enjoyed writing code in Nim, and really liked that it was ‘low enough’ to make it easy to do things like syscalls and memory operations, while being ‘high enough’ to have nice things baked in like argument parsing, slices, and string manipulation.

I’m unsure how much I’ll keep writing in Nim as my main goal of programming nowadays is to share information, and that means sharing code in a language more people use.

Using Tetragon

Tetragon from a programmer’s perspective Tetragon is really cool, easy to run and attach Kprobes to observe all kinds of activity. I could see how you could create some really crafty TracingPolicies to catch all sort of unusual behaviour.

Like Sysmon for Windows and other observability tools, it doesn’t come with its own detections, which means it’s on the users and community to create and share new policies to detect and prevent suspicious behaviour. But writing new policies for Tetragon requires someone with close-to-expert knowledge and experience in Linux and Tetragon.

It reminded me a bit of AuditD rules, where there people who get the most out of AuditD are those whole deeply understand the Linux kernel, or who are lucky enough that someone else’s rules work in their environment without being flooded by false positives. But those without the luck or skills end up feeling frustrated, or simply just don’t use it, losing out on what should be a great source of secutity data.

I hope the Tetragon team can create a community of detection writers, as I feel without that many people won’t be able to get the most out of it.

A good addition to the work would be to be able to automatically translate Sigma rules to and from Tetragon TracePolicies. This would enable people to share detections in a common format beyond Tetragon, and convert rules they may be using with other tools and systems into Tetragon-specific policies.

Example Code

I’ve updated my Commandline Cloaking repository with all the Nim examples, the new Chroot example, as well as the Tetragon TracePolicies.

Tool - Using AWS Lambdas to distribute WebRequests

2022-07-21T12:00:00+00:00

At times I have the need to spread out multiple web requets across many IPs or Geographic locations.

To do this I’ve been using Terraform, AWS, And Python templating to auto create and run Lambda Functions to handle the requests for me, sending the data back to an AWS SQS Queue to read asynchornously.

I recently got around to cleaning up my code and making it public, which you can find here: https://github.com/pathtofile/aws-lambda-webrunner.

This is far from the first system to do this, but perhaps the code may prove useful to others.

SIEMCraft - Security detection monitoring using Minecraft

2022-06-10T12:00:00+00:00

tl;dr

I spent way too much time creating a Security Information and Event Monitoring experience (SIEM) in Minecraft code, demo video. The process was actually more complex and fun than I was expecting it to be, and could serve as a blueprint to understanding the detection lifecycle end-to-end.

tl;dr
From boredom, big things grow
Bursting at the SIEM
1. Collecting raw events
2. Generating Detections
3. Send detects into Minecraft
4. Allow players to take action
- PlayerMessage
Shaving the GameTest
Finishing SIEMCraft

From boredom, big things grow

At the start of the year, I found myself on PTO without the ability to travel to the place I had booked the PTO for.

With my kiddo in daycare, I had nothing of importance to do, so while idly browsing the internet I came across the awesome KubeCraftAdmin by Eric Jadi, a project that allows you to manage Kubernetes clusters from Minecraft. To be clear, Eric had put the correct amount of effort into KubeCraftAdmin, making it functional and portable enough for the lighthearted project that it was.

But for me, with more time than sense, noticed a few ‘rough edges’ to the project that bugged me, namely that while it could send data from Kubernetes into the Minecraft server, it couldn’t receive and data back from the Minecraft player, at least not directly. Again for a project like Eric’s, this is fine, but I had nothing but time to devote to pedantic things. And besides, it couldn’t be that hard, right? right….?

Bursting at the SIEM

With the Kubernetes controller idea already being taken, I wanted to look for another project with the same basic idea of ‘controlling something unusual from Minecraft’. Back in 2014, I had created a controller in Minecraft for Metasploit, although the code has since been lost to time, and was written in LUA for a now very outdated Minecraft version.

So instead I turned from red to blue and decided to write a Security Information and Event Monitoring system, aka a SIEM. But I wanted to write the project end-to-end, from raw events to detection, visualisation, and human interaction. So I came up with the following requirements, putting on my (very ill-fitting) Business Analyst hat (TM) that I used to wear many years ago:

Requirement #1: Must have the ability to collect raw events

Business people tend to focus on CEOs, so I decided to focus solely on Windows, as the only CEO I know personally that uses Linux just rants about parasites all day. I also chose to only focus on Process Creation events because I’m lazy. However, as a stretch goal, I wanted to be able to collect raw events from an entire domain of Machines, not just a single box.

Requirement #2: Must generate detections based on raw events

Displaying a firehose of raw events isn’t very useful to anyone, so I wanted to be able to write detections based on the raw events, and only alert the User/Player if something suspicious is detected.

Requirement #3: Must display detections to player

Pretty obvious, Player must be able to see the detection, be able to tell the difference between a high- and low-severity detection, and be able to see all relevant process information to help them decide if it is a True or False Positive.

Requirement #4: Must allow a player to take action

I thought it’d be hilarious to make it so that a player can kill the offending process from Minecraft if they detected it was a True Positive. Spoiler for the rest of the blog, this turned out to be way more complex than I expected it to be.

1. Collecting raw events

This turned out to be the easiest of steps. In the real world, the best free solution would be to use Sysmon for Windows, which generates events about Process activity and writes these events to the Windows Event log.

But to truly create an end-to-end solution, I made my own event collector (or really used an event collector I already made) called Sealighter. Sealighter uses ETW to collect raw information from Windows, so I used this basic Sealighter config to collect ‘Process Start’ events from the Kernel ETW Provider and send the events to the Windows Event Log as a JSON object:

{
  "session_properties": {
    "session_name": "Sealighter-Kernel-Process-Start",
    "output_format": "event_log"
  },
  "kernel_traces": [{
    "trace_name": "kernel_proc_start_trace",
    "provider_name": "process",
    "filters": { "any_of": { "opcode_is": 1 } }
  }]
}

By using Event logs to store the raw events, we can also make use of another built-in feature of Windows, Windows Event Forwarding. This means that if we deploy Sealighter to every machine in a Windows Domain, we can forward all the events to a single PC, where we can do the heavier and more complex detection parts of SIEMCraft from a single machine, without writing any extra code:

Once WEF is set up, a SIEMCraft binary can make use of the EvtSubscribe API to collect and process all the Sealighter events as they come in. I had planned to write SIEMCraft in Go and found this awesome library from RawSec that made it easy to collect and parse Windows Event logs, ready to generate detections from.

2. Generating Detections

The correct way to write and generate Open Source detections is to make use of the amazing Sigma project, which led by the incredibly dedicated Florian Roth.

Sigma defines a standard schema in which people can create and share detections across multiple operating systems and applications. SIEMCraft needed to be able to ingest the 100s of free Sigma detection rules (as well as any other rules people create) for its detection engine, rather than attempting to create my own proprietary format.

Thankfully Bradley Kemp had already created a really neat open source Go library that could run and check Sigma rules against arbitrary data. I had to do some data formatting to ingest a list of rules from the filesystem, convert the Data from Sealighter to Sigma-Go, and account for some differences of opinion on the Sigma schema, but otherwise, Florian and Bradley also made this step super easy for me.

With these two pieces of the puzzle, I could write a basic SIGMA rule to detect the most well-known of attacker tools - whoami.exe:

title: Whoami Execution
id: 36de6a23-651e-485a-ba69-3966d66707af
status: experimental
description: Whoami.exe runs
references:
    - https://blog.tofile.dev
tags:
    - attack.execution
author: pathtofile
date: 2022/01/15
logsource:
    category: process_creation
    product: windows
detection:
    selection:
        Image|endswith: '\whoami.exe'
    condition: selection
falsepositives:
    - unknown
level: high

Whoami is used in almost every single intrusion I’ve observed, from eCrime to APT. Just look at the MiTRE page!. And what real user doesn’t already know who they are, right?

So now we had raw Process events flowing from every domain PC into SIEMCraft, which uses SIGMA to look for the dastardly whoami.exe (as well as any other actual rules we might create). Once SIEMCraft detects the whoami intrusion, it was time to send that data into Minecraft.

3. Send detects into Minecraft

This step would have been easier if I had used the old, Java version of Minecraft, which allows Mods to do basically whatever they want.

But the multiplayer server of the modern, better version of Minecraft (“Bedrock edition”, send Minecraft opinions to @maybe-sybr) isn’t designed to allow custom data into or out of a Minecraft server. Instead of ‘Mods’, Minecraft Bedrock has ‘Addons’, allowing you to do lots of cool things inside the world (new monsters, animations, etc.), while not allowing the Addons to read or write anything outside of Minecraft. This is great for security, but not great for SIEMCraft, which both needs to read Windows Events Logs and spawn creatures in Minecraft.

I could have written/used a custom server program, but after hanging out of the Minecraft hacker discords I learnt this would prevent other Mods from running at the same time as SIEMCraft, which wasn’t ideal.

But it turns out that in a very roundabout way, I could make use of a WebSocket designed for “Minecraft Educational Edition” to poke a hole in the security boundary. Anand describes the process best, but essentially Minecraft and a remote process can communicate over a WebSocket to send a receive several undocumented message types. This means we can run a SIEMCraft process ‘outside’ of Minecraft, to do the things Addons can’t do and use it to control an Addon inside of Minecraft.

The first step was to get the detection data into Minecraft, so I first made use of the ‘Command Request’ message type. This Message can be sent to a Minecraft server, allowing you to run any /command console from the player’s perspective. For example, to spawn a pig named ‘snuffles’ right next to the Player, you send this JSON command:

{
      "header": {
        "version": 1,
        "requestId": "36de6a23-651e-485a-ba69-3966d66707af",
        "messagePurpose": "commandRequest",
        "messageType": "commandRequest"
      },
      "body": {
        "version": 1,
        "commandLine": "summon pig \"snuffles\" ~ ~ ~",
        "origin": { "type": "player" }
      }
    }

Using these messages (via a minecraft websocket library by Sandertv), I could Spawn an animal near the player with their ‘name’ containing all the important detection details:

I decided to make different animals correspond to different detection severities, with “low” severity spawning cows and chickens, and “high” spawning spiders, pandas, and bears.

I needed to create and use a new Addon to get this all to work, to prevent the animal names/detection details from being hidden too quickly and to add bears to the game, as Minecraft only has polar bears.

But now, thanks to the Websocket, A player can be cruising around a Minecraft server doing Minecraft things, and if a detection comes in a chicken, spider, or bear will helpfully spawn nearby, alerting the user that a whoami.exe attack is afoot.

4. Allow players to take action

The final piece of the puzzle was to allow the player to take action, and kill a process identified by a detection if they deem it suspicious enough. This would be accomplished by having the play kill the detection-named animal specifically with a diamond sword (any other type of death would be treated as the player ignoring the alert).

In theory, this should have been easy, as the WebSocket exposes a MobKilled event which is sent from Minecraft to the remote Process whenever an animal or monster is killed. However, the message lacks any information to tell exactly which animal was killed, only the fact they were killed by the player. This means that while SIEMCraft could tell the player wants to respond to a detection, if it has sent multiple detections to the player it can’t tell which one.

A Minecraft Addon can get more information from inside the game, however. Addons can run small JavaScript programs which have access to a wider array of message types they can subscribe to, including the minecraft:entity_death message. This message type does contain all the information we need about exactly what mob was killed by who and with what. But as we know Addons can’t send this data out of Minecraft, so after many days of hacking, I decided on a really dodgy workaround: the PlayerMessage event.

PlayerMessage

This message is sent from the Minecraft WebSocket to the remote process and contains any text players (or other entities) send each other. The messages contain a bunch of information about who was talking to who, where they were talking from, and most importantly the exact text they typed. So I simply made the internal Javascript Addo ‘whisper’ the data to the player:

// Listen for entity_death events
var system = server.registerSystem(0,0);
system.initialize = function() {
  this.listenForEvent("minecraft:entity_death", (eventData) => this.onEntityDeath(eventData));
};

system.onEntityDeath = function (eventData) {
  // Get the name of the mob
  let killed_name = this.getComponent(eventData.data.entity, "minecraft:nameable");

  // Get weapon in main hand, assume it was what was used to kill it
  let handContainer = system.getComponent(eventData.data.killer, "minecraft:hand_container");
  let mainHandItem = handContainer.data[0];

  // Create JSON string to send
  less message = 'killed_name='+ killed_name + ', weapon=' + mainHandItem.item;

  // Send message by used the /say command
  let ExecuteEventData = this.createEventData("minecraft:execute_command");
  ExecuteEventData.data.command = "/tell @a \"" + message + "\"";
  this.broadcastEvent("minecraft:execute_command", ExecuteEventData);
};

And this results in the following message being sent to the SIEMCraft process over the Websocket:

{
  "header": {
    "messagePurpose": "event",
    "requestId": "00000000-0000-0000-0000-000000000000",
    "version": 1
  },
  "body": {
    "eventName": "PlayerMessage",
    "measurements": null,
    "properties": {
      // ...
      "Sender": "Script Engine",
      "Message": "killed_name=snuffles, weapon=diamond_sword",
      "MessageType": "tell",
      // ...
    }
  }
}

After some tweaking to do things like base64 and JSON encode the data to avoid parsing errors, I now had the full loop of data going from SIEMCraft -> Minecraft -> SIEMCraft. I could then read the weapon name to check it was a diamond sword, and check the mob name to extract the Process ID of the process the player wished to kill. I could have implemented logic to kill remote processes across the domain, but for now, I’m happy with just killing processes locally.

Shaving the GameTest

Everything was working fine, and I began writing up this blog and sharing screenshots with mates.

Unfortunetly one of them came across this blog from the Minecraft developers, stating they were removing the Addons Javascript engine in … 1 month’s time.

This would completely destroy the addon’s ability to talk back to SIEMCraft to kill processes, so I had to find a new solution. The replacement Javascript engine Game Test framework didn’t have the same events and had no events related to a mob’s death. But just as I was ready to throw the feature in the bin, a new beta version of Minecraft dropped, and deep within its patch notes contained a new event, beforeDataDrivenEntityTriggerEvent:

I have never Yak Shaved harder to solve a problem in my life.

In…short:

Combining this feature with a new Addon allowed me to define a new type of animal that looks like a chicken/cow/spider/etc. These animals would only ever be created by SIEMCraft, and unlike a normal animal, they had a special “event”, that trigged only when they were specifically killed by a player holding a diamond sword. Events are used by Minecraft to trigger behaviour, e.g. give an egg to a player when they pet a chicken. I created an event that didn’t actually do anything, but it was enough to trigger some Game Test Framework Javascript code that was listening to the new beforeDataDrivenEntityTriggerEvent event. This event contained the event and animal names, which I could then (by filtering only on my custom ‘killed by a diamond sword’ event) send back to SIEMCraft using the same playerMessage technique as before.

In diagram form, it looks like this:

While technically we’re past the deprecation date and the old version of the Javascript addon still works, SIEMCraft is now protected from future releases of Minecraft…almost.

Finishing SIEMCraft

By this stage, SIEMCraft had consumed not only my PTO but many weeks of free time beyond that. I think I yak-shaved too hard and ended up shaving part of my brain, so after I shared my journey of bullshit at my local security meetup, I took a long break before finalising the code and writing this blog. During this time Minecraft made another breaking change, requiring me to fork and fix the Minecraft Websocket library I was using.

But it’s now all completed and working (for now). You can get SIEMCraft code here: https://github.com/pathtofile/siemcraft

Instructions on how to set up Minecraft and run it are in the README, but please don’t use it for real things. Or do, and send me the screenshots of the ensuring dumpster fire. If Minecraft changes something again and breaks SIEMCRAFT, I am unlikley to fix it, as it has already taken up way too much of my time for a such a silly thing. So try it out while you can.

Many, many thanks to all the library authors I used for this project (Bradley Kemp, RawSec, Florian Roth, Sander Ten Veldhuis), S Anand’s blog as well as everyone who puts up with this garbage.

Oh, did I mention it works in Minecraft VR? https://youtu.be/8Vyf8Y5wcRY

Is this the first Metaverse SIEM?

Tool - Create run VPNs in various clouds

2022-06-02T12:00:00+00:00

For a while I’ve been using Terraform to quickly create simple VPN endpoints on various Cloud providers.

I recently cleaned up the documentation and variables to make it easy to spin up the VMs in various locations, and uploaded the code to GitHub: https://github.com/pathtofile/tf_wireguard.

As I state in the README, other projects like Algo are probably a better choice, but for me I wanted to have more understanding and control over what I was using.

Tool - Use Terraform and Bitcoin to run VMs

2022-04-24T12:00:00+00:00

Bitlaunch is a way to pay for DigitalOcean VMs using Bitcoin.

I was interested in using Bitlaunch, and wanted to learn how to create a Terraform Provider, so while I’m 100% unafilliated with Bitlaunch I made a Terraform Provider to make it easier to create and manage Vms.

The provider is here, and the sourcecode is here.

The repository is under my pathtofile-tf GitHub account, as in order to publish a Terraform Provider you must give Hashicorp permission to use your GitHub account, which I didn’t want to do on my main account.

Example:

terraform {
  required_providers {
    bitlaunch = {
      source  = "pathtofile-tf/bitlaunch"
      version = "0.4.0"
    }
  }
}

variable "token" { sensitive = true }
variable "ssh_pubkey" { type = string }
variable "host" { default = "DigitalOcean" }

provider "bitlaunch" {
  token = var.token
}

// Data
data "bitlaunch_image" "image" {
  host         = var.host
  distro_name  = "Ubuntu"
  version_name = "20.04 (LTS) x64"
}

data "bitlaunch_region" "region" {
  host        = var.host
  region_name = "San Francisco"
  slug        = "sfo2"
}

data "bitlaunch_size" "size" {
  host      = var.host
  cpu_count = 1
  memory_mb = 1024
}

// Resources
resource "bitlaunch_sshkey" "sshkey" {
  name    = "tf_sshkeys"
  content = var.ssh_pubkey
}

resource "bitlaunch_server" "server" {
  host        = var.host
  name        = "tf_server"
  image_id    = data.bitlaunch_image.image.id
  size_id     = data.bitlaunch_size.size.id
  region_id   = data.bitlaunch_region.region.id
  ssh_keys    = [bitlaunch_sshkey.sshkey.id]
  wait_for_ip = true
}

// Outputs
output "ip_address" {
  value = bitlaunch_server.server.ipv4
}

Commandline Cloaking and Sysmon for Linux

2022-01-04T12:00:00+00:00

tl;dr

I investigated running Sysmon for Linux against a number of techniques to fake a program’s commandline. Code and Sysmon config available on GitHub.

tl;dr
Sysmon for Linux
Attack Scenario
Lab setup
Tests
Conclusions and final thoughts
- Sysmon for Linux
- Writing low-level Go
Example Code

Sysmon for Linux

Earlier this year, Microsoft released Sysmon for Linux, which is an awesome new addition to free Linux monitoring tools. Typically, sysadmins have used AuditD (or tools that use AuditD under the hood) to monitor Linux systems. However, using AuditD can be tricky to configure, and can require a level of knowledge of Linux internals that might put off administrators that just want basic process monitoring.

Sysmon for Linux conversely uses the same event schema as Sysmon on Windows, which abstracts away the internals in how it generates events, instead groping them into “ProcessCreate”, “NetworkConnect “, “FileDelete”, etc. Using the same schema as Windows is quite useful for people already familiar with the Windows version of Sysmon, although documentation could be improved to help those who aren’t familiar, or only know AuditD.

I was interested in exploring what it’s like to set up and run Sysmon for Linux, and how it might fare against some common techniques malware uses to hide from system administrators.

Attack Scenario

A lot of common malware detections on Linux are based upon on pattern matching commandlines and program filenames. I wanted to explore the various ways a program can alter or hide commandlines from Audit tools, and see what is recorded by Sysmon for Linux.

It was also an excuse to learn more about writing low-level programs in Go, which I’ve always wanted to learn more about, instead of defaulting to C all the time.

Lab setup

To set things up, I used both Roberto Rodriguez’s dope Sysmon Overview and the official instructions.

As I was only interested in process events, I used this simple config to just get process creations:

 schemaversion="4.70">
  
     name="" groupRelation="or">
       onmatch="exclude"/>

Once Sysmon was installed and running, to make things simple I didn’t forward the events to a SIEM, I just used the sysmonLogView helper tool to print events out to the console in a human-readable format:

sudo bash -c '
    tail -f /var/log/syslog |
    /opt/sysmon/sysmonLogView -e 1
'

Tests

Test #0 - No cloaking

To start, I made a simple program named basic that just prints out the arguments passed into it:

func main() {
  pid := os.Getpid()
  ppid := os.Getppid()

  argc := len(os.Args)
  fmt.Printf("  PID     %d\n", pid)
  fmt.Printf("  PPID    %d\n", ppid)
  fmt.Printf("  argc    %d\n", argc)
  for i := 0; i < argc; i++ {
    fmt.Printf("  argv[%d] %s\n", i, os.Args[i])
  }

  fmt.Print("  Sleeping for 60 seconds so you can lookup the PID")
  time.Sleep(60 * time.Second)
}

When run with a single argument, basic prints out:

$> /path/to/basic AAAA
  PID     984173
  PPID    1054354
  argc    2
  argv[0] /path/to/basic
  argv[1] AAAA
  Sleeping for 60 seconds so you can lookup the PID

This generates the following Sysmon Event as expected:

Event SYSMONEVENT_CREATE_PROCESS
  RuleName: -
  UtcTime: 2021-11-30 23:51:08.592
  ProcessGuid: {35dd0383-b8ec-61a6-b8d0-480000000000}
  ProcessId: 984173
  Image: /path/to/basic
  FileVersion: -
  Description: -
  Product: -
  Company: -
  OriginalFileName: -
  CommandLine: /path/to/basic AAAA
  CurrentDirectory: /path/to
  User: path
  LogonGuid: {35dd0383-9d97-61a4-e803-000002000000}
  LogonId: 1000
  TerminalSessionId: 347
  IntegrityLevel: no level
  Hashes: -
  ParentProcessGuid: {35dd0383-9a98-61a4-fddb-e43082550000}
  ParentProcessId: 1054354
  ParentImage: /usr/bin/bash
  ParentCommandLine: -bash
  ParentUser: path

We can see basic was launched by bash, the process IDs match up, and the commandline is the same. All as expected.

A really neat aspect of Sysmon for Linux is how it extracts the commandline arguments from a process. Other tools use the arguments passed into the execve syscall, but these arguments can be vulnerable to a race condition, such as the one detailed by Rex Guo and Junyuan Zeng at DEF CON 29, due to the fact they exist in the parent process’ userspace memory. Sysmon for Linux however extracts the arguments from the task struct within the kernel’s memory, which (at the time Sysmon reads it) is inaccessible to the parent process, and will always be the real arguments used to start the process.

We can also compare Sysmon’s output to other sysadmin tools, proving the commandline, process ID, and filename match up:

# Use ps to look for 'basic' process
$> ps u -C "basic"
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
path     1066830  0.0  0.1 702896  8968 pts/2    Sl+  11:23   0:00 /path/to/basic AAAA

Under the hood, ps parses data from the /proc/ pseudo-filesystem, so we can also do that:

# Look up the commandline using the PID and the proc psudo filesystem
$> xxd -g 1 -c 6 /proc/1066830/cmdline
00000000: 2e 2f 62 69 6e 2f  /path/to/
00000006: 62 61 73 69 63 00  basic.
0000000c: 41 41 41 41 00     AAAA.

By reading /proc//cmdline we can see the strings in argv, each ending in a 00 NULL character. This data exists within basic’s memory, which means that a dodgy program could alter this data.

Test #1 - Post-execution cloaking

Next, I wrote a program named dodgy to alter the contents of it’s argv after it has started to run. While this is easy to do in either C or Go, the Go code is harder to read, so I created dodgy in C, which looks like this:

int main(int argc, char* argv[]) {
  // Overwrite address of argv to fool 'ps'
  memset(argv[0], 'F', strlen(argv[0]));
  if (argc > 1) {
      memset(argv[1], 'B', strlen(argv[1]));
  }

  // Print arguments just like 'basic' did
  printf("  PID     %d\n", getpid());
  printf("  PPID    %d\n", getppid());
  printf("  argc    %d\n", argc);
  for (int i = 0; i < argc; i++) {
    printf("  argv[%d] %s\n", i, argv[i]);
  }
  printf("Sleeping for 60 seconds so you can lookup the PID\n");
  sleep(60);
  return 0;
}

Now when we run the program we get:

$> /path/to/dodgy AAAA
  PID     1067696
  PPID    1054354
  argc    2
  argv[0] FFFFFFFFFFF
  argv[1] BBBB
Sleeping for 60 seconds so you can lookup the PID

# Then in another tab look for all 'dodgy' processes
$> ps u -C "dodgy"
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
path     1067696  0.0  0.0   2304   556 pts/2    S+   12:06   0:00 FFFFFFFFFFF BBBB

# Look up the commandline using the PID and the proc psudo filesystem
$> xxd -g 1 -c 6 /proc/1067696/cmdline
00000000: 46 46 46 46 46 46  FFFFFF
00000006: 46 46 46 46 46 00  FFFFF.
0000000c: 42 42 42 42 00     BBBB.

There are similar techniques programs can use to fake information once it has executed, including:

Using prctl and PR_SET_NAME to change the process name in /proc/pid/comm
Fork twice and kill parents, which sets the process’ Parent PID to PID 1

But to Sysmon all these efforts are in vain - As it logs the process information before it runs, it sees the original commandline that the process was started with:

Event SYSMONEVENT_CREATE_PROCESS
  RuleName: -
  UtcTime: 2021-11-30 23:51:08.592
  ProcessGuid: {35dd0383-b8ec-61a6-b8d0-480000000000}
  ProcessId: 1067696
  Image: /path/to/dodgy
  FileVersion: -
  Description: -
  Product: -
  Company: -
  OriginalFileName: -
  CommandLine: /path/to/dodgy AAAA
  CurrentDirectory: /path/to
  User: path
  LogonGuid: {35dd0383-9d97-61a4-e803-000002000000}
  LogonId: 1000
  TerminalSessionId: 347
  IntegrityLevel: no level
  Hashes: -
  ParentProcessGuid: {35dd0383-9a98-61a4-fddb-e43082550000}
  ParentProcessId: 1054354
  ParentImage: /usr/bin/bash
  ParentCommandLine: -bash
  ParentUser: path

Test #2 - Piping input

Some programs support reading data in from standard input, which means it can either be | piped in or read in using < file redirection. One example is Python, where the following examples are all equivalent:

# Run the file
python3 quietquokka.py
python3 $(bash -c "echo cXVpZXRxdW9ra2EucHkK | base64 -d")
# Pipe file in
cat quietquokka.py | python3
# File redirection
python3 < quietquokka.py

The first two generate Syslog events matching what you expect (the $(bash -c ... part gets evaluated prior to the syscall):

Event SYSMONEVENT_CREATE_PROCESS
  RuleName: -
  UtcTime: 2021-12-01 02:57:11.135
  ProcessGuid: {35dd0383-e487-61a6-fdbb-b3691d560000}
  ProcessId: 1068450
  Image: /usr/bin/bash
  FileVersion: -
  Description: -
  Product: -
  Company: -
  OriginalFileName: -
  CommandLine: bash -c echo cXVpZXRxdW9ra2EucHkK | base64 -d
  CurrentDirectory: /path/to
  User: path
  LogonGuid: {35dd0383-9d97-61a4-e803-000002000000}
  LogonId: 1000
  TerminalSessionId: 347
  IntegrityLevel: no level
  Hashes: -
  ParentProcessGuid: {35dd0383-9a98-61a4-fddb-e43082550000}
  ParentProcessId: 1054354
  ParentImage: /usr/bin/bash
  ParentCommandLine: -bash
  ParentUser: path
Event SYSMONEVENT_CREATE_PROCESS
  RuleName: -
  UtcTime: 2021-12-01 02:57:11.137
  ProcessGuid: {35dd0383-e487-61a6-69ee-60001a560000}
  ProcessId: 1068452
  Image: /usr/bin/base64
  FileVersion: -
  Description: -
  Product: -
  Company: -
  OriginalFileName: -
  CommandLine: base64 -d
  CurrentDirectory: /path/to
  User: path
  LogonGuid: {35dd0383-9d97-61a4-e803-000002000000}
  LogonId: 1000
  TerminalSessionId: 347
  IntegrityLevel: no level
  Hashes: -
  ParentProcessGuid: {35dd0383-e487-61a6-fdbb-b3691d560000}
  ParentProcessId: 1068450
  ParentImage: /usr/bin/bash
  ParentCommandLine: bash
  ParentUser: path
Event SYSMONEVENT_CREATE_PROCESS
  RuleName: -
  UtcTime: 2021-12-01 02:57:11.137
  ProcessGuid: {35dd0383-e487-61a6-ed63-6a0000000000}
  ProcessId: 1068453
  Image: /usr/bin/python3.9
  FileVersion: -
  Description: -
  Product: -
  Company: -
  OriginalFileName: -
  CommandLine: python3 quietquokka.py
  CurrentDirectory: /path/to
  User: path
  LogonGuid: {35dd0383-9d97-61a4-e803-000002000000}
  LogonId: 1000
  TerminalSessionId: 347
  IntegrityLevel: no level
  Hashes: -
  ParentProcessGuid: {35dd0383-9a98-61a4-fddb-e43082550000}
  ParentProcessId: 1054354
  ParentImage: /usr/bin/bash
  ParentCommandLine: -bash
  ParentUser: path

However, in the other two examples (piping in data and file redirection), the Python commandline is empty:

Event SYSMONEVENT_CREATE_PROCESS
  RuleName: -
  UtcTime: 2021-12-01 02:58:58.656
  ProcessGuid: {35dd0383-e4f2-61a6-ed63-6a0000000000}
  ProcessId: 1068470
  Image: /usr/bin/python3.9
  FileVersion: -
  Description: -
  Product: -
  Company: -
  OriginalFileName: -
  CommandLine: python3
  CurrentDirectory: /path/to
  User: path
  LogonGuid: {35dd0383-9d97-61a4-e803-000002000000}
  LogonId: 1000
  TerminalSessionId: 347
  IntegrityLevel: no level
  Hashes: -
  ParentProcessGuid: {35dd0383-9a98-61a4-fddb-e43082550000}
  ParentProcessId: 1054354
  ParentImage: /usr/bin/bash
  ParentCommandLine: -bash
  ParentUser: path

I’ve covered this technique before, including thoughts on how to make use of eBPF to detect it. And to be clear Sysmon isn’t wrong - the Python was started with just the commandline “python3”.

You can’t pipe in all arguments to a program, however, only in specific cases where the program supports it. But this does include a number of tools like python, curl, and bash, and is important to think about when looking to make commandline-focussed detection.

Test #3 - In-memory loader

Another common technique used by malware is to execute a program from memory. This can be easily done by using memfd_create:

func main() {
  // Print PID of loader program
  fmt.Printf("  Ldr PID %d\n", os.Getpid())

  // Use memfd to create in-memory file
  fd, _ := unix.MemfdCreate("", 0)

  // open created in-memory file
  fp := fmt.Sprintf("/proc/self/fd/%d", fd)
  memfd_file := os.NewFile(uintptr(fd), fp)
  defer memfd_file.Close()

  // Read ELF from disk and write into in-memory file
  // ELF could also be read from a C2 server, encrypted
  // on disk, hardcoded, piped in from stdin, etc.
  data, _ := ioutil.ReadFile("/path/to/basic")
  memfd_file.Write(data)

  // Execute in-memory binary, fake argv[0] filename
  argv := []string{"from_loader", "AAAA"}
  unix.Exec(fp, argv, os.Environ())
}

This loader program can be then be run to launch basic from in-memory:

$> /path/to/loader
  Ldr PID 17436
  PID     17436
  PPID    15924
  argc    2
  argv[0] from_loader
  argv[1] AAAA
  Sleeping for 60 seconds so you can lookup the PID

# Look up pid
$> ps u | grep 17436
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
path       17436  0.0  0.0 703064  2872 pts/0    Sl+  15:13   0:00 from_loader AAAA

Due to how the execve syscall works, basic’s PID is the same as loader’s. This is because execve doesn’t really create a new process per sey, but instead replaces the program running from loader to basic. But to Sysmon, both programs are recorded:

Event SYSMONEVENT_CREATE_PROCESS
  RuleName: -
  UtcTime: 2022-01-02 04:19:04.901
  ProcessGuid: {35dd0383-27b8-61d1-8cdf-480000000000}
  ProcessId: 17436
  Image: /path/to//bin/loader
  FileVersion: -
  Description: -
  Product: -
  Company: -
  OriginalFileName: -
  CommandLine: /path/to/loader /path/to/basic AAAA
  CurrentDirectory: /path/to/
  User: path
  LogonGuid: {35dd0383-21ff-61d1-e803-000000000000}
  LogonId: 1000
  TerminalSessionId: 177
  IntegrityLevel: no level
  Hashes: -
  ParentProcessGuid: {00000000-0000-0000-0000-000000000000}
  ParentProcessId: 15924
  ParentImage: -
  ParentCommandLine: -
  ParentUser: -
Event SYSMONEVENT_CREATE_PROCESS
  RuleName: -
  UtcTime: 2022-01-02 04:19:04.901
  ProcessGuid: {35dd0383-27b8-61d1-1000-480000000000}
  ProcessId: 17436
  Image: /memfd:
  FileVersion: -
  Description: -
  Product: -
  Company: -
  OriginalFileName: -
  CommandLine: from_loader AAAA
  CurrentDirectory: /path/to/
  User: path
  LogonGuid: {35dd0383-21ff-61d1-e803-000000000000}
  LogonId: 1000
  TerminalSessionId: 177
  IntegrityLevel: no level
  Hashes: -
  ParentProcessGuid: {00000000-0000-0000-0000-000000000000}
  ParentProcessId: 15924
  ParentImage: -
  ParentCommandLine: -
  ParentUser: -

Sysmon can only tell the second ELF executed was from in-memory (marking it as /memfd), and it records the ‘fake’ from_loader commandline. It cannot tell that the binary run was actually /path/to/basic.

But depending on the machine, user, or parent process, it is probably highly suspicious to see a /memfd in-memory ELF being run. So while you cannot tell what binary it was, this alone might be enough to investigate.

Test #4 - LD_PRELOAD

One common technique used by malware to hide is to make use of LD_PRELOAD and similar environment variables. By setting LD_PRELOAD to the path to a library on disk, the Linux dynamic linker will inject this library into all new programs, and will look in that library first for any exported functions that the program may have instead wanted to use from other libraries. This means a malicious library and can hook and alter the program, and is commonly used as a backdoor to alter functions that relate to file reading or network connections, so the malware can hide data on disk, or allow a normally blocked connection through.

But another thing a malicious LD_PRELOAD library can do is hook the program just as it starts, and alter the argv arguments before they are seen and used. For programs written in C using libc, this means hooking the __libc_start_main function.

In C, this type of hooking is simple:

int __libc_start_main(
  int (*main)(int, char **, char **),
  int argc,
  char **argv,
  int (*init)(int, char **, char **),
  void (*fini)(void),
  void (*rtld_fini)(void),
  void *stack_end)
{
  // Overwrite args like in 'dodgy'
  memset(argv[0], 'F', strlen(argv[0]));
  if (argc > 1) {
    memset(argv[1], 'B', strlen(argv[1]));
  }

  // Lookup the real libc function
  typeof(&__libc_start_main) orig_start_main = dlsym(RTLD_NEXT, "__libc_start_main");

  // Call real __libc_start_main
  return orig_start_main(main, argc, argv, init, fini, rtld_fini, stack_end);
}

After writing a C version of the basic program to be hooked, I ran with and set the LD_PRELOAD variable to be my malicious library:

# Set the LD_PRELOAD variable to path to preload
# NOTE: this won't work on Go programs, so use the C version of Basic
LD_PRELOAD=/path/to/preload.so /path/to/basic_c AAAA
  PID     26687
  PPID    16961
  argc    2
  argv[0] FFFFFFFFFFFFF
  argv[1] BBBB
  Sleeping for 60 seconds so you can lookup the PID

# Check ps output for pid
$> ps aux | grep 26687
path       26687  0.0  0.0   2360   532 pts/1    S+   15:47   0:00 FFFFFFFFFFFFF BBBB

# Extract LD_PRELOAD env variable from ps output
$> ps ue | grep 26687 | grep -o "LD_PRELOAD=.*" | awk '{print $1}'
LD_PRELOAD=/path/to/preload.so

As preload’s shenanigans run after the process starts, Sysmon only see’s the original arguments, and not what the effective arguments to basic_c are:

Event SYSMONEVENT_CREATE_PROCESS
  RuleName: -
  UtcTime: 2021-12-02 03:29:01.320
  ProcessGuid: {35dd0383-3d7d-61a8-9d12-65fb78550000}
  ProcessId: 26687
  Image: /path/to/basic_c
  FileVersion: -
  Description: -
  Product: -
  Company: -
  OriginalFileName: -
  CommandLine: /path/to/basic_c AAAA
  CurrentDirectory: /path/to
  User: path
  LogonGuid: {35dd0383-4d8c-61a4-e803-000000000000}
  LogonId: 1000
  TerminalSessionId: 334
  IntegrityLevel: no level
  Hashes: -
  ParentProcessGuid: {35dd0383-4d8c-61a4-fd3b-d4af46560000}
  ParentProcessId: 1052430
  ParentImage: /usr/bin/bash
  ParentCommandLine: -bash
  ParentUser: path

This is accurate, technically basic_c was launched with AAAA, and preload runs after the process has launched. But the basic_c program will act as if the fake BBBB argument was passed into it.

To add to the confusion, instead of overwriting argv, the preload library could instead point __libc_start_main to a new argv array, which will fool ps as well:

// Create our own argv and argc
static char* new_argv[] = {
    "from_preload",
    "BBBB"
};
static int new_argc = 2;

int __libc_start_main(
  int (*main)(int, char **, char **),
  int argc,
  char **argv,
  int (*init)(int, char **, char **),
  void (*fini)(void),
  void (*rtld_fini)(void),
  void *stack_end)
{
  // Lookup the real libc function
  typeof(&__libc_start_main) orig_start_main = dlsym(RTLD_NEXT, "__libc_start_main");

  // Call real __libc_start_main, note we instead pass it our new argc and argv
  // instead of the original ones
  return orig_start_main(main, new_argc, new_argv, init, fini, rtld_fini, stack_end);
}

When run this looks like:

# Set the LD_PRELOAD variable to path to preload
# NOTE: this won't work on Go programs, so use the C version of Basic
LD_PRELOAD=/path/to/preload.so /path/to/basic_c AAAA
  PID     28152
  PPID    16961
  argc    2
  argv[0] from_preload
  argv[1] BBBB
  Sleeping for 60 seconds so you can lookup the PID

# Check ps output for pid, note the original arguments and not what basic effectivly ran with
$> ps aux | grep 28152
path       28152  0.0  0.0   2360   536 pts/1    S+   15:57   0:00 /path/to/basic_c AAAA

This blog was also meant to be about how to do things in Go and not C, so thanks to Awgh on the Bloodhound Slack’s Go channel, I was able to create a similar library in Go:

//export __libc_start_main
func __libc_start_main(main *C.void, argc C.int, argv **C.char,    init *C.void,    fini *C.void,    rtld_fini *C.void,) C.int {
  // Create and use new argv[0] and argv[1]
  offset := unsafe.Sizeof(uintptr(0))
  argv_copy := argv
  new_argv_0 := C.CString("from_preload")
  *argv_copy = new_argv_0
  if argc > 1 {
    new_argv_1 := C.CString("BBBB")
    argv_copy = (**C.char)(unsafe.Pointer(uintptr(unsafe.Pointer(argv_copy)) + offset))
    *argv_copy = new_argv_1
  }

  // Find and call real __libc_start_main function
  fp := C.dlopen(C.CString("libc.so.6"), C.RTLD_LAZY)
  orig_func := C.dlsym(fp, C.CString("__libc_start_main"))
  val, err := cdecl.Call(
    uintptr(orig_func),
    uintptr(unsafe.Pointer(main)),
    uintptr(argc),
    uintptr(unsafe.Pointer(argv)),
    uintptr(unsafe.Pointer(init)),
    uintptr(unsafe.Pointer(fini)),
    uintptr(unsafe.Pointer(rtld_fini)),
  )

  return (C.int)(val)
}

(see the full code here)

Unfortunately, Sysmon doesn’t log the environment variables a program was launched with, most likely due to size constraints, but if it did so you should be able to see the LD_PRELOAD, which is not commonly used in production environments, and will be pointing to the malicious library.

Test #5 - In-memory patching

The final technique to discuss is making use of in-memory patching, to achieve the same goal as LD_PRELOAD, but without using the environment variable.

Back in 1998, Silvio Cesare wrote a paper on Unix ELF parasites and viruses. In the paper he wrote about how to patch ELF binaries to inject code at the program’s entrypoint, running custom shellcode in between when the program has been execve‘d, and the program’s “main” function.

The awesome Binjection library has implemented Silvio’s technique in Go, so I was able to use their library and my in-memory loader code to:

Read a binary to run from disk
Patch the entrypoint with custom shellcode that alters argv before returning to the real entrypoint
Run the patched binary from memory using memfd.

The full code I called injector, and is available here. The shellcode to alter the arguments at the ELF entrypoint is actually very simple, as at the start of the ELF execution the kernel will place the memory addresses to the arguments on the top of the stack, so as long as you don’t care about memory corruption, to overwrite the first argument from AAAA to BBBB the shellcode simply looks like:

; Get address of to argv[1] string
mov  rax, [rsp+0x10]

; overwrite argv[1] string to "BBBB\0"
; which in hex is: 42 42 42 42 00
mov [rax],   dword 0x42424242
mov [rax+4], byte 0x00

; Reset rax to 0
xor rax, rax

The end result looks like this:

$> /path/to/injector /path/to/shellcode.bin /path/to/basic AAAA
  PID     28638
  PPID    16961
  argc    2
  argv[0] from_injector
  argv[1] BBBB
  Sleeping for 60 seconds so you can lookup the PID

# Check ps output for pid, note the original arguments and not what `basic` saw
$> ps aux | grep 28152
path       28638  0.0  0.0 703064  2876 pts/1    Sl+  16:34   0:00 from_injector BBBB

Sysmon for Linux will log the loader fully, but again will have some trouble with the loaded binary, only recording /memfd:

Event SYSMONEVENT_CREATE_PROCESS
 RuleName: -
 UtcTime: 2022-01-02 05:36:48.387
 ProcessGuid: {35dd0383-39f0-61d1-a01b-4b0000000000}
 ProcessId: 28638
 Image: /path/to/injector
 FileVersion: -
 Description: -
 Product: -
 Company: -
 OriginalFileName: -
 CommandLine: /path/to/injector /path/to/shellcode.bin /path/to/basic AAAA
 CurrentDirectory: /path/to/
 User: path
 LogonGuid: {35dd0383-25d1-61d1-e803-000001000000}
 LogonId: 1000
 TerminalSessionId: 178
 IntegrityLevel: no level
 Hashes: -
 ParentProcessGuid: {00000000-0000-0000-0000-000000000000}
 ParentProcessId: 16961
 ParentImage: -
 ParentCommandLine: -
 ParentUser: -
Event SYSMONEVENT_CREATE_PROCESS
 RuleName: -
 UtcTime: 2022-01-02 05:36:48.387
 ProcessGuid: {35dd0383-39f0-61d1-2700-480000000000}
 ProcessId: 28638
 Image: /memfd:
 FileVersion: -
 Description: -
 Product: -
 Company: -
 OriginalFileName: -
 CommandLine: from_injector AAAA
 CurrentDirectory: /path/to/
 User: path
 LogonGuid: {35dd0383-25d1-61d1-e803-000001000000}
 LogonId: 1000
 TerminalSessionId: 178
 IntegrityLevel: no level
 Hashes: -
 ParentProcessGuid: {00000000-0000-0000-0000-000000000000}
 ParentProcessId: 16961
 ParentImage: -
 ParentCommandLine: -
 ParentUser: -

Sysmon for Linux doesn’t currently log the hash of the image being executed. If it did, the Sha256 hash of the binary launched would be the hash for the original binary+shellcode:

$> sha256sum /path/to/basic
9cadc2121a28ffe4c9c30afe093794206fdc429b8d56b4b0ddae70ac9dc993cd  /path/to/basic
# the patched binary can be read from the proc filesystem:
$> sha256sum /proc/28152/exe
0a21c7e1bb346db045e2d182ec2678f13d431689ff2d0a8ab83e641bd94a5b4f  /proc/28152/exe

The patched binary hash would be extremely uncommon, possibly unique, and doesn’t match the original binary on disk. Depending on the environment, the uncommonness of the hash may be enough to investigate.

Conclusions and final thoughts

Sysmon for Linux

Sysmon for Linux is looking like a great free monitoring tool for Linux. For people familiar with the Windows version of Sysmon, it provides a common configuration language that will definitely be useful for admin to maintain, and it’s process creation events would provide excellent insight into a lot of standard Linux malware attacks.

It is missing a few features that might catch more advanced attacks, such a file hashing and environment variable logging, and I didn’t explore its performance on a production-sized machine, but it’s a solid monitor to rival using AuditD, or other free monitors such as Tracee.

One thing not covered in this blog is the ability for an attacker to tamper with Sysmon’s configuration. Sysmon uses eBPF under the hood, and uses eBPF Maps to store the configuration (What event types to record, processes to ignore, etc.), which as I’ve written about in the past can be susceptible to tampering by an attacker with admin/root level privileges. While it is difficult for a program to protect itself for such a highly privileged attacker, it’s worth noting that the attack I described in my previous blog does work on Sysmon without generating any tamper-detection events. This is something the Sysmon team might consider detecting in the future, but it’s not an easy thing to prevent, and they may rightfully consider it out of scope for a free monitoring tool.

Writing low-level Go

Thanks to the Awgh, the Bloodhound Slack, and the Binjection project, I was able to pretty easily get up to speed and write Go code to do raw syscalls, LD_PRELOADing, and other low-level tricks pretty easily. Raw pointer manipulation looks pretty ugly in Go compared to C, but doing things like argument parsing and safe string manipulation was much nicer in Go.

Example Code

I’ve uploaded the source for all the different programs mentioned in this blog here: https://github.com/pathtofile/commandline_cloaking. With examples in both C and Go, it has these programs:

Basic’s argument printing
Dodgy’s argv manipulation
LD_PRELOAD libraries
In-memory loading to alter argv after execution
ELF Patching to alter argv after execution
Sample Sysmon for Linux configuration and setup

pat_h/to/file

Investigating realtime detections on iOS using Unified Logging

Table of Contents

Current Detection Efforts

1. Snapshots give time for attackers to hide

2. You need to know when to make a snapshot

Enter Lockdown

Working Remotely

Unified Logging

Configuration Profiles

Collecting Data

Process Creation

Networking

Networking - Using PCAPs

File Access

Messaging

iMessage

SMS

Other Activity

Enabling Ad Tracking

Location

Entitlements

KeyChain access

Cookies

XPC

Conclusions and Future work

Epilogue - Offensive Uses

Gaining Threat-Intelligence the REALLY dodgy way

tl;dr

Back to ETW-TI

Event Tracing for Windows (ETW)

KrabsETW and Sealighter

Protected Processes, PPL, and Early Launch Anti-Malware

Microsft-Security-Threat-Intelligence

Previous attempts

Test-Signing Mode

Exploitation

Even Dodgier Method - Vulnerable Drivers

Enter the Kernel Driver Utility

KDU and Sealighter

Conclusion

Linux cloud memory forensics tutorial

tl;dr

Volatility and Linux

A tale of two halves

1. A Memory capture

2. An Intermediate Symbol File (ISF)

Auto-Generate ISF

Manually Generating

Ubuntu:

Debian:

Centos8, Fedora, Amazon Linux:

Azure CBL-Mariner:

Generating an ISF

Using ISF

Conclusion

Tool - Use TouchID and the Secure Enclave from the commandline

Commandline Cloaking 2 - Tetragon and Nim

tl;dr

Tetragon

Nim

Lab setup

Tests

Test #0 - No cloaking

Nim

Tetragon

Test #1 - Post-execution cloaking

Nim

Tetragon

Tests #2 and #3 - In-memory loader and Piping input

Nim

Tetragon

Test #4 - LD_PRELOAD

Nim

Tetragon

Test #5 - In-memory patching

Nim

Tetragon

Test #6 - Chroot dorking

Conclusions and final thoughts