1 - Overview

Discover Tetragon and its capabilities

Cilium’s new Tetragon component enables powerful realtime, eBPF-based Security Observability and Runtime Enforcement.

Tetragon detects and is able to react to security-significant events, such as

  • Process execution events
  • System call activity
  • I/O activity including network & file access

When used in a Kubernetes environment, Tetragon is Kubernetes-aware - that is, it understands Kubernetes identities such as namespaces, pods and so-on - so that security event detection can be configured in relation to individual workloads.

A diagram showing Tetragon capabilities and how it interacts with Kubernetes, the kernel and other metrics, logging, tracing or events systems

Tetragon Overview Diagram

Functionality Overview

eBPF Real-Time

Tetragon is a runtime security enforcement and observability tool. What this means is Tetragon applies policy and filtering directly in eBPF in the kernel. It performs the filtering, blocking, and reacting to events directly in the kernel instead of sending events to a user space agent.

For an observability use case, applying filters directly in the kernel drastically reduces observation overhead. By avoiding expensive context switching and wake-ups, especially for high frequency events, such as send, read, or write operations, eBPF reduces required resources. Instead, Tetragon provides rich filters (file, socket, binary names, namespace/capabilities, etc.) in eBPF, which allows users to specify the important and relevant events in their specific context, and pass only those to the user-space agent.

eBPF Flexibility

Tetragon can hook into any function in the Linux kernel and filter on its arguments, return value, associated metadata that Tetragon collects about processes (e.g., executable names), files, and other properties. By writing tracing policies users can solve various security and observability use cases. We provide a number of examples for these in the repository and highlight some below in the ‘Getting Started Guide’, but users are encouraged to create new policies that match their use cases. The examples are just that, jumping off points that users can then use to create new and specific policy deployments even potentially tracing kernel functions we did not consider. None of the specifics about which functions are traced and what filters are applied are hard-coded in the engine itself.

Critically, Tetragon allows hooking deep in the kernel where data structures can not be manipulated by user space applications avoiding common issues with syscall tracing where data is incorrectly read, maliciously altered by attackers, or missing due to page faults and other user/kernel boundary errors.

Many of the Tetragon developers are also kernel developers. By leveraging this knowledge base Tetragon has created a set of tracing policies that can solve many common observability and security use cases.

eBPF Kernel Aware

Tetragon, through eBPF, has access to the Linux kernel state. Tetragon can then join this kernel state with Kubernetes awareness or user policy to create rules enforced by the kernel in real time. This allows annotating and enforcing process namespace and capabilities, sockets to processes, process file descriptor to filenames and so on. For example, when an application changes its privileges we can create a policy to trigger an alert or even kill the process before it has a chance to complete the syscall and potentially run additional syscalls.

What’s next?

2 - Getting Started

How to quickly get started with Tetragon and learn how to install, deploy and configure it

2.1 - Try Tetragon on Linux

Discover and experiment with Tetragon on your local Linux host

Start Tetragon

The easiest way to start experimenting with Tetragon is to run it via Docker using the released container images. If you prefer running directly the binaries without Docker, please refer to the development setup to see how to build the binaries.

docker run --name tetragon-container --rm --pull always \
    --pid=host --cgroupns=host --privileged             \
    -v /sys/kernel/btf/vmlinux:/var/lib/tetragon/btf    \
    quay.io/cilium/tetragon-ci:latest

Let’s break down the previous command:

  • --name tetragon-container names our container so that we can refer to it later via a simple name.
  • --rm will make Docker delete the container once it exists.
  • --pull always will force Docker to download the latest existing image corresponding to the latest mutable tag.
  • --pid=host is needed to read the procfs for gathering context on processes started before Tetragon.
  • --cgroupns=host is needed to detect container feature.
  • --privileged is needed for Tetragon to load its BPF programs.
  • -v /sys/kernel/btf/vmlinux:/var/lib/tetragon/btf mounts the BTF file inside the Tetragon container filesystem. Tetragon needs a BTF file corresponding to your kernel version to load its BPF programs.
  • quay.io/cilium/tetragon-ci:latest is the latest version of Tetragon built from the main branch.

Observe Tetragon base events

With this default configuration, Tetragon already loaded its base sensors to perform process lifecycle observability.

To quickly see the events, you can use the tetra CLI already shipped in the Tetragon container that was just started, it will connect to the Tetragon gRPC server listening on localhost:54321. In another terminal, type the following command:

docker exec tetragon-container tetra getevents -o compact

In a different terminal, you can start a shell like sh and start executing programs. Here, the commands executed in the shell where ls, uname -a, whoami, and exit, the output should be similar to this:

🚀 process  /usr/bin/sh
🚀 process  /usr/bin/ls
💥 exit     /usr/bin/ls  0
🚀 process  /usr/bin/uname -a
💥 exit     /usr/bin/uname -a 0
🚀 process  /usr/bin/whoami
💥 exit     /usr/bin/whoami  0
💥 exit     /usr/bin/sh  0

You can see the start of sh first, then the execution of the multiple programs, ls, uname and whoami, and finally the exit of sh.

Write a TracingPolicy to extend Tetragon events

Tetragon by default observes only process lifecycles. More advanced use-cases, require configuring Tetragon via TracingPolicies. A TracingPolicy is a user-configurable Kubernetes custom resource (CR) that allows users to access additional functionality, such as tracing arbitrary events in the kernel and, optionally, define actions to take on a match. These actions can be, for example, used to implement enforcement.

Observability with TracingPolicy

Let’s start with a simple policy:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "cat-open"
spec:
  kprobes:
  - call: "sys_openat"
    syscall: true
    args:
    - index: 0
      type: "int"
    - index: 1
      type: "string"
    - index: 2
      type: "int"
    selectors:
    - matchBinaries:
      - operator: In
        values:
        - "/usr/bin/cat"

Copy this policy in a file named tracing_policy.yaml in your current working directory and restart Tetragon with the following command:

docker run --name tetragon-container --rm --pull always \
    --pid=host --cgroupns=host --privileged             \
    -v $PWD/tracing_policy.yaml:/tracing_policy.yaml    \
    -v /sys/kernel/btf/vmlinux:/var/lib/tetragon/btf    \
    quay.io/cilium/tetragon-ci:latest                   \
    --tracing-policy /tracing_policy.yaml

We added two flags to our commands:

  • -v $(pwd)/tracing_policy.yaml:/tracing_policy.yaml mounts the created local file into the container filesystem.
  • --tracing-policy /tracing_policy.yaml indicates Tetragon to load the policy in the provided file.

Now again, in another terminal, open the tetra CLI to see the new events generated by Tetragon, monitoring the openat system call.

docker exec tetragon-container tetra getevents -o compact

And in the third terminal, just read the tracing_policy.yaml file we created using cat with:

cat tracing_policy.yaml

The output from the tetra CLI should be similar to:

🚀 process  /usr/bin/cat tracingpolicy.yaml
📬️ openat   /usr/bin/cat /etc/ld.so.cache
📬️ openat   /usr/bin/cat /lib/aarch64-linux-gnu/libc.so.6
📬️ openat   /usr/bin/cat tracingpolicy.yaml
💥 exit     /usr/bin/cat tracingpolicy.yaml 0

We can see that during the execution of cat, because it is dynamically linked, it used the openat syscall to open:

  • the cache file for shared libraries ld.so.cache;
  • the libc.so.6 library;
  • the file we actually wanted to read tracing_policy.yaml.

This is the kind of information we could obtain by executing the program with strace for example, but here Tetragon obtains this information directly from the kernel, transparently for the executed program.

Please note that tetra CLI, with the -o compact output format, automatically tried to decode the second argument as a string and printed it because it has the hardcoded knowledge that the openat syscall has an interesting string as second argument.

You can use tetra getevents to retrieve the whole JSON events and see the complete fields and values retrieved from the event. In our situation the output should be similar to the following.

{
  "process_kprobe": {
    "process": {
      "exec_id": "OjEwMTgzOTUyNjg1NjcwNDoyNjIzNw==",
      "pid": 26237,
      "uid": 1000,
      "cwd": "/home/ubuntu",
      "binary": "/usr/bin/cat",
      "arguments": "tracing_policy.yaml",
      "flags": "execve clone",
      "start_time": "2023-04-06T18:30:03.405730761Z",
      "auid": 1000,
      "parent_exec_id": "OjEwMTczMDEyNTQ3MjUxMDoyNjIwOA==",
      "refcnt": 1
    },
    "parent": {
      "exec_id": "OjEwMTczMDEyNTQ3MjUxMDoyNjIwOA==",
      "pid": 26208,
      "uid": 1000,
      "cwd": "/home/ubuntu",
      "binary": "/bin/bash",
      "flags": "execve clone",
      "start_time": "2023-04-06T18:28:14.004347210Z",
      "auid": 1000,
      "parent_exec_id": "OjEwMTczMDEwMjc2NTg0NjoyNjIwNw==",
      "refcnt": 2
    },
    "function_name": "__x64_sys_openat",
    "args": [
      {
        "int_arg": -100
      },
      {
        "string_arg": "tracing_policy.yaml"
      },
      {
        "int_arg": 0
      }
    ],
    "action": "KPROBE_ACTION_POST"
  },
  "time": "2023-04-06T18:30:03.406205829Z"
}

You can see that Tetragon collected information on the process itself, with for example, the uid, the binary name, the start_time and much more. It also indicates process parent, /bin/bash in this case. This is the common base of information we also get from the process lifecycle sensors. Finally, we can see that Tetragon hooked the function __x64_sys_openat (for an arm64 host it would be __arm64_sys_openat) and that it also collected the arguments’ value.

Enforcement with TracingPolicy

Let’s see how we can modify the previous TracingPolicy to do enforcement with Tetragon.

Overriding return values

First, let’s say we don’t want to stop any program that would try to open the tracing_policy.yaml but just stop them from actually using the syscall to open the file, as if they were not unauthorized to access the file. We could write the following TracingPolicy.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "file-access"
spec:
  kprobes:
  - call: "sys_openat"
    syscall: true
    args:
    - index: 0
      type: "int"
    - index: 1
      type: "string"
    selectors:
    - matchBinaries:
      - operator: In
        values:
        - "/usr/bin/cat"
      matchArgs:
      - index: 1
        operator: "Equal"
        values:
        - "tracing_policy.yaml\0"
      matchActions:
      - action: Override
        argError: -1

In the above TracingPolicy, we removed the third argument because we don’t need to extract it specifically, and we added two filters in our existing selector. One matchArgs to filter on the value of the index 1 argument, which contains the pathname to access, and one matchActions to perform on action on a such match.

Replace the content of the policy in the tracing_policy.yaml file with the above and restart Tetragon with the same command as before. If we open tetra with docker exec tetragon-container tetra getevents -o compact and perform a cat tracing_policy.yaml on the side, we should now observe an output similar to:

🚀 process  /usr/bin/cat tracing_policy.yaml
📬️ openat   /usr/bin/cat tracing_policy.yaml
💥 exit     /usr/bin/cat tracing_policy.yaml 1

And the cat tracing_policy.yaml should return the following error:

cat: tracing_policy.yaml: Operation not permitted

Synchronously stopping the process

Now, let’s say that accessing this file in this manner is critical and we want to stop any process that tries to perform such action immediately. We just need to modify the action in the matchActions filter from Override to Sigkill.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "file-access"
spec:
  kprobes:
  - call: "sys_openat"
    syscall: true
    args:
    - index: 0
      type: "int"
    - index: 1
      type: "string"
    selectors:
    - matchBinaries:
      - operator: In
        values:
        - "/usr/bin/cat"
      matchArgs:
      - index: 1
        operator: "Equal"
        values:
        - "tracing_policy.yaml\0"
      matchActions:
      - action: Sigkill

Again, replace the content of the policy in the tracing_policy.yaml file with the above and restart Tetragon with the same command as before. If we open tetra with docker exec tetragon-container tetra getevents -o compact and perform a cat tracing_policy.yaml on the side, we should now observe an output similar to:

🚀 process  /usr/bin/cat tracing_policy.yaml
📬️ openat   /usr/bin/cat tracing_policy.yaml
💥 exit     /usr/bin/cat tracing_policy.yaml SIGKILL

And the cat tracing_policy.yaml should return an error (that is actually returned by the shell that invoked the program):

Killed

Now the process will not have the time to catch the error and will be synchronously terminated as soon as he triggers the kernel function we hooked with our specified argument values.

What’s next

2.2 - Kubernetes quickstart guide

Deploy Tetragon on a Kubernetes cluster

This quickstart guide uses a Kind cluster and a helm-based installation to provide a simple way to get a hands on experience with Tetragon and the generated events. These events include monitoring process execution, network sockets, and file access to see what binaries are executing and making network connections or writing to sensitive files.

In this scenario, we are going to install a demo application,

  • observe all process execution happening inside a Kubernetes workload
  • detect file access and writes
  • observe network connections that a Kubernetes workload is making
  • detect privileged processes inside a Kubernetes workload

While, we use a Kubernetes Kind cluster in this guide, users can also apply the same concepts in other Kubernetes platforms, bare-metal, or VM environments.

Requirements

The base kernel should support BTF or the BTF file should be placed where Tetragon can read it.

For reference, the examples below use this Vagrantfile and we created our Kind cluster using the defaults options.

Create a cluster

Create a Kubernetes cluster using Kind or GKE.

Kind

Run the following command to create the Kubernetes cluster:

kind create cluster

GKE

Run the following command to create a GKE cluster:

export NAME="$(whoami)-$RANDOM"
gcloud container clusters create "${NAME}" \
  --zone us-west2-a \
  --num-nodes 1

Deploy Tetragon

To install and deploy Tetragon, run the following commands:

helm repo add cilium https://helm.cilium.io
helm repo update
helm install tetragon cilium/tetragon -n kube-system
kubectl rollout status -n kube-system ds/tetragon -w

By default, kube-system pods are filtered. For the examples below, we use the demo deployment from Cilium to generate events.

Deploy the demo application

Once Tetragon is installed, you can use our demo application to explore the Security Observability Events:

kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.11/examples/minikube/http-sw-app.yaml

Before going forward, verify that all pods are up and running - it might take several seconds for some pods until they satisfy all the dependencies:

kubectl get pods

The output should be similar to:

NAME                         READY   STATUS    RESTARTS   AGE
deathstar-6c94dcc57b-7pr8c   1/1     Running   0          10s
deathstar-6c94dcc57b-px2vw   1/1     Running   0          10s
tiefighter                   1/1     Running   0          10s
xwing                        1/1     Running   0          10s

What’s next

2.3 - Explore security observability events

Learn how to start exploring the Tetragon events

After Tetragon and the demo application is up and running you can examine the security and observability events produced by Tetragon in different ways.

Raw JSON events

The first way is to observe the raw json output from the stdout container log:

kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f

The raw JSON events provide Kubernetes API, identity metadata, and OS level process visibility about the executed binary, its parent and the execution time.

tetra CLI

A second way is to pretty print the events using the tetra CLI. The tool also allows filtering by process, pod, and other fields.

If you are using homebrew, you can install the latest release with:

brew install tetra

Or you can download and install the latest release with the following commands:

GOOS=$(go env GOOS)
GOARCH=$(go env GOARCH)
curl -L --remote-name-all https://github.com/cilium/tetragon/releases/latest/download/tetra-${GOOS}-${GOARCH}.tar.gz{,.sha256sum}
sha256sum --check tetra-${GOOS}-${GOARCH}.tar.gz.sha256sum
sudo tar -C /usr/local/bin -xzvf tetra-${GOOS}-${GOARCH}.tar.gz
rm tetra-${GOOS}-${GOARCH}.tar.gz{,.sha256sum}

See the latest release page for supported GOOS/GOARCH binary releases.

To start printing events run:

kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f | tetra getevents -o compact

The tetra CLI is also available inside tetragon container.

kubectl exec -it -n kube-system ds/tetragon -c tetragon -- tetra getevents -o compact

2.4 - Deployment

Learn different ways to deploy and manage Tetragon

2.4.1 - Deploy on Kubernetes

Deploy and manage Tetragon on Kubernetes

The recommended way to deploy Tetragon on a Kubernetes cluster is to use the Helm chart with Helm 3. Tetragon uses the helm.cilium.io repository to release the helm chart.

Install

To install the latest release of the Tetragon helm chart, use the following command.

helm repo add cilium https://helm.cilium.io
helm repo update
helm install tetragon cilium/tetragon -n kube-system

To wait until Tetragon deployment is ready, use the following kubectl command:

kubectl rollout status -n kube-system ds/tetragon -w

Configuration

You can then make modifications to the Tetragon configuration using helm upgrade, see the following example.

helm upgrade tetragon cilium/tetragon -n kube-system --set tetragon.grpc.address=localhost:1337

You can also edit the tetragon-config ConfigMap directly and restart the Tetragon daemonset with:

kubectl edit cm tetragon-config -n kube-system
kubectl rollout restart ds/tetragon -n kube-system

Upgrade

Upgrade Tetragon using a new specific version of the helm chart.

helm upgrade tetragon cilium/tetragon -n kube-system --version 0.9.0

Uninstall

Uninstall Tetragon using the following command.

helm uninstall tetragon -n kube-system

2.4.2 - Deploy as a container

Install and manage Tetragon as a container without a Kubernetes cluster

Install

Stable versions

To run a stable version, please check Tetragon quay repository and select which version you want. For example if you want to run the latest version which is v0.11.0 currently.

docker run --name tetragon --rm -d                   \
    --pid=host --cgroupns=host --privileged          \
    -v /sys/kernel/btf/vmlinux:/var/lib/tetragon/btf \
    quay.io/cilium/tetragon:v0.11.0

Unstable-development versions

To run unstable development versions of Tetragon, use the latest tag from Tetragon-CI quay repository. This will run the image that was built from the latest commit available on the Tetragon main branch.

docker run --name tetragon --rm -d                  \
   --pid=host --cgroupns=host --privileged          \
   -v /sys/kernel/btf/vmlinux:/var/lib/tetragon/btf \
   quay.io/cilium/tetragon-ci:latest

Configuration

There are multiple ways to set configuration options:

  1. Append Tetragon controlling settings at the end of the command

    As an example set the file where to export JSON events with --export-filename argument:

    docker run --name tetragon --rm -d \
        --pid=host --cgroupns=host --privileged \
        -v /sys/kernel:/sys/kernel \
        quay.io/cilium/tetragon:v0.11.0 \
        /usr/bin/tetragon --export-filename /var/log/tetragon/tetragon.log
    

    For a complete list of CLI arguments, please check Tetragon daemon configuration.

  2. Environment variables

    docker run --name tetragon --rm -d \
        --pid=host --cgroupns=host --privileged \
        --env "TETRAGON_EXPORT_FILENAME=/var/log/tetragon/tetragon.log" \
        -v /sys/kernel:/sys/kernel \
        quay.io/cilium/tetragon:v0.11.0
    

    Every controlling setting can be set using environment variables. Prefix it with the key word TETRAGON_ then upper case the controlling setting. As an example to set where to export JSON events: --export-filename will be TETRAGON_EXPORT_FILENAME.

    For a complete list of all controlling settings, please check tetragon daemon configuration.

  3. Configuration files mounted as volumes

    On the host machine set the configuration drop-ins inside /etc/tetragon/tetragon.conf.d/ directory according to the configuration examples, then mount it as volume:

    docker run --name tetragon --rm -d \
        --pid=host --cgroupns=host --privileged \
        -v /sys/kernel:/sys/kernel \
        -v /etc/tetragon/tetragon.conf.d/:/etc/tetragon/tetragon.conf.d/ \
        quay.io/cilium/tetragon:v0.11.0
    

    This will map the /etc/tetragon/tetragon.conf.d/ drop-in directory from the host into the container.

See Tetragon daemon configuration reference for further details.

What’s next

2.4.3 - Deploy with a package

Install and manage Tetragon via released packages.

Install

Tetragon will be managed as a systemd service. Tarballs are built and distributed along the assets in the releases.

  1. First download the latest binary tarball, using curl for example to download the amd64 release:

    curl -LO https://github.com/cilium/tetragon/releases/download/v0.11.0/tetragon-v0.11.0-amd64.tar.gz
    
  2. Extract the downloaded archive, and start the install script to install Tetragon. Feel free to inspect the script before starting it.

    tar -xvf tetragon-v0.11.0-amd64.tar.gz
    cd tetragon-v0.11.0-amd64/
    sudo ./install.sh
    

    If Tetragon was successfully installed, the final output should be similar to:

    Tetragon installed successfully!
    
  3. Finally, you can check the Tetragon systemd service.

    sudo systemctl status tetragon
    

    The output should be similar to:

    ● tetragon.service - Tetragon eBPF-based Security Observability and Runtime Enforcement
     Loaded: loaded (/lib/systemd/system/tetragon.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2023-01-23 20:08:16 CET; 5s ago
       Docs: https://github.com/cilium/tetragon/
    Main PID: 138819 (tetragon)
      Tasks: 17 (limit: 18985)
     Memory: 151.7M
        CPU: 913ms
     CGroup: /system.slice/tetragon.service
             └─138819 /usr/local/bin/tetragon
    

Configuration

The default Tetragon configuration shipped with the Tetragon package will be installed in /usr/local/lib/tetragon/tetragon.conf.d/. Local administrators can change the configuration by adding drop-ins inside /etc/tetragon/tetragon.conf.d/ to override the default settings or use the command line flags. To restore default settings, remove any added configuration inside /etc/tetragon/tetragon.conf.d/.

See Tetragon daemon configuration for further details.

Upgrade

To upgrade Tetragon:

  1. Download the new tarball.

    curl -LO https://github.com/cilium/tetragon/releases/download/v0.11.0/tetragon-v0.11.0-amd64.tar.gz
    
  2. Stop the Tetragon service.

    sudo systemctl stop tetragon
    
  3. Remove the old Tetragon version.

    sudo rm -fr /usr/lib/systemd/system/tetragon.service
    sudo rm -fr /usr/local/bin/tetragon
    sudo rm -fr /usr/local/lib/tetragon/
    
  4. Install the upgraded Tetragon version.

    tar -xvf tetragon-v0.11.0-amd64.tar.gz
    cd tetragon-v0.11.0-amd64/
    sudo ./install.sh
    

Uninstall

To completely remove Tetragon run the uninstall.sh script that is provided inside the tarball.

sudo ./uninstall.sh

Or remove it manually.

sudo systemctl stop tetragon
sudo systemctl disable tetragon
sudo rm -fr /usr/lib/systemd/system/tetragon.service
sudo systemctl daemon-reload
sudo rm -fr /usr/local/bin/tetragon
sudo rm -fr /usr/local/bin/tetra
sudo rm -fr /usr/local/lib/tetragon/

To just purge custom settings:

sudo rm -fr /etc/tetragon/

Operating

gRPC API access

To access the gRPC API with tetra client, set --server-address to point to the corresponding address:

sudo tetra --server-address unix:///var/run/tetragon/tetragon.sock getevents

See restrict gRPC API access for further details.

Tetragon Events

By default JSON events are logged to /var/log/tetragon/tetragon.log unless this location is changed. Logs are always rotated into the same directory.

To read real-time JSON events, tailing the logs file is enough.

sudo tail -f /var/log/tetragon/tetragon.log

Tetragon also ships a gRPC client that can be used to receive events.

  1. To print events in json format using tetra gRPC client:

    sudo tetra --server-address "unix:///var/run/tetragon/tetragon.sock" getevents
    
  2. To print events in human compact format:

    sudo tetra --server-address "unix:///var/run/tetragon/tetragon.sock" getevents -o compact
    

What’s next

See Explore security observability events to learn more about how to see the Tetragon events.

2.5 - Install tetra CLI

To interact with Tetragon, install the Tetragon client CLI tetra

This guide presents various methods to install tetra in your environment.

Install the latest release

Autodetect your environment

This shell script autodetects the OS and the architecture, downloads the archive of the binary and its SHA 256 digest, compares that the actual digest with the supposed one, installs the binary, and removes the download artifacts.

GOOS=$(go env GOOS)
GOARCH=$(go env GOARCH)
curl -L --remote-name-all https://github.com/cilium/tetragon/releases/latest/download/tetra-${GOOS}-${GOARCH}.tar.gz{,.sha256sum}
sha256sum --check tetra-${GOOS}-${GOARCH}.tar.gz.sha256sum
sudo tar -C /usr/local/bin -xzvf tetra-${GOOS}-${GOARCH}.tar.gz
rm tetra-${GOOS}-${GOARCH}.tar.gz{,.sha256sum}

Quick install for each environment

This installation method retrieves the adapted archived for your environment, extract it and install it in the /usr/local/bin directory.

curl -L https://github.com/cilium/tetragon/releases/latest/download/tetra-linux-amd64.tar.gz | tar -xz
sudo mv tetra /usr/local/bin
curl -L https://github.com/cilium/tetragon/releases/latest/download/tetra-linux-arm64.tar.gz | tar -xz
sudo mv tetra /usr/local/bin
curl -L https://github.com/cilium/tetragon/releases/latest/download/tetra-darwin-amd64.tar.gz | tar -xz
sudo mv tetra /usr/local/bin
curl -L https://github.com/cilium/tetragon/releases/latest/download/tetra-darwin-arm64.tar.gz | tar -xz
sudo mv tetra /usr/local/bin
curl -L https://github.com/cilium/tetragon/releases/latest/download/tetra-windows-amd64.tar.gz
tar -xz tetra-windows-amd64.tar.gz
# move the binary in a directory in your PATH
curl -L https://github.com/cilium/tetragon/releases/latest/download/tetra-windows-arm64.tar.gz
tar -xz tetra-windows-arm64.tar.gz
# move the binary in a directory in your PATH

Install using homebrew

Homebrew is a package manager for macOS and Linux. A formulae is available to fetch precompiled binaries. You can also use it to build from sources (using the --build-from-source flag) with a Go dependency.

brew install tetra

Install a specific release

You can retrieve the release of tetra along the release of Tetragon on GitHub at the following URL: https://github.com/cilium/tetragon/releases.

To download a specific release you can use the following script, replacing the OS, ARCH and TAG values with your desired options.

OS=linux
ARCH=amd64
TAG=v0.9.0
curl -LO https://github.com/cilium/tetragon/releases/download/${TAG}/tetra-${OS}-${ARCH}.tar.gz | tar -xz

3 - Concepts

The concepts section helps you understand various Tetragon abstractions and mechanisms.

3.1 - Tracing Policy

Documentation for the TracingPolicy custom resource

Tetragon’s TracingPolicy is a user-configurable Kubernetes custom resource (CR) that allows users to trace arbitrary events in the kernel and optionally define actions to take on a match. Policies consist of a hook point (kprobes, tracepoints, and uprobes are supported), and selectors for in-kernel filtering and specifying actions. For more details, see hook points page and the selectors page.

For the complete custom resource definition (CRD) refer to the YAML file cilium.io_tracingpolicies.yaml. One practical way to explore the CRD is to use kubectl explain against a Kubernetes API server on which it is installed, for example kubectl explain tracingpolicy.spec.kprobes provides field-specific documentation and details on kprobe spec.

Tracing Policies can be loaded and unloaded at runtime in Tetragon, or on startup using flags.

  • With Kubernetes, you can use kubectl to add and remove a TracingPolicy.
  • You can use tetra gRPC CLI to add and remove a TracingPolicy.
  • You can use the --tracing-policy and --tracing-policy-dir flags, see more in the daemon configuration page.

3.1.1 - Example

Learn the basics of Tracing Policy via an example

To discover TracingPolicy, let’s understand via an example that will be explained, part by part, in this document:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "fd-install"
spec:
  kprobes:
  - call: "fd_install"
    syscall: false
    args:
    - index: 0
      type: "int"
    - index: 1
      type: "file"
    selectors:
    - matchArgs:
      - index: 1
        operator: "Equal"
        values:
        - "/tmp/tetragon"
      matchActions:
      - action: Sigkill

Required fields

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "fd-install"

The first part follows a common pattern among all Cilium Policies or more widely Kubernetes object. It first declares the Kubernetes API used, then the kind of Kubernetes object it is in this API and an arbitrary name for the object that has to comply with Kubernetes naming convention.

Hook point

spec:
  kprobes:
  - call: "fd_install"
    syscall: false
    args:
    - index: 0
      type: "int"
    - index: 1
      type: "file"

The beginning of the specification describe the hook point to use. Here we are using a kprobe, hooking on the kernel function fd_install. That’s the kernel function that gets called when a new file descriptor needs to be created. We indicate that it’s not a syscall, but a regular kernel function. We then specify the argument of the specified function symbol to be able to extract and optionally perform filtering on them.

See the hook points page for further information on the various hook points available and arguments.

Selectors

    selectors:
    - matchArgs:
      - index: 1
        operator: "Equal"
        values:
        - "/tmp/tetragon"
      matchActions:
      - action: Sigkill

Selectors allow you to filter on the events to extract only a subset of the events based on different properties and optionally take an enforcement action.

In the example, we filter on the argument at index 1, passing a file struct to the function. Tetragon has the knowledge on how to apply the Equal operator over a Linux kernel file struct and you can basically match on the path of the file.

Then we add the Sigkill action, meaning, that any match of the selector should send a SIGKILL signal to the process that initiated the event.

Learn more about the various selectors in the dedicated selectors page.

Policy effect

First, let’s create the /tmp/tetragon file with some content:

echo eBPF! > /tmp/tetragon

Starting Tetragon with the above TracingPolicy, for example putting the policy in the example.yaml file, compiling the project locally and starting Tetragon with (you can do similar things with container image releases, see the docker run command in the Try Tetragon on Linux guide:

sudo ./tetragon --bpf-lib bpf/objs --tracing-policy example.yaml

Reading the /tmp/tetragon file with cat:

cat /tmp/tetragon

Should result in the following events:

🚀 process  /usr/bin/cat /tmp/tetragon
📬 open     /usr/bin/cat /tmp/tetragon
💥 exit     /usr/bin/cat /tmp/tetragon SIGKILL

And the shell will return:

Killed

See more

For more examples of tracing policies, take a look at the examples/tracingpolicy folder in the Tetragon repository. Also read the following sections on hook points and selectors.

3.1.2 - Hook points

Hook points for Tracing Policies and arguments description

The spec or specification of a TracingPolicy can be composed of kprobes, tracepoints or uprobes at the moment. It describes the arbitrary event in the kernel that Tetragon will trace. These hook points have arguments and return values that can be specified using the args and returnArg fields detailed in the following sections.

Kprobes

Kprobes enables you to dynamically break into any kernel routine and collect debugging and performance information non-disruptively. kprobes are highly tight to your kernel version and might not be portable since the kernel symbols depend on your build.

Conveniently, you can list all kernel symbols reading the /proc/kallsyms file. For example to search for the write syscall kernel function, you can execute sudo grep sys_write /proc/kallsyms, the output should be similar to this, minus the architecture specific prefixes.

ffffdeb14ea712e0 T __arm64_sys_writev
ffffdeb14ea73010 T ksys_write
ffffdeb14ea73140 T __arm64_sys_write
ffffdeb14eb5a460 t proc_sys_write
ffffdeb15092a700 d _eil_addr___arm64_sys_writev
ffffdeb15092a740 d _eil_addr___arm64_sys_write

You can see that the exact name of the symbol for the write syscall on our kernel version is __arm64_sys_write. Note that on x86_64, the prefix should be __x64_ instead of __arm64_.

In our example, we will explore a kprobe hooking into the fd_install kernel function. The fd_install kernel function is called each time a file descriptor is installed into the file descriptor table of a process, typically referenced within system calls like open or openat. Hooking fd_install has its benefits and limitations, which are out of the scope of this document.

spec:
  kprobes:
  - call: "fd_install"
    syscall: false

As usual, kprobes calls can be defined independently in different policies, or together in the same Policy. For example, we can define trace multiple kprobes under the same tracing policy:

spec:
  kprobes:
  - call: "sys_read"
    syscall: true
    # [...]
  - call: "sys_write"
    syscall: true
    # [...]

Lists

It’s possible to define list of functions and use it in the kprobe’s call field.

Following example traces all sys_dup[23] syscalls.

spec:
  lists:
  - name: "dups"
    type: "syscalls"
    values:
    - "sys_dup"
    - "sys_dup2"
    - "sys_dup3"
  kprobes:
  - call: "list:dups"

It is basically a shortcut for following policy:

spec:
  kprobes:
  - call: "sys_dup"
    syscall: true
  - call: "sys_dup2"
    syscall: true
  - call: "sys_dup3"
    syscall: true

As shown in subsequent examples, its main benefit is allowing a single definition for calls that have the same filters.

The list is defined under lists field with arbitrary values for name and values fields.

spec:
  lists:
  - name: "dups"
    type: "syscalls"
    values:
    - "sys_dup"
    - "sys_dup2"
    - "sys_dup3"
    ...

It’s possible to define multiple lists.

spec:
  lists:
  - name: "dups"
    type: "syscalls"
    values:
    - "sys_dup"
    - "sys_dup2"
    - "sys_dup3"
    name: "another"
    - "sys_open"
    - "sys_close"

Specific list can be referenced in kprobe’s call field with "list:NAME" value.

spec:
  lists:
  - name: "dups"
  ...

  kprobes:
  - call: "list:dups"

The kprobe definition creates a kprobe for each item in the list and shares the rest of the config specified for kprobe.

List can also specify type field that implies extra checks on the values (like for syscall type) or denote that the list is generated automatically (see below). User must specify syscall type for list with syscall functions. Also syscall functions can’t be mixed with regular functions in the list.

The additional selector configuration is shared with all functions in the list. In following example we create 3 kprobes that share the same pid filter.

spec:
  lists:
  - name: "dups"
    type: "syscalls"
    values:
    - "sys_dup"
    - "sys_dup2"
    - "sys_dup3"
  kprobes:
  - call: "list:dups"
    selectors:
    - matchPIDs:
      - operator: In
        followForks: true
        isNamespacePID: false
        values:
        - 12345

It’s possible to use argument filter together with the list.

It’s important to understand that the argument will be retrieved by using the specified argument type for all the functions in the list.

Following example adds argument filter for first argument on all functions in dups list to match value 9999.

spec:
  lists:
  - name: "dups"
    type: "syscalls"
    values:
    - "sys_dup"
    - "sys_dup2"
    - "sys_dup3"
  kprobes:
  - call: "list:dups"
    args:
    - index: 0
      type: int
    selectors:
    - matchArgs:
      - index: 0
        operator: "Equal"
        values:
        - 9999

There are two additional special types of generated lists.

The generated_syscalls type of list that generates list with all possible syscalls on the system.

Following example traces all syscalls for /usr/bin/kill binary.

spec:
  lists:
  - name: "all-syscalls"
    type: "generated_syscalls"
  kprobes:
  - call: "list:all-syscalls"
    selectors:
    - matchBinaries:
      - operator: "In"
        values:
        - "/usr/bin/kill"

The generated_ftrace type of list that generates functions from ftrace available_filter_functions file with specified filter. The filter is specified with pattern field and expects regular expression.

Following example traces all kernel ksys_* functions for /usr/bin/kill binary.

spec:
  lists:
  - name: "ksys"
    type: "generated_ftrace"
    pattern: "^ksys_*"
  kprobes:
  - call: "list:ksys"
    selectors:
    - matchBinaries:
      - operator: "In"
        values:
        - "/usr/bin/kill"

Uprobes

This hook point method lacks documentation, see issue #878.

Tracepoints

A tracepoint placed in the Linux kernel code provides a hook to call a function that you can provide at runtime using Tetragon. Tracepoints have the advantage of being stable across kernel versions and thus more portable than kprobes.

To see the list of tracepoints available on your kernel, you can list them using sudo ls /sys/kernel/debug/tracing/events, the output should be similar to this.

alarmtimer    ext4            iommu           page_pool     sock
avc           fib             ipi             pagemap       spi
block         fib6            irq             percpu        swiotlb
bpf_test_run  filelock        jbd2            power         sync_trace
bpf_trace     filemap         kmem            printk        syscalls
bridge        fs_dax          kvm             pwm           task
btrfs         ftrace          libata          qdisc         tcp
cfg80211      gpio            lock            ras           tegra_apb_dma
cgroup        hda             mctp            raw_syscalls  thermal
clk           hda_controller  mdio            rcu           thermal_power_allocator
cma           hda_intel       migrate         regmap        thermal_pressure
compaction    header_event    mmap            regulator     thp
cpuhp         header_page     mmap_lock       rpm           timer
cros_ec       huge_memory     mmc             rpmh          tlb
dev           hwmon           module          rseq          tls
devfreq       i2c             mptcp           rtc           udp
devlink       i2c_slave       napi            sched         vmscan
dma_fence     initcall        neigh           scmi          wbt
drm           interconnect    net             scsi          workqueue
emulation     io_uring        netlink         signal        writeback
enable        iocost          oom             skb           xdp
error_report  iomap           page_isolation  smbus         xhci-hcd

You can then choose the subsystem that you want to trace, and look the tracepoint you want to use and its format. For example, if we choose the netif_receive_skb tracepoints from the net subsystem, we can read its format with sudo cat /sys/kernel/debug/tracing/events/net/netif_receive_skb/format, the output should be similar to the following.

name: netif_receive_skb
ID: 1398
format:
        field:unsigned short common_type;       offset:0;       size:2; signed:0;
        field:unsigned char common_flags;       offset:2;       size:1; signed:0;
        field:unsigned char common_preempt_count;       offset:3;       size:1; signed:0;
        field:int common_pid;   offset:4;       size:4; signed:1;

        field:void * skbaddr;   offset:8;       size:8; signed:0;
        field:unsigned int len; offset:16;      size:4; signed:0;
        field:__data_loc char[] name;   offset:20;      size:4; signed:0;

print fmt: "dev=%s skbaddr=%px len=%u", __get_str(name), REC->skbaddr, REC->len

Similarly to kprobes, tracepoints can also hook into system calls. For more details, see the raw_syscalls and syscalls subysystems.

An example of tracepoints TracingPolicy could be the following, observing all syscalls and getting the syscall ID from the argument at index 4:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "raw-syscalls"
spec:
  tracepoints:
  - subsystem: "raw_syscalls"
    event: "sys_enter"
    args:
    - index: 4
      type: "int64"

Arguments

Kprobes, uprobes and tracepoints all share a needed arguments fields called args. It’s a list of function arguments to include in the trace output. Indeed the BPF code that runs on the hook point requires information about the types of arguments of the function being traced to properly read, print and filter on its arguments. Currently, this information needs to be provided by the user under the args section. For the available types, see directly in the TracingPolicy CRD.

Following our example, here is the part that defines the arguments:

args:
- index: 0
  type: "int"
- index: 1
  type: "file"

Each argument can optionally include a ’label’ parameter, which will be included in the output. This can be used to annotate the arguments to help with understanding and processing the output. As an example, here is the same definition, with an appropriate label on the int argument:

args:
- index: 0
  type: "int"
  label: "FD"
- index: 1
  type: "file"

To properly read and hook onto the fd_install(unsigned int fd, struct file *file) function, the YAML snippet above tells the BPF code that the first argument is an int and the second argument is a file, which is the struct file of the kernel. In this way, the BPF code and its printer can properly collect and print the arguments.

These types are sorted by the index field, where you can specify the order. The indexing starts with 0. So, index: 0 means, this is going to be the first argument of the function, index: 1 means this is going to be the second argument of the function, etc.

Note that for some args types, char_buf and char_iovec, there are additional fields named returnCopy and sizeArgIndex available:

  • returnCopy indicates that the corresponding argument should be read later (when the kretprobe for the symbol is triggered) because it might not be populated when the kprobe is triggered at the entrance of the function. For example, a buffer supplied to read(2) won’t have content until kretprobe is triggered.
  • sizeArgIndex indicates the (1-based, see warning below) index of the arguments that represents the size of the char_buf or iovec. For example, for write(2), the third argument, size_t count is the number of char element that we can read from the const void *buf pointer from the second argument. Similarly, if we would like to capture the __x64_sys_writev(long, iovec *, vlen) syscall, then iovec has a size of vlen, which is going to be the third argument.

These flags can be combined, see the example below.

- call: "sys_write"
  syscall: true
  args:
  - index: 0
    type: "int"
  - index: 1
    type: "char_buf"
    returnCopy: true
    sizeArgIndex: 3
  - index: 2
    type: "size_t"

Note that you can specify which arguments you would like to print from a specific syscall. For example if you don’t care about the file descriptor, which is the first int argument with index: 0 and just want the char_buf, what is written, then you can leave this section out and just define:

args:
- index: 1
  type: "char_buf"
  returnCopy: true
  sizeArgIndex: 3
- index: 2
  type: "size_t"

This tells the printer to skip printing the int arg because it’s not useful.

For char_buf type up to the 4096 bytes are stored. Data with bigger size are cut and returned as truncated bytes.

You can specify maxData flag for char_buf type to read maximum possible data (currently 327360 bytes), like:

args:
- index: 1
  type: "char_buf"
  maxData: true
  sizeArgIndex: 3
- index: 2
  type: "size_t"

This field is only used for char_buff data. When this value is false (default), the bpf program will fetch at most 4096 bytes. In later kernels (>=5.4) tetragon supports fetching up to 327360 bytes if this flag is turned on.

The maxData flag does not work with returnCopy flag at the moment, so it’s usable only for syscalls/functions that do not require return probe to read the data.

Return values

A TracingPolicy spec can specify that the return value should be reported in the tracing output. To do this, the return parameter of the call needs to be set to true, and the returnArg parameter needs to be set to specify the type of the return argument. For example:

- call: "sk_alloc"
  syscall: false
  return: true
  args:
  - index: 1
    type: int
    label: "family"
  returnArg:
    type: sock

In this case, the sk_alloc hook is specified to return a value of type sock (a pointer to a struct sock). Whenever the sk_alloc hook is hit, not only will it report the family parameter in index 1, it will also report the socket that was created.

Return values for socket tracking

A unique feature of a sock being returned from a hook such as sk_alloc is that the socket it refers to can be tracked. Most networking hooks in the network stack are run in a context that is not that of the process that owns the socket for which the actions relate; this is because networking happens asynchronously and not entirely in-line with the process. The sk_alloc hook does, however, occur in the context of the process, such that the task, the PID, and the TGID are of the process that requested that the socket was created.

Specifying socket tracking tells Tetragon to store a mapping between the socket and the process’ PID and TGID; and to use that mapping when it sees the socket in a sock argument in another hook to replace the PID and TGID of the context with the process that actually owns the socket. This can be done by adding a returnArgAction to the call. Available actions are TrackSock and UntrackSock. See TrackSock and UntrackSock.

- call: "sk_alloc"
  syscall: false
  return: true
  args:
  - index: 1
    type: int
    label: "family"
  returnArg:
    type: sock
  returnArgAction: TrackSock

Socket tracking is only available on kernels >=5.3.

3.1.3 - Selectors

Perform in-kernel BPF filtering and actions on events

Selectors are a way to perform in-kernel BPF filtering on the events to export, or on the events on which to apply an action.

A TracingPolicy can contain from 0 to 5 selectors. A selector is composed of 1 or more filters. The available filters are the following:

Arguments filter

Arguments filters can be specified under the matchArgs field and provide filtering based on the value of the function’s argument.

In the next example, a selector is defined with a matchArgs filter that tells the BPF code to process only the function call for which the second argument, index equal to 1, concerns the file under the path /etc/passwd or /etc/shadow. It’s using the operator Equal to match against the value of the argument.

Note that conveniently, we can match against a path directly when the argument is of type file.

selectors:
- matchArgs:
  - index: 1
    operator: "Equal"
    values:
    - "/etc/passwd"
    - "/etc/shadow"

The available operators for matchArgs are:

  • Equal
  • NotEqual
  • Prefix
  • Postfix
  • Mask

Further examples

In the previous example, we used the operator Equal, but we can also use the Prefix operator and match against all files under /etc with:

selectors:
- matchArgs:
  - index: 1
    operator: "Prefix"
    values:
    - "/etc"

In this situation, an event will be created every time a process tries to access a file under /etc.

Although it makes less sense, you can also match over the first argument, to only detect events that will use the file descriptor 4, which is usually the first that come afters stdin, stdout and stderr in process. And combine that with the previous example.

- matchArgs:
  - index: 0
    operator: "Equal"
    values:
    - "3"
  - index: 1
    operator: "Prefix"
    values:
    - "/etc"

Return args filter

Arguments filters can be specified under the returnMatchArgs field and provide filtering based on the value of the function return value. It allows you to filter on the return value, thus success, error or value returned by a kernel call.

matchReturnArgs:
- operator: "NotEqual"
  values:
  - 0

The available operators for matchReturnArgs are:

  • Equal
  • NotEqual
  • Prefix
  • Postfix

A use case for this would be to detect the failed access to certain files, like /etc/shadow. Doing cat /etc/shadow will use a openat syscall that will returns -1 for a failed attempt with an unprivileged user.

PIDs filter

PIDs filters can be specified under the matchPIDs field and provide filtering based on the value of host pid of the process. For example, the following matchPIDs filter tells the BPF code that observe only hooks for which the host PID is equal to either pid1 or pid2 or pid3:

- matchPIDs:
  - operator: "In"
    followforks: true
    values:
    - "pid1"
    - "pid2"
    - "pid3"

The available operators for matchPIDs are:

  • In
  • NotIn

Further examples

Another example can be to collect all processes not associated with a container’s init PID, which is equal to 1. In this way, we are able to detect if there was a kubectl exec performed inside a container because processes created by kubectl exec are not children of PID 1.

- matchPIDs:
  - operator: NotIn
    followforks: false
    isNamespacePID: true
    values:
    - 1

Binaries filter

Binary filters can be specified under the matchBinaries field and provide filtering based on the value of a certain binary name. For example, the following matchBinaries selector tells the BPF code to process only system calls and kernel functions that are coming from cat or tail.

- matchBinaries:
  - operator: "In"
    values:
    - "/usr/bin/cat"
    - "/usr/bin/tail"

Currently, only the In operator type is supported and the values field has to be a map of strings. The default behaviour is followForks: true, so all the child processes are followed. The current limitation is 4 values.

Further examples

One example can be to monitor all the sys_write system calls which are coming from the /usr/sbin/sshd binary and its child processes and writing to stdin/stdout/stderr.

This is how we can monitor what was written to the console by different users during different ssh sessions. The matchBinaries selector in this case is the following:

- matchBinaries:
  - operator: "In"
    values:
    - "/usr/sbin/sshd"

while the whole kprobe call is the following:

- call: "sys_write"
  syscall: true
  args:
  - index: 0
    type: "int"
  - index: 1
    type: "char_buf"
    sizeArgIndex: 3
  - index: 2
    type: "size_t"
  selectors:
  # match to /sbin/sshd
  - matchBinaries:
    - operator: "In"
      values:
      - "/usr/sbin/sshd"
  # match to stdin/stdout/stderr
    matchArgs:
    - index: 0
      operator: "Equal"
      values:
      - "1"
      - "2"
      - "3"

Namespaces filter

Namespaces filters can be specified under the matchNamespaces field and provide filtering of calls based on Linux namespace. You can specify the namespace inode or use the special host_ns keyword, see the example and description for more information.

An example syntax is:

- matchNamespaces:
  - namespace: Pid
    operator: In
    values:
    - "4026531836"
    - "4026531835"

This will match if: [Pid namespace is 4026531836] OR [Pid namespace is 4026531835]

  • namespace can be: Uts, Ipc, Mnt, Pid, PidForChildren, Net, Cgroup, or User. Time and TimeForChildren are also available in Linux >= 5.6.
  • operator can be In or NotIn
  • values can be raw numeric values (i.e. obtained from lsns) or "host_ns" which will automatically be translated to the appropriate value.

Limitations

  1. We can have up to 4 values. These can be both numeric and host_ns inside a single namespace.
  2. We can have up to 4 namespace values under matchNamespaces in Linux kernel < 5.3. In Linux >= 5.3 we can have up to 10 values (i.e. the maximum number of namespaces that modern kernels provide).

Further examples

We can have multiple namespace filters:

selectors:
- matchNamespaces:
  - namespace: Pid
    operator: In
    values:
    - "4026531836"
    - "4026531835"
  - namespace: Mnt
    operator: In
    values:
    - "4026531833"
    - "4026531834"

This will match if: ([Pid namespace is 4026531836] OR [Pid namespace is 4026531835]) AND ([Mnt namespace is 4026531833] OR [Mnt namespace is 4026531834])

Use cases examples

Generate a kprobe event if /etc/shadow was opened by /bin/cat which either had host Net or Mnt namespace access

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "example_ns_1"
spec:
  kprobes:
    - call: "fd_install"
      syscall: false
      args:
        - index: 0
          type: int
        - index: 1
          type: "file"
      selectors:
        - matchBinaries:
          - operator: "In"
            values:
            - "/bin/cat"
          matchArgs:
          - index: 1
            operator: "Equal"
            values:
            - "/etc/shadow"
          matchNamespaces:
          - namespace: Mnt
            operator: In
            values:
            - "host_ns"
        - matchBinaries:
          - operator: "In"
            values:
            - "/bin/cat"
          matchArgs:
          - index: 1
            operator: "Equal"
            values:
            - "/etc/shadow"
          matchNamespaces:
          - namespace: Net
            operator: In
            values:
            - "host_ns"

This example has 2 selectors. Note that each selector starts with -.

Selector 1:

        - matchBinaries:
          - operator: "In"
            values:
            - "/bin/cat"
          matchArgs:
          - index: 1
            operator: "Equal"
            values:
            - "/etc/shadow"
          matchNamespaces:
          - namespace: Mnt
            operator: In
            values:
            - "host_ns"

Selector 2:

        - matchBinaries:
          - operator: "In"
            values:
            - "/bin/cat"
          matchArgs:
          - index: 1
            operator: "Equal"
            values:
            - "/etc/shadow"
          matchNamespaces:
          - namespace: Net
            operator: In
            values:
            - "host_ns"

We have [Selector1 OR Selector2]. Inside each selector we have filters. Both selectors have 3 filters (i.e. matchBinaries, matchArgs, and matchNamespaces) with different arguments. Adding a - in the beginning of a filter will result in a new selector.

So the previous CRD will match if:

[binary == /bin/cat AND arg1 == /etc/shadow AND MntNs == host] OR [binary == /bin/cat AND arg1 == /etc/shadow AND NetNs is host]

We can modify the previous example as follows:

Generate a kprobe event if /etc/shadow was opened by /bin/cat which has host Net and Mnt namespace access

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "example_ns_2"
spec:
  kprobes:
    - call: "fd_install"
      syscall: false
      args:
        - index: 0
          type: int
        - index: 1
          type: "file"
      selectors:
        - matchBinaries:
          - operator: "In"
            values:
            - "/bin/cat"
          matchArgs:
          - index: 1
            operator: "Equal"
            values:
            - "/etc/shadow"
          matchNamespaces:
          - namespace: Mnt
            operator: In
            values:
            - "host_ns"
          - namespace: Net
            operator: In
            values:
            - "host_ns"

Here we have a single selector. This CRD will match if:

[binary == /bin/cat AND arg1 == /etc/shadow AND (MntNs == host AND NetNs == host) ]

Capabilities filter

Capabilities filters can be specified under the matchCapabilities field and provide filtering of calls based on Linux capabilities in the specific sets.

An example syntax is:

- matchCapabilities:
  - type: Effective
    operator: In
    values:
    - "CAP_CHOWN"
    - "CAP_NET_RAW"

This will match if: [Effective capabilities contain CAP_CHOWN] OR [Effective capabilities contain CAP_NET_RAW]

  • type can be: Effective, Inheritable, or Permitted.
  • operator can be In or NotIn
  • values can be any supported capability. A list of all supported capabilities can be found in /usr/include/linux/capability.h.

Limitations

  1. There is no limit in the number of capabilities listed under values.
  2. Only one type field can be specified under matchCapabilities.

Namespace changes filter

Namespace changes filter can be specified under the matchNamespaceChanges field and provide filtering based on calls that are changing Linux namespaces. This filter can be useful to track execution of code in a new namespace or even container escapes that change their namespaces.

For instance, if an unprivileged process creates a new user namespace, it gains full privileges within that namespace. This grants the process the ability to perform some privileged operations within the context of this new namespace that would otherwise only be available to privileged root user. As a result, such filter is useful to track namespace creation, which can be abused by untrusted processes.

To keep track of the changes, when a process_exec happens, the namespaces of the process are recorded and these are compared with the current namespaces on the event with a matchNamespaceChanges filter.

matchNamespaceChanges:
- operator: In
  values:
  - "Mnt"

The unshare command, or executing in the host namespace using nsenter can be used to test this feature. See a demonstration example of this feature.

Capability changes filter

Capability changes filter can be specified under the matchCapabilityChanges field and provide filtering based on calls that are changing Linux capabilities.

To keep track of the changes, when a process_exec happens, the capabilities of the process are recorded and these are compared with the current capabilities on the event with a matchCapabilityChanges filter.

matchCapabilityChanges:
- type: Effective
  operator: In
  isNamespaceCapability: false
  values:
  - "CAP_SETUID"

See a demonstration example of this feature.

Actions filter

Actions filters are a list of actions that execute when an appropriate selector matches. They are defined under matchActions and currently, the following action types are supported:

All actions can be rate limited by adding the rateLimit parameter with a time value. This value defaults to seconds, but post-fixing ’m’ or ‘h’ will cause the value to be interpreted in minutes or hours. When this parameter is specified for an action, that action will check if the same action has fired within the time window, with the same inspected arguments. (Only the first 16 bytes of each inspected argument is used in the matching. Only supported on kernels v5.3 onwards.)

matchActions:
- action: Post
  rateLimit: 5m

Sigkill action

Sigkill action terminates synchronously the process that made the call that matches the appropriate selectors from the kernel. In the example below, every sys_write system call with a PID not equal to 1 or 0 attempting to write to /etc/passwd will be terminated. Indeed when using kubectl exec, a new process is spawned in the container PID namespace and is not a child of PID 1.

- call: "sys_write"
  syscall: true
  args:
  - index: 0
    type: "fd"
  - index: 1
    type: "char_buf"
    sizeArgIndex: 3
  - index: 2
    type: "size_t"
  selectors:
  - matchPIDs:
    - operator: NotIn
      followForks: true
      isNamespacePID: true
      values:
      - 0
      - 1
    matchArgs:
    - index: 0
      operator: "Prefix"
      values:
      - "/etc/passwd"
    matchActions:
    - action: Sigkill

Signal action

Signal action sends specified signal to current process. The signal number is specified with argSig value.

Following example is equivalent to the Sigkill action example above. The difference is to use the signal action with SIGKILL(9) signal.

- call: "sys_write"
  syscall: true
  args:
  - index: 0
    type: "fd"
  - index: 1
    type: "char_buf"
    sizeArgIndex: 3
  - index: 2
    type: "size_t"
  selectors:
  - matchPIDs:
    - operator: NotIn
      followForks: true
      isNamespacePID: true
      values:
      - 0
      - 1
    matchArgs:
    - index: 0
      operator: "Prefix"
      values:
      - "/etc/passwd"
    matchActions:
    - action: Signal
      argSig: 9

Override action

Override action allows to modify the return value of call. While Sigkill will terminate the entire process responsible for making the call, Override will run in place of the original kprobed function and return the value specified in the argError field. It’s then up to the code path or the user space process handling the returned value to whether stop or proceed with the execution.

For example, you can create a TracingPolicy that intercepts sys_symlinkat and will make it return -1 every time the first argument is equal to the string /etc/passwd:

kprobes:
- call: "sys_symlinkat"
  syscall: true
  args:
  - index: 0
    type: "string"
  - index: 1
    type: "int"
  - index: 2
    type: "string"
  selectors:
  - matchArgs:
    - index: 0
      operator: "Equal"
      values:
      - "/etc/passwd\0"
    matchActions:
    - action: Override
      argError: -1

FollowFD action

The FollowFD action allows to create a mapping using a BPF map between file descriptors numbers and filenames. It however needs to maintain a state correctly, see UnfollowFD and CopyFD related actions.

The fd_install kernel function is called each time a file descriptor must be installed into the file descriptor table of a process, typically referenced within system calls like open or openat. It is a good place for tracking file descriptor and filename matching.

Let’s take a look at the following example:

- call: "fd_install"
  syscall: false
  args:
  - index: 0
    type: int
  - index: 1
    type: "file"
  selectors:
  - matchPIDs:
      # [...]
    matchArgs:
      # [...]
    matchActions:
    - action: FollowFD
      argFd: 0
      argName: 1

This action uses the dedicated argFd and argName fields to get respectively the index of the file descriptor argument and the index of the name argument in the call.

UnfollowFD action

The UnfollowFD action takes a file descriptor from a system call and deletes the corresponding entry from the BPF map, where it was put under the FollowFD action.

Let’s take a look at the following example:

- call: "sys_close"
  syscall: true
  args:
  - index: 0
    type: "int"
  selectors:
  - matchPIDs:
    - operator: NotIn
      followForks: true
      isNamespacePID: true
      values:
      - 0
      - 1
    matchActions:
    - action: UnfollowFD
      argFd: 0

Similar to the FollowFD action, the index of the file descriptor is described under argFd:

matchActions:
- action: UnfollowFD
  argFd: 0

In this example, argFD is 0. So, the argument from the sys_close system call at index: 0 will be deleted from the BPF map whenever a sys_close is executed.

- index: 0
  type: "int"

CopyFD action

The CopyFD action is specific to duplication of file descriptor use cases. Similary to FollowFD, it takes an argFd and argName arguments. It can typically be used tracking the dup, dup2 or dup3 syscalls.

See the following example for illustration:

- call: "sys_dup2"
  syscall: true
  args:
  - index: 0
    type: "fd"
  - index: 1
    type: "int"
  selectors:
  - matchPIDs:
    # [...]
    matchActions:
    - action: CopyFD
      argFd: 0
      argName: 1
- call: "sys_dup3"
  syscall: true
  args:
  - index: 0
    type: "fd"
  - index: 1
    type: "int"
  - index: 2
    type: "int"
  selectors:
  - matchPIDs:
    # [...]
    matchActions:
    - action: CopyFD
      argFd: 0
      argName: 1

GetUrl action

The GetUrl action can be used to perform a remote interaction such as triggering Thinkst canaries or any system that can be triggered via an URL request. It uses the argUrl field to specify the URL to request using GET method.

matchActions:
- action: GetUrl
  argUrl: http://ebpf.io

DnsLookup action

The DnsLookup action can be used to perform a remote interaction such as triggering Thinkst canaries or any system that can be triggered via an DNS entry request. It uses the argFqdn field to specify the domain to lookup.

matchActions:
- action: DnsLookup
  argFqdn: ebpf.io

Post action

The Post action is intended to create an event but at the moment should be considered as deprecated as all TracingPolicy will generate an event by default.

NoPost action

The NoPost action can be used to suppress the event to be generated, but at the same time all its defined actions are performed.

It’s useful when you are not interested in the event itself, just in the action being performed.

Following example override openat syscall for “/etc/passwd” file but does not generate any event about that.

- call: "sys_openat"
  return: true
  syscall: true
  args:
  - index: 0
    type: int
  - index: 1
    type: "string"
  - index: 2
    type: "int"
  returnArg:
    type: "int"
  selectors:
  - matchPIDs:
    matchArgs:
    - index: 1
      operator: "Equal"
      values:
      - "/etc/passwd"
    matchActions:
    - action: Override
      argError: -2
    - action: NoPost

TrackSock action

The TrackSock action allows to create a mapping using a BPF map between sockets and processes. It however needs to maintain a state correctly, see UntrackSock related action. TrackSock works similarly to FollowFD, specifying the argument with the sock type using argSock instead of specifying the FD argument with argFd.

It is however more likely that socket tracking will be performed on the return value of sk_alloc as described above.

Socket tracking is only available on kernel >=5.3.

UntrackSock action

The UntrackSock action takes a struct sock pointer from a function call and deletes the corresponding entry from the BPF map, where it was put under the TrackSock action.

Let’s take a look at the following example:

- call: "__sk_free"
  syscall: false
  args:
    - index: 0
      type: sock
  selectors:
    - matchActions:
      - action: UntrackSock
        argSock: 0

Similar to the TrackSock action, the index of the sock is described under argSock:

- matchActions:
  - action: UntrackSock
    argSock: 0

In this example, argSock is 0. So, the argument from the __sk_free function call at index: 0 will be deleted from the BPF map whenever a __sk_free is executed.

- index: 0
  type: "sock"

Socket tracking is only available on kernel >=5.3.

Selector Semantics

The selector semantics of the CiliumTracingPolicy follows the standard Kubernetes semantics and the principles that are used by Cilium to create a unified policy definition.

To explain deeper the structure and the logic behind it, let’s consider first the following example:

selectors:
 - matchPIDs:
   - operator: In
     followForks: true
     values:
     - pid1
     - pid2
     - pid3
  matchArgs:
  - index: 0
    operator: "Equal"
    values:
    - fdString1

In the YAML above matchPIDs and matchArgs are logically AND together giving the expression:

(pid in {pid1, pid2, pid3} AND arg0=fdstring1)

Multiple values

When multiple values are given, we apply the OR operation between them. In case of having multiple values under the matchPIDs selector, if any value matches with the given pid from pid1, pid2 or pid3 then we accept the event:

pid==pid1 OR pid==pid2 OR pid==pid3

As an example, we can filter for sys_read() syscalls that were not part of the container initialization and the main pod process and tried to read from the /etc/passwd file by using:

selectors:
 - matchPIDs:
   - operator: NotIn
     followForks: true
     values:
     - 0
     - 1
  matchArgs:
  - index: 0
    operator: "Equal"
    values:
    - "/etc/passwd"

Similarly, we can use multiple values under the matchArgs selector:

(pid in {pid1, pid2, pid3} AND arg0={fdstring1, fdstring2})

If any value matches with fdstring1 or fdstring2, specifically (string==fdstring1 OR string==fdstring2) then we accept the event.

For example, we can monitor sys_read() syscalls accessing both the /etc/passwd or the /etc/shadow files:

selectors:
 - matchPIDs:
   - operator: NotIn
     followForks: true
     values:
     - 0
     - 1
  matchArgs:
  - index: 0
    operator: "Equal"
    values:
    - "/etc/passwd"
    - "/etc/shadow"

Multiple operators

When multiple operators are supported under matchPIDs or matchArgs, they are logically AND together. In case if we have multiple operators under matchPIDs:

selectors:
  - matchPIDs:
    - operator: In
      followForks: true
      values:
      - pid1
    - operator: NotIn
      followForks: true
      values:
      - pid2

then we would build the following expression on the BPF side:

(pid == 0[following forks]) && (pid != 1[following forks])

In case of having multiple matchArgs:

selectors:
 - matchPIDs:
   - operator: In
     followForks: true
     values:
     - pid1
     - pid2
     - pid3
  matchArgs:
  - index: 0
    operator: "Equal"
    values:
    - 1
  - index: 2
    operator: "lt"
    values:
    - 500

Then we would build the following expression on the BPF side

(pid in {pid1, pid2, pid3} AND arg0=1 AND arg2 < 500)

Operator types

There are different types supported for each operator. In case of matchArgs:

  • Equal
  • NotEqual
  • Prefix
  • Postfix
  • Mask
  • GreaterThan (aka GT)
  • LessThan (aka LT)
  • SPort - Source Port
  • NotSPort - Not Source Port
  • SPortPriv - Source Port is Privileged (0-1023)
  • NotSPortPriv - Source Port is Not Privileged (Not 0-1023)
  • DPort - Destination Port
  • NotDPort - Not Destination Port
  • DPortPriv - Destination Port is Privileged (0-1023)
  • NotDPortPriv - Destination Port is Not Privileged (Not 0-1023)
  • SAddr - Source Address, can be IPv4/6 address or IPv4/6 CIDR (for ex 1.2.3.4/24 or 2a1:56::1/128)
  • NotSAddr - Not Source Address
  • DAddr - Destination Address
  • NotDAddr - Not Destination Address
  • Protocol
  • Family
  • State

The operator types Equal and NotEqual are used to test whether the certain argument of a system call is equal to the defined value in the CR.

For example, the following YAML snippet matches if the argument at index 0 is equal to /etc/passwd:

      matchArgs:
      - index: 0
        operator: "Equal"
        values:
        - "/etc/passwd"

Both Equal and NotEqual are set operations. This means if multiple values are specified, they are ORd together in case of Equal, and ANDd together in case of NotEqual.

For example, in case of Equal the following YAML snippet matches if the argument at index 0 is in the set of {arg0, arg1, arg2}.

matchArgs:
- index: 0
  operator: "Equal"
  values:
  - "arg0"
  - "arg1"
  - "arg2"

The above would be executed in the kernel as

arg == arg0 OR arg == arg1 OR arg == arg2

In case of NotEqual the following YAML snippet matches if the argument at index 0 is not in the set of {arg0, arg1}.

matchArgs:
- index: 0
  operator: "NotEqual"
  values:
  - "arg0"
  - "arg1"

The above would be executed in the kernel as

arg != arg0 AND arg != arg1

The operator type Mask performs and bitwise operation on the argument value and defined values. The argument type needs to be one of the value types.

For example in following YAML snippet we match second argument for bits 1 and 9 (0x200 value). We could use single value 0x201 as well.

matchArgs:
- index: 2
  operator: "Mask"
  values:
  - 1
  - 0x200

The above would be executed in the kernel as

arg & 1 OR arg & 0x200

The value can be specified as hexadecimal (with 0x prefix) octal (with 0 prefix) or decimal value (no prefix).

The operator Prefix checks if the certain argument starts with the defined value, while the operator Postfix compares if the argument matches to the defined value as trailing.

The operators relating to ports, addresses and protocol are used with sock or skb types. Port operators can accept a range of ports specified as min:max as well as lists of individual ports. Address operators can accept IPv4/6 CIDR ranges as well as lists of individual addresses.

The Protocol operator can accept integer values to match against, or the equivalent IPPROTO_ enumeration. For example, UDP can be specified as either IPPROTO_UDP or 17; TCP can be specified as either IPPROTO_TCP or 6.

The Family operator can accept integer values to match against or the equivalent AF_ enumeration. For example, IPv4 can be specified as either AF_INET or 2; IPv6 can be specified as either AF_INET6 or 10.

The State operator can accept integer values to match against or the equivalent TCP_ enumeration. For example, an established socket can be matched with TCP_ESTABLISHED or 1; a closed socket with TCP_CLOSE or 7.

In case of matchPIDs:

  • In
  • NotIn

The operator types In and NotIn are used to test whether the pid of a system call is found in the provided values list in the CR. Both In and NotIn are set operations, which means if multiple values are specified they are ORd together in case of In and ANDd together in case of NotIn.

For example, in case of In the following YAML snippet matches if the pid of a certain system call is being part of the list of {0, 1}:

- matchPIDs:
  - operator: In
    followForks: true
    isNamespacePID: true
    values:
    - 0
    - 1

The above would be executed in the kernel as

pid == 0 OR pid == 1

In case of NotIn the following YAML snippet matches if the pid of a certain system call is not being part of the list of {0, 1}:

- matchPIDs:
  - operator: NotIn
    followForks: true
    isNamespacePID: true
    values:
    - 0
    - 1

The above would be executed in the kernel as

pid != 0 AND pid != 1

In case of matchBinaries:

  • In

The In operator type is used to test whether a binary name of a system call is found in the provided values list. For example, the following YAML snippet matches if the binary name of a certain system call is being part of the list of {binary0, binary1, binary2}:

- matchBinaries:
  - operator: "In"
    values:
    - "binary0"
    - "binary1"
    - "binary2"

Multiple selectors

When multiple selectors are configured they are logically ORd together.

selectors:
 - matchPIDs:
   - operator: In
     followForks: true
     values:
     - pid1
     - pid2
     - pid3
  matchArgs:
  - index: 0
    operator: "Equal"
    values:
    - 1
  - index: 2
    operator: "lt"
    values:
    -  500
 - matchPIDs:
   - operator: In
     followForks: true
     values:
     - pid1
     - pid2
     - pid3
  matchArgs:
  - index: 0
    operator: "Equal"
    values:
    - 2

The above would be executed in kernel as:

(pid in {pid1, pid2, pid3} AND arg0=1 AND arg2 < 500) OR
(pid in {pid1, pid2, pid3} AND arg0=2)

Limitations

Those limitations might be outdated, see issue #709.

Because BPF must be bounded we have to place limits on how many selectors can exist.

  • Max Selectors 8.
  • Max PID values per selector 4
  • Max MatchArgs per selector 5 (one per index)
  • Max MatchArg Values per MatchArgs 1 (limiting initial implementation can bump to 16 or so)

3.2 - K8s namespace and pod label filtering

Tetragon in-kernel filtering based on Kubernetes namespaces and pod label filters

Motivation

Tetragon is configured via TracingPolicies. Broadly speaking, TracingPolicies define what situations Tetragon should react to and how. The what can be, for example, specific system calls with specific argument values. The how defines what action the Tetragon agent should perform when the specified situation occurs. The most common action is generating an event, but there are others (e.g., returning an error without executing a function, or killing the corresponding process).

Here we discuss how to apply tracing policies only on a subset of pods running on the system via two mechanisms: namespaced policies, and pod-label filters. Tetragon implements both mechanisms in-kernel via eBPF. This is important for both observability and enforcement use-cases. For observability, copying only the relevant events from kernel- to user-space reduces overhead. For enforcement, performing the enforcement action in the kernel avoids the race-condition of doing it in user-space. For example, let us consider the case where we want to block an application from performing a system call. Performing the filtering in-kernel means that the application will never finish executing the system call, which is not possible if enforcement happens in user-space (after the fact).

To ensure that namespaced tracing policies are always correctly applied, Tetragon needs to perform actions before containers start executing. Tetragon supports this via OCI runtime hooks. If such hooks are not added, Tetragon will apply policies in a best-effort manner using information from the k8s API server.

Namespace filtering

For namespace filtering we use TracingPolicyNamespaced which has the same contents as a TracingPolicy, but it is defined in a specific namespace and it is only applied to pods of that namespace.

Pod label filters

For pod label filters, we use the PodSelector field of tracing policies to select the pods that the policy is applied to.

Demo

Setup

For this demo, we use containerd and configure appropriate run-time hooks using minikube.

First, let us start minikube, build and load images, and install Tetragon and OCI hooks:

minikube start --container-runtime=containerd
./contrib/rthooks/minikube-containerd-install-hook.sh
make image image-operator
minikube image load --daemon=true cilium/tetragon:latest cilium/tetragon-operator:latest
minikube ssh -- sudo mount bpffs -t bpf /sys/fs/bpf
helm install --namespace kube-system \
	--set tetragonOperator.image.override=cilium/tetragon-operator:latest \
	--set tetragon.image.override=cilium/tetragon:latest  \
	--set tetragon.enablePolicyFilter="true" \
	--set tetragon.grpc.address="unix:///var/run/cilium/tetragon/tetragon.sock" \
	tetragon ./install/kubernetes

Once the tetragon pod is up and running, we can get its name.

tetragon_pod=$(kubectl -n kube-system get pods -l app.kubernetes.io/name=tetragon -o custom-columns=NAME:.metadata.name --no-headers)

Next, we check the tetragon-operator logs and tetragon agent logs to ensure that everything is in order.

First, we check if the operator installed the TracingPolicyNamespaced CRD.

kubectl -n kube-system logs -c tetragon-operator $tetragon_pod

The expected output is:

level=info msg="Tetragon Operator: " subsys=tetragon-operator
level=info msg="CRD (CustomResourceDefinition) is installed and up-to-date" name=TracingPolicy/v1alpha1 subsys=k8s
level=info msg="Creating CRD (CustomResourceDefinition)..." name=TracingPolicyNamespaced/v1alpha1 subsys=k8s
level=info msg="CRD (CustomResourceDefinition) is installed and up-to-date" name=TracingPolicyNamespaced/v1alpha1 subsys=k8s
level=info msg="Initialization complete" subsys=tetragon-operator

Next, we check that policyfilter (the low level mechanism that implements the desired functionality) is indeed enabled.

kubectl -n kube-system logs -c tetragon $tetragon_pod

The output should include:

level=info msg="Enabling policy filtering"

Namespaced policies

For illustration purposes, we will use the lseek system call with an invalid argument. Specifically a file descriptor (the first argument) of -1. Normally, this operation would return a “Bad file descriptor error”.

Let us start a pod in the default namespace:

kubectl -n default run test --image=python -it --rm --restart=Never  -- python

Above command will result in the following python shell:

If you don't see a command prompt, try pressing enter.
>>>

There is no policy installed so attempting to do the lseek operation will just return an error. So using the python shell we can execute an lseek and see the returned error.

>>> import os
>>> os.lseek(-1,0,0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 9] Bad file descriptor
>>>

In another terminal, we install a policy in the default namespace:

cat << EOF | kubectl apply -n default -f -
apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
  name: "lseek-namespaced"
spec:
  kprobes:
  - call: "sys_lseek"
    syscall: true
    args:
    - index: 0
      type: "int"
    selectors:
    - matchArgs:
      - index: 0
        operator: "Equal"
        values:
        - "-1"
      matchActions:
      - action: Sigkill
EOF

The above tracing policy will kill the process that performs an lseek system call with a file descriptor of -1. Note that we use a SigKill action only for illustration purposes because it’s easier to observe its effects.

Then, attempting the lseek operation on the previous terminal, will result in the process getting killed:

>>> os.lseek(-1, 0, 0)
pod "test" deleted
pod default/test terminated (Error)

The same is true for a newly started container:

kubectl -n default run test --image=python -it --rm --restart=Never  -- python
If you don't see a command prompt, try pressing enter.
>>> import os
>>> os.lseek(-1, 0, 0)
pod "test" deleted
pod default/test terminated (Error)

Doing the same on another namespace:

kubectl create namespace test
kubectl -n test run test --image=python -it --rm --restart=Never  -- python

Will not kill the process and result in an error:

If you don't see a command prompt, try pressing enter.
>>> import os
>>> os.lseek(-1, 0, 0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 9] Bad file descriptor

Pod label filters

Let’s install a tracing policy with a pod label filter.

cat << EOF | kubectl apply -f -
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "lseek-podfilter"
spec:
  podSelector:
    matchLabels:
      app: "lseek-test"
  kprobes:
  - call: "sys_lseek"
    syscall: true
    args:
    - index: 0
      type: "int"
    selectors:
    - matchArgs:
      - index: 0
        operator: "Equal"
        values:
        - "-1"
      matchActions:
      - action: Sigkill
EOF

Pods without the label will not be affected:

kubectl run test  --image=python -it --rm --restart=Never  -- python
If you don't see a command prompt, try pressing enter.
>>> import os
>>> os.lseek(-1, 0, 0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  OSError: [Errno 9] Bad file descriptor
  >>>

But pods with the label will:

kubectl run test --labels "app=lseek-test" --image=python -it --rm --restart=Never  -- python
If you don't see a command prompt, try pressing enter.
>>> import os
>>> os.lseek(-1, 0, 0)
pod "test" deleted
pod default/test terminated (Error)

4 - Use Cases

This section presents various use cases on process, files, network and security monitoring and enforcement.

By default, Tetragon monitors process lifecycle, learn more about that in the dedicated use cases.

For more advanced use cases, Tetragon can observe tracepoints and arbitrary kernel calls via kprobes. For that, Tetragon must be extended and configured with custom resources objects named TracingPolicy. It can then generates process_tracepoint and process_kprobes events.

4.1 - Process lifecycle

Tetragon observes by default the process lifecycle via exec and exit

Tetragon observes process creation and termination with default configuration and generates process_exec and process_exit events:

  • The process_exec events include useful information about the execution of binaries and related process information. This includes the binary image that was executed, command-line arguments, the UID context the process was executed with, the process parent information, the capabilities that a process had while executed, the process start time, the Kubernetes Pod, labels and more.
  • The process_exit events, as the process_exec event shows how and when a process started, indicate how and when a process is removed. The information in the event includes the binary image that was executed, command-line arguments, the UID context the process was executed with, process parent information, process start time, the status codes and signals on process exit. Understanding why a process exited and with what status code helps understand the specifics of that exit.

Both these events include Linux-level metadata (UID, parents, capabilities, start time, etc.) but also Kubernetes-level metadata (Kubernetes namespace, labels, name, etc.). This data make the connection between node-level concepts, the processes, and Kubernetes or container environments.

These events enable a full lifecycle view into a process that can aid an incident investigation, for example, we can determine if a suspicious process is still running in a particular environment. For concrete examples of such events, see the next use case on process execution.

4.1.1 - Process execution

Monitor process lifecycle with process_exec and process_exit

This first use case is monitoring process execution, which can be observed with the Tetragon process_exec and process_exit JSON events. These events contain the full lifecycle of processes, from fork/exec to exit, including metadata such as:

  • Binary name: Defines the name of an executable file
  • Parent process: Helps to identify process execution anomalies (e.g., if a nodejs app forks a shell, this is suspicious)
  • Command-line argument: Defines the program runtime behavior
  • Current working directory: Helps to identify hidden malware execution from a temporary folder, which is a common pattern used in malwares
  • Kubernetes metadata: Contains pods, labels, and Kubernetes namespaces, which are critical to identify service owners, particularly in a multitenant environments
  • exec_id: A unique process identifier that correlates all recorded activity of a process

As a first step, let’s start monitoring the events from the xwing pod:

kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f | tetra getevents -o compact --namespace default --pod xwing

Then in another terminal, let’s kubectl exec into the xwing pod and execute some example commands:

kubectl exec -it xwing -- /bin/bash
whoami

If you observe, the output in the first terminal should be:

🚀 process default/xwing /bin/bash
🚀 process default/xwing /usr/bin/whoami
💥 exit    default/xwing /usr/bin/whoami 0

Here you can see the binary names along with its arguments, the pod info, and return codes in a compact one-line view of the events.

For more details use the raw JSON events to get detailed information, you can stop the Tetragon CLI by Crl-C and parse the tetragon.log file by executing:

kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f | jq 'select(.process_exec.process.pod.name=="xwing" or .process_exit.process.pod.name=="xwing")'

Example process_exec and process_exit events can be:

Process Exec Event

{
  "process_exec": {
    "process": {
      "exec_id": "a2luZC1jb250cm9sLXBsYW5lOjExNDI4NjE1NjM2OTAxOjUxNTgz",
      "pid": 51583,
      "uid": 0,
      "cwd": "/",
      "binary": "/usr/bin/whoami",
      "arguments": "--version",
      "flags": "execve rootcwd clone",
      "start_time": "2022-05-11T12:54:45.615Z",
      "auid": 4294967295,
      "pod": {
        "namespace": "default",
        "name": "xwing",
        "container": {
          "id": "containerd://1fb931d2f6e5e4cfdbaf30fdb8e2fdd81320bdb3047ded50120a4f82838209ce",
          "name": "spaceship",
          "image": {
            "id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
            "name": "docker.io/tgraf/netperf:latest"
          },
          "start_time": "2022-05-11T10:07:33Z",
          "pid": 50
        }
      },
      "docker": "1fb931d2f6e5e4cfdbaf30fdb8e2fdd",
      "parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjkwNzkyMjU2MjMyNjk6NDM4NzI=",
      "refcnt": 1
    },
    "parent": {
      "exec_id": "a2luZC1jb250cm9sLXBsYW5lOjkwNzkyMjU2MjMyNjk6NDM4NzI=",
      "pid": 43872,
      "uid": 0,
      "cwd": "/",
      "binary": "/bin/bash",
      "flags": "execve rootcwd clone",
      "start_time": "2022-05-11T12:15:36.225Z",
      "auid": 4294967295,
      "pod": {
        "namespace": "default",
        "name": "xwing",
        "container": {
          "id": "containerd://1fb931d2f6e5e4cfdbaf30fdb8e2fdd81320bdb3047ded50120a4f82838209ce",
          "name": "spaceship",
          "image": {
            "id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
            "name": "docker.io/tgraf/netperf:latest"
          },
          "start_time": "2022-05-11T10:07:33Z",
          "pid": 43
        }
      },
      "docker": "1fb931d2f6e5e4cfdbaf30fdb8e2fdd",
      "parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjkwNzkxODU5NTMzOTk6NDM4NjE=",
      "refcnt": 1
    }
  },
  "node_name": "kind-control-plane",
  "time": "2022-05-11T12:54:45.615Z"
}

Process Exit Event

{
  "process_exit": {
    "process": {
      "exec_id": "a2luZC1jb250cm9sLXBsYW5lOjExNDI4NjE1NjM2OTAxOjUxNTgz",
      "pid": 51583,
      "uid": 0,
      "cwd": "/",
      "binary": "/usr/bin/whoami",
      "arguments": "--version",
      "flags": "execve rootcwd clone",
      "start_time": "2022-05-11T12:54:45.615Z",
      "auid": 4294967295,
      "pod": {
        "namespace": "default",
        "name": "xwing",
        "container": {
          "id": "containerd://1fb931d2f6e5e4cfdbaf30fdb8e2fdd81320bdb3047ded50120a4f82838209ce",
          "name": "spaceship",
          "image": {
            "id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
            "name": "docker.io/tgraf/netperf:latest"
          },
          "start_time": "2022-05-11T10:07:33Z",
          "pid": 50
        }
      },
      "docker": "1fb931d2f6e5e4cfdbaf30fdb8e2fdd",
      "parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjkwNzkyMjU2MjMyNjk6NDM4NzI="
    },
    "parent": {
      "exec_id": "a2luZC1jb250cm9sLXBsYW5lOjkwNzkyMjU2MjMyNjk6NDM4NzI=",
      "pid": 43872,
      "uid": 0,
      "cwd": "/",
      "binary": "/bin/bash",
      "flags": "execve rootcwd clone",
      "start_time": "2022-05-11T12:15:36.225Z",
      "auid": 4294967295,
      "pod": {
        "namespace": "default",
        "name": "xwing",
        "container": {
          "id": "containerd://1fb931d2f6e5e4cfdbaf30fdb8e2fdd81320bdb3047ded50120a4f82838209ce",
          "name": "spaceship",
          "image": {
            "id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
            "name": "docker.io/tgraf/netperf:latest"
          },
          "start_time": "2022-05-11T10:07:33Z",
          "pid": 43
        }
      },
      "docker": "1fb931d2f6e5e4cfdbaf30fdb8e2fdd",
      "parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjkwNzkxODU5NTMzOTk6NDM4NjE="
    }
  },
  "node_name": "kind-control-plane",
  "time": "2022-05-11T12:54:45.616Z"
}

4.1.2 - Advanced Process execution

Advanced Process Execution monitoring using Tracing Policies

Monitor ELF or Flat binaries execution

Advanced process execution can be performed by using Tracing Policies to monitor the execve system call path.

If we want to monitor execution of Executable and Linkable Format (ELF) or flat binaries before they are actually executed. Then the process-exec-elf-begin tracing policy is a good first choice.

Before going forward, verify that all pods are up and running, ensure you deploy our Demo Application to explore the Security Observability Events:

kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.11/examples/minikube/http-sw-app.yaml

It might take several seconds for some pods until they satisfy all the dependencies:

kubectl get pods -A

The output should be similar to:

NAMESPACE            NAME                                         READY   STATUS    RESTARTS   AGE
default              deathstar-54bb8475cc-6c6lc                   1/1     Running   0          2m54s
default              deathstar-54bb8475cc-zmfkr                   1/1     Running   0          2m54s
default              tiefighter                                   1/1     Running   0          2m54s
default              xwing                                        1/1     Running   0          2m54s
kube-system          tetragon-sdwv6                               2/2     Running   0          27m

Let’s apply the process-exec-elf-begin Tracing Policy.

kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/process-exec/process-exec-elf-begin.yaml

Then start monitoring events with the tetra CLI:

kubectl exec -it -n kube-system ds/tetragon -c tetragon -- tetra getevents

In another terminal, kubectl exec into the xwing Pod:

kubectl exec -it xwing -- /bin/bash

And execute some commands:

id

The tetra CLI will generate the following ProcessKprobe events:

{
  "process_kprobe": {
    "process": {
      "exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE2NjY0MDI4MTA4MzcxOjM2NDk5",
      "pid": 36499,
      "uid": 0,
      "cwd": "/",
      "binary": "/bin/bash",
      "flags": "execve",
      "start_time": "2023-08-02T11:58:53.618461573Z",
      "auid": 4294967295,
      "pod": {
        "namespace": "default",
        "name": "xwing",
        "container": {
          "id": "containerd://775beeb1a25a95e10dc149d6eb166bf45dd5e6039e8af3b64e8fb4d29669f349",
          "name": "spaceship",
          "image": {
            "id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
            "name": "docker.io/tgraf/netperf:latest"
          },
          "start_time": "2023-08-02T07:24:54Z",
          "pid": 13
        },
        "pod_labels": {
          "app.kubernetes.io/name": "xwing",
          "class": "xwing",
          "org": "alliance"
        }
      },
      "docker": "775beeb1a25a95e10dc149d6eb166bf",
      "parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE2NjYyNzg3ODI1MTQ4OjM2NDkz",
      "refcnt": 1,
      "tid": 36499
    },
      "parent": {
      "exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE2NjYyNzg3ODI1MTQ4OjM2NDkz",
      "pid": 36493,
      "uid": 0,
      "cwd": "/",
      "binary": "/bin/bash",
      "flags": "execve rootcwd clone",
      "start_time": "2023-08-02T11:58:52.378178852Z",
      "auid": 4294967295,
      "pod": {
        "namespace": "default",
        "name": "xwing",
        "container": {
          "id": "containerd://775beeb1a25a95e10dc149d6eb166bf45dd5e6039e8af3b64e8fb4d29669f349",
          "name": "spaceship",
          "image": {
            "id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
            "name": "docker.io/tgraf/netperf:latest"
          },
          "start_time": "2023-08-02T07:24:54Z",
          "pid": 13
        },
        "pod_labels": {
          "app.kubernetes.io/name": "xwing",
          "class": "xwing",
          "org": "alliance"
        }
      },
      "docker": "775beeb1a25a95e10dc149d6eb166bf",
      "parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE2NjYyNzE2OTU0MjgzOjM2NDg0",
      "tid": 36493
    },
    "function_name": "security_bprm_creds_from_file",
    "args": [
      {
        "file_arg": {
          "path": "/bin/busybox"
        }
      }
    ],
    "action": "KPROBE_ACTION_POST"
  },
  "node_name": "kind-control-plane",
  "time": "2023-08-02T11:58:53.624096751Z"
}

In addition to the Kubernetes Identity and process metadata, ProcessKprobe events contain the binary being executed. In the above case they are:

  • function_name: where we are hooking into the kernel to read the binary that is being executed.
  • file_arg: that includes the path being executed, and here it is /bin/busybox that is the real binary being executed, since on the xwing pod the container is running busybox. The binary /usr/bin/id -> /bin/busybox points to busybox.

To disable the process-exec-elf-being Tracing Policy run:

kubectl delete -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/process-exec/process-exec-elf-begin.yaml

4.1.3 - Privileged execution

Monitor process capabilities and kernel namespace access

Tetragon also provides the ability to check process capabilities and kernel namespaces access.

This information would help us determine which process or Kubernetes pod has started or gained access to privileges or host namespaces that it should not have. This would help us answer questions like:

Which Kubernetes pods are running with CAP_SYS_ADMIN in my cluster?

Which Kubernetes pods have host network or pid namespace access in my cluster?

As a first step let’s enable visibility to capability and namespace changes via the configmap by setting enable-process-cred and enable-process-ns from false to true:

kubectl edit cm -n kube-system tetragon-config
# change "enable-process-cred" from "false" to "true"
# change "enable-process-ns" from "false" to "true"
# then save and exit

Restart the Tetragon daemonset:

kubectl rollout restart -n kube-system ds/tetragon

As a second step, let’s start monitoring the Security Observability events from the privileged test-pod workload:

kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f | tetra getevents --namespace default --pod test-pod

In another terminal let’s apply the privileged PodSpec:

kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/testdata/specs/testpod.yaml

If you observe the output in the first terminal, you can see the container start with CAP_SYS_ADMIN:

🚀 process default/test-pod /bin/sleep 365d                🛑 CAP_SYS_ADMIN
🚀 process default/test-pod /usr/bin/jq -r .bundle         🛑 CAP_SYS_ADMIN
🚀 process default/test-pod /usr/bin/cp /kind/product_name /kind/product_uuid /run/containerd/io.containerd.runtime.v2.task/k8s.io/7c7e513cd4d506417bc9d97dd9af670d94d9e84161c8c8 fdc9fa3a678289a59/rootfs/ 🛑 CAP_SYS_ADMIN

4.2 - Filename access

Monitor filename access using kprobe hooks

This page shows how you can create a tracing policy to monitor filename access. For general information about tracing policies, see the tracing policy page.

There are two aspects of the tracing policy: (i) what hooks you can use to monitor specific types of access, and (ii) how you can filter at the kernel level for only specific events.

Hooks

There are different ways applications can access and modify files, and for this tracing policy we focus in three different types.

The first is read and write accesses, which the most common way that files are accessed by applications. Applications can perform this type of accesses with a variety of different system calls: read and write, optimized system calls such as copy_file_range and sendfile, as well as asynchronous I/O system call families such as the ones provided by aio and io_uring. Instead of monitoring every system call, we opt to hook into the security_file_permission hook, which is a common execution point for all the above system calls.

Applications can also access files by mapping them directly into their virtual address space. Since it is difficult to caught the accesses themselves in this case, our policy will instead monitor the point when the files are mapped into the application’s virtual memory. To do so, we use the security_mmap_file hook.

Lastly, there is a family of system calls (e.g,. truncate) that allow to indirectly modify the contents of the file by changing its size. To catch these types of accesses we will hook into security_path_truncate.

Filtering

Using the hooks above, you can monitor all accesses in the system. This will create a large number of events, however, and it is frequently the case that you are only interested in a specific subset those events. It is possible to filter the events after their generation, but this induces unnecessary overhead. Tetragon, using BPF, allows filtering these events directly in the kernel.

For example, the following snippet shows how you can limit the events from the security_file_permission hook only for the /etc/shadow file. For this, you need to specify the arguments of the function that you hooking into, as well as their type.

  - call: "security_file_permission"
    syscall: false
    args:
    - index: 0
      type: "file" # (struct file *) used for getting the path
    - index: 1
      type: "int" # 0x04 is MAY_READ, 0x02 is MAY_WRITE
    selectors:
    - matchArgs:      
      - index: 0
        operator: "Equal"
        values:
        - "/etc/passwd" # filter by filename (/etc/passwd)
      - index: 1
        operator: "Equal"
        values:
        - "2" # filter by type of access (MAY_WRITE)

The previous example uses the Equal operator. Similarly, you can use the Prefix operator to filter events based on the prefix of a filename.

Examples

In this example, we monitor if a process inside a Kubernetes workload performs an read or write in the /etc/ directory. The policy may be extended with additional directories or specific files if needed.

As a first step, we apply the following policy that uses the three hooks mentioned previously as well as appropriate filtering:

kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/filename_monitoring.yaml

Next, you can start monitoring the events from the xwing pod:

kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f | tetra getevents -o compact --namespace default --pod xwing

In another terminal, kubectl exec into the xwing pod:

kubectl exec -it xwing -- /bin/bash

and edit the /etc/passwd file:

vi /etc/passwd

If you observe, the output in the first terminal should be:

🚀 process default/xwing /bin/bash
📚 read    default/xwing /bin/bash /etc/passwd
📚 read    default/xwing /bin/bash /etc/passwd
💥 exit    default/xwing /bin/bash  127
🚀 process default/xwing /usr/bin/vi /etc/passwd
📚 read    default/xwing /usr/bin/vi /etc/passwd
📚 read    default/xwing /usr/bin/vi /etc/passwd
📝 write   default/xwing /usr/bin/vi /etc/passwd
📝 write   default/xwing /usr/bin/vi /etc/passwd
📝 trunc   default/xwing /usr/bin/vi /etc/passwd
💥 exit    default/xwing /usr/bin/vi /etc/passwd 0

Note, that read and writes are only generated for /etc/ files based on BPF in-kernel filtering specified in the policy. The default CRD additionally filters events associated with the pod init process to filter init noise from pod start.

Similarly to the previous example, reviewing the JSON events provides additional data. An example process_kprobe event observing a write can be:

{
  "process_kprobe": {
    "process": {
      "exec_id": "a2luZC1jb250cm9sLXBsYW5lOjMyMDY2OTEyNDkyMTA4MToyMDk3ODM=",
      "pid": 209783,
      "uid": 0,
      "cwd": "/",
      "binary": "/usr/bin/vi",
      "arguments": "/etc/passwd",
      "flags": "execve rootcwd clone",
      "start_time": "2023-06-26T11:23:43.774969054Z",
      "auid": 4294967295,
      "pod": {
        "namespace": "default",
        "name": "xwing",
        "container": {
          "id": "containerd://1f1d1ed09d56f04c17857cc5154f11d22d738b04b28941accd08569183caa4b5",
          "name": "spaceship",
          "image": {
            "id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
            "name": "docker.io/tgraf/netperf:latest"
          },
          "start_time": "2023-06-26T11:11:01Z",
          "pid": 37
        },
        "pod_labels": {
          "app.kubernetes.io/name": "xwing",
          "class": "xwing",
          "org": "alliance"
        }
      },
      "docker": "1f1d1ed09d56f04c17857cc5154f11d",
      "parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjMyMDY2MDM3OTEzODY1MzoyMDk3NTE=",
      "refcnt": 1,
      "tid": 209783
    },
    "parent": {
      "exec_id": "a2luZC1jb250cm9sLXBsYW5lOjMyMDY2MDM3OTEzODY1MzoyMDk3NTE=",
      "pid": 209751,
      "uid": 0,
      "cwd": "/",
      "binary": "/bin/bash",
      "flags": "execve rootcwd clone",
      "start_time": "2023-06-26T11:23:35.029187062Z",
      "auid": 4294967295,
      "pod": {
        "namespace": "default",
        "name": "xwing",
        "container": {
          "id": "containerd://1f1d1ed09d56f04c17857cc5154f11d22d738b04b28941accd08569183caa4b5",
          "name": "spaceship",
          "image": {
            "id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
            "name": "docker.io/tgraf/netperf:latest"
          },
          "start_time": "2023-06-26T11:11:01Z",
          "pid": 28
        },
        "pod_labels": {
          "app.kubernetes.io/name": "xwing",
          "class": "xwing",
          "org": "alliance"
        }
      },
      "docker": "1f1d1ed09d56f04c17857cc5154f11d",
      "parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjMyMDY2MDM0NTk5MzM2NToyMDk3NDE=",
      "refcnt": 2,
      "tid": 209751
    },
    "function_name": "security_file_permission",
    "args": [
      {
        "file_arg": {
          "path": "/etc/passwd"
        }
      },
      {
        "int_arg": 2
      }
    ],
    "return": {
      "int_arg": 0
    },
    "action": "KPROBE_ACTION_POST"
  },
  "node_name": "kind-control-plane",
  "time": "2023-06-26T11:23:49.019160599Z"
}

In addition to the Kubernetes Identity and process metadata from exec events, process_kprobe events contain the arguments of the observed system call. In the above case they are

  • file_arg.path: the observed file path
  • int_arg: is the type of the operation (2 for a write and 4 for a read)
  • return.int_arg: is 0 if the operation is allowed

To disable the TracingPolicy run:

kubectl delete -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/filename_monitoring.yaml

Another example of a similar policy can be found in our examples folder.

Limitations

Note that this policy has certain limitations because it matches on the filename that the application uses to accesses. If an application accesses the same file via a hard link or a different bind mount, no event will be generated.

4.3 - Network observability

Monitor TCP connect using kprobe hooks

To view TCP connect events, apply the example TCP connect TracingPolicy:

kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/tcp-connect.yaml

To start monitoring events in the xwing pod run the Tetragon CLI:

kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f | tetra getevents -o compact --namespace default --pod xwing

In another terminal, start generate a TCP connection. Here we use curl.

kubectl exec -it xwing -- curl http://cilium.io

The output in the first terminal will capture the new connect and write,

🚀 process default/xwing /usr/bin/curl http://cilium.io
🔌 connect default/xwing /usr/bin/curl tcp 10.244.0.6:34965 -> 104.198.14.52:80
📤 sendmsg default/xwing /usr/bin/curl tcp 10.244.0.6:34965 -> 104.198.14.52:80 bytes 73
🧹 close   default/xwing /usr/bin/curl tcp 10.244.0.6:34965 -> 104.198.14.52:80
💥 exit    default/xwing /usr/bin/curl http://cilium.io 0

To disable the TracingPolicy run:

kubectl delete -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/tcp-connect.yaml

4.4 - Linux process credentials

Monitor Linux process credentials

On Linux each process has various associated user, group IDs, capabilities, secure management flags, keyring, LSM security that are used part of the security checks upon acting on other objects. These are called the task privileges or process credentials.

Changing the process credentials is a standard operation to perform privileged actions or to execute commands as another user. The obvious example is sudo that allows to gain high privileges and run commands as root or another user. An other example is services or containers that can gain high privileges during execution to perform restricted operations.

Composition of Linux process credentials

Traditional UNIX credentials

  • Real User ID
  • Real Group ID
  • Effective, Saved and FS User ID
  • Effective, Saved and FS Group ID
  • Supplementary groups

Linux Capabilities

  • Set of permitted capabilities: a limiting superset for the effective capabilities.
  • Set of inheritable capabilities: the set that may get passed across execve(2).
  • Set of effective capabilities: the set of capabilities a task is actually allowed to make use of itself.
  • Set of bounding capabilities: limits the capabilities that may be inherited across execve(2), especially when a binary is executed that will execute as UID 0.

Secure management flags (securebits).

These govern the way the UIDs/GIDs and capabilities are manipulated and inherited over certain operations such as execve(2).

Linux Security Module (LSM)

The LSM framework provides a mechanism for various security checks to be hooked by new kernel extensions. Tasks can have extra controls part of LSM on what operations they are allowed to perform.

Tetragon Process Credentials monitoring

Monitoring Linux process credentials is a good practice to idenfity programs running with high privileges. Tetragon allows retrieving Linux process credentials as a process_credentials object.

Changes to credentials can be monitored either in system calls or in internal kernel functions.

Generally it is better to monitor in internal kernel functions. For further details please read Advantages and disadvantages of kernel layer monitoring compared to the system call layer section.

4.4.1 - Monitor Process Credentials changes at the System Call layer

Monitor system calls that change Process Credentials

Tetragon can hook at the system calls that directly manipulate the credentials. This allows us to determine which process is trying to change its credentials and the new credentials that could be applied by the kernel.

This answers the questions:

Which process or container is trying to change its UIDs/GIDs in my cluster?

Which process or container is trying to change its capabilities in my cluster?

Before going forward, verify that all pods are up and running, ensure you deploy our Demo Application to explore the Security Observability Events:

kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.11/examples/minikube/http-sw-app.yaml

It might take several seconds for some pods until they satisfy all the dependencies:

kubectl get pods -A
NAMESPACE            NAME                                         READY   STATUS    RESTARTS   AGE
default              deathstar-54bb8475cc-6c6lc                   1/1     Running   0          2m54s
default              deathstar-54bb8475cc-zmfkr                   1/1     Running   0          2m54s
default              tiefighter                                   1/1     Running   0          2m54s
default              xwing                                        1/1     Running   0          2m54s
kube-system          tetragon-sdwv6                               2/2     Running   0          27m

Monitor UIDs/GIDs credential changes

We use the process.credentials.changes.at.syscalls Tracing Policy that hooks the setuid system calls family:

  • setuid
  • setgid
  • setfsuid
  • setfsgid
  • setreuid
  • setregid
  • setresuid
  • setresgid

Let’s apply the process.credentials.changes.at.syscalls Tracing Policy.

kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/process-credentials/process.credentials.changes.at.syscalls.yaml

Then start monitoring events with the tetra CLI:

kubectl exec -it -n kube-system ds/tetragon -c tetragon -- tetra getevents

In another terminal, kubectl exec into the xwing Pod:

kubectl exec -it xwing -- /bin/bash

And execute su as this will call the related setuid system calls:

su root

The tetra CLI will generate the following ProcessKprobe events:

{
  "process_kprobe": {
    "process": {
      "exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQwNzk4ODc2MDI2NTk4OjEyNTc5OA==",
      "pid": 125798,
      "uid": 0,
      "cwd": "/",
      "binary": "/bin/su",
      "arguments": "root",
      "flags": "execve rootcwd clone",
      "start_time": "2023-07-05T19:14:30.918693157Z",
      "auid": 4294967295,
      "pod": {
        "namespace": "default",
        "name": "xwing",
        "container": {
          "id": "containerd://55936e548de63f77ceb595d64966dd8e267b391ff0ef63b26c17eb8c2f6510be",
          "name": "spaceship",
          "image": {
            "id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
            "name": "docker.io/tgraf/netperf:latest"
          },
          "start_time": "2023-07-05T18:45:16Z",
          "pid": 19
        },
        "pod_labels": {
          "app.kubernetes.io/name": "xwing",
          "class": "xwing",
          "org": "alliance"
        }
      },
      "docker": "55936e548de63f77ceb595d64966dd8",
      "parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQwNzk1NjYyMDM3MzMyOjEyNTc5Mg==",
      "refcnt": 1,
      "tid": 125798
    },
    "parent": {
      "exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQwNzk1NjYyMDM3MzMyOjEyNTc5Mg==",
      "pid": 125792,
      "uid": 0,
      "cwd": "/",
      "binary": "/bin/bash",
      "flags": "execve rootcwd clone",
      "start_time": "2023-07-05T19:14:27.704703805Z",
      "auid": 4294967295,
      "pod": {
        "namespace": "default",
        "name": "xwing",
        "container": {
          "id": "containerd://55936e548de63f77ceb595d64966dd8e267b391ff0ef63b26c17eb8c2f6510be",
          "name": "spaceship",
          "image": {
            "id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
            "name": "docker.io/tgraf/netperf:latest"
          },
          "start_time": "2023-07-05T18:45:16Z",
          "pid": 13
        },
        "pod_labels": {
          "app.kubernetes.io/name": "xwing",
          "class": "xwing",
          "org": "alliance"
        }
      },
      "docker": "55936e548de63f77ceb595d64966dd8",
      "parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQwNzk1NjE2MTU0NzA2OjEyNTc4Mw==",
      "refcnt": 2,
      "tid": 125792
    },
    "function_name": "__x64_sys_setgid",
    "args": [
      {
        "int_arg": 0
      }
    ],
    "action": "KPROBE_ACTION_POST"
  },
  "node_name": "kind-control-plane",
  "time": "2023-07-05T19:14:30.918977160Z"
}
{
  "process_kprobe": {
    "process": {
      "exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQwNzk4ODc2MDI2NTk4OjEyNTc5OA==",
      "pid": 125798,
      "uid": 0,
      "cwd": "/",
      "binary": "/bin/su",
      "arguments": "root",
      "flags": "execve rootcwd clone",
      "start_time": "2023-07-05T19:14:30.918693157Z",
      "auid": 4294967295,
      "pod": {
        "namespace": "default",
        "name": "xwing",
        "container": {
          "id": "containerd://55936e548de63f77ceb595d64966dd8e267b391ff0ef63b26c17eb8c2f6510be",
          "name": "spaceship",
          "image": {
            "id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
            "name": "docker.io/tgraf/netperf:latest"
          },
          "start_time": "2023-07-05T18:45:16Z",
          "pid": 19
        },
        "pod_labels": {
          "app.kubernetes.io/name": "xwing",
          "class": "xwing",
          "org": "alliance"
        }
      },
      "docker": "55936e548de63f77ceb595d64966dd8",
      "parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQwNzk1NjYyMDM3MzMyOjEyNTc5Mg==",
      "refcnt": 1,
      "tid": 125798
    },
    "parent": {
      "exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQwNzk1NjYyMDM3MzMyOjEyNTc5Mg==",
      "pid": 125792,
      "uid": 0,
      "cwd": "/",
      "binary": "/bin/bash",
      "flags": "execve rootcwd clone",
      "start_time": "2023-07-05T19:14:27.704703805Z",
      "auid": 4294967295,
      "pod": {
        "namespace": "default",
        "name": "xwing",
        "container": {
          "id": "containerd://55936e548de63f77ceb595d64966dd8e267b391ff0ef63b26c17eb8c2f6510be",
          "name": "spaceship",
          "image": {
            "id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
            "name": "docker.io/tgraf/netperf:latest"
          },
          "start_time": "2023-07-05T18:45:16Z",
          "pid": 13
        },
        "pod_labels": {
          "app.kubernetes.io/name": "xwing",
          "class": "xwing",
          "org": "alliance"
        }
      },
      "docker": "55936e548de63f77ceb595d64966dd8",
      "parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQwNzk1NjE2MTU0NzA2OjEyNTc4Mw==",
      "refcnt": 2,
      "tid": 125792
    },
    "function_name": "__x64_sys_setuid",
    "args": [
      {
        "int_arg": 0
      }
    ],
    "action": "KPROBE_ACTION_POST"
  },
  "node_name": "kind-control-plane",
  "time": "2023-07-05T19:14:30.918990583Z"
}

In addition to the Kubernetes Identity and process metadata from exec events, ProcessKprobe events contain the arguments of the observed system call. In the above case they are:

  • function_name: the system call, __x64_sys_setuid or __x64_sys_setgid
  • int_arg: the uid or gid to use, in our case it’s 0 which corresponds to the root user.

To disable the process.credentials.changes.at.syscalls Tracing Policy run:

kubectl delete -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/process-credentials/process.credentials.changes.at.syscalls.yaml

4.4.2 - Monitor Process Credentials changes at the Kernel layer

Monitor Process Credentials changes at the kernel layer

Monitoring Process Credentials changes at the kernel layer is also possible. This allows to capture the new process_credentials that should be applied.

This process-creds-installed tracing policy can be used to answer the following questions:

Which process or container is trying to change its own UIDs/GIDs in the cluster?

Which process or container is trying to change its own capabilities in the cluster?

In which user namespace the credentials are being changed?

How to monitor process_credentials changes?

Advantages and disadvantages of kernel layer monitoring compared to the system call layer

The main advantages of monitoring at the kernel layer compared to the system call layer:

  • Not vulnerable to user space arguments tampering.

  • Ability to display the full new credentials to be applied.

  • It is more reliable since it has full context on where and how the new credentials should be applied including the user namespace.

  • A catch all layer for all system calls, and every normal kernel path that manipulate credentials.

  • One potential disadvantage is that this approach may generate a lot of events, so appropriate filtering must be applied to reduce the noise.

Kubernetes Environments

First, verify that your k8s environment is all setup and that all pods are up and running, and deploy the Demo Application:

kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.11/examples/minikube/http-sw-app.yaml

It might take several seconds until all pods are Running:

kubectl get pods -A
NAMESPACE            NAME                                         READY   STATUS    RESTARTS   AGE
default              deathstar-54bb8475cc-6c6lc                   1/1     Running   0          2m54s
default              deathstar-54bb8475cc-zmfkr                   1/1     Running   0          2m54s
default              tiefighter                                   1/1     Running   0          2m54s
default              xwing                                        1/1     Running   0          2m54s
kube-system          tetragon-sdwv6                               2/2     Running   0          27m

Monitor Process Credentials installation

We use the process-creds-installed Tracing Policy that hooks the kernel layer when credentials are being installed.

So let’s apply the process-creds-installed Tracing Policy.

kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/process-credentials/process-creds-installed.yaml

Then we start monitoring for events with tetra cli:

kubectl exec -it -n kube-system ds/tetragon -c tetragon -- tetra getevents

In another terminal, inside a pod and as a non root user we will execute a setuid binary (suid):

/tmp/su -

The tetra cli will generate the following ProcessKprobe events:

{
 "process_kprobe": {
    "process": {
      "exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE0MDczMDQyODk3MTc6MjIzODY=",
      "pid": 22386,
      "uid": 11,
      "cwd": "/",
      "binary": "/tmp/su",
      "arguments": "-",
      "flags": "execve rootcwd clone",
      "start_time": "2023-07-25T12:04:59.359333454Z",
      "auid": 4294967295,
      "pod": {
        "namespace": "default",
        "name": "xwing",
        "container": {
          "id": "containerd://2e58c8357465961fd96f758e87d0269dfb5f97c536847485de9d7ec62be34a64",
          "name": "spaceship",
          "image": {
            "id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
            "name": "docker.io/tgraf/netperf:latest"
          },
          "start_time": "2023-07-25T11:44:48Z",
          "pid": 43
        },
        "pod_labels": {
          "app.kubernetes.io/name": "xwing",
          "class": "xwing",
          "org": "alliance"
        }
      },
      "docker": "2e58c8357465961fd96f758e87d0269",
      "parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjEzOTU0MDY5OTA1ODc6MjIzNTI=",
      "refcnt": 1,
      "tid": 22386
    },
    "parent": {
      "exec_id": "a2luZC1jb250cm9sLXBsYW5lOjEzOTU0MDY5OTA1ODc6MjIzNTI=",
      "pid": 22352,
      "uid": 11,
      "cwd": "/",
      "binary": "/bin/sh",
      "flags": "execve rootcwd",
      "start_time": "2023-07-25T12:04:47.462035587Z",
      "auid": 4294967295,
      "pod": {
        "namespace": "default",
                "name": "xwing",
        "container": {
          "id": "containerd://2e58c8357465961fd96f758e87d0269dfb5f97c536847485de9d7ec62be34a64",
          "name": "spaceship",
          "image": {
            "id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
            "name": "docker.io/tgraf/netperf:latest"
          },
          "start_time": "2023-07-25T11:44:48Z",
          "pid": 41
        },
        "pod_labels": {
          "app.kubernetes.io/name": "xwing",
          "class": "xwing",
          "org": "alliance"
        }
      },
      "docker": "2e58c8357465961fd96f758e87d0269",
      "parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjEzOTU0MDQ3NzY5NzI6MjIzNTI=",
      "refcnt": 2,
      "tid": 22352
    },
    "function_name": "commit_creds",
    "args": [
      {
        "process_credentials_arg": {
          "uid": 0,
          "gid": 0,
          "euid": 0,
          "egid": 0,
          "suid": 0,
          "sgid": 0,
          "fsuid": 0,
          "fsgid": 0,
          "caps": {
            "permitted": [
              "CAP_CHOWN",
              "DAC_OVERRIDE",
              "CAP_FOWNER",
              "CAP_FSETID",
              "CAP_KILL",
              "CAP_SETGID",
              "CAP_SETUID",
              "CAP_SETPCAP",
              "CAP_NET_BIND_SERVICE",
              "CAP_NET_RAW",
              "CAP_SYS_CHROOT",
              "CAP_MKNOD",
              "CAP_AUDIT_WRITE",
              "CAP_SETFCAP"
            ],
            "effective": [
              "CAP_CHOWN",
              "DAC_OVERRIDE",
              "CAP_FOWNER",
              "CAP_FSETID",
              "CAP_KILL",
              "CAP_SETGID",
              "CAP_SETUID",
              "CAP_SETPCAP",
              "CAP_NET_BIND_SERVICE",
              "CAP_NET_RAW",
              "CAP_SYS_CHROOT",
              "CAP_MKNOD",
              "CAP_AUDIT_WRITE",
              "CAP_SETFCAP"
            ]
          },
          "user_ns": {
            "level": 0,
            "uid": 0,
            "gid": 0,
            "ns": {
              "inum": 4026531837,
              "is_host": true
            }
          }
        }
      }
    ],
    "action": "KPROBE_ACTION_POST"
  },
  "node_name": "kind-control-plane",
  "time": "2023-07-25T12:05:01.410834172Z"
}

In addition to the Kubernetes Identity and process metadata from exec events, ProcessKprobe events contain the arguments of the observed system call. In the above case they are:

  • function_name: the kernel commit_creds() function to install new credentials.
  • process_credentials_arg: the new process_credentials to be installed on the current process. It includes the UIDs/GIDs, the capabilities and the target user namespace.

Here we can clearly see that the suid binary is being executed by a user ID 11 in order to elevate its privileges to user ID 0 including capabilities.

To disable the process-creds-installed Tracing Policy run:

kubectl delete -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/process-credentials/process-creds-installed.yaml

4.5 - Host System Changes

Monitor Host System changes

Some pods need to change the host system or kernel parameters in order to perform administrative tasks, obvious examples are pods loading a kernel module to extend the operating system functionality, or pods managing the network.

However, there are also other cases where a compromised container may want to load a kernel module to hide its behaviour.

In this aspect, monitoring such host system changes helps to identify pods and containers that affect the host system.

Monitor Linux kernel modules

A kernel module is a code that can be loaded into the kernel image at runtime, without rebooting. These modules, which can be loaded by pods and containers, can modify the host system. The Monitor Linux kernel modules guide will assist you in observing such events.

4.5.1 - Monitor Linux Kernel Modules

Monitor Linux Kernel Modules operations

Monitoring kernel modules helps to identify processes that load kernel modules to add features, to the operating system, to alter host system functionality or even hide their behaviour. This can be used to answer the following questions:

Which process or container is changing the kernel?

Which process or container is loading or unloading kernel modules in the cuslter?

Which process or container requested a feature that triggered the kernel to automatically load a module?

Are the loaded kernel modules signed?

Monitor Loading kernel modules

Kubernetes Environments

After deploying Tetragon, use the monitor-kernel-modules tracing policy which generates ProcessKprobe events to trace kernel module operations.

Apply the monitor-kernel-modules tracing policy:

kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/host-changes/monitor-kernel-modules.yaml

Then start monitoring for events with tetra CLI:

kubectl exec -it -n kube-system ds/tetragon -c tetragon -- tetra getevents

When loading an out of tree module named kernel_module_hello.ko with the command insmod, tetra CLI will generate the following ProcessKprobe events:

1. Reading the kernel module from the file system

{
  "process_kprobe": {
    "process": {
      "exec_id": "OjEzMTg4MTQwNDUwODkwOjgyMDIz",
      "pid": 82023,
      "uid": 0,
      "cwd": "/home/tixxdz/tetragon",
      "binary": "/usr/sbin/insmod",
      "arguments": "contrib/tester-progs/kernel_module_hello.ko",
      "flags": "execve clone",
      "start_time": "2023-08-30T11:01:22.846516679Z",
      "auid": 1000,
      "parent_exec_id": "OjEzMTg4MTM4MjY2ODQyOjgyMDIy",
      "refcnt": 1,
      "tid": 82023
    },
    "parent": {
      "exec_id": "OjEzMTg4MTM4MjY2ODQyOjgyMDIy",
      "pid": 82022,
      "uid": 1000,
      "cwd": "/home/tixxdz/tetragon",
      "binary": "/usr/bin/sudo",
      "arguments": "insmod contrib/tester-progs/kernel_module_hello.ko",
      "flags": "execve",
      "start_time": "2023-08-30T11:01:22.844332959Z",
      "auid": 1000,
      "parent_exec_id": "OjEzMTg1NTE3MTgzNDM0OjgyMDIx",
      "refcnt": 1,
      "tid": 0
    },
    "function_name": "security_kernel_read_file",
    "args": [
      {
        "file_arg": {
          "path": "/home/tixxdz/tetragon/contrib/tester-progs/kernel_module_hello.ko"
        }
      },
      {
        "int_arg": 2
      }
    ],
    "return": {
      "int_arg": 0
    },
    "action": "KPROBE_ACTION_POST"
  },
  "time": "2023-08-30T11:01:22.847554295Z"
}

In addition to the process metadata from exec events, ProcessKprobe events contain the arguments of the observed call. In the above case they are:

  • security_kernel_read_file: the kernel security hook when the kernel loads file specified by user space.
  • file_arg: the full path of the kernel module on the file system.

2. Finalize loading of kernel modules

{
  "process_kprobe": {
    "process": {
      "exec_id": "OjEzMTg4MTQwNDUwODkwOjgyMDIz",
      "pid": 82023,
      "uid": 0,
      "cwd": "/home/tixxdz/tetragon",
      "binary": "/usr/sbin/insmod",
      "arguments": "contrib/tester-progs/kernel_module_hello.ko",
      "flags": "execve clone",
      "start_time": "2023-08-30T11:01:22.846516679Z",
      "auid": 1000,
      "parent_exec_id": "OjEzMTg4MTM4MjY2ODQyOjgyMDIy",
      "refcnt": 1,
      "tid": 82023
    },
    "parent": {
      "exec_id": "OjEzMTg4MTM4MjY2ODQyOjgyMDIy",
      "pid": 82022,
      "uid": 1000,
      "cwd": "/home/tixxdz/tetragon",
      "binary": "/usr/bin/sudo",
      "arguments": "insmod contrib/tester-progs/kernel_module_hello.ko",
      "flags": "execve",
      "start_time": "2023-08-30T11:01:22.844332959Z",
      "auid": 1000,
      "parent_exec_id": "OjEzMTg1NTE3MTgzNDM0OjgyMDIx",
      "refcnt": 1,
      "tid": 0
    },
    "function_name": "do_init_module",
    "args": [
      {
        "module_arg": {
          "name": "kernel_module_hello",
          "tainted": [
            "TAINT_OUT_OF_TREE_MODULE",
            "TAINT_UNSIGNED_MODULE"
          ]
        }
      }
    ],
    "action": "KPROBE_ACTION_POST"
  },
  "time": "2023-08-30T11:01:22.847638990Z"
}

This ProcessKprobe event contains:

  • do_init_module: the function call where the module is finaly loaded.
  • module_arg: the kernel module information, it contains:
    • name: the name of the kernel module as a string.
    • tainted: the module tainted flags that will be applied on the kernel. In the example above, it indicates we are loading an out-of-tree module, that is unsigned module which may compromise the integrity of our system.

Monitor Kernel Modules Signature

Kernels compiled with CONFIG_MODULE_SIG option will check if the modules being loaded were cryptographically signed. This allows to assert that:

  • If the module being loaded is signed, the kernel has its key and the signature verification succeeded.

  • The integrity of the system or the kernel was not compromised.

Kubernetes Environments

After deploying Tetragon, use the monitor-signed-kernel-modules tracing policy which generates ProcessKprobe events to identify if kernel modules are signed or not.

Apply the monitor-signed-kernel-modules tracing policy:

kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/host-changes/monitor-signed-kernel-modules.yaml

Before going forward, deploy the test-pod into the demo-app namespace, which has its security context set to privileged. This allows to run the demo by mountig an xfs file system inside the test-pod which requires privileges, but will also trigger an automatic xfs module loading operation.

kubectl create namespace demo-app
kubectl apply -n demo-app -f https://raw.githubusercontent.com/cilium/tetragon/main/testdata/specs/testpod.yaml

Start monitoring for events with tetra CLI:

kubectl exec -it -n kube-system ds/tetragon -c tetragon -- tetra getevents

In another terminal, kubectl exec into the test-pod and run the following commands to create an xfs filesystem:

kubectl exec -it -n demo-app test-pod -- /bin/sh
apk update
dd if=/dev/zero of=loop.xfs bs=1 count=0 seek=32M
ls -lha loop.xfs
apk add xfsprogs
mkfs.xfs -q loop.xfs
mkdir /mnt/xfs.volume
mount -o loop -t xfs loop.xfs /mnt/xfs.volume/
losetup -a | grep xfs

Now the xfs filesystem should be mounted at /mnt/xfs.volume. To unmount it and release the loop device run:

umount /mnt/xfs.volume/

tetra CLI will generate the following events:

1. Automatic loading of kernel modules

First the mount command will trigger an automatic operation to load the xfs kernel module.

{
  "process_kprobe": {
    "process": {
      "exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQxMjc1NTA0OTk5NTcyOjEzMDg3Ng==",
      "pid": 130876,
      "uid": 0,
      "cwd": "/",
      "binary": "/bin/mount",
      "arguments": "-o loop -t xfs loop.xfs /mnt/xfs.volume/",
      "flags": "execve rootcwd clone",
      "start_time": "2023-09-09T23:27:42.732039059Z",
      "auid": 4294967295,
      "pod": {
        "namespace": "demo-app",
        "name": "test-pod",
        "container": {
          "id": "containerd://1e910d5cc8d8d68c894934170b162ef93aea5652867ed6bd7c620c7e3f9a10f1",
          "name": "test-pod",
          "image": {
            "id": "docker.io/cilium/starwars@sha256:f92c8cd25372bac56f55111469fe9862bf682385a4227645f5af155eee7f58d9",
            "name": "docker.io/cilium/starwars:latest"
          },
          "start_time": "2023-09-09T22:46:09Z",
          "pid": 45672
        },
        "workload": "test-pod"
      },
      "docker": "1e910d5cc8d8d68c894934170b162ef",
      "parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQxMjYyOTc1MjI1MDkzOjEzMDgwOQ==",
      "refcnt": 1,
      "tid": 130876
    },
    "parent": {
      "exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQxMjYyOTc1MjI1MDkzOjEzMDgwOQ==",
      "pid": 130809,
      "uid": 0,
      "cwd": "/",
      "binary": "/bin/sh",
      "flags": "execve rootcwd clone",
      "start_time": "2023-09-09T23:27:30.202263472Z",
      "auid": 4294967295,
      "pod": {
        "namespace": "demo-app",
        "name": "test-pod",
        "container": {
          "id": "containerd://1e910d5cc8d8d68c894934170b162ef93aea5652867ed6bd7c620c7e3f9a10f1",
          "name": "test-pod",
          "image": {
            "id": "docker.io/cilium/starwars@sha256:f92c8cd25372bac56f55111469fe9862bf682385a4227645f5af155eee7f58d9",
            "name": "docker.io/cilium/starwars:latest"
          },
          "start_time": "2023-09-09T22:46:09Z",
          "pid": 45612
        },
        "workload": "test-pod"
      },
      "docker": "1e910d5cc8d8d68c894934170b162ef",
      "parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQxMjYyOTEwMjM3OTQ2OjEzMDgwMA==",
      "tid": 130809
    },
    "function_name": "security_kernel_module_request",
    "args": [
      {
        "string_arg": "fs-xfs"
      }
    ],
    "return": {
      "int_arg": 0
    },
    "action": "KPROBE_ACTION_POST"
  },
  "node_name": "kind-control-plane",
  "time": "2023-09-09T23:27:42.751151233Z"
}

In addition to the process metadata from exec events, ProcessKprobe event contains the arguments of the observed call. In the above case they are:

  • security_kernel_module_request: the kernel security hook where modules are loaded on-demand.
  • string_arg: the name of the kernel module. When modules are automatically loaded, for security reasons, the kernel prefixes the module with the name of the subsystem that requested it. In our case, it’s requested by the file system subsystem, hence the name is fs-xfs.

2. Kernel calls modprobe to load the kernel module

The kernel will then call user space modprobe to load the kernel module.

{
  "process_exec": {
    "process": {
      "exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQxMjc1NTI0MjYzMjIxOjEzMDg3Nw==",
      "pid": 130877,
      "uid": 0,
      "cwd": "/",
      "binary": "/sbin/modprobe",
      "arguments": "-q -- fs-xfs",
      "flags": "execve rootcwd clone",
      "start_time": "2023-09-09T23:27:42.751301124Z",
      "auid": 4294967295,
      "parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE6MA==",
      "tid": 130877
    },
    "parent": {
      "exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE6MA==",
      "pid": 0,
      "uid": 0,
      "binary": "<kernel>",
      "flags": "procFS",
      "start_time": "2023-09-09T11:59:47.227037763Z",
      "auid": 0,
      "parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE6MA==",
      "tid": 0
    }
  },
  "node_name": "kind-control-plane",
  "time": "2023-09-09T23:27:42.751300984Z"
}

The ProcessExec event where modprobe tries to load the xfs module.

3. Reading the kernel module from the file system

modprobe will read the passed xfs kernel module from the host file system.

{
  "process_kprobe": {
    "process": {
      "exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQxMjc1NTI0MjYzMjIxOjEzMDg3Nw==",
      "pid": 130877,
      "uid": 0,
      "cwd": "/",
      "binary": "/sbin/modprobe",
      "arguments": "-q -- fs-xfs",
      "flags": "execve rootcwd clone",
      "start_time": "2023-09-09T23:27:42.751301124Z",
      "auid": 4294967295,
      "parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE6MA==",
      "refcnt": 1,
      "tid": 130877
    },
    "parent": {
      "exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE6MA==",
      "pid": 0,
      "uid": 0,
      "binary": "<kernel>",
      "flags": "procFS",
      "start_time": "2023-09-09T11:59:47.227037763Z",
      "auid": 0,
      "parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE6MA==",
      "tid": 0
    },
    "function_name": "security_kernel_read_file",
    "args": [
      {
        "file_arg": {
          "path": "/usr/lib/modules/6.2.0-32-generic/kernel/fs/xfs/xfs.ko"
        }
      },
      {
        "int_arg": 2
      }
    ],
    "return": {
      "int_arg": 0
    },
    "action": "KPROBE_ACTION_POST"
  },
  "node_name": "kind-control-plane",
  "time": "2023-09-09T23:27:42.752425825Z"
}

This ProcessKprobe event contains:

  • security_kernel_read_file: the kernel security hook when the kernel loads file specified by user space.
  • file_arg: the full path of the kernel module on the host file system.

4. Kernel module signature and sections are parsed

The final event is when the kernel is parsing the module sections. If all succeed the module will be loaded.

{
  "process_kprobe": {
    "process": {
      "exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQxMjc1NTI0MjYzMjIxOjEzMDg3Nw==",
      "pid": 130877,
      "uid": 0,
      "cwd": "/",
      "binary": "/sbin/modprobe",
      "arguments": "-q -- fs-xfs",
      "flags": "execve rootcwd clone",
      "start_time": "2023-09-09T23:27:42.751301124Z",
      "auid": 4294967295,
      "parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE6MA==",
      "refcnt": 1,
      "tid": 130877
    },
    "parent": {
      "exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE6MA==",
      "pid": 0,
      "uid": 0,
      "binary": "<kernel>",
      "flags": "procFS",
      "start_time": "2023-09-09T11:59:47.227037763Z",
      "auid": 0,
      "parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE6MA==",
      "tid": 0
    },
    "function_name": "find_module_sections",
    "args": [
      {
        "module_arg": {
          "name": "xfs",
          "signature_ok": true
        }
      }
    ],
    "action": "KPROBE_ACTION_POST"
  },
  "node_name": "kind-control-plane",
  "time": "2023-09-09T23:27:42.760880332Z"
}

This ProcessKprobe event contains the module argument.

  • find_module_sections: the function call where the kernel parses the module sections.
  • module_arg: the kernel module information, it contains:
    • name: the name of the kernel module as a string.
    • signature_ok: a boolean value, if set to true then module signature was successfully verified by the kernel. If it is false or missing then the signature verification was not performed or probably failed. In all cases this means the integrity of the system has been compromised. Depends on kernels compiled with CONFIG_MODULE_SIG option.

Monitor Unloading of kernel modules

Using the same monitor-kernel-modules tracing policy allows to monitor unloading of kernel modules.

The following ProcessKprobe event will be generated:

Removing kernel modules event

{
  "process_kprobe": {
    "process": {
      "exec_id": "OjMzNzQ4NzY1MDAyNDk5OjI0OTE3NQ==",
      "pid": 249175,
      "uid": 0,
      "cwd": "/home/tixxdz/tetragon",
      "binary": "/usr/sbin/rmmod",
      "arguments": "kernel_module_hello",
      "flags": "execve clone",
      "start_time": "2023-08-30T16:44:03.471068355Z",
      "auid": 1000,
      "parent_exec_id": "OjMzNzQ4NzY0MjQ4MTY5OjI0OTE3NA==",
      "refcnt": 1,
      "tid": 249175
    },
    "parent": {
      "exec_id": "OjMzNzQ4NzY0MjQ4MTY5OjI0OTE3NA==",
      "pid": 249174,
      "uid": 1000,
      "cwd": "/home/tixxdz/tetragon",
      "binary": "/usr/bin/sudo",
      "arguments": "rmmod kernel_module_hello",
      "flags": "execve",
      "start_time": "2023-08-30T16:44:03.470314558Z",
      "auid": 1000,
      "parent_exec_id": "OjMzNzQ2MjA5OTUxODI4OjI0OTE3Mw==",
      "refcnt": 1,
      "tid": 0
    },
    "function_name": "free_module",
    "args": [
      {
        "module_arg": {
          "name": "kernel_module_hello",
          "tainted": [
            "TAINT_OUT_OF_TREE_MODULE",
            "TAINT_UNSIGNED_MODULE"
          ]
        }
      }
    ],
    "action": "KPROBE_ACTION_POST"
  },
  "time": "2023-08-30T16:44:03.471984676Z"
}

To disable the monitor-kernel-modules run:

kubectl delete -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/host-changes/monitor-kernel-modules.yaml

4.6 - Security Profiles

Observe and record security events

Tetragon is able to observe various security events and even enforce security policies.

The Record Linux Capabilities Usage guide shows how to monitor and record Capabilities checks conducted by the kernel on behalf of applications during privileged operations. This can be used to inspect and produce security profiles for pods and containers.

4.6.1 - Record Linux Capabilities Usage

Record a capability profile of pods and containers

When the kernel needs to perform a privileged operation on behalf of a process, it checks the Capabilities of the process and issues a verdict to allow or deny the operation.

Tetragon is able to record these checks performed by the kernel. This can be used to answer the following questions:

What is the capabilities profile of pods or containters running in the cluster?

What capabilities to add or remove when configuring a security context for a pod or container?

Kubernetes Environments

First, verify that your k8s environment is set up and that all pods are up and running, and deploy the demo application:

kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.11/examples/minikube/http-sw-app.yaml

It might take several seconds until all pods are Running:

kubectl get pods -A

The output should be similar to:

NAMESPACE            NAME                                         READY   STATUS    RESTARTS   AGE
default              deathstar-54bb8475cc-6c6lc                   1/1     Running   0          2m54s
default              deathstar-54bb8475cc-zmfkr                   1/1     Running   0          2m54s
default              tiefighter                                   1/1     Running   0          2m54s
default              xwing                                        1/1     Running   0          2m54s
kube-system          tetragon-sdwv6                               2/2     Running   0          27m

Monitor Capability Checks

We use the creds-capability-usage tracing policy which generates ProcessKprobe events.

Apply the creds-capability-usage policy:

kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/process-credentials/creds-capability-usage.yaml

Start monitoring for events with tetra cli, but match only events of xwing pod:

kubectl exec -it -n kube-system ds/tetragon -c tetragon -- tetra getevents --namespaces default --pods xwing

In another terminal, kubectl exec into the xwing pod:

kubectl exec -it xwing -- /bin/bash

As an example execute dmesg to print the kernel ring buffer. This requires the special capability CAP_SYSLOG:

dmesg

The output should be similar to:

dmesg: klogctl: Operation not permitted

The tetra cli will generate the following ProcessKprobe events:

{
  "process_kprobe": {
    "process": {
      "exec_id": "a2luZC1jb250cm9sLXBsYW5lOjEyODQyNzgzMzUwNjg0OjczODYw",
      "pid": 73860,
      "uid": 0,
      "cwd": "/",
      "binary": "/bin/dmesg",
      "flags": "execve rootcwd clone",
      "start_time": "2023-07-06T10:13:33.834390020Z",
      "auid": 4294967295,
      "pod": {
        "namespace": "default",
        "name": "xwing",
        "container": {
          "id": "containerd://cfb961400ff25811d22d139a10f6a62efef53c2ecc11af47bc911a7f9a2ac1f7",
          "name": "spaceship",
          "image": {
            "id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
            "name": "docker.io/tgraf/netperf:latest"
          },
          "start_time": "2023-07-06T08:07:30Z",
          "pid": 171
        },
        "pod_labels": {
          "app.kubernetes.io/name": "xwing",
          "class": "xwing",
          "org": "alliance"
        }
      },
      "docker": "cfb961400ff25811d22d139a10f6a62",
      "parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjEyODQyMTI3MTIwOTcyOjczODUw",
      "refcnt": 1,
      "ns": {
        "uts": {
          "inum": 4026534655
        },
        "ipc": {
          "inum": 4026534656
        },
        "mnt": {
          "inum": 4026534731
        },
        "pid": {
          "inum": 4026534732
        },
        "pid_for_children": {
          "inum": 4026534732
        },
        "net": {
          "inum": 4026534512
        },
        "time": {
          "inum": 4026531834,
          "is_host": true
        },
        "time_for_children": {
          "inum": 4026531834,
          "is_host": true
        },
        "cgroup": {
          "inum": 4026534733
        },
        "user": {
          "inum": 4026531837,
          "is_host": true
        }
      },
      "tid": 73860
    },
    "parent": {
      "exec_id": "a2luZC1jb250cm9sLXBsYW5lOjEyODQyMTI3MTIwOTcyOjczODUw",
      "pid": 73850,
      "uid": 0,
      "cwd": "/",
      "binary": "/bin/bash",
      "flags": "execve rootcwd clone",
      "start_time": "2023-07-06T10:13:33.178160018Z",
      "auid": 4294967295,
      "pod": {
        "namespace": "default",
        "name": "xwing",
        "container": {
          "id": "containerd://cfb961400ff25811d22d139a10f6a62efef53c2ecc11af47bc911a7f9a2ac1f7",
          "name": "spaceship",
          "image": {
            "id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
            "name": "docker.io/tgraf/netperf:latest"
          },
          "start_time": "2023-07-06T08:07:30Z",
          "pid": 165
        },
        "pod_labels": {
          "app.kubernetes.io/name": "xwing",
          "class": "xwing",
          "org": "alliance"
        }
      },
      "docker": "cfb961400ff25811d22d139a10f6a62",
      "parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjEyODQyMDgxNTA3MzUzOjczODQx",
      "refcnt": 2,
      "tid": 73850
    },
    "function_name": "cap_capable",
    "args": [
      {
        "user_ns_arg": {
          "level": 0,
          "uid": 0,
          "gid": 0,
          "ns": {
            "inum": 4026531837,
            "is_host": true
          }
        }
      },
      {
        "capability_arg": {
          "value": 34,
          "name": "CAP_SYSLOG"
        }
      }
    ],
    "return": {
      "int_arg": -1
    },
    "action": "KPROBE_ACTION_POST"
  },
  "node_name": "kind-control-plane",
  "time": "2023-07-06T10:13:33.834882128Z"
}

In addition to the Kubernetes Identity and process metadata from exec events, ProcessKprobe events contain the arguments of the observed system call. In the above case they are:

  • function_name: that is the cap_capable kernel function.

  • user_ns_arg: is the user namespace where the capability is required.

    • level: is the nested level of the user namespace. Here it is zero which indicates the initial user namespace.
    • uid: is the user ID of the owner of the user namespace.
    • gid: is the group ID of the owner of the user namespace.
    • ns: details the information about the namespace. is_host indicates that the target user namespace where the capability is required is the host namespace.
  • capability_arg: is the capability required to perform the operation. In this example reading the kernel ring buffer.

    • value: is the integer number of the required capability.
    • name: is the name of the required capability. Here it is the CAP_SYSLOG.
  • return: indicates via the int_arg if the capability check succeeded or failed. 0 means it succeeded and the access was granted while -1 means it failed and the operation was denied.

To disable the creds-capability-usage run:

kubectl delete -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/process-credentials/creds-capability-usage.yaml

5 - Tutorials

Tutorials for various tasks related to Tetragon

5.1 - Debugging Tetragon

Diagnosing Tetragon problems

Enable debug log level

When debugging, it might be useful to change the log level. The default log level is controlled by the log-level option:

  • Enable debug level with --log-level=debug

  • Enable trace level with --log-level=trace

Change log level dynamically

It is possible to change the log level dynamically by sending the corresponding signal to tetragon process.

  • Change log level to debug level by sending the SIGRTMIN+20 signal to tetragon pid:

    sudo kill -s SIGRTMIN+20 $(pidof tetragon)
    
  • Change log level to trace level by sending the SIGRTMIN+21 signal to tetragon pid:

    sudo kill -s SIGRTMIN+21 $(pidof tetragon)
    
  • To Restore the original log level send the SIGRTMIN+22 signal to tetragon pid:

    sudo kill -s SIGRTMIN+22 $(pidof tetragon)
    

5.2 - Verify Tetragon image signatures

Learn how to verify Tetragon container images signatures.

Prerequisites

You will need to install cosign.

Verify Signed Container Images

Since version 0.8.4, all Tetragon container images are signed using cosign.

Let’s verify a Tetragon image’s signature using the cosign verify command:

COSIGN_EXPERIMENTAL=1 cosign verify --certificate-github-workflow-repository cilium/tetragon --certificate-oidc-issuer https://token.actions.githubusercontent.com <Image URL> | jq

Note

COSIGN_EXPERIMENTAL=1 is used to allow verification of images signed in KEYLESS mode. To learn more about keyless signing, please refer to Keyless Signatures.

5.3 - Software bill of materials

Download and verify the signature of the software bill of materials

A Software Bill of Materials (SBOM) is a complete, formally structured list of components that are required to build a given piece of software. SBOM provides insight into the software supply chain and any potential concerns related to license compliance and security that might exist.

Starting with version 0.8.4, all Tetragon images include an SBOM. The SBOM is generated in SPDX format using the bom tool. If you are new to the concept of SBOM, see what an SBOM can do for you.

Download SBOM

The SBOM can be downloaded from the supplied Tetragon image using the cosign download sbom command.

cosign download sbom --output-file sbom.spdx <Image URL>

Verify SBOM Image Signature

To ensure the SBOM is tamper-proof, its signature can be verified using the cosign verify command.

COSIGN_EXPERIMENTAL=1 cosign verify --certificate-github-workflow-repository cilium/tetragon --certificate-oidc-issuer https://token.actions.githubusercontent.com --attachment sbom <Image URL> | jq

It can be validated that the SBOM image was signed using Github Actions in the Cilium repository from the Issuer and Subject fields of the output.

6 - Reference

Low level reference documentation for Tetragon

6.1 - gRPC API

This reference is generated from the protocol buffer specification and documents the gRPC API of Tetragon.

The Tetragon API is an independant Go module that can be found in the Tetragon repository under api. The version 1 of this API is defined in github.com/cilium/tetragon/api/v1/tetragon.

tetragon/capabilities.proto

CapabilitiesType

NameNumberDescription
CAP_CHOWN0In a system with the [_POSIX_CHOWN_RESTRICTED] option defined, this overrides the restriction of changing file ownership and group ownership.
DAC_OVERRIDE1Override all DAC access, including ACL execute access if [_POSIX_ACL] is defined. Excluding DAC access covered by CAP_LINUX_IMMUTABLE.
CAP_DAC_READ_SEARCH2Overrides all DAC restrictions regarding read and search on files and directories, including ACL restrictions if [_POSIX_ACL] is defined. Excluding DAC access covered by "$1"_LINUX_IMMUTABLE.
CAP_FOWNER3Overrides all restrictions about allowed operations on files, where file owner ID must be equal to the user ID, except where CAP_FSETID is applicable. It doesn't override MAC and DAC restrictions.
CAP_FSETID4Overrides the following restrictions that the effective user ID shall match the file owner ID when setting the S_ISUID and S_ISGID bits on that file; that the effective group ID (or one of the supplementary group IDs) shall match the file owner ID when setting the S_ISGID bit on that file; that the S_ISUID and S_ISGID bits are cleared on successful return from chown(2) (not implemented).
CAP_KILL5Overrides the restriction that the real or effective user ID of a process sending a signal must match the real or effective user ID of the process receiving the signal.
CAP_SETGID6Allows forged gids on socket credentials passing.
CAP_SETUID7Allows forged pids on socket credentials passing.
CAP_SETPCAP8Without VFS support for capabilities: Transfer any capability in your permitted set to any pid, remove any capability in your permitted set from any pid With VFS support for capabilities (neither of above, but) Add any capability from current's capability bounding set to the current process' inheritable set Allow taking bits out of capability bounding set Allow modification of the securebits for a process
CAP_LINUX_IMMUTABLE9Allow modification of S_IMMUTABLE and S_APPEND file attributes
CAP_NET_BIND_SERVICE10Allows binding to ATM VCIs below 32
CAP_NET_BROADCAST11Allow broadcasting, listen to multicast
CAP_NET_ADMIN12Allow activation of ATM control sockets
CAP_NET_RAW13Allow binding to any address for transparent proxying (also via NET_ADMIN)
CAP_IPC_LOCK14Allow mlock and mlockall (which doesn't really have anything to do with IPC)
CAP_IPC_OWNER15Override IPC ownership checks
CAP_SYS_MODULE16Insert and remove kernel modules - modify kernel without limit
CAP_SYS_RAWIO17Allow sending USB messages to any device via /dev/bus/usb
CAP_SYS_CHROOT18Allow use of chroot()
CAP_SYS_PTRACE19Allow ptrace() of any process
CAP_SYS_PACCT20Allow configuration of process accounting
CAP_SYS_ADMIN21Allow everything under CAP_BPF and CAP_PERFMON for backward compatibility
CAP_SYS_BOOT22Allow use of reboot()
CAP_SYS_NICE23Allow setting cpu affinity on other processes
CAP_SYS_RESOURCE24Control memory reclaim behavior
CAP_SYS_TIME25Allow setting the real-time clock
CAP_SYS_TTY_CONFIG26Allow vhangup() of tty
CAP_MKNOD27Allow the privileged aspects of mknod()
CAP_LEASE28Allow taking of leases on files
CAP_AUDIT_WRITE29Allow writing the audit log via unicast netlink socket
CAP_AUDIT_CONTROL30Allow configuration of audit via unicast netlink socket
CAP_SETFCAP31Set or remove capabilities on files
CAP_MAC_OVERRIDE32Override MAC access. The base kernel enforces no MAC policy. An LSM may enforce a MAC policy, and if it does and it chooses to implement capability based overrides of that policy, this is the capability it should use to do so.
CAP_MAC_ADMIN33Allow MAC configuration or state changes. The base kernel requires no MAC configuration. An LSM may enforce a MAC policy, and if it does and it chooses to implement capability based checks on modifications to that policy or the data required to maintain it, this is the capability it should use to do so.
CAP_SYSLOG34Allow configuring the kernel's syslog (printk behaviour)
CAP_WAKE_ALARM35Allow triggering something that will wake the system
CAP_BLOCK_SUSPEND36Allow preventing system suspends
CAP_AUDIT_READ37Allow reading the audit log via multicast netlink socket
CAP_PERFMON38Allow system performance and observability privileged operations using perf_events, i915_perf and other kernel subsystems
CAP_BPF39CAP_BPF allows the following BPF operations: - Creating all types of BPF maps - Advanced verifier features - Indirect variable access - Bounded loops - BPF to BPF function calls - Scalar precision tracking - Larger complexity limits - Dead code elimination - And potentially other features - Loading BPF Type Format (BTF) data - Retrieve xlated and JITed code of BPF programs - Use bpf_spin_lock() helper CAP_PERFMON relaxes the verifier checks further: - BPF progs can use of pointer-to-integer conversions - speculation attack hardening measures are bypassed - bpf_probe_read to read arbitrary kernel memory is allowed - bpf_trace_printk to print kernel memory is allowed CAP_SYS_ADMIN is required to use bpf_probe_write_user. CAP_SYS_ADMIN is required to iterate system wide loaded programs, maps, links, BTFs and convert their IDs to file descriptors. CAP_PERFMON and CAP_BPF are required to load tracing programs. CAP_NET_ADMIN and CAP_BPF are required to load networking programs.
CAP_CHECKPOINT_RESTORE40Allow writing to ns_last_pid

SecureBitsType

NameNumberDescription
SecBitNotSet0
SecBitNoRoot1When set UID 0 has no special privileges. When unset, inheritance of root-permissions and suid-root executable under compatibility mode is supported. If the effective uid of the new process is 0 then the effective and inheritable bitmasks of the executable file is raised. If the real uid is 0, the effective (legacy) bit of the executable file is raised.
SecBitNoRootLocked2Make bit-0 SecBitNoRoot immutable
SecBitNoSetUidFixup4When set, setuid to/from uid 0 does not trigger capability-"fixup". When unset, to provide compatiblility with old programs relying on set*uid to gain/lose privilege, transitions to/from uid 0 cause capabilities to be gained/lost.
SecBitNoSetUidFixupLocked8Make bit-2 SecBitNoSetUidFixup immutable
SecBitKeepCaps16When set, a process can retain its capabilities even after transitioning to a non-root user (the set-uid fixup suppressed by bit 2). Bit-4 is cleared when a process calls exec(); setting both bit 4 and 5 will create a barrier through exec that no exec()'d child can use this feature again.
SecBitKeepCapsLocked32Make bit-4 SecBitKeepCaps immutable
SecBitNoCapAmbientRaise64When set, a process cannot add new capabilities to its ambient set.
SecBitNoCapAmbientRaiseLocked128Make bit-6 SecBitNoCapAmbientRaise immutable

tetragon/tetragon.proto

Capabilities

FieldTypeLabelDescription
permittedCapabilitiesTyperepeatedPermitted set indicates what capabilities the process can use. This is a limiting superset for the effective capabilities that the thread may assume. It is also a limiting superset for the capabilities that may be added to the inheritable set by a thread without the CAP_SETPCAP in its effective set.
effectiveCapabilitiesTyperepeatedEffective set indicates what capabilities are active in a process. This is the set used by the kernel to perform permission checks for the thread.
inheritableCapabilitiesTyperepeatedInheritable set indicates which capabilities will be inherited by the current process when running as a root user.

Container

FieldTypeLabelDescription
idstringIdentifier of the container.
namestringName of the container.
imageImageImage of the container.
start_timegoogle.protobuf.TimestampStart time of the container.
pidgoogle.protobuf.UInt32ValueProcess identifier in the container namespace.
maybe_exec_probeboolIf this is set true, it means that the process might have been originated from a Kubernetes exec probe. For this field to be true, the following must be true: 1. The binary field matches the first element of the exec command list for either liveness or readiness probe excluding the basename. For example, "/bin/ls" and "ls" are considered a match. 2. The arguments field exactly matches the rest of the exec command list.

CreateContainer

CreateContainer informs the agent that a container was created This is intented to be used by OCI hooks (but not limited to them) and corresponds to the CreateContainer hook: https://github.com/opencontainers/runtime-spec/blob/main/config.md#createcontainer-hooks.

FieldTypeLabelDescription
cgroupsPathstringcgroupsPath is the cgroups path for the container. The path is expected to be relative to the cgroups mountpoint. See: https://github.com/opencontainers/runtime-spec/blob/58ec43f9fc39e0db229b653ae98295bfde74aeab/specs-go/config.go#L174
rootDirstringrootDir is the absolute path of the root directory of the container. See: https://github.com/opencontainers/runtime-spec/blob/main/specs-go/config.go#L174
annotationsCreateContainer.AnnotationsEntryrepeatedannotations are the run-time annotations for the container see https://github.com/opencontainers/runtime-spec/blob/main/config.md#annotations

CreateContainer.AnnotationsEntry

FieldTypeLabelDescription
keystring
valuestring

GetHealthStatusRequest

FieldTypeLabelDescription
event_setHealthStatusTyperepeated

GetHealthStatusResponse

FieldTypeLabelDescription
health_statusHealthStatusrepeated

HealthStatus

FieldTypeLabelDescription
eventHealthStatusType
statusHealthStatusResult
detailsstring

Image

FieldTypeLabelDescription
idstringIdentifier of the container image composed of the registry path and the sha256.
namestringName of the container image composed of the registry path and the tag.

KernelModule

FieldTypeLabelDescription
namestringKernel module name
signature_okgoogle.protobuf.BoolValueIf true the module signature was verified successfully. Depends on kernels compiled with CONFIG_MODULE_SIG option, for details please read: https://www.kernel.org/doc/Documentation/admin-guide/module-signing.rst
taintedTaintedBitsTyperepeatedThe module tainted flags that will be applied on the kernel. For further details please read: https://docs.kernel.org/admin-guide/tainted-kernels.html

KprobeArgument

FieldTypeLabelDescription
string_argstring
int_argint32
skb_argKprobeSkb
size_arguint64
bytes_argbytes
path_argKprobePath
file_argKprobeFile
truncated_bytes_argKprobeTruncatedBytes
sock_argKprobeSock
cred_argKprobeCred
long_argint64
bpf_attr_argKprobeBpfAttr
perf_event_argKprobePerfEvent
bpf_map_argKprobeBpfMap
uint_arguint32
user_namespace_argKprobeUserNamespaceDeprecated.
capability_argKprobeCapability
process_credentials_argProcessCredentials
user_ns_argUserNamespace
module_argKernelModule
labelstring

KprobeBpfAttr

FieldTypeLabelDescription
ProgTypestring
InsnCntuint32
ProgNamestring

KprobeBpfMap

FieldTypeLabelDescription
MapTypestring
KeySizeuint32
ValueSizeuint32
MaxEntriesuint32
MapNamestring

KprobeCapability

FieldTypeLabelDescription
valuegoogle.protobuf.Int32Value
namestring

KprobeCred

FieldTypeLabelDescription
permittedCapabilitiesTyperepeated
effectiveCapabilitiesTyperepeated
inheritableCapabilitiesTyperepeated

KprobeFile

FieldTypeLabelDescription
mountstring
pathstring
flagsstring

KprobePath

FieldTypeLabelDescription
mountstring
pathstring
flagsstring

KprobePerfEvent

FieldTypeLabelDescription
KprobeFuncstring
Typestring
Configuint64
ProbeOffsetuint64

KprobeSkb

FieldTypeLabelDescription
hashuint32
lenuint32
priorityuint32
markuint32
saddrstring
daddrstring
sportuint32
dportuint32
protouint32
sec_path_lenuint32
sec_path_olenuint32
protocolstring
familystring

KprobeSock

FieldTypeLabelDescription
familystring
typestring
protocolstring
markuint32
priorityuint32
saddrstring
daddrstring
sportuint32
dportuint32
cookieuint64
statestring

KprobeTruncatedBytes

FieldTypeLabelDescription
bytes_argbytes
orig_sizeuint64

KprobeUserNamespace

FieldTypeLabelDescription
levelgoogle.protobuf.Int32Value
ownergoogle.protobuf.UInt32Value
groupgoogle.protobuf.UInt32Value
nsNamespace

Namespace

FieldTypeLabelDescription
inumuint32Inode number of the namespace.
is_hostboolIndicates if namespace belongs to host.

Namespaces

FieldTypeLabelDescription
utsNamespaceHostname and NIS domain name.
ipcNamespaceSystem V IPC, POSIX message queues.
mntNamespaceMount points.
pidNamespaceProcess IDs.
pid_for_childrenNamespaceProcess IDs for children processes.
netNamespaceNetwork devices, stacks, ports, etc.
timeNamespaceBoot and monotonic clocks.
time_for_childrenNamespaceBoot and monotonic clocks for children processes.
cgroupNamespaceCgroup root directory.
userNamespaceUser and group IDs.

Pod

FieldTypeLabelDescription
namespacestringKubernetes namespace of the Pod.
namestringName of the Pod.
labelsstringrepeatedDeprecated. Cilium identity labels of the Pod. This field is deprecated. Use pod_labels instead.
containerContainerContainer of the Pod from which the process that triggered the event originates.
pod_labelsPod.PodLabelsEntryrepeatedContains all the labels of the pod. Note that the labels field contains Cilium identity labels, which is a subset of pod labels.
workloadstringKubernetes workload of the Pod.
workload_kindstringKubernetes workload kind (e.g. "Deployment", "DaemonSet") of the Pod.

Pod.PodLabelsEntry

FieldTypeLabelDescription
keystring
valuestring

Process

FieldTypeLabelDescription
exec_idstringExec ID uniquely identifies the process over time across all the nodes in the cluster.
pidgoogle.protobuf.UInt32ValueProcess identifier from host PID namespace.
uidgoogle.protobuf.UInt32ValueUser identifier associated with the process.
cwdstringCurrent working directory of the process.
binarystringAbsolute path of the executed binary.
argumentsstringArguments passed to the binary at execution.
flagsstringFlags are for debugging purposes only and should not be considered a reliable source of information. They hold various information about which syscalls generated events, use of internal Tetragon buffers, errors and more. - execve This event is generated by an execve syscall for a new process. See procFs for the other option. A correctly formatted event should either set execve or procFS (described next). - procFS This event is generated from a proc interface. This happens at Tetragon init when existing processes are being loaded into Tetragon event buffer. All events should have either execve or procFS set. - truncFilename Indicates a truncated processes filename because the buffer size is too small to contain the process filename. Consider increasing buffer size to avoid this. - truncArgs Indicates truncated the processes arguments because the buffer size was too small to contain all exec args. Consider increasing buffer size to avoid this. - taskWalk Primarily useful for debugging. Indicates a walked process hierarchy to find a parent process in the Tetragon buffer. This may happen when we did not receive an exec event for the immediate parent of a process. Typically means we are looking at a fork that in turn did another fork we don't currently track fork events exactly and instead push an event with the original parent exec data. This flag can provide this insight into the event if needed. - miss An error flag indicating we could not find parent info in the Tetragon event buffer. If this is set it should be reported to Tetragon developers for debugging. Tetragon will do its best to recover information about the process from available kernel data structures instead of using cached info in this case. However, args will not be available. - needsAUID An internal flag for Tetragon to indicate the audit has not yet been resolved. The BPF hooks look at this flag to determine if probing the audit system is necessary. - errorFilename An error flag indicating an error happened while reading the filename. If this is set it should be reported to Tetragon developers for debugging. - errorArgs An error flag indicating an error happened while reading the process args. If this is set it should be reported to Tetragon developers for debugging - needsCWD An internal flag for Tetragon to indicate the current working directory has not yet been resolved. The Tetragon hooks look at this flag to determine if probing the CWD is necessary. - noCWDSupport Indicates that CWD is removed from the event because the buffer size is too small. Consider increasing buffer size to avoid this. - rootCWD Indicates that CWD is the root directory. This is necessary to inform readers the CWD is not in the event buffer and is '/' instead. - errorCWD An error flag indicating an error occurred while reading the CWD of a process. If this is set it should be reported to Tetragon developers for debugging. - clone Indicates the process issued a clone before exec*. This is the general flow to exec* a new process, however its possible to replace the current process with a new process by doing an exec* without a clone. In this case the flag will be omitted and the same PID will be used by the kernel for both the old process and the newly exec'd process.
start_timegoogle.protobuf.TimestampStart time of the execution.
auidgoogle.protobuf.UInt32ValueAudit user ID, this ID is assigned to a user upon login and is inherited by every process even when the user's identity changes. For example, by switching user accounts with su - john.
podPodInformation about the the Kubernetes Pod where the event originated.
dockerstringThe 15 first digits of the container ID.
parent_exec_idstringExec ID of the parent process.
refcntuint32Reference counter from the Tetragon process cache.
capCapabilitiesSet of capabilities that define the permissions the process can execute with.
nsNamespacesLinux namespaces of the process, disabled by default, can be enabled by the --enable-process-ns flag.
tidgoogle.protobuf.UInt32ValueThread ID, note that for the thread group leader, tid is equal to pid.

ProcessCredentials

FieldTypeLabelDescription
uidgoogle.protobuf.UInt32ValueThe real user ID
gidgoogle.protobuf.UInt32ValueThe real group ID
euidgoogle.protobuf.UInt32ValueThe effective user ID
egidgoogle.protobuf.UInt32ValueThe effective group ID
suidgoogle.protobuf.UInt32ValueThe saved user ID
sgidgoogle.protobuf.UInt32ValueThe saved group ID
fsuidgoogle.protobuf.UInt32Valuethe filesystem user ID
fsgidgoogle.protobuf.UInt32ValueThe filesystem group ID
securebitsSecureBitsTyperepeatedSecure management flags
capsCapabilitiesSet of capabilities that define the permissions the process can execute with.
user_nsUserNamespaceUser namespace where the UIDs, GIDs and capabilities are relative to.

ProcessExec

FieldTypeLabelDescription
processProcessProcess that triggered the exec.
parentProcessImmediate parent of the process.
ancestorsProcessrepeatedAncestors of the process beyond the immediate parent.

ProcessExit

FieldTypeLabelDescription
processProcessProcess that triggered the exit.
parentProcessImmediate parent of the process.
signalstringSignal that the process received when it exited, for example SIGKILL or SIGTERM (list all signal names with kill -l). If there is no signal handler implemented for a specific process, we report the exit status code that can be found in the status field.
statusuint32Status code on process exit. For example, the status code can indicate if an error was encountered or the program exited successfully.
timegoogle.protobuf.TimestampDate and time of the event.

ProcessKprobe

FieldTypeLabelDescription
processProcessProcess that triggered the kprobe.
parentProcessImmediate parent of the process.
function_namestringSymbol on which the kprobe was attached.
argsKprobeArgumentrepeatedArguments definition of the observed kprobe.
returnKprobeArgumentReturn value definition of the observed kprobe.
actionKprobeActionAction performed when the kprobe matched.

ProcessLoader

loader sensor event triggered for loaded binary/library

FieldTypeLabelDescription
processProcess
pathstring
buildidbytes

ProcessTracepoint

FieldTypeLabelDescription
processProcessProcess that triggered the tracepoint.
parentProcessImmediate parent of the process.
subsysstringSubsystem of the tracepoint.
eventstringEvent of the subsystem.
argsKprobeArgumentrepeatedArguments definition of the observed tracepoint. TODO: once we implement all we want, rename KprobeArgument to GenericArgument

ProcessUprobe

FieldTypeLabelDescription
processProcess
parentProcess
pathstring
symbolstring

RuntimeHookRequest

RuntimeHookRequest synchronously propagates information to the agent about run-time state.

FieldTypeLabelDescription
createContainerCreateContainer

RuntimeHookResponse

Test

FieldTypeLabelDescription
arg0uint64
arg1uint64
arg2uint64
arg3uint64

UserNamespace

FieldTypeLabelDescription
levelgoogle.protobuf.Int32ValueNested level of the user namespace. Init or host user namespace is at level 0.
uidgoogle.protobuf.UInt32ValueThe owner user ID of the namespace
gidgoogle.protobuf.UInt32ValueThe owner group ID of the namepace.
nsNamespaceThe user namespace details that include the inode number of the namespace.

HealthStatusResult

NameNumberDescription
HEALTH_STATUS_UNDEF0
HEALTH_STATUS_RUNNING1
HEALTH_STATUS_STOPPED2
HEALTH_STATUS_ERROR3

HealthStatusType

NameNumberDescription
HEALTH_STATUS_TYPE_UNDEF0
HEALTH_STATUS_TYPE_STATUS1

KprobeAction

NameNumberDescription
KPROBE_ACTION_UNKNOWN0Unknown action
KPROBE_ACTION_POST1Post action creates an event (default action).
KPROBE_ACTION_FOLLOWFD2Post action creates a mapping between file descriptors and file names.
KPROBE_ACTION_SIGKILL3Sigkill action synchronously terminates the process.
KPROBE_ACTION_UNFOLLOWFD4Post action removes a mapping between file descriptors and file names.
KPROBE_ACTION_OVERRIDE5Override action modifies the return value of the call.
KPROBE_ACTION_COPYFD6Post action dupplicates a mapping between file descriptors and file names.
KPROBE_ACTION_GETURL7GetURL action issue an HTTP Get request against an URL from userspace.
KPROBE_ACTION_DNSLOOKUP8GetURL action issue a DNS lookup against an URL from userspace.
KPROBE_ACTION_NOPOST9NoPost action suppresses the transmission of the event to userspace.
KPROBE_ACTION_SIGNAL10Signal action sends specified signal to the process.

TaintedBitsType

Tainted bits to indicate if the kernel was tainted. For further details: https://docs.kernel.org/admin-guide/tainted-kernels.html

NameNumberDescription
TAINT_UNSET0
TAINT_PROPRIETARY_MODULE1A proprietary module was loaded.
TAINT_FORCED_MODULE2A module was force loaded.
TAINT_FORCED_UNLOAD_MODULE4A module was force unloaded.
TAINT_STAGED_MODULE1024A staging driver was loaded.
TAINT_OUT_OF_TREE_MODULE4096An out of tree module was loaded.
TAINT_UNSIGNED_MODULE8192An unsigned module was loaded. Supported only on kernels built with CONFIG_MODULE_SIG option.
TAINT_KERNEL_LIVE_PATCH_MODULE32768The kernel has been live patched.
TAINT_TEST_MODULE262144Loading a test module.

tetragon/events.proto

AggregationInfo

AggregationInfo contains information about aggregation results.

FieldTypeLabelDescription
countuint64Total count of events in this aggregation time window.

AggregationOptions

AggregationOptions defines configuration options for aggregating events.

FieldTypeLabelDescription
window_sizegoogle.protobuf.DurationAggregation window size. Defaults to 15 seconds if this field is not set.
channel_buffer_sizeuint64Size of the buffer for the aggregator to receive incoming events. If the buffer becomes full, the aggregator will log a warning and start dropping incoming events.

FieldFilter

FieldTypeLabelDescription
event_setEventTyperepeatedEvent types to filter or undefined to filter over all event types.
fieldsgoogle.protobuf.FieldMaskFields to include or exclude.
actionFieldFilterActionWhether to include or exclude fields.
invert_event_setgoogle.protobuf.BoolValueWhether or not the event set filter should be inverted.

Filter

FieldTypeLabelDescription
binary_regexstringrepeated
namespacestringrepeated
health_checkgoogle.protobuf.BoolValue
piduint32repeated
pid_setuint32repeatedFilter by the PID of a process and any of its descendants. Note that this filter is intended for testing and development purposes only and should not be used in production. In particular, PID cycling in the OS over longer periods of time may cause unexpected events to pass this filter.
event_setEventTyperepeated
pod_regexstringrepeatedFilter by process.pod.name field using RE2 regular expression syntax: https://github.com/google/re2/wiki/Syntax
arguments_regexstringrepeatedFilter by process.arguments field using RE2 regular expression syntax: https://github.com/google/re2/wiki/Syntax
labelsstringrepeatedFilter events by pod labels using Kubernetes label selector syntax: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors Note that this filter never matches events without the pod field (i.e. host process events).

GetEventsRequest

FieldTypeLabelDescription
allow_listFilterrepeatedallow_list specifies a list of filters to apply to only return certain events. If multiple filters are specified, at least one of them has to match for an event to be included in the results.
deny_listFilterrepeateddeny_list specifies a list of filters to apply to exclude certain events from the results. If multiple filters are specified, at least one of them has to match for an event to be excluded. If both allow_list and deny_list are specified, the results contain the set difference allow_list - deny_list.
aggregation_optionsAggregationOptionsaggregation_options configures aggregation options for this request. If this field is not set, responses will not be aggregated. Note that currently only process_accept and process_connect events are aggregated. Other events remain unaggregated.
field_filtersFieldFilterrepeatedFields to include or exclude for events in the GetEventsResponse. Omitting this field implies that all fields will be included. Exclusion always takes precedence over inclusion in the case of conflicts.

GetEventsResponse

FieldTypeLabelDescription
process_execProcessExecProcessExec event includes information about the execution of binaries and other related process metadata.
process_exitProcessExitProcessExit event indicates how and when a process terminates.
process_kprobeProcessKprobeProcessKprobe event contains information about the pre-defined functions and the process that invoked them.
process_tracepointProcessTracepointProcessTracepoint contains information about the pre-defined tracepoint and the process that invoked them.
process_loaderProcessLoader
process_uprobeProcessUprobe
testTest
rate_limit_infoRateLimitInfo
node_namestringName of the node where this event was observed.
timegoogle.protobuf.TimestampTimestamp at which this event was observed. For an aggregated response, this field to set to the timestamp at which the event was observed for the first time in a given aggregation time window.
aggregation_infoAggregationInfoaggregation_info contains information about aggregation results. This field is set only for aggregated responses.

RateLimitInfo

FieldTypeLabelDescription
number_of_dropped_process_eventsuint64

EventType

Represents the type of a Tetragon event.

NOTE: EventType constants must be in sync with the numbers used in the GetEventsResponse event oneof.

NameNumberDescription
UNDEF0
PROCESS_EXEC1
PROCESS_EXIT5
PROCESS_KPROBE9
PROCESS_TRACEPOINT10
PROCESS_LOADER11
PROCESS_UPROBE12
TEST40000
RATE_LIMIT_INFO40001

FieldFilterAction

Determins the behaviour of a field filter

NameNumberDescription
INCLUDE0
EXCLUDE1

tetragon/stack.proto

StackAddress

FieldTypeLabelDescription
addressuint64
symbolstring

StackTrace

FieldTypeLabelDescription
addressesStackAddressrepeated

StackTraceLabel

FieldTypeLabelDescription
keystring
countuint64

StackTraceNode

FieldTypeLabelDescription
addressStackAddress
countuint64
labelsStackTraceLabelrepeated
childrenStackTraceNoderepeated

tetragon/sensors.proto

AddTracingPolicyRequest

FieldTypeLabelDescription
yamlstring

AddTracingPolicyResponse

DeleteTracingPolicyRequest

FieldTypeLabelDescription
namestring

DeleteTracingPolicyResponse

DisableSensorRequest

FieldTypeLabelDescription
namestring

DisableSensorResponse

EnableSensorRequest

FieldTypeLabelDescription
namestring

EnableSensorResponse

GetStackTraceTreeRequest

FieldTypeLabelDescription
namestring

GetStackTraceTreeResponse

FieldTypeLabelDescription
rootStackTraceNode

GetVersionRequest

GetVersionResponse

FieldTypeLabelDescription
versionstring

ListSensorsRequest

ListSensorsResponse

FieldTypeLabelDescription
sensorsSensorStatusrepeated

ListTracingPoliciesRequest

ListTracingPoliciesResponse

FieldTypeLabelDescription
policiesTracingPolicyStatusrepeated

RemoveSensorRequest

FieldTypeLabelDescription
namestring

RemoveSensorResponse

SensorStatus

FieldTypeLabelDescription
namestringname is the name of the sensor
enabledboolenabled marks whether the sensor is enabled
collectionstringcollection is the collection the sensor belongs to (typically a tracing policy)

TracingPolicyStatus

FieldTypeLabelDescription
iduint64id is the id of the policy
namestringname is the name of the policy
namespacestringnamespace is the namespace of the policy (or empty of the policy is global)
infostringinfo is additional information about the policy
sensorsstringrepeatedsensors loaded in the scope of this policy

FineGuidanceSensors

Method NameRequest TypeResponse TypeDescription
GetEventsGetEventsRequestGetEventsResponse stream
GetHealthGetHealthStatusRequestGetHealthStatusResponse
AddTracingPolicyAddTracingPolicyRequestAddTracingPolicyResponse
DeleteTracingPolicyDeleteTracingPolicyRequestDeleteTracingPolicyResponse
RemoveSensorRemoveSensorRequestRemoveSensorResponse
ListTracingPoliciesListTracingPoliciesRequestListTracingPoliciesResponse
ListSensorsListSensorsRequestListSensorsResponse
EnableSensorEnableSensorRequestEnableSensorResponse
DisableSensorDisableSensorRequestDisableSensorResponse
GetStackTraceTreeGetStackTraceTreeRequestGetStackTraceTreeResponse
GetVersionGetVersionRequestGetVersionResponse
RuntimeHookRuntimeHookRequestRuntimeHookResponse

Scalar Value Types

.proto TypeNotesC++JavaPythonGoC#PHPRuby
doubledoubledoublefloatfloat64doublefloatFloat
floatfloatfloatfloatfloat32floatfloatFloat
int32Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead.int32intintint32intintegerBignum or Fixnum (as required)
int64Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead.int64longint/longint64longinteger/stringBignum
uint32Uses variable-length encoding.uint32intint/longuint32uintintegerBignum or Fixnum (as required)
uint64Uses variable-length encoding.uint64longint/longuint64ulonginteger/stringBignum or Fixnum (as required)
sint32Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s.int32intintint32intintegerBignum or Fixnum (as required)
sint64Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s.int64longint/longint64longinteger/stringBignum
fixed32Always four bytes. More efficient than uint32 if values are often greater than 2^28.uint32intintuint32uintintegerBignum or Fixnum (as required)
fixed64Always eight bytes. More efficient than uint64 if values are often greater than 2^56.uint64longint/longuint64ulonginteger/stringBignum
sfixed32Always four bytes.int32intintint32intintegerBignum or Fixnum (as required)
sfixed64Always eight bytes.int64longint/longint64longinteger/stringBignum
boolboolbooleanbooleanboolboolbooleanTrueClass/FalseClass
stringA string must always contain UTF-8 encoded or 7-bit ASCII text.stringStringstr/unicodestringstringstringString (UTF-8)
bytesMay contain any arbitrary sequence of bytes.stringByteStringstr[]byteByteStringstringString (ASCII-8BIT)

6.2 - Helm chart

This reference is generated from the Tetragon Helm chart values.

The Tetragon Helm chart source is available under github.io/cilium/tetragon/install/kubernetes and is distributed from the Cilium helm charts repository helm.cilium.io.

To deploy Tetragon using this Helm chart you can run the following commands:

helm repo add cilium https://helm.cilium.io
helm repo update
helm install tetragon cilium/tetragon -n kube-system

To use the values available, with helm install or helm upgrade, use --set key=value.

Values

KeyTypeDefaultDescription
affinityobject{}
daemonSetAnnotationsobject{}
daemonSetLabelsOverrideobject{}
dnsPolicystring"Default"
enabledbooltrue
export.filenames[0]string"tetragon.log"
export.modestring"stdout"
export.resourcesobject{}
export.securityContextobject{}
export.stdout.argsOverridelist[]
export.stdout.commandOverridelist[]
export.stdout.enabledArgsbooltrue
export.stdout.enabledCommandbooltrue
export.stdout.extraEnvlist[]
export.stdout.extraVolumeMountslist[]
export.stdout.image.overridestringnil
export.stdout.image.repositorystring"quay.io/cilium/hubble-export-stdout"
export.stdout.image.tagstring"v1.0.3"
exportDirectorystring"/var/run/cilium/tetragon"
exportFileCreationIntervalstring"120s"
extraConfigmapMountslist[]
extraHostPathMountslist[]
extraVolumeslist[]
hostNetworkbooltrue
imagePullPolicystring"IfNotPresent"
imagePullSecretslist[]
nodeSelectorobject{}
podAnnotationsobject{}
podLabelsOverrideobject{}
podSecurityContextobject{}
podWatcher.enabledboolfalse
selectorLabelsOverrideobject{}
serviceAccount.annotationsobject{}
serviceAccount.createbooltrue
serviceAccount.namestring""
serviceLabelsOverrideobject{}
tetragon.argsOverridelist[]
tetragon.btfstring""
tetragon.commandOverridelist[]
tetragon.enableCiliumAPIboolfalse
tetragon.enableK8sAPIbooltrue
tetragon.enableMsgHandlingLatencyboolfalse
tetragon.enablePolicyFilterboolfalse
tetragon.enablePolicyFilterDebugboolfalse
tetragon.enableProcessCredboolfalse
tetragon.enableProcessNsboolfalse
tetragon.enabledbooltrue
tetragon.exportAllowListstring"{\"event_set\":[\"PROCESS_EXEC\", \"PROCESS_EXIT\", \"PROCESS_KPROBE\", \"PROCESS_UPROBE\"]}"
tetragon.exportDenyListstring"{\"health_check\":true}\n{\"namespace\":[\"\", \"cilium\", \"kube-system\"]}"
tetragon.exportFileCompressboolfalse
tetragon.exportFileMaxBackupsint5
tetragon.exportFileMaxSizeMBint10
tetragon.exportFilenamestring"tetragon.log"
tetragon.exportRateLimitint-1
tetragon.extraArgsobject{}
tetragon.extraEnvlist[]
tetragon.extraVolumeMountslist[]
tetragon.fieldFiltersstring"{}"
tetragon.gops.addressstring"localhost"The address at which to expose gops.
tetragon.gops.portint8118The port at which to expose gops.
tetragon.grpc.addressstring"localhost:54321"The address at which to expose gRPC. Examples: localhost:54321, unix:///var/run/tetragon/tetragon.sock
tetragon.grpc.enabledbooltrueWhether to enable exposing Tetragon gRPC.
tetragon.image.overridestringnil
tetragon.image.repositorystring"quay.io/cilium/tetragon"
tetragon.image.tagstring"v0.11.0"
tetragon.processCacheSizeint65536
tetragon.prometheus.addressstring""The address at which to expose metrics. Set it to "" to expose on all available interfaces.
tetragon.prometheus.enabledbooltrueWhether to enable exposing Tetragon metrics.
tetragon.prometheus.portint2112The port at which to expose metrics.
tetragon.prometheus.serviceMonitor.enabledboolfalseWhether to create a ‘ServiceMonitor’ resource targeting the ’tetragon’ pods.
tetragon.prometheus.serviceMonitor.labelsOverrideobject{}The set of labels to place on the ‘ServiceMonitor’ resource.
tetragon.resourcesobject{}
tetragon.securityContext.privilegedbooltrue
tetragonOperator.enabledbooltrueEnable the tetragon-operator component (required).
tetragonOperator.imageobject{"override":null,"repository":"quay.io/cilium/tetragon-operator","suffix":"","tag":"v0.11.0"}tetragon-operator image.
tetragonOperator.skipCRDCreationboolfalse
tolerations[0].operatorstring"Exists"
updateStrategyobject{}

6.3 - Tetragon Daemon Configuration

Configure Tetragon daemon

Tetragon default controlling settings are set during compilation, so configuration is only needed when it is necessary to deviate from those defaults. This document lists those controlling settings and how they can be set as a CLI arguments or as configuration options from YAML files.

Tetragon Controlling Settings

Tetragon CLI arguments:

Tetragon - eBPF-based Security Observability and Runtime Enforcement

Usage:
  tetragon [flags]

Flags:
      --bpf-lib string                            Location of Tetragon libs (btf and bpf files) (default "/var/lib/tetragon/")
      --btf string                                Location of btf
      --cilium-bpf string                         Cilium BPF directory
      --config-dir string                         Configuration directory that contains a file for each option
      --data-cache-size int                       Size of the data events cache (default 1024)
  -d, --debug                                     Enable debug messages. Equivalent to '--log-level=debug'
      --disable-kprobe-multi                      Allow to disable kprobe multi interface
      --enable-cilium-api                         Access Cilium API to associate Tetragon events with Cilium endpoints and DNS cache
      --enable-export-aggregation                 Enable JSON export aggregation
      --enable-k8s-api                            Access Kubernetes API to associate Tetragon events with Kubernetes pods
      --enable-msg-handling-latency               Enable metrics for message handling latency
      --enable-pid-set-filter                     Enable pidSet export filters. Not recommended for production use
      --enable-policy-filter                      Enable policy filter code (beta)
      --enable-policy-filter-debug                Enable policy filter debug messages
      --enable-process-ancestors                  Include ancestors in process exec events (default true)
      --enable-process-cred                       Enable process_cred events
      --enable-process-ns                         Enable namespace information in process_exec and process_kprobe events
      --event-queue-size uint                     Set the size of the internal event queue. (default 10000)
      --export-aggregation-buffer-size uint       Aggregator channel buffer size (default 10000)
      --export-aggregation-window-size duration   JSON export aggregation time window (default 15s)
      --export-allowlist string                   JSON export allowlist
      --export-denylist string                    JSON export denylist
      --export-file-compress                      Compress rotated JSON export files
      --export-file-max-backups int               Number of rotated JSON export files to retain (default 5)
      --export-file-max-size-mb int               Size in MB for rotating JSON export files (default 10)
      --export-file-rotation-interval duration    Interval at which to rotate JSON export files in addition to rotating them by size
      --export-filename string                    Filename for JSON export. Disabled by default
      --export-rate-limit int                     Rate limit (per minute) for event export. Set to -1 to disable (default -1)
      --field-filters string                      Field filters for event exports
      --force-large-progs                         Force loading large programs, even in kernels with < 5.3 versions
      --force-small-progs                         Force loading small programs, even in kernels with >= 5.3 versions
      --gops-address string                       gops server address (e.g. 'localhost:8118'). Disabled by default
  -h, --help                                      help for tetragon
      --k8s-kubeconfig-path string                Absolute path of the kubernetes kubeconfig file
      --kernel string                             Kernel version
      --log-format string                         Set log format (default "text")
      --log-level string                          Set log level (default "info")
      --metrics-server string                     Metrics server address (e.g. ':2112'). Disabled by default
      --netns-dir string                          Network namespace dir (default "/var/run/docker/netns/")
      --process-cache-size int                    Size of the process cache (default 65536)
      --procfs string                             Location of procfs to consume existing PIDs (default "/proc/")
      --rb-size int                               Set perf ring buffer size for single cpu (default 65k)
      --rb-size-total int                         Set perf ring buffer size in total for all cpus (default 65k per cpu)
      --release-pinned-bpf                        Release all pinned BPF programs and maps in Tetragon BPF directory. Enabled by default. Set to false to disable (default true)
      --server-address string                     gRPC server address (e.g. 'localhost:54321' or 'unix:///var/run/tetragon/tetragon.sock' (default "localhost:54321")
      --tracing-policy string                     Tracing policy file to load at startup
      --tracing-policy-dir string                 Directory from where to load Tracing Policies (default "/etc/tetragon/tetragon.tp.d/")
      --verbose int                               set verbosity level for eBPF verifier dumps. Pass 0 for silent, 1 for truncated logs, 2 for a full dump

Configuration precedence

Tetragon controlling settings can also be loaded from YAML configuration files according to this order:

  1. From the drop-in configuration snippets inside the following directories where each filename maps to one controlling setting and the content of the file to its corresponding value:

    • /usr/lib/tetragon/tetragon.conf.d/*
    • /usr/local/lib/tetragon/tetragon.conf.d/*
  2. From the configuration file /etc/tetragon/tetragon.yaml if available, overriding previous settings.

  3. From the drop-in configuration snippets inside /etc/tetragon/tetragon.conf.d/*, similarly overriding previous settings.

  4. If the config-dir setting is set, Tetragon loads its settings from the files inside the directory pointed by this option, overriding previous controlling settings. The config-dir is also part of Kubernetes ConfigMap.

When reading configuration from directories, each filename maps to one controlling setting. If the same controlling setting is set multiple times, then the last value or content of that file overrides the previous ones.

To summarize the configuration precedence:

  1. Drop-in directory pointed by --config-dir.

  2. Drop-in directory /etc/tetragon/tetragon.conf.d/*.

  3. Configuration file /etc/tetragon/tetragon.yaml.

  4. Drop-in directories:

    • /usr/local/lib/tetragon/tetragon.conf.d/*
    • /usr/lib/tetragon/tetragon.conf.d/*

Configuration examples

The examples/configuration/tetragon.yaml file contains example entries showing the defaults as a guide to the administrator. Local overrides can be created by editing and copying this file into /etc/tetragon/tetragon.yaml, or by editing and copying “drop-ins” from the examples/configuration/tetragon.conf.d directory into the /etc/tetragon/tetragon.conf.d/ subdirectory. The latter is generally recommended.

Each filename maps to a one controlling setting and the content of the file to its corresponding value. This is the recommended way.

Changing configuration example:

  • /etc/tetragon/tetragon.conf.d/bpf-lib with a corresponding value of:

    /var/lib/tetragon/
    
  • /etc/tetragon/tetragon.conf.d/log-format with a corresponding value of:

    text
    
  • /etc/tetragon/tetragon.conf.d/export-filename with a corresponding value of:

    /var/log/tetragon/tetragon.log
    

Restrict gRPC API access

The gRPC API supports unix sockets, it can be set using one of the following methods:

  • Use the --server-address flag:

    --server-address unix:///var/run/tetragon/tetragon.sock
    
  • Or use the drop-in configuration file /etc/tetragon/tetragon.conf.d/server-address containing:

    unix:///var/run/tetragon/tetragon.sock
    

Then to access the gRPC API with tetra client, set --server-address to point to the corresponding address:

sudo tetra --server-address unix:///var/run/tetragon/tetragon.sock getevents

Configure Tracing Policies location

Tetragon daemon automatically loads Tracing policies from the default /etc/tetragon/tetragon.tp.d/ directory. Tracing policies can be organized in directories such: /etc/tetragon/tetragon.tp.d/file-access, /etc/tetragon/tetragon.tp.d/network-access, etc.

The --tracing-policy-dir controlling setting can be used to change the default directory from where Tracing policies are loaded.

The --tracing-policy controlling setting can be used to specify the path of one tracing policy to load.

7 - Contribution Guide

How to contribute to the project

We’re happy you’re interested in contributing to the Tetragon project.

This section of the Tetragon documentation will help you make sure you have an environment capable of testing changes to the Tetragon source code, and that you understand the workflow of getting these changes reviewed and merged upstream.

Clone and provision an environment

  1. Make sure you have a GitHub account.

  2. Fork the Tetragon repository to your GitHub user or organization.

  3. Turn off GitHub actions for your fork as described in the GitHub Docs.

    This is recommended to avoid unnecessary CI notification failures on the fork.

  4. Clone your ${YOUR_GITHUB_USERNAME_OR_ORG}/tetragon fork into your GOPATH, and set up the base repository as upstream remote:

    mkdir -p "${GOPATH}/src/github.com/cilium"
    cd "${GOPATH}/src/github.com/cilium"
    git clone https://github.com/${YOUR_GITHUB_USERNAME_OR_ORG}/tetragon.git
    cd tetragon
    git remote add upstream https://github.com/cilium/tetragon.git
    
  5. Prepare your Development setup, see section below.

  6. Check the GitHub issues for good tasks to get started.

  7. Follow the steps in Making changes to start contributing. Welcome :)!

7.1 - Development setup

This will help you getting started with your development setup to build Tetragon

Building and Running Tetragon

For local development, you will likely want to build and run bare-metal Tetragon.

Requirements:

  • go 1.18
  • GNU make
  • A running docker service
  • libcap and libelf (in Debian systems, e.g., install libelf-dev and libcap-dev)

You can build Tetragon as follows:

make

If you want to use podman instead of docker, you can do the following (assuming you need to use sudo with podman):

CONTAINER_ENGINE='sudo podman' make

You can ignore /bin/sh: docker: command not found in the output.

To build using the local clang, you can use:

CONTAINER_ENGINE='sudo podman' LOCAL_CLANG=1 LOCAL_CLANG_FORMAT=1 make

See Dockerfile.clang for the minimal required version of clang.

You should now have a ./tetragon binary, which can be run as follows:

sudo ./tetragon --bpf-lib bpf/objs

Notes:

  1. The --bpf-lib flag tells Tetragon where to look for its compiled BPF programs (which were built in the make step above).

  2. If Tetragon fails with an error "BTF discovery: candidate btf file does not exist", then make sure that your kernel support BTF, otherwise place a BTF file where Tetragon can read it and specify its path with the --btf flag.

Running Code Generation

Tetragon uses code generation based on protoc to generate large amounts of boilerplate code based on our protobuf API. We similarly use automatic generation to maintain our k8s CRDs. Whenever you make changes to these files, you will be required to re-run code generation before your PR can be accepted.

To run codegen from protoc, run the following command from the root of the repository:

make codegen

And to run k8s CRD generation, run the following command from the root of the repository:

make generate

Finally, should you wish to modify any of the resulting codegen files (ending in .pb.go), do not modify them directly. Instead, you can edit the files in cmd/protoc-gen-go-tetragon and then re-run make codegen.

Building and running a Docker image

The base kernel should support BTF or a BTF file should be bind mounted on top of /var/lib/tetragon/btf inside container.

To build Tetragon image:

make image

To run the image:

docker run --name tetragon \
   --rm -it -d --pid=host \
   --cgroupns=host --privileged \
   -v /sys/kernel/btf/vmlinux:/var/lib/tetragon/btf \
   cilium/tetragon:latest \
   bash -c "/usr/bin/tetragon"

Run the tetra binary to get Tetragon events:

docker exec -it tetragon \
   bash -c "/usr/bin/tetra getevents -o compact"

Building and running as a systemd service

To build Tetragon tarball:

make tarball

The produced tarball will be inside directory ./build/, then follow the package deployment guide to install it as a systemd service.

Running Tetragon in kind

The scripts in contrib/localdev will help you run Tetragon locally in a kind cluster. First, ensure that docker, kind, kubectl, and helm are installed on your system. Then, run the following commands:

# Build Tetragon agent and operator images
make LOCAL_CLANG=0 image image-operator

# Bootstrap the cluster
contrib/localdev/bootstrap-kind-cluster.sh

# Install Tetragon
contrib/localdev/install-tetragon.sh --image cilium/tetragon:latest --operator cilium/tetragon-operator:latest

Verify that Tetragon is installed by running:

kubectl get pods -n kube-system

Local Development in Vagrant Box

If you are on a Mac, use Vagrant to create a dev VM:

vagrant up
vagrant ssh
make

If you are getting an error, you can try to run sudo launchctl load /Library/LaunchDaemons/org.virtualbox.startup.plist (from a Stackoverflow answer).

Local Development in Minikube

You can also run the tetragon agent directly (instead of in a pod). Here we describe how this can be done in minikube:

minikube start --driver=kvm2
minikube mount $HOME:$HOME # so that we can use .kube/config
./tetragon-operator --kube-config ~/.kube/config
make STATIC=1 tetragon
minikube ssh --  'sudo mkdir -p /var/run/cilium/tetragon'
minikube ssh sudo "sh -c 'NODE_NAME=minikube /home/kkourt/src/tetragon/tetragon --bpf-lib /home/kkourt/src/tetragon/bpf/objs --server-address unix:///var/run/cilium/tetragon/tetragon.sock --enable-k8s-api --k8s-kubeconfig-path /home/kkourt/.kube/config'"

What’s next

7.2 - Making changes

Learn how to make your first changes to the project
  1. Make sure the main branch of your fork is up-to-date:

    git fetch upstream
    git checkout main
    git merge upstream/main
    

    For further reference read github syncing a fork documentation.

  2. Create a PR branch with a descriptive name, branching from main:

    git switch -c pr/${GITHUB_USERNAME_OR_ORG}/changes-to-something main
    
  3. Make the changes you want.

  4. Separate the changes into logical commits.

    • Describe the changes in the commit messages. Focus on answering the question why the change is required and document anything that might be unexpected.
    • If any description is required to understand your code changes, then those instructions should be code comments instead of statements in the commit description.
    • For submitting PRs, all commits need to be signed off (git commit -s). See the section Developer’s Certificate of Origin
  5. Make sure your changes meet the following criteria:

    • New code is covered by Integration Testing.
    • End to end integration / runtime tests have been extended or added. If not required, mention in the commit message what existing test covers the new code.
    • Follow-up commits are squashed together nicely. Commits should separate logical chunks of code and not represent a chronological list of changes.
  6. Run git diff --check to catch obvious white space violations

  7. Build Tetragon with your changes included.

What’s next

7.3 - Running tests

Learn how to run the tests of the project

See https://github.com/cilium/tetragon/tree/main/tests/vmtests.

What’s next

7.4 - Submitting a pull request

Learn how to submit a pull request to the project

Submitting a pull request

Contributions must be submitted in the form of pull requests against the upstream GitHub repository at https://github.com/cilium/tetragon.

  1. Fork the Tetragon repository.

  2. Push your changes to the topic branch in your fork of the repository.

  3. Submit a pull request on https://github.com/cilium/tetragon.

Before hitting the submit button, please make sure that the following requirements have been met:

  1. Each commit compiles and is functional on its own to allow for bisecting of commits.

  2. All code is covered by unit and/or runtime tests where feasible.

  3. All changes have been tested and checked for regressions by running the existing testsuite against your changes.

  4. All commits contain a well written commit description including a title, description and a Fixes: #XXX line if the commit addresses a particular GitHub issue identified by its number. Note that the GitHub issue will be automatically closed when the commit is merged.

    doc: add contribution guideline and how to submit pull requests
    
    Tetragon Open Source project was just released and it does not include
    default contributing guidelines.
    
    This patch fixes this by adding:
    
    1. CONTRIBUTING.md file in the root directory as suggested by github documentation: https://docs.github.com/en/communities/setting-up-your-project-for-healthy-contributions/setting-guidelines-for-repository-contributors
    
    2. Development guide under docs directory with a section on how to submit pull requests.
    
    3. Moves the DEVELOP.md file from root directory to the `docs/contributing/development/` one.
    
    Fixes: #33
    
    Signed-off-by: Djalal Harouni <djalal@cilium.io>
    

    Note: Make sure to include a blank line in between commit title and commit description.

  5. All commits are signed off. See the section Developer’s Certificate of Origin.

  6. All important steps in Making changes have been followed.

7.5 - Developer's certificate of origin

Learn about the “sign-off” procedure

To improve tracking of who did what, we’ve introduced a “sign-off” procedure, make sure to read and apply the Developer’s Certificate of Origin.

8 - FAQ

List of frequently asked questions

What is the minimum Linux kernel version to run Tetragon?

Tetragon needs Linux kernel version 4.19 or greater.

We currently run tests on stable long-term support kernels 4.19, 5.4, 5.10, 5.15 and bpf-next, see this test workflow for up to date information. Not all Tetragon features work with older kernel versions. BPF evolves rapidly and we recommend you use the most recent stable kernel possible to get the most out of Tetragon’s features.

Note that Tetragon also needs BTF support which might take some work on older kernels.

See the recommended Linux kernel configuration options

  # CORE BPF
  CONFIG_BPF
  CONFIG_BPF_JIT
  CONFIG_BPF_JIT_DEFAULT_ON
  CONFIG_BPF_EVENTS
  CONFIG_BPF_SYSCALL
  CONFIG_HAVE_BPF_JIT
  CONFIG_HAVE_EBPF_JIT
  CONFIG_FTRACE_SYSCALLS

  # BTF
  CONFIG_DEBUG_INFO_BTF
  CONFIG_DEBUG_INFO_BTF_MODULES

  # Enforcement
  CONFIG_BPF_KPROBE_OVERRIDE

  # CGROUP and Process tracking
  CONFIG_CGROUPS=y        Control Group support
  CONFIG_MEMCG=y          Memory Control group
  CONFIG_BLK_CGROUP=y     Generic block IO controller
  CONFIG_CGROUP_SCHED=y
  CONFIG_CGROUP_PIDS=y    Process Control group
  CONFIG_CGROUP_FREEZER=y Freeze and unfreeze tasks controller
  CONFIG_CPUSETS=y        Manage CPUSETs
  CONFIG_PROC_PID_CPUSET=y
  CONFIG_CGROUP_DEVICE=Y  Devices Control group
  CONFIG_CGROUP_CPUACCT=y CPU accouting controller
  CONFIG_CGROUP_PERF=y
  CONFIG_CGROUP_BPF=y     Attach eBPF programs to a cgroup
  CGROUP_FAVOR_DYNMODS=y  (optional)  >= 6.0
    Reduces the latencies of dynamic cgroup modifications at the
    cost of making hot path operations such as forks and exits
    more expensive.
    Platforms with frequent cgroup migrations could enable this
    option as a potential alleviation for pod and containers
    association issues.

Tetragon failed to start complaining about a missing BTF file

You might have encountered the following issues:

level=info msg="BTF discovery: default kernel btf file does not exist" btf-file=/sys/kernel/btf/vmlinux
level=info msg="BTF discovery: candidate btf file does not exist" btf-file=/var/lib/tetragon/metadata/vmlinux-5.15.49-linuxkit
level=info msg="BTF discovery: candidate btf file does not exist" btf-file=/var/lib/tetragon/btf
[...]
level=fatal msg="Failed to start tetragon" error="tetragon, aborting kernel autodiscovery failed: Kernel version \"5.15.49-linuxkit\" BTF search failed kernel is not included in supported list. Please check Tetragon requirements documentation, then use --btf option to specify BTF path and/or '--kernel' to specify kernel version"

Tetragon needs BTF (BPF Type Format) support to load its BPF programs using CO-RE (Compile Once - Run Everywhere). In brief, CO-RE is useful to load BPF programs that have been compiled on a different kernel version than the target kernel. Indeed, kernel structures change between versions and BPF programs need to access fields in them. So CO-RE uses the BTF file of the kernel in which you are loading the BPF program to know the differences between the struct and patch the fields offset in the accessed structures. CO-RE allows portability of the BPF programs but requires a kernel with BTF enabled.

Most of the common Linux distributions now ship with BTF enabled and do not require any extra work, this is kernel option CONFIG_DEBUG_INFO_BTF=y. To check if BTF is enabled on your Linux system and see the BTF data file of your kernel, the standard location is /sys/kernel/btf/vmlinux. By default, Tetragon will look for this file (this is the first line in the log output above).

If your kernel does not support BTF you can:

  • Retrieve the BTF file for your kernel version from an external source.
  • Build the BTF file from your kernel debug symbols. You will need pahole to add BTF metadata to the debugging symbols and LLVM minimize the medata size.
  • Rebuild your kernel with CONFIG_DEBUG_INFO_BTF to y.

Tetragon will also look into /var/lib/tetragon/btf for the vmlinux file (this is the third line in the log output above). Or you can use the --btf flag to directly indicate Tetragon where to locate the file.

If you encounter this issue while using Docker Desktop on macOS, please refer to can I run Tetragon on Mac computers.

Can I install and use Tetragon in standalone mode (outside of k8s)?

Yes! Check the Container deployment or Package deployment guides for alternative install methods.

Otherwise you can build Tetragon from source by running make to generate standalone binaries. Make sure to take a look at the Development Setup guide for the build requirements. Then use sudo ./tetragon --bpf-lib bpf/objs to run Tetragon.

CI is complaining about Go module vendoring, what do I do?

You can run make vendor then add and commit your changes.

CI is complaining about a missing “signed-off-by” line. What do I do?

You need to add a signed-off-by line to your commit messages. The easiest way to do this is with git fetch origin/main && git rebase --signoff origin/main. Then push your changes.

Can I run Tetragon on Mac computers?

Yes! You can run Tetragon locally by running a Linux virtual machine on your Mac.

On macOS running on amd64 (also known as Intel Mac) and arm64 (also know as Apple Silicon Mac), open source and commercial solutions exists to run virtual machines, here is a list of popular open source projects that you can use:

You can use these solutions to run a recent Linux distribution that ships with BTF debug information support.

Please note that you cannot use Docker Desktop on macOS, because the Linux virtual machine provided by Docker lacks support for the BTF debug information. The BTF debug information file is needed for CO-RE in order to load sensors of Tetragon. While it is technically possible to manually generate the BTF metadata, you will need the kernel debug symbols, and Docker stopped pushing the kernel images containing those debug symbols.

Issues are open on the GitHub repository for Docker Desktop on macOS and linuxkit, the project used to build the kernel used by Docker Desktop. If you would like Docker Desktop to support BTF, feel free to support those issues.

Go compiler is complaining about missing symbols in pkg/bpf

You might end up with the output of the Go compiler looking similar to this:

# github.com/cilium/tetragon/pkg/bpf
pkg/bpf/map_linux.go:174:9: undefined: bpfAttrObjOp
pkg/bpf/map_linux.go:266:19: undefined: bpfAttrMapOpElem
pkg/bpf/map_linux.go:310:19: undefined: bpfAttrMapOpElem
pkg/bpf/map_linux.go:318:16: undefined: bpfAttrMapOpElem
pkg/bpf/map_linux.go:415:9: undefined: bpfAttrMapOpElem
pkg/bpf/map_linux.go:422:9: undefined: UpdateElementFromPointers
pkg/bpf/map_linux.go:455:9: undefined: bpfAttrMapOpElem
pkg/bpf/map_linux.go:466:9: undefined: bpfAttrMapOpElem
pkg/bpf/map_linux.go:496:9: undefined: bpfAttrMapOpElem
pkg/bpf/map_linux.go:515:9: undefined: bpfAttrMapOpElem
pkg/bpf/map_linux.go:515:9: too many errors

It is likely because this package uses CGO and you might be missing a C compiler or the Go toolchain might fail detecting one. Also note from the CGO documentation that CGO is disabled by default when cross-compiling:

The cgo tool is enabled by default for native builds on systems where it is expected to work. It is disabled by default when cross-compiling as well as when the CC environment variable is unset and the default C compiler (typically gcc or clang) cannot be found on the system PATH. You can override the default by setting the CGO_ENABLED environment variable when running the go tool: set it to 1 to enable the use of cgo, and to 0 to disable it. […]

If you encounter the issue under Linux:

  • Check your Go environment information with go env CC. The output should be similar to gcc. Verify that gcc is installed with gcc --version. If it’s not the case, install gcc.
  • If you prefer to use clang, you can manually set the CC env variable to clang, for example building tetragon with CC=clang make tetragon. To persistently change the value of the CC go environment variable you can use go env -w "CC=clang".

9 - Resources

Additional resources to learn about Tetragon

Conference talks

TitleAuthorsConferenceDate
The Next Log4jshell?! Preparing for CVEs with eBPF!John Fastabend & Natália Réka IvánkóKubeCon EU2023
Tutorial: Getting Familiar with Security Observability Using eBPF & Cilium TetragonDuffie Cooley & Raphaël PinsonKubeCon EU2023
Securing the Superpowers: Who Loaded That eBPF Program?John Fastabend & Natália Réka IvánkóCloudNative SecurityCon NA2023
Container Security and Runtime Enforcement with TetragonDjalal HarounieBPF Summit2022
You and Your Security Profiles; Generating Security Policies with the Help of eBPFJohn Fastabend & Natália Réka IvánkóeBPF Day North America2022
Keeping your cluster safe from attacks with eBPFJed Salazar & Natália Réka IvánkóeBPF Summit2021
Uncovering a Sophisticated Kubernetes Attack in Real Time Part II.Jed Salazar & Natália Réka IvánkóO’Reilly Superstream Series, Infrastructure & Ops2021
Uncovering a Sophisticated Kubernetes Attack in Real-TimeJed Salazar & Natália Réka IvánkóKubeCon EU2020

Book

Security Observability with eBPF - Jed Salazar & Natália Réka Ivánkó, OReilly, 2022

Blog posts

Detecting a Container Escape with Cilium and eBPF - Natália Réka Ivánkó, 2021

Detecting and Blocking log4shell with Isovalent Cilium Enterprise - Jed Salazar, 2021

Hands-on lab

Security Observability with eBPF and Tetragon - Natália Réka Ivánkó, Roland Wolters, Raphaël Pinson