Linux Kernel Detection & Response: let's try to block some stuff (Part 3/3)

In Part 1 of this series, we covered how to configure the Linux audit framework — auditd rules, syscall monitoring, and kernel-level telemetry that captures what’s actually happening on your systems. In Part 2, we built the collection pipeline — centralizing those logs, normalizing them, and getting them into a place where they can be analyzed at scale.

Now comes the part that actually matters: turning all of that telemetry into detections that catch attackers.

Most organizations collect logs and never look at them. They have terabytes of audit trails sitting in Elasticsearch, rotated weekly, queried only during post-incident forensics when it’s already too late. The gap between “we have logs” and “we detect threats” is where detection engineering lives — and it’s the most underinvested discipline in most security teams.

Understanding What Your Logs Are Telling You

Before writing a single detection rule, you need to understand the patterns in your data. Raw auditd logs are dense, but they follow predictable structures when attacks occur.

A legitimate administrator session looks like this:

# Normal: user authenticates via SSH, runs expected commands
type=USER_LOGIN msg=audit(1713600000.000:1234): pid=5678 uid=0 auid=1000 ses=42
  subj=unconfined acct="admin" addr=10.0.1.50 terminal=ssh res=success
type=SYSCALL msg=audit(1713600001.000:1235): arch=c000003e syscall=59
  success=yes exit=0 a0=... comm="systemctl" exe="/usr/bin/systemctl"
  key="service_management"

A privilege escalation attack looks fundamentally different:

# Suspicious: unexpected SUID binary execution, followed by uid change
type=SYSCALL msg=audit(1713600500.000:4567): arch=c000003e syscall=59
  success=yes exit=0 a0=... comm="find" exe="/usr/bin/find"
  key="suid_execution" euid=0 uid=1001
type=SYSCALL msg=audit(1713600501.000:4568): arch=c000003e syscall=59
  success=yes exit=0 a0=... comm="bash" exe="/bin/bash" euid=0 uid=1001

The key difference: context and sequence. A single event is rarely conclusive. It’s the correlation of multiple events — a user authenticating from an unusual IP, executing a known-vulnerable SUID binary, then spawning a root shell — that constitutes a detection.

Detection Engineering as a Discipline

Detection engineering is not “writing SIEM rules.” It’s a systematic, engineering-driven practice that treats detections as code — versioned, tested, measured, and continuously improved.

According to the 2025 State of Detection Engineering Report, 80% of organizations now invest seriously in detection engineering, with that number climbing to 85% in large enterprises. The shift is clear: reactive alert-writing is being replaced by proactive, CI/CD-driven detection pipelines.

The core principles:

1. Detection-as-Code — Rules live in Git, go through pull requests, and are deployed via CI/CD pipelines. No more clicking around in a SIEM GUI.
2. Coverage-Driven — Measure what percentage of MITRE ATT&CK techniques you can detect. Identify gaps systematically, not by waiting for an incident to expose them.
3. Test with Real Attack Data — Every rule must be validated against known-good attack samples. If you haven’t tested it, it doesn’t work.
4. Tune Continuously — A rule that fires 500 times a day is worse than no rule at all. False positive rates are a first-class metric.

Sigma Rules: Portable Detection Logic

Sigma is to detection what YARA is to malware — a vendor-agnostic format for expressing detection logic. Write a rule once, compile it to any SIEM backend.

Sigma Rule Structure

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

title: Suspicious SUID Binary Execution
id: a1b2c3d4-e5f6-7890-abcd-ef1234567890
status: experimental
description: |
  Detects execution of commonly abused SUID binaries
  that can lead to privilege escalation.
references:
  - https://gtfobins.github.io/
author: Security Team
date: 2026/04/15
tags:
  - attack.privilege_escalation
  - attack.t1548.001   # Abuse Elevation Control Mechanism: SUID/SGID
logsource:
  product: linux
  service: auditd
  category: process_creation
detection:
  selection:
    type: SYSCALL
    syscall: execve
    key: suid_execution
  filter_known:
    exe:
      - /usr/bin/sudo
      - /usr/bin/passwd
      - /usr/bin/ping
  condition: selection and not filter_known
falsepositives:
  - Legitimate use of SUID binaries by administrators
level: high

Compiling Sigma to Wazuh

The Sigma CLI can convert rules to multiple backends:

1
2
3
4
5
6
7
8

# Install Sigma CLI and the Wazuh backend
pip install sigma-cli sigma-backend-wazuh

# Convert a single rule
sigma convert -t wazuh -p wazuh-linux rules/linux/privilege_escalation/suid_abuse.yml

# Batch convert an entire directory
sigma convert -t wazuh -p wazuh-linux rules/linux/ --output wazuh_rules/

Sigma to Other Backends

The power of Sigma is that your detection investment is portable. When you switch SIEMs (and you will, eventually), your detection logic moves with you.

Wazuh: Open-Source Detection and Response

Wazuh has become the de-facto open-source SIEM/XDR platform for Linux environments. It combines rule-based and anomaly-based detection, file integrity monitoring, vulnerability scanning, and active response — all without vendor lock-in.

Wazuh Architecture for Linux Detection

┌─────────────────┐     ┌─────────────────┐     ┌──────────────────┐
│  Linux Endpoint  │     │  Wazuh Manager   │     │  Wazuh Indexer   │
│  ┌─────────────┐│     │  ┌─────────────┐ │     │  (OpenSearch)    │
│  │ Wazuh Agent ││────▶│  │ Analysis     │ │────▶│  ┌────────────┐ │
│  │ - auditd    ││     │  │ Engine       │ │     │  │ Dashboards │ │
│  │ - syslog    ││     │  │ - Decoders   │ │     │  │ Alerts     │ │
│  │ - FIM       ││     │  │ - Rules      │ │     │  │ Compliance │ │
│  │ - osquery   ││     │  │ - Active Resp│ │     │  └────────────┘ │
│  └─────────────┘│     │  └─────────────┘ │     └──────────────────┘
└─────────────────┘     └─────────────────┘

Built-In Linux Rules

Wazuh ships with over 4,000 rules out of the box. The ones relevant to Linux kernel security:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

<!-- Built-in: PAM authentication failure (rule 5503) -->
<rule id="5503" level="5">
  <if_sid>5500</if_sid>
  <match>authentication failure</match>
  <description>PAM: User login failed.</description>
  <mitre>
    <id>T1110</id>  <!-- Brute Force -->
  </mitre>
</rule>

<!-- Built-in: Multiple authentication failures (rule 5712) -->
<rule id="5712" level="10" frequency="8" timeframe="120">
  <if_matched_sid>5503</if_matched_sid>
  <description>PAM: Possible brute force attack.</description>
  <mitre>
    <id>T1110</id>
  </mitre>
</rule>

Writing Custom Wazuh Rules

The built-in rules are a baseline. Real detection requires custom rules tailored to your environment. Here’s how to build them for common Linux attack patterns.

Custom decoder for auditd SYSCALL events:

1
2
3
4
5
6

<!-- /var/ossec/etc/decoders/local_decoder.xml -->
<decoder name="auditd-syscall-custom">
  <parent>auditd</parent>
  <regex>type=SYSCALL msg=audit\(\S+\): \.+ syscall=(\d+) \.+ exe="(\S+)" \.+ key="(\S+)"</regex>
  <order>audit.syscall, audit.exe, audit.key</order>
</decoder>

Custom rule for kernel module loading (rootkit detection):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

<!-- /var/ossec/etc/rules/local_rules.xml -->
<rule id="100010" level="14">
  <decoded_as>auditd-syscall-custom</decoded_as>
  <field name="audit.key">^kernel_module$</field>
  <description>Kernel module loaded — possible rootkit installation.</description>
  <mitre>
    <id>T1547.006</id>  <!-- Boot or Logon Autostart: Kernel Modules -->
  </mitre>
  <group>rootkit,kernel,</group>
</rule>

MITRE ATT&CK Mapping: Making Detections Meaningful

Every detection rule should map to at least one MITRE ATT&CK technique. Without this mapping, you have alerts with no context — and analysts waste time figuring out what the alert means instead of responding to it.

Here’s the mapping for common Linux attack vectors:

Building Detection Rules for Common Linux Attacks

Privilege Escalation: Sudo Abuse and SUID Exploitation

Sudo abuse is the most common privilege escalation path on Linux. Attackers look for misconfigured sudoers entries that allow command execution as root.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

# Sigma rule: Suspicious sudo usage
title: Sudo to Root Shell
status: stable
logsource:
  product: linux
  service: auditd
detection:
  selection:
    type: EXECVE
    a0: sudo
  suspicious_commands:
    a1|contains:
      - '/bin/bash'
      - '/bin/sh'
      - 'su -'
      - '/usr/bin/env'
  condition: selection and suspicious_commands
level: high
tags:
  - attack.privilege_escalation
  - attack.t1548.003

For SUID exploitation, monitor the execution of binaries listed on GTFOBins:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

<!-- Wazuh rule: SUID binary abuse -->
<rule id="100020" level="12">
  <decoded_as>auditd-syscall-custom</decoded_as>
  <field name="audit.key">^suid_execution$</field>
  <field name="audit.exe">find|nmap|vim|python|perl|ruby|php|env|awk</field>
  <description>Potentially abusable SUID binary executed: $(audit.exe)</description>
  <mitre>
    <id>T1548.001</id>
  </mitre>
</rule>

Persistence: Cron Jobs, SSH Keys, and Systemd Services

Attackers need persistence to survive reboots and session terminations. The three most common mechanisms on Linux:

Wazuh File Integrity Monitoring configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

<!-- /var/ossec/etc/ossec.conf — FIM for persistence paths -->
<syscheck>
  <!-- Cron persistence -->
  <directories check_all="yes" realtime="yes">/etc/crontab</directories>
  <directories check_all="yes" realtime="yes">/etc/cron.d</directories>
  <directories check_all="yes" realtime="yes">/var/spool/cron</directories>

  <!-- SSH persistence -->
  <directories check_all="yes" realtime="yes">/root/.ssh</directories>
  <directories check_all="yes" realtime="yes" restrict=".ssh/authorized_keys">
    /home
  </directories>

  <!-- Systemd persistence -->
  <directories check_all="yes" realtime="yes">/etc/systemd/system</directories>
  <directories check_all="yes" realtime="yes">/usr/lib/systemd/system</directories>
</syscheck>

Custom rule to alert on new authorized_keys entries:

1
2
3
4
5
6
7
8

<rule id="100030" level="10">
  <if_sid>554</if_sid>  <!-- FIM file modified -->
  <field name="file">authorized_keys</field>
  <description>SSH authorized_keys file modified — possible persistence mechanism.</description>
  <mitre>
    <id>T1098.004</id>
  </mitre>
</rule>

Lateral Movement: SSH Brute Force and Unusual Connections

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

<!-- Wazuh: SSH brute force with geographic context -->
<rule id="100040" level="12" frequency="6" timeframe="60">
  <if_matched_sid>5503</if_matched_sid>
  <same_source_ip />
  <description>SSH brute force detected: 6+ failures in 60s from same IP.</description>
  <mitre>
    <id>T1110.001</id>
  </mitre>
  <group>authentication_failures,brute_force,</group>
</rule>

<!-- Unusual outbound connection from server -->
<rule id="100041" level="10">
  <decoded_as>auditd-syscall-custom</decoded_as>
  <field name="audit.key">^network_connect$</field>
  <field name="audit.exe">nc|ncat|socat|telnet|curl|wget</field>
  <description>Unusual network tool executed: $(audit.exe) — possible lateral movement or data exfiltration.</description>
  <mitre>
    <id>T1021</id>
  </mitre>
</rule>

Defense Evasion: Log Tampering, Timestomping, Fileless Execution

These are the techniques that make incident response difficult. If an attacker can erase their tracks, your audit trail becomes unreliable.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

<!-- Log file deletion or truncation -->
<rule id="100050" level="14">
  <decoded_as>auditd-syscall-custom</decoded_as>
  <field name="audit.key">^log_tampering$</field>
  <description>Log file tampering detected — audit trail integrity compromised.</description>
  <mitre>
    <id>T1070.002</id>
  </mitre>
  <group>log_tampering,critical,</group>
</rule>

<!-- Timestomping detection via auditd -->
<rule id="100051" level="12">
  <decoded_as>auditd-syscall-custom</decoded_as>
  <field name="audit.key">^timestomp$</field>
  <description>File timestamp modification detected — possible timestomping (T1070.006).</description>
  <mitre>
    <id>T1070.006</id>
  </mitre>
</rule>

The corresponding auditd rules that generate these events:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# /etc/audit/rules.d/detection.rules

# Log tampering — monitor deletions and truncations in /var/log
-a always,exit -F arch=b64 -S unlinkat -S renameat -F dir=/var/log -F key=log_tampering
-a always,exit -F arch=b64 -S truncate -S ftruncate -F dir=/var/log -F key=log_tampering

# Timestomping — monitor timestamp modification syscalls
-a always,exit -F arch=b64 -S utimensat -S futimesat -F key=timestomp

# Fileless execution — monitor /proc/pid/mem writes
-a always,exit -F arch=b64 -S openat -F path=/proc -F perm=w -F key=proc_mem_write

# Kernel module loading
-a always,exit -F arch=b64 -S init_module -S finit_module -S delete_module -F key=kernel_module

The Detection Toolchain

Detection engineering on Linux isn’t a single-tool problem. Each tool in the chain serves a specific purpose:

The recommended stack for most Linux environments:

auditd (telemetry) → Wazuh Agent (collection) → Wazuh Manager (detection) → OpenSearch (storage/search)
                                                       ↑
                                              Sigma rules (portable logic)
                                              YARA rules (malware scanning)
                                              osquery (endpoint queries)

Active Response: Automated Containment

Detection without response is just expensive logging. Wazuh’s active response capability lets you automate containment actions when high-confidence detections fire.

Block an IP after brute force detection:

1
2
3
4
5
6
7

<!-- /var/ossec/etc/ossec.conf -->
<active-response>
  <command>firewall-drop</command>
  <location>local</location>
  <rules_id>100040</rules_id>  <!-- Our SSH brute force rule -->
  <timeout>3600</timeout>       <!-- Block for 1 hour -->
</active-response>

Disable a compromised user account:

1
2
3
4
5
6

<active-response>
  <command>disable-account</command>
  <location>local</location>
  <rules_id>100010</rules_id>  <!-- Kernel module loading -->
  <timeout>no</timeout>         <!-- Permanent until manual review -->
</active-response>

The disable-account script:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

#!/bin/bash
# /var/ossec/active-response/bin/disable-account.sh
# Wazuh active response: disable compromised user account

LOCAL=$(dirname $0)
USER=$1
ACTION=$2

if [ "$ACTION" = "add" ]; then
    # Lock the account immediately
    passwd -l "$USER" 2>/dev/null
    # Kill all user sessions
    pkill -KILL -u "$USER" 2>/dev/null
    logger -t wazuh-ar "Active response: disabled account $USER"
elif [ "$ACTION" = "delete" ]; then
    # Re-enable (manual review completed)
    passwd -u "$USER" 2>/dev/null
    logger -t wazuh-ar "Active response: re-enabled account $USER"
fi

The Detection Engineering Lifecycle

Detection engineering is iterative. No rule is ever “done.” Follow this cycle:

    ┌──────────────┐
    │  Hypothesis   │  "Attackers escalating via SUID find on our servers"
    └──────┬───────┘
           ▼
    ┌──────────────┐
    │  Write Rule   │  Sigma/Wazuh rule targeting SUID find execution
    └──────┬───────┘
           ▼
    ┌──────────────┐
    │  Test Rule    │  Run against attack simulation (Atomic Red Team)
    └──────┬───────┘
           ▼
    ┌──────────────┐
    │  Tune Rule    │  Exclude legitimate admin usage, reduce false positives
    └──────┬───────┘
           ▼
    ┌──────────────┐
    │  Deploy Rule  │  Push via CI/CD to Wazuh Manager
    └──────┬───────┘
           ▼
    ┌──────────────┐
    │  Measure      │  Track: true positive rate, MTTD, ATT&CK coverage
    └──────┬───────┘
           │
           └──────────▶ Back to Hypothesis (new intelligence, new gaps)

Testing with Atomic Red Team:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

# Install Atomic Red Team
git clone https://github.com/redcanaryco/atomic-red-team.git

# Execute a specific ATT&CK technique to test your detection
# T1548.001 — SUID/SGID abuse
bash atomic-red-team/atomics/T1548.001/T1548.001.sh

# T1053.003 — Cron job persistence
bash atomic-red-team/atomics/T1053.003/T1053.003.sh

# T1070.002 — Log deletion
bash atomic-red-team/atomics/T1070.002/T1070.002.sh

# After each test: verify your Wazuh alerts fired correctly
/var/ossec/bin/ossec-logtest < test_log_sample.log

Key metrics for your detection program:

Incident Response Workflow

When a detection fires, the collected telemetry from Parts 1 and 2 becomes your investigation foundation. Here’s a practical workflow:

1. Triage (0-5 minutes)

• Assess alert severity and MITRE ATT&CK context
• Check if the alert correlates with other recent events on the same host
• Determine if active response already contained the threat

2. Scope (5-15 minutes)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Query Wazuh API for all alerts from the affected host in the last hour
curl -k -X GET "https://wazuh-manager:55000/alerts?agents_list=003&limit=100" \
  -H "Authorization: Bearer $TOKEN"

# Use osquery to snapshot the current system state
osqueryi "SELECT pid, name, cmdline, uid, start_time FROM processes
          WHERE uid = 0 AND start_time > (strftime('%s','now') - 3600);"

# Check auditd for the full execution chain
ausearch --start recent -k suid_execution -i
ausearch --start recent -k network_connect -i

3. Contain (15-30 minutes)

• Isolate the host from the network (Wazuh active response or manual)
• Preserve volatile evidence (memory dump, process list, network connections)
• Rotate compromised credentials

4. Investigate (30 minutes - hours)

• Reconstruct the full attack chain using auditd logs
• Identify initial access vector, persistence mechanisms, lateral movement
• Determine data exposure

5. Remediate and Improve

• Remove persistence mechanisms
• Patch the exploited vulnerability
• Update detection rules based on lessons learned
• Add new coverage for any gaps the attack exposed

Conclusion

The trilogy is complete. Auditing gives you visibility. Collection gives you centralized data. Detection and response give you the ability to act on that data before an attacker achieves their objective.

The key takeaways:

• Detection engineering is a discipline, not a task. Treat your rules like production code — version them, test them, measure them, and iterate.
• Sigma gives you portability. Invest in Sigma rules and you’re never locked into a single SIEM vendor.
• Wazuh is a capable, open-source detection platform for Linux environments. Its combination of built-in rules, custom decoders, FIM, and active response covers the full detection-to-response pipeline.
• Map everything to MITRE ATT&CK. It forces you to think about what you’re detecting and — more importantly — what you’re not.
• Automate response carefully. Active response is powerful but dangerous. Start with alerting, validate your rules, then graduate to automated containment for high-confidence detections.

The gap between “we have security tools” and “we detect and respond to threats” is filled by engineering rigor, not more products. Build the pipeline, test it relentlessly, and iterate.

References:

• Sigma Rules — SigmaHQ GitHub
• Wazuh Documentation — Custom Rules and Decoders
• MITRE ATT&CK Linux Matrix
• Atomic Red Team — Red Canary
• Neo23x0 Auditd Best Practice Configuration
• 2025 State of Detection Engineering Report — Anvilogic
• Wazuh: Open Source SIEM/XDR
• Tetragon — eBPF-based Security Observability
• GTFOBins — SUID Binary Abuse Reference