Systemd is the default init system in most modern Linux distributions, responsible for managing services, boot processes, logging, and system events. Its logging component, journalctl, stores extensive system and application logs in a structured, binary format. For forensic investigators, systemd and journalctl provide critical insights into system activity, startup behavior, service execution, process failures, and potential malicious activity.

This chapter covers how systemd works, how services create persistence, and how journalctl logs can be analyzed for forensic investigations.

Understanding Systemd

Systemd is the service manager that boots the system, controls services, manages dependencies, and provides logging through the journal. It replaces older init systems like SysVinit.

Systemd components important for forensics:

Systemd Services – define persistent or boot-time processes
Systemd Timers – scheduled tasks replacing cron in many cases
Systemd Targets – define system states
Systemd Journal – centralized logging system

Attackers often target systemd because services can run with root privileges and persist across reboots.

Systemd Service Files

Systemd services define how programs start, stop, restart, and auto-run.

Service file locations:

1. System-Wide Services

/etc/systemd/system/
/lib/systemd/system/
/usr/lib/systemd/system/

2. User-Level Services

~/.config/systemd/user/

Types of suspicious services:

Newly added services
Services with strange names (misspellings of legitimate services)
Services pointing to odd directories (/tmp, /var/tmp, hidden folders)
Services executing scripts instead of binaries

Structure of a Systemd Service File

Example service:

[Unit]
Description=System Update Service

[Service]
ExecStart=/usr/bin/python3 /etc/system-update/.update.py
Restart=always

[Install]
WantedBy=multi-user.target

Indicators of compromise:

ExecStart pointing to hidden or unusual paths
Startup scripts in /tmp/.hidden/, /dev/shm/, custom folders
Unexpected Restart=always for persistence
Services enabled automatically without admin knowledge

Commands for Investigating Systemd Services

List all active services:

systemctl list-units --type=service

List enabled services (boot-time):

systemctl list-unit-files | grep enabled

View service details:

systemctl cat <service-name>

Check service status:

systemctl status <service-name>

Forensic value:

Detect unauthorized services
Identify persistence mechanisms
View execution paths
Review last execution logs

Systemd Timers (Cron Alternative)

Systemd timers can replace cron jobs. Attackers may use timers to schedule malicious tasks.

Timer file locations:

/etc/systemd/system/
/usr/lib/systemd/system/

List timers:

systemctl list-timers

Look for unusual, suspicious, or newly added timers.

Example malicious timer:

[Timer]
OnBootSec=5min
OnUnitActiveSec=1min

Introduction to journalctl

Journalctl is the primary tool to read systemd logs.
Logs are stored in a binary format in:

/var/log/journal/

or (volatile):

/run/log/journal/

These logs include:

Boot logs
Service logs
Kernel logs
Login attempts
SSH connections
Process crashes
System messages
Suspicious task executions

Because systemd logs EVERYTHING in one place, journalctl is extremely useful for forensic investigations.

Essential Journalctl Commands for Forensics

View entire journal:

journalctl

View logs since a specific time:

journalctl --since "2025-01-01"

View boot logs:

journalctl -b

or older boots:

journalctl -b -1
journalctl -b -2

View logs for a specific service:

journalctl -u <service-name>

View kernel logs:

journalctl -k

View logs by user ID:

journalctl _UID=1000

Useful for mapping user-level activity.

View failures:

journalctl -p err
journalctl -p warning

Forensic Value of Journalctl

Journalctl logs reveal:

Service execution history
Suspicious restarts or crashes
Unauthorized services being launched
Failed login attempts
sudo activity
SSH connections
System modifications
Time-based anomalies
Malware execution paths

Because systemd logs everything with timestamps and process IDs, timeline reconstruction becomes highly accurate.

Detecting Malicious Systemd Activity

1. Suspicious Services

Look for:

Strange names (typosquatting): sysnetd, network-mgr, etc.
Services pointing to hidden directories
Services created recently (check timestamps)

2. Services Executing Scripts Instead of Binaries

Example:

ExecStart=/bin/bash /tmp/malware.sh

Legitimate services usually run binaries, not scripts.

3. High-Frequency Restarts

Using:

Restart=always

This keeps malware running even after crashing.

4. Logs Showing Repeated Execution

journalctl -u suspicious.service

If it runs every minute, it’s likely malicious.

Detecting Malware in Journal Logs

Look for patterns such as:

Strange commands being executed:

wget/curl downloads
Base64 decoding
Python reverse shells
Netcat listeners

Unauthorized user creation:

useradd attacker
passwd attacker

Unauthorized sudo usage:

journalctl -u sudo

Manipulation of system files:

Modifying startup scripts
Changing service configurations
Editing SSH keys

Journalctl Log Persistence & Tampering

Logs may exist in two places:

Persistent: /var/log/journal/
Volatile: /run/log/journal/

Attackers may disable persistent logging:

systemctl disable systemd-journald

or clear logs:

rm -rf /var/log/journal/*

Evidence of missing logs itself indicates tampering.

Best Practices for Systemd & Journalctl Forensics

Check all service files for unauthorized modifications.
Compare systemd services with known-good baselines.
Analyze journalctl logs for:
- Suspicious commands
- Failed logins
- Service restarts
- Unexpected boots/shutdowns
Verify systemd timers for stealth persistence.
Look for scripts stored in unusual directories.
Check file modification timestamps of service files.
Use tools like find to detect recently modified files:

find /etc/systemd -type f -mtime -3

Summary

Systemd and journalctl provide powerful forensic insights into Linux system behavior. Systemd services and timers are commonly abused by attackers for persistence, while journalctl stores detailed logs covering everything from boots, logins, and crashes to service-level execution. By examining service files, analyzing journal logs, and correlating timestamps, investigators can uncover unauthorized access, malware execution, persistence techniques, and system tampering. Understanding systemd and journalctl is essential for accurate and effective Linux forensic investigations.