Writing systemd Services That Actually Work

#linux#systemd#infrastructure

At some point you write a script or a server that needs to run forever. You could use screen, tmux, nohup, or a cron @reboot hack. Or you could write a systemd service and get automatic restarts, logging, resource limits, and dependency management for free.

The Minimal Service

Create /etc/systemd/system/myapp.service:

[Unit]
Description=My Application
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/node /opt/myapp/server.js
Restart=unless-stopped
RestartSec=5
WorkingDirectory=/opt/myapp

[Install]
WantedBy=multi-user.target

Then:

sudo systemctl daemon-reload
sudo systemctl enable myapp
sudo systemctl start myapp

That is it. The app starts on boot, restarts on crash, and logs to journald.

Service Types

Type=simple is correct for most things. The process you start is the service. But there are others:

# App forks into background (like old-school daemons)
Type=forking
PIDFile=/var/run/myapp.pid

# App signals readiness via sd_notify
Type=notify

# App is a one-shot script, not a long-running process
Type=oneshot
RemainAfterExit=yes

Use simple unless you have a specific reason not to. oneshot is useful for startup scripts that need to run once.

Environment Variables

Three ways to pass them:

# Inline (good for a few vars)
[Service]
Environment=PORT=3000
Environment=NODE_ENV=production

# From a file (good for secrets)
[Service]
EnvironmentFile=/opt/myapp/.env

# Multiple files
[Service]
EnvironmentFile=/opt/myapp/.env
EnvironmentFile=/opt/myapp/.env.local

The EnvironmentFile approach keeps secrets out of the unit file. The unit file itself might be in version control or readable by any user. The .env file can be locked down with permissions.

Running As a Non-Root User

[Service]
User=myapp
Group=myapp
ExecStart=/usr/bin/node /opt/myapp/server.js

Create the user first:

sudo useradd -r -s /usr/sbin/nologin -d /opt/myapp myapp
sudo chown -R myapp:myapp /opt/myapp

The -r flag creates a system user. -s /usr/sbin/nologin prevents interactive login.

Resource Limits

Prevent a runaway process from eating all your RAM or CPU:

[Service]
MemoryMax=512M
CPUQuota=50%
TasksMax=100
LimitNOFILE=65535

MemoryMax kills the process if it exceeds the limit. CPUQuota throttles but does not kill. TasksMax limits the number of threads and child processes. LimitNOFILE sets the open file descriptor limit.

Restart Policies

# Restart on any exit (crash, signal, clean exit)
Restart=always

# Restart on failure only (non-zero exit, signal)
Restart=on-failure

# Restart unless you explicitly stopped it
Restart=unless-stopped

I use unless-stopped for everything. It restarts on crashes but stays down when I manually stop the service for debugging.

Add rate limiting to prevent restart loops:

RestartSec=5
StartLimitIntervalSec=60
StartLimitBurst=5

This allows 5 restarts within 60 seconds. After that, the service enters a failed state and stops trying. Check what happened with journalctl -u myapp.

Logging

systemd captures stdout and stderr automatically:

# View logs
journalctl -u myapp

# Follow in real time
journalctl -u myapp -f

# Last 50 lines
journalctl -u myapp -n 50

# Since last boot
journalctl -u myapp -b

# JSON output for parsing
journalctl -u myapp -o json-pretty

No log rotation config needed. journald handles it automatically based on disk space.

Dependencies and Ordering

[Unit]
Description=My App
After=network.target postgresql.service
Requires=postgresql.service

After controls startup order. Requires means if PostgreSQL stops, myapp stops too. Use Wants instead of Requires for soft dependencies where your app can handle the other service being down.

Pre and Post Commands

[Service]
ExecStartPre=/opt/myapp/migrate.sh
ExecStart=/usr/bin/node /opt/myapp/server.js
ExecStartPost=/usr/bin/curl -s http://localhost:3000/health
ExecStopPost=/opt/myapp/cleanup.sh

ExecStartPre runs before the main process. Good for database migrations or config validation. If it fails, the service does not start.

Debugging

# Check service status
systemctl status myapp

# View full logs on failure
journalctl -u myapp -e --no-pager

# Check the actual unit file being used
systemctl cat myapp

# Verify unit file syntax
systemd-analyze verify myapp.service

The systemd-analyze verify command catches syntax errors without starting anything. Use it before reloading.

systemd is verbose and opinionated, but once you get past the initial learning curve, it handles all the edge cases that screen and nohup do not. Crashes, reboots, logging, resource limits, dependencies. Write the unit file once and forget about it.

← all articles wleeaf.dev →