Pattern 1: Print a Column#

awk splits each line into fields. $1 is the first, $2 is the second, $0 is the whole line:

# Print the first column of ps output
ps aux | awk '{print $1}'

# Print columns 1 and 11 (user and command)
ps aux | awk '{print $1, $11}'

The default field separator is whitespace. Change it with -F:

# Parse /etc/passwd (colon-separated)
awk -F: '{print $1, $7}' /etc/passwd

# Parse CSV
awk -F, '{print $2}' data.csv

Pattern 2: Filter Rows#

Put a condition before the action:

# Lines where column 3 is greater than 100
awk '$3 > 100' data.txt

# Lines containing "ERROR"
awk '/ERROR/' app.log

# Lines where the first field matches a pattern
awk '$1 ~ /^192\.168/' access.log

Combine conditions:

# Memory usage over 50% AND the process is not root
ps aux | awk '$4 > 50.0 && $1 != "root" {print $1, $4, $11}'

Pattern 3: Sum a Column#

# Total bytes transferred (column 10 in access logs)
awk '{sum += $10} END {print sum}' access.log

# Average response time
awk '{sum += $NF; n++} END {print sum/n}' response_times.log

END runs after all lines are processed. NF is the number of fields on the current line, so $NF is the last field.

Pattern 4: Count and Group#

# Count HTTP status codes
awk '{count[$9]++} END {for (c in count) print c, count[c]}' access.log | sort -rn -k2

# Output:
# 200 45231
# 304 12044
# 404 891
# 500 23

This uses an associative array. count[$9]++ increments the count for each unique value in column 9.

Top 10 IP addresses by request count:

awk '{count[$1]++} END {for (ip in count) print count[ip], ip}' access.log | sort -rn | head -10

Pattern 5: Transform Fields#

# Convert bytes to megabytes
awk '{printf "%s %.1fMB\n", $1, $2/1048576}' sizes.txt

# Add a prefix to every line
awk '{print "[INFO]", $0}' app.log

# Swap two columns
awk '{temp=$1; $1=$2; $2=temp; print}' data.txt

Real-World Examples#

Parse Docker ps output#

docker ps --format '{{.Names}}\t{{.Status}}\t{{.Ports}}' | \
  awk -F'\t' '{printf "%-20s %-15s %s\n", $1, $2, $3}'

Find the largest Docker images#

docker images --format '{{.Repository}}:{{.Tag}}\t{{.Size}}' | \
  awk -F'\t' '{
    size = $2
    if (size ~ /GB/) {
      gsub(/GB/, "", size)
      mb = size * 1024
    } else if (size ~ /MB/) {
      gsub(/MB/, "", size)
      mb = size
    } else {
      mb = 0
    }
    printf "%8.0fMB  %s\n", mb, $1
  }' | sort -rn | head -10

Calculate uptime percentage#

Given a log with timestamps and status:

awk '
  /UP/   { up++ }
  /DOWN/ { down++ }
  END {
    total = up + down
    if (total > 0)
      printf "Uptime: %.2f%% (%d/%d checks)\n", (up/total)*100, up, total
  }
' monitoring.log

Extract unique error messages#

awk '/ERROR/ {
  msg = ""
  for (i=4; i<=NF; i++) msg = msg " " $i
  errors[msg]++
}
END {
  for (e in errors) printf "%5d %s\n", errors[e], e
}' app.log | sort -rn | head -20

Built-in Variables#

When to Use awk vs. Other Tools#

• cut: When you just need a column and nothing else. cut -d: -f1 /etc/passwd is simpler than awk for trivial extraction.
• grep: When you just need to filter lines by a pattern.
• sed: When you need to do find-and-replace on text.
• awk: When you need to do math, grouping, conditional logic, or work with multiple fields at once.

The mental model is simple. awk reads lines, splits them into fields, and lets you write rules about what to do with each line. Once that clicks, you start seeing awk solutions everywhere.

Quick HTTP Server#

The classic. Serve the current directory:

python3 -m http.server 8080

Bind to a specific interface:

python3 -m http.server 8080 --bind 127.0.0.1

This is genuinely useful for transferring files between machines, testing static sites, or sharing a directory with a coworker.

JSON Pretty Printing#

When you do not have jq installed:

curl -s https://api.example.com/data | python3 -m json.tool

Or with color:

echo '{"a":1,"b":[2,3]}' | python3 -c "
import json, sys
data = json.load(sys.stdin)
print(json.dumps(data, indent=2, sort_keys=True))
"

Parse Timestamps#

Convert epoch timestamps that show up in logs:

python3 -c "
from datetime import datetime
import sys
for line in sys.stdin:
    ts = int(line.strip())
    print(datetime.fromtimestamp(ts).isoformat())
" <<< "1711708800"
# 2024-03-29T12:00:00

Find Large Files#

find works, but if you want more control over the output:

python3 -c "
import os
files = []
for root, dirs, fnames in os.walk('/var/log'):
    for f in fnames:
        path = os.path.join(root, f)
        try:
            size = os.path.getsize(path)
            if size > 100_000_000:  # 100MB
                files.append((size, path))
        except OSError:
            pass
for size, path in sorted(files, reverse=True):
    print(f'{size / 1e9:.1f}G  {path}')
"

Base64 Decode#

Decode those opaque values in Kubernetes secrets or JWTs:

python3 -c "
import base64, json, sys
token = sys.argv[1]
parts = token.split('.')
for part in parts[:2]:
    padded = part + '=' * (4 - len(part) % 4)
    decoded = base64.urlsafe_b64decode(padded)
    print(json.dumps(json.loads(decoded), indent=2))
" "eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxMjM0In0.signature"

Port Scanner#

Quick check which ports are open without installing nmap:

python3 -c "
import socket, sys
host = sys.argv[1]
for port in range(1, 1025):
    s = socket.socket()
    s.settimeout(0.1)
    if s.connect_ex((host, port)) == 0:
        print(f'{port}/tcp open')
    s.close()
" 10.0.0.1

Monitor a Log File With Filtering#

Like tail -f | grep but with more logic:

python3 -c "
import sys, re, time

pattern = re.compile(r'(ERROR|WARN|FATAL)', re.IGNORECASE)
seen = set()

for line in sys.stdin:
    if pattern.search(line):
        key = line.strip()
        if key not in seen:
            seen.add(key)
            print(line, end='')
" < /var/log/syslog

This deduplicates repeated errors so you see each unique message only once.

CSV to JSON#

Convert that spreadsheet export into something usable:

python3 -c "
import csv, json, sys
reader = csv.DictReader(sys.stdin)
print(json.dumps(list(reader), indent=2))
" < data.csv

URL Encode and Decode#

# Encode
python3 -c "from urllib.parse import quote; print(quote('hello world & friends'))"
# hello%20world%20%26%20friends

# Decode
python3 -c "from urllib.parse import unquote; print(unquote('hello%20world%20%26%20friends'))"
# hello world & friends

Generate Random Passwords#

python3 -c "
import secrets, string
alphabet = string.ascii_letters + string.digits + '!@#$%'
password = ''.join(secrets.choice(alphabet) for _ in range(24))
print(password)
"

IP Address Math#

Check if an address is in a subnet:

python3 -c "
from ipaddress import ip_address, ip_network
net = ip_network('10.0.0.0/24')
print(ip_address('10.0.0.42') in net)  # True
print(ip_address('10.14.0.1') in net)    # False
print(f'Network: {net.network_address}')
print(f'Broadcast: {net.broadcast_address}')
print(f'Hosts: {net.num_addresses - 2}')
"

The standard library covers most sysadmin needs without pip. json, csv, http.server, ipaddress, secrets, urllib, base64, socket. It is all there. When the bash one-liner turns into a bash five-liner with nested quotes and escaped dollars signs, switch to Python. Your future self will thank you.

The Minimal Service#

Create /etc/systemd/system/myapp.service:

[Unit]
Description=My Application
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/node /opt/myapp/server.js
Restart=unless-stopped
RestartSec=5
WorkingDirectory=/opt/myapp

[Install]
WantedBy=multi-user.target

Then:

sudo systemctl daemon-reload
sudo systemctl enable myapp
sudo systemctl start myapp

That is it. The app starts on boot, restarts on crash, and logs to journald.

Service Types#

Type=simple is correct for most things. The process you start is the service. But there are others:

# App forks into background (like old-school daemons)
Type=forking
PIDFile=/var/run/myapp.pid

# App signals readiness via sd_notify
Type=notify

# App is a one-shot script, not a long-running process
Type=oneshot
RemainAfterExit=yes

Use simple unless you have a specific reason not to. oneshot is useful for startup scripts that need to run once.

Environment Variables#

Three ways to pass them:

# Inline (good for a few vars)
[Service]
Environment=PORT=3000
Environment=NODE_ENV=production

# From a file (good for secrets)
[Service]
EnvironmentFile=/opt/myapp/.env

# Multiple files
[Service]
EnvironmentFile=/opt/myapp/.env
EnvironmentFile=/opt/myapp/.env.local

The EnvironmentFile approach keeps secrets out of the unit file. The unit file itself might be in version control or readable by any user. The .env file can be locked down with permissions.

Running As a Non-Root User#

[Service]
User=myapp
Group=myapp
ExecStart=/usr/bin/node /opt/myapp/server.js

Create the user first:

sudo useradd -r -s /usr/sbin/nologin -d /opt/myapp myapp
sudo chown -R myapp:myapp /opt/myapp

The -r flag creates a system user. -s /usr/sbin/nologin prevents interactive login.

Resource Limits#

Prevent a runaway process from eating all your RAM or CPU:

[Service]
MemoryMax=512M
CPUQuota=50%
TasksMax=100
LimitNOFILE=65535

MemoryMax kills the process if it exceeds the limit. CPUQuota throttles but does not kill. TasksMax limits the number of threads and child processes. LimitNOFILE sets the open file descriptor limit.

Restart Policies#

# Restart on any exit (crash, signal, clean exit)
Restart=always

# Restart on failure only (non-zero exit, signal)
Restart=on-failure

# Restart unless you explicitly stopped it
Restart=unless-stopped

I use unless-stopped for everything. It restarts on crashes but stays down when I manually stop the service for debugging.

Add rate limiting to prevent restart loops:

RestartSec=5
StartLimitIntervalSec=60
StartLimitBurst=5

This allows 5 restarts within 60 seconds. After that, the service enters a failed state and stops trying. Check what happened with journalctl -u myapp.

Logging#

systemd captures stdout and stderr automatically:

# View logs
journalctl -u myapp

# Follow in real time
journalctl -u myapp -f

# Last 50 lines
journalctl -u myapp -n 50

# Since last boot
journalctl -u myapp -b

# JSON output for parsing
journalctl -u myapp -o json-pretty

No log rotation config needed. journald handles it automatically based on disk space.

Dependencies and Ordering#

[Unit]
Description=My App
After=network.target postgresql.service
Requires=postgresql.service

After controls startup order. Requires means if PostgreSQL stops, myapp stops too. Use Wants instead of Requires for soft dependencies where your app can handle the other service being down.

Pre and Post Commands#

[Service]
ExecStartPre=/opt/myapp/migrate.sh
ExecStart=/usr/bin/node /opt/myapp/server.js
ExecStartPost=/usr/bin/curl -s http://localhost:3000/health
ExecStopPost=/opt/myapp/cleanup.sh

ExecStartPre runs before the main process. Good for database migrations or config validation. If it fails, the service does not start.

Debugging#

# Check service status
systemctl status myapp

# View full logs on failure
journalctl -u myapp -e --no-pager

# Check the actual unit file being used
systemctl cat myapp

# Verify unit file syntax
systemd-analyze verify myapp.service

The systemd-analyze verify command catches syntax errors without starting anything. Use it before reloading.

systemd is verbose and opinionated, but once you get past the initial learning curve, it handles all the edge cases that screen and nohup do not. Crashes, reboots, logging, resource limits, dependencies. Write the unit file once and forget about it.

docker exec#

The obvious one. Drop into a shell:

docker exec -it mycontainer sh

Use sh not bash because Alpine images do not have bash. Once inside, you can poke around:

# Check what processes are running
ps aux

# Look at environment variables
env | sort

# Test network connectivity
wget -qO- http://other-service:3000/health

# Check disk usage
df -h

But what if the container has no shell? Minimal images built FROM scratch or distroless images strip everything out.

docker logs#

Before exec-ing into anything, check the logs:

# Last 100 lines
docker logs --tail 100 mycontainer

# Follow in real time
docker logs -f mycontainer

# Logs since a specific time
docker logs --since 2026-03-29T10:00:00 mycontainer

# With timestamps
docker logs -t mycontainer

The --since flag is underrated. When a container has been running for weeks, you do not want to scroll through megabytes of old logs.

docker inspect#

This tells you everything about the container’s configuration:

# Full JSON dump
docker inspect mycontainer

# Just the IP address
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' mycontainer

# Environment variables
docker inspect -f '{{json .Config.Env}}' mycontainer | jq .

# Mount points
docker inspect -f '{{json .Mounts}}' mycontainer | jq .

# Port mappings
docker inspect -f '{{json .NetworkSettings.Ports}}' mycontainer | jq .

# Health check status
docker inspect -f '{{json .State.Health}}' mycontainer | jq .

docker cp#

Copy files in or out of a running container:

# Copy a config file out for inspection
docker cp mycontainer:/etc/nginx/nginx.conf ./nginx.conf

# Copy a fix in
docker cp ./fixed-config.json mycontainer:/app/config.json

This works even if the container has no shell or text editor.

nsenter: The Nuclear Option#

When docker exec is not enough or the container has no shell, nsenter lets you enter the container’s namespaces from the host:

# Get the container's PID
PID=$(docker inspect -f '{{.State.Pid}}' mycontainer)

# Enter its network namespace
sudo nsenter -t $PID -n ss -tlnp

# Enter all namespaces (like exec but from the host)
sudo nsenter -t $PID -m -u -i -n -p -- /bin/sh

This is useful for network debugging. You can run tcpdump, ss, or ip inside the container’s network namespace even if those tools are not installed in the container.

docker diff#

See what files changed since the container started:

docker diff mycontainer

Output looks like:

C /app
A /app/data/cache.db
C /tmp
A /tmp/something-unexpected

A means added, C means changed, D means deleted. This is great for finding containers that are writing to unexpected places.

Debugging Network Issues#

When containers cannot talk to each other:

# Check which networks a container is on
docker inspect mycontainer -f '{{json .NetworkSettings.Networks}}' | jq 'keys'

# Ping from one container to another
docker exec web ping -c 3 api

# Check DNS resolution
docker exec web nslookup api

# List all containers on a network
docker network inspect mynetwork -f '{{range .Containers}}{{.Name}} {{.IPv4Address}}{{end}}'

One-Liner Health Checks#

# Which containers are unhealthy?
docker ps --filter health=unhealthy --format '{{.Names}}: {{.Status}}'

# Which containers restarted recently?
docker ps --format '{{.Names}}: {{.Status}}' | grep -i restarting

# Resource usage snapshot
docker stats --no-stream --format 'table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}'

The instinct to restart a broken container is strong. But nine times out of ten, the information you need to fix the actual problem is sitting right there in the logs, the inspect output, or the filesystem diff. Restarting just resets the evidence.

This means changing your DNS server does nothing. The ISP sees the domain name in the TLS handshake before the connection is encrypted and drops the packet. You need a different approach. Mine has two parts: zapret, which splits up the TLS ClientHello so the DPI system cannot read the SNI field in time, and DNS-over-TLS, so DNS queries stop travelling in plain text.

How DPI Blocking Works#

When you connect to a website over HTTPS, the first thing your browser sends is a TLS ClientHello message. This message is not encrypted. It contains the Server Name Indication field, which tells the server which domain you are trying to reach. This is necessary because multiple domains can share the same IP address, and the server needs to know which certificate to present.

Your ISP inspects this field. If the domain matches their blocklist, they inject a TCP RST packet or simply drop the connection. From your perspective, the site just times out or you get a connection reset.

The Zapret Approach#

There is a tool called zapret that manipulates outgoing packets to confuse DPI systems. It does not encrypt anything or route traffic through a proxy. Instead, it exploits the fact that DPI systems have to make real-time decisions about packets and often take shortcuts.

The basic techniques include fragmenting the ClientHello across multiple TCP segments, adding padding to push the SNI field into a second packet, or modifying TCP flags in ways that confuse stateful inspection engines but are tolerated by the destination server.

I run zapret as a systemd service. It hooks into the outgoing packet flow using nfqueue and modifies specific packets before they leave my machine. The destination server receives the same data, just split up differently, and handles it fine. The ISP DPI system sees fragments that it cannot reassemble quickly enough and lets them through.

DNS-over-TLS Complement#

Zapret handles the TLS handshake problem, but DNS queries can still leak information. Even if the ISP cannot block the connection, they can see which domains you are resolving and log that data.

I force all DNS through DNS-over-TLS using systemd-resolved. The configuration tells resolved to only use TLS-encrypted DNS and to require valid certificates from the upstream resolver. I use Cloudflare and Google as upstream servers since both support DNS-over-TLS.

The resolved config looks straightforward. You set the DNS servers and tell it to require encrypted transport. After restarting the service, all DNS queries from the machine go through TLS on port 853 instead of plaintext on port 53.

The VPN Conflict#

This setup conflicts with split-tunnel VPN configurations. When the VPN is active, DNS queries for private services need to go to the VPN DNS server over the tunnel. But the DPI bypass forces all DNS through TLS, which means queries to the VPN DNS server also try to use TLS.

The VPN DNS server is just dnsmasq. It does not speak TLS. So the queries fail and private domain names do not resolve.

The fix is a PostUp hook in the WireGuard configuration that disables DNS-over-TLS specifically on the VPN interface. The system still uses DoT for all other DNS queries, but queries routed through the VPN go as plain DNS over the encrypted tunnel. Since the tunnel itself is encrypted, this is fine from a privacy perspective.

Getting this right took some trial and error. The symptoms were confusing: public websites worked, VPN connected successfully, but private subdomains just would not resolve. Once I traced it to the DoT setting, the fix was one line.

Performance Impact#

Zapret adds essentially zero latency. The packet manipulation happens in kernel space and takes microseconds. DNS-over-TLS adds a small amount of latency to the first query for each domain, maybe 10 to 20 milliseconds. Subsequent queries are cached by resolved so you do not notice it.

I have been running this setup for months and have never noticed any performance impact in normal usage. Streaming, gaming, and large downloads all work at full speed. The only time I notice it is when I forget to check if the service is running and a blocked site fails to load.

Reliability#

The DPI bypass works about 99 percent of the time. Occasionally the ISP updates their inspection rules and a specific technique stops working. Zapret has multiple strategies and you can configure fallbacks. Usually updating to the latest version fixes any regressions within a day or two.

The DNS-over-TLS setup is rock solid. I have never had it fail. Systemd-resolved handles reconnections and failovers between upstream servers automatically.

Legal and Ethical Notes#

This is not about accessing illegal content. My ISP blocks legitimate websites, often because a previous user of the domain or IP address did something that triggered a broad block. The blocking is opaque, there is no notification, and there is no process for getting a site unblocked.

Bypassing DPI to access legal content that your ISP has incorrectly or broadly blocked is a reasonable thing to do. The tools involved are all open source and widely used. Nobody has ever gotten in trouble for running zapret on their own machine.

If you are in a situation where your ISP does DPI blocking, this setup gives you back control over your own internet connection without the overhead and privacy implications of routing all your traffic through a third-party VPN.

jq is the tool that makes JSON usable from the command line. Once you learn the basics, you will never go back.

Basics#

Pretty-print some JSON:

curl -s https://api.github.com/users/torvalds | jq .

Extract a single field:

echo '{"name": "Linus", "repos": 5}' | jq .name
# "Linus"

Drop the quotes with -r:

echo '{"name": "Linus"}' | jq -r .name
# Linus

Arrays#

Get the first element:

echo '[1, 2, 3]' | jq '.[0]'
# 1

Iterate over all elements:

echo '[{"name": "a"}, {"name": "b"}]' | jq '.[].name'
# "a"
# "b"

Filtering#

This is where it gets powerful. Say you have a list of GitHub repos:

curl -s https://api.github.com/users/torvalds/repos | \
  jq '[.[] | select(.stargazers_count > 1000)] | length'

That filters repos with more than 1000 stars and counts them.

Constructing Objects#

You can reshape JSON on the fly:

curl -s https://api.github.com/users/torvalds/repos | \
  jq '.[] | {name: .name, stars: .stargazers_count, lang: .language}'

This outputs a new object for each repo with only the fields you care about.

Real World: Parsing Docker#

List container names and their memory usage:

docker stats --no-stream --format '{{json .}}' | \
  jq -r '[.Name, .MemUsage] | @tsv'

Find containers using more than 100MB:

docker stats --no-stream --format '{{json .}}' | \
  jq -r 'select(.MemPerc | gsub("%"; "") | tonumber > 1) | .Name'

Working With Nested Data#

Given deeply nested JSON:

{
  "data": {
    "users": [
      {"id": 1, "profile": {"email": "a@b.com"}},
      {"id": 2, "profile": {"email": "c@d.com"}}
    ]
  }
}

Extract all emails:

jq '[.data.users[].profile.email]' data.json
# ["a@b.com", "c@d.com"]

Combining With Other Tools#

jq plays well with the rest of the Unix toolkit:

# Get all unique languages across repos
curl -s https://api.github.com/users/torvalds/repos | \
  jq -r '.[].language // empty' | sort -u

# Convert JSON array to lines for xargs
echo '["file1.txt", "file2.txt"]' | jq -r '.[]' | xargs rm

# Merge multiple JSON files
jq -s 'add' file1.json file2.json > merged.json

The Cheat Sheet#

jq has a learning curve, but it is one of those tools where the investment pays off within a week. Every time you find yourself writing a Python script just to parse some JSON, remember that jq probably does it in one line.

The Basics#

The SSH config lives at ~/.ssh/config. Here is a minimal example:

Host myserver
  HostName 203.0.113.50
  User root
  IdentityFile ~/.ssh/id_ed25519

Now ssh myserver is all you need. But it gets better.

Jump Hosts#

If you need to reach a server through a bastion:

Host bastion
  HostName bastion.example.com
  User admin

Host internal
  HostName 10.0.1.50
  User deploy
  ProxyJump bastion

ssh internal automatically tunnels through the bastion. No manual port forwarding, no two-step login.

Wildcard Hosts#

This is the one most people miss:

Host *.prod
  User deploy
  IdentityFile ~/.ssh/prod_key
  StrictHostKeyChecking accept-new

Host web.prod
  HostName 10.0.1.10

Host api.prod
  HostName 10.0.1.11

Host db.prod
  HostName 10.0.1.12

The wildcard block sets defaults for anything ending in .prod. Individual host blocks just add the hostname. You get consistent configuration without repetition.

Keep Connections Alive#

Add this to your Host * block:

Host *
  ServerAliveInterval 60
  ServerAliveCountMax 3
  AddKeysToAgent yes
  IdentitiesOnly yes

ServerAliveInterval sends a keepalive every 60 seconds. This stops firewalls and NATs from killing idle connections. AddKeysToAgent automatically loads keys into ssh-agent after first use.

Connection Multiplexing#

This is the real speed trick:

Host *
  ControlMaster auto
  ControlPath ~/.ssh/sockets/%r@%h-%p
  ControlPersist 600

The first SSH connection to a host creates a socket. Subsequent connections to the same host reuse it. No new TCP handshake, no new key exchange. The second ssh myserver connects instantly.

ControlPersist 600 keeps the socket alive for 10 minutes after you disconnect. So if you SSH in, run a command, disconnect, and SSH back in within 10 minutes, the second connection is instant.

Create the socket directory first:

mkdir -p ~/.ssh/sockets

Port Forwarding Shortcuts#

Host db-tunnel
  HostName myserver.com
  User admin
  LocalForward 5432 localhost:5432
  RequestTTY no
  ExitOnForwardFailure yes

Now ssh db-tunnel opens a tunnel to the remote Postgres. No remembering the -L syntax. ExitOnForwardFailure means if the port is already in use, SSH exits instead of connecting without the tunnel.

SSHFP for Host Verification#

Tired of the “Are you sure you want to continue connecting?” messages? If you control the DNS, add SSHFP records:

ssh-keygen -r myserver.com

This outputs DNS records you can add to your zone. Then set:

Host *
  VerifyHostKeyDNS yes

SSH verifies the host key against DNS automatically. No more blindly accepting fingerprints.

The SSH config file is probably the highest-leverage dotfile you can invest time in. Five minutes of setup saves thousands of keystrokes.

That is when I started using traps. trap cleanup EXIT runs a cleanup function however the script ends: a normal exit, an error or a Ctrl-C.

The Problem#

Bash scripts are fragile by default. If a command fails, the script keeps going. If you hit Ctrl-C, whatever state you left things in is where they stay. If your SSH connection drops mid-script, same thing.

Most people know about set -e which exits on the first error. But that only covers one failure mode. What about cleanup?

Enter trap#

The trap builtin lets you run a command when the script receives a signal or exits:

#!/bin/bash
set -euo pipefail

TMPDIR=$(mktemp -d)

cleanup() {
  echo "Cleaning up $TMPDIR..."
  rm -rf "$TMPDIR"
}

trap cleanup EXIT

# Your actual work here
cp important-files "$TMPDIR/"
process "$TMPDIR/important-files"
mv "$TMPDIR/important-files" /final/destination/

The EXIT trap fires no matter how the script ends. Normal exit, error, Ctrl-C, even kill. The temp directory always gets cleaned up.

A Real Deploy Script#

Here is the pattern I use for deployments:

#!/bin/bash
set -euo pipefail

SERVER="myserver"
REMOTE_DIR="/opt/app"
BACKUP_DIR="/opt/app-backup-$(date +%s)"
ROLLED_BACK=false

rollback() {
  if [ "$ROLLED_BACK" = true ]; then return; fi
  ROLLED_BACK=true
  echo "Deploy failed. Rolling back..."
  ssh "$SERVER" "rm -rf $REMOTE_DIR && mv $BACKUP_DIR $REMOTE_DIR"
  echo "Rollback complete."
}

trap rollback ERR

# Backup current version
ssh "$SERVER" "cp -r $REMOTE_DIR $BACKUP_DIR"

# Deploy
rsync -a --delete ./dist/ "$SERVER:$REMOTE_DIR/"
ssh "$SERVER" "cd $REMOTE_DIR && docker compose up -d"

# Success - remove backup
ssh "$SERVER" "rm -rf $BACKUP_DIR"
echo "Deploy complete."

If anything after the backup fails, the old version gets restored automatically. No manual intervention, no half-broken state.

set -euo pipefail#

This trio should be at the top of every script you write:

• set -e exits on any command failure
• set -u treats unset variables as errors instead of empty strings
• set -o pipefail makes pipes fail if any command in the chain fails, not just the last one

Without pipefail, this silently succeeds even when curl fails:

curl https://example.com/data | jq .result
echo "Done"

jq gets an empty input, outputs null, and the script continues. With pipefail, the whole pipe fails and set -e kills the script.

Trapping Multiple Signals#

You can trap specific signals separately:

trap "echo Interrupted; cleanup; exit 130" INT
trap "echo Terminated; cleanup; exit 143" TERM
trap cleanup EXIT

But usually just trapping EXIT is enough. It covers everything.

The key insight is that bash does not protect you from yourself. You have to opt into safety. But once you do, scripts become surprisingly reliable.