So I built a runtime state editor instead. It attaches to a running Suzerain.exe, finds the live in-game Lua variables, and lets me read or rewrite any of them without restarting the game. The result is WolknerProtocol, and the path to it ran through three dead ends and one good idea. Worth writing up, because the good idea is more general than a single game.

What I was working with#

Suzerain is a Unity 6 game shipped with IL2CPP. The build I had was version 3.1.0.1.153, running through Proton on Linux. IL2CPP compiles C# to C++ at build time, so there is no Mono runtime metadata to query and no easy way to enumerate types from outside the process. The only obvious entry points were GameAssembly.dll (the compiled native code) and il2cpp_data/Metadata/global-metadata.dat (the game’s own type information).

The game state lives in C# heap memory once the game is running. The save file confirmed it: a single JSON document with a giant variables field that turned out to be a Lua-table-style serialization of around 800,000 characters. Things like ["BaseGame.GovernmentBudget"]=9, ["BaseGame.PersonalWealth"]=4, ... going on forever.

Dead end one: classic memory scanning#

The first instinct is always the same. Find the value 9 in the running process, change it in-game, scan again, narrow. Standard scanmem workflow, no metadata required.

This works for finding a value of 9 in memory. It does not work for finding the value of 9 that the game uses, because Suzerain has hundreds of int-valued things. Even after several rounds of narrowing you end up with thousands of candidates, and even when you find one and write to it, the game ignores you. The address you found is a buffer, a render cache, an undo snapshot, anything but the live state.

I confirmed this by overwriting the literal save-file string ["BaseGame.GovernmentBudget"]=9 to =5 in memory. The change held, but the game UI kept showing 9, and at the next autosave my edit was gone. Whatever I was editing, it was not what the game read.

Dead end two: save editing#

The save file is JSON. In principle you can edit it in a text editor, set budget to whatever you want, and reload the save. The community has guides for this.

Two reasons it is not enough. First, you need to restart the game on every change. Second, the variables BaseGame.Sordland_HUDStat_GovernmentBudget_Max = 30 and similar set hard caps. Editing budget to 999 triggers a debuff cascade because the value is out of bounds. You can edit the cap variables too, but now you are doing two restarts to make one change.

I wanted to edit values mid-play, freely, with no consequences other than the game state itself.

The dump that changed everything#

I gave up on blind scanning and dumped the IL2CPP metadata. The right tool for Unity 6 (metadata version 31) is Cpp2IL. Run it against the game directory, ask for diffable-cs output, and you get a directory full of decompiled C# pseudocode showing every class, every field, every offset.

Two things jumped out:

First, there is a Language.Lua namespace with classes like LuaTable, LuaString, LuaNumber, LuaBoolean, LuaInterpreter. Suzerain has an embedded Lua VM written in C#.

Second, there is PixelCrushers.DialogueSystem.DialogueLua with methods like GetVariable(string) and SetVariable(string, object). So the game uses PixelCrushers Dialogue System, a popular Unity asset for narrative games, which embeds its own Lua interpreter for variable storage.

The data model became clear. The game has a global Lua table called Variable. Every game variable is stored in it as a key-value pair where the key is a LuaString (or sometimes a plain System.String) and the value is a LuaNumber, LuaBoolean, or LuaString. The save file is just this table serialized.

The dump also gave me the field layouts. LuaNumber has a double Number field at offset 16 in the object. LuaTable has its main hash dictionary at offset 24. Every C# Dictionary<TKey, TValue> has a contiguous entries array with each entry shaped like [hashCode(4), next(4), key(8), value(8)] for reference types.

Finding the LuaNumber class at runtime#

The metadata told me what to look for. The next problem was actually finding it without hardcoding addresses. Class pointers in IL2CPP are at fixed offsets within GameAssembly.dll once it is loaded, but the loaded base address varies between launches due to ASLR. I needed runtime detection.

My approach was a heuristic that worked the first time and has been reliable since. I sample 8-byte aligned positions in writable memory. For each position whose 8 bytes interpret as a finite, non-zero, integer-valued double in a sane range, I look at the 16 preceding bytes (the type pointer for IL2CPP objects always sits at offset 0). I cluster all the type pointers I see and pick the most common one. That is the LuaNumber class.

for (size_t off = 16; off + 8 <= toRead; off += 8) {
    double d;
    std::memcpy(&d, buf.data() + off, 8);
    if (!std::isfinite(d) || d == 0.0) continue;
    if (std::fabs(d) > 1e6) continue;
    if (d != std::trunc(d)) continue;
    uint64_t typePtr;
    std::memcpy(&typePtr, buf.data() + off - 16, 8);
    if (typePtr < 0x100000 || typePtr > 0x0000800000000000ULL) continue;
    counts[typePtr]++;
}

Suzerain has hundreds of LuaNumber instances representing live game variables, each one preceded by the same type pointer. Other types of objects with double values are much rarer. The cluster wins clearly every time.

The dict entries#

Once you have the LuaNumber class pointer, finding dict entries is straightforward in two passes.

Pass one: scan writable memory for any 8-byte position whose value equals the LuaNumber class pointer. Each hit is a LuaNumber object’s start address. Collect all of these into a hash set.

Pass two: scan writable memory again for 8-byte positions whose value is in that set. Each hit is a place that points to a LuaNumber. The 8 bytes immediately before the hit are usually the key pointer for that dict entry. Dereference the key pointer as either a System.String or a LuaString (which itself wraps a System.String), and you have a name for the value.

I filter for keys that start with known prefixes like BaseGame., RiziaDLC., or GameCondition.. After the second pass I have a complete catalog of every numeric in-game variable, keyed by its real name. About 428 of them in a typical save.

Total runtime, parallelized across CPUs, around 20 seconds for a 4 GB process. The bottleneck is the kernel serializing reads from /proc/<pid>/mem, not the scanning logic.

The actual hard part#

This is where I almost gave up.

After locating a dict entry pointing to the LuaNumber for BaseGame.GovernmentBudget, I wrote 11.0 to that LuaNumber’s Number field. The game UI showed 11. I high-fived myself. Then I played for a minute, the value changed naturally to 10, and the next time I read from that same address I got 2.42092e-322.

A denormalized double is the smell of pointer-sized garbage being interpreted as a float. The address still existed, the read succeeded, but whatever was at that address was no longer the budget LuaNumber.

Suzerain’s Lua treats values as immutable. When Variable["BaseGame.GovernmentBudget"] changes from 9 to 10, the runtime allocates a new LuaNumber(10) and updates the dict entry’s value pointer to point at the new instance. The old LuaNumber(9) becomes garbage, eventually collected, eventually overwritten by something else.

So caching LuaNumber addresses is hopeless. They go stale on every game tick.

What is stable is the dict entry itself. The dict’s entries array is allocated once and lives at a fixed address for the process lifetime. IL2CPP uses Boehm GC, which is non-moving for non-managed types, so allocations do not relocate. The 8 bytes inside the entry that hold the LuaNumber pointer (the value slot) sit at a stable address. The pointer they hold changes; the slot does not.

The fix is one extra dereference. Instead of caching valueLuaNumberAddr, cache valueSlotAddr. Reads become a two-hop operation:

double readValue(uint64_t valueSlotAddr) {
    uint64_t luaPtr;
    mem.readInto(valueSlotAddr, &luaPtr, 8);
    double v;
    mem.readInto(luaPtr + 16, &v, 8);
    return v;
}

Writes work the same way: dereference the slot to get the current LuaNumber, then write to its Number field. If the game later allocates a fresh LuaNumber for that variable, our next read picks up the new address automatically through the slot.

This single insight made the rest of the project work. Live polling at 1 Hz, manual edits that stick across game events, autosaves that do not break tracking, all of it falls out of the value-slot indirection.

The build#

Once the data model was working I wrapped it in a Qt6 GUI. C++20, frameless window, dark slate theme, sliding mode toggle for the Sordland and Rizia campaigns, sortable table of all 428 variables, search box that supports name substrings and value comparisons (>10, =5, etc.), 1 Hz live polling with the editing cell exempted so typed input is not clobbered.

The platform abstraction is small. One ProcessMem class with an #ifdef _WIN32 split. On Linux it uses pread and pwrite against /proc/<pid>/mem after parsing /proc/<pid>/maps to find writable regions. On Windows it uses OpenProcess, VirtualQueryEx to enumerate regions, and ReadProcessMemory and WriteProcessMemory for I/O. About fifty lines of platform-specific code total.

For distribution I cross compile from Linux to Windows. mingw-w64 for the toolchain, aqtinstall to fetch Qt6 binaries for both targets, a small CMake toolchain file, and windres to embed the icon and version info into the PE. The Windows build is a self-contained 28 MB folder with the executable and the necessary Qt DLLs. Defender flags it as a HackTool on first run, which is standard for memory-editing binaries; whitelist the folder and move on.

The repo has the toolchain file and full instructions if you want to learn how to ship a Qt6 desktop app for Windows from a Linux machine. That path was less documented than I expected.

What I learned#

Three things, in order of usefulness for future projects.

Do not fight the game’s data model. Classic memory scanning works against simple games where values are stored as primitives in stable C structs. Modern Unity games are not those games. Once you accept that there is a managed runtime with its own object lifecycle, the right move is to reverse engineer the runtime, not to scan around it.

Cache slots, not values. Anywhere a runtime might replace objects (Lua tables, immutable strings, GC-managed values), pointers to those objects are inherently fragile. Pointers to the slots that hold those pointers are stable. One extra dereference per access is a tiny cost for full correctness across every game state mutation.

IL2CPP is more legible than its reputation. Cpp2IL produces readable pseudocode. The runtime layout is consistent and predictable. Boehm GC is non-moving. Compared to JIT-based Mono, where addresses can move on you, IL2CPP is actually friendly to outside-process introspection once you accept that you are working at the heap level.

If you have ever wanted to mess with the runtime state of a single-player Unity game without touching save files, the playbook above generalizes. Find the scripting layer (Lua, custom DSL, or just C# itself), dump the metadata, locate live values through one stable indirection, and stop fighting the GC.

The repo is at github.com/ts-solidarity/WolknerProtocol. Linux and Windows binaries on the releases page, MIT licensed, single-player only, please be kind to the small studio that made the game.

The Record Types That Matter#

A and AAAA#

The most basic. Maps a domain to an IP address:

example.dev.    A      203.0.113.50
example.dev.    AAAA   2001:db8::1

A is IPv4. AAAA is IPv6. You want both if your server supports v6.

CNAME#

An alias. Points one name to another:

www.example.dev.    CNAME    example.dev.

This says “www is the same as the bare domain.” The DNS resolver follows the chain and ends up at the A record for example.dev.

The critical rule: you cannot have a CNAME alongside other records for the same name. This means the zone apex (bare domain) usually cannot be a CNAME. Use an A record there instead.

MX#

Where to deliver email:

example.dev.    MX    10 mail.example.dev.

The number is priority. Lower means preferred. If you have a backup mail server:

example.dev.    MX    10 mail.example.dev.
example.dev.    MX    20 backup-mail.example.dev.

Senders try priority 10 first, fall back to 20 if it is down.

TXT#

Arbitrary text. Used for verification and email authentication:

example.dev.    TXT    "v=spf1 a mx ip4:203.0.113.50 ~all"

SPF tells receiving mail servers which IPs are allowed to send email for your domain. DKIM and DMARC are also TXT records. Google and other services use TXT for domain ownership verification too.

NS#

Nameserver delegation. Usually set by your registrar:

example.dev.    NS    ada.ns.cloudflare.com.
example.dev.    NS    greg.ns.cloudflare.com.

This tells the internet “Cloudflare knows the DNS records for example.dev.” You rarely touch these directly.

Debugging With dig#

dig is the standard DNS debugging tool:

# Basic lookup
dig example.dev

# Specific record type
dig MX example.dev

# Short output
dig +short example.dev
# 203.0.113.50

# Use a specific DNS server
dig @8.8.8.8 example.dev

# Trace the full resolution path
dig +trace example.dev

The +trace option follows the entire DNS chain from the root servers down. This is invaluable when debugging propagation issues.

TTL#

Time To Live. How long DNS resolvers cache a record:

example.dev.    300    IN    A    203.0.113.50

300 means 5 minutes. After that, resolvers ask again. Lower TTL means faster propagation when you change records, but more DNS queries hitting your nameservers.

Before a migration:

1. Lower TTL to 60 seconds, 24 hours ahead
2. Wait for the old TTL to expire
3. Change the record
4. Verify
5. Raise TTL back to 3600 or higher

This way, when you change the IP, everyone picks up the new address within a minute.

Common Patterns#

Subdomain to a different server#

api.example.com.    A    203.0.113.50

Wildcard#

*.example.com.    A    93.184.216.34

Matches any subdomain that does not have a more specific record. Useful for catch-all routing.

Email setup#

Minimum viable email DNS:

example.com.         MX   10 mail.example.com.
mail.example.com.    A    93.184.216.34
example.com.         TXT  "v=spf1 a mx ~all"
_dmarc.example.com.  TXT  "v=DMARC1; p=quarantine"
sel._domainkey.example.com.  TXT  "v=DKIM1; k=rsa; p=MIIBIj..."

Cloudflare proxy vs DNS-only#

If you use Cloudflare:

# Proxied (orange cloud) - traffic goes through Cloudflare CDN
example.dev.    A    203.0.113.50    (proxied)

# DNS-only (gray cloud) - direct connection to your server
mail.example.dev.    A    203.0.113.50    (DNS-only)

Proxied records hide your real IP and add DDoS protection. But they break non-HTTP protocols. Mail servers, game servers, and anything that is not HTTP/HTTPS needs DNS-only.

Why Is My DNS Not Working?#

In order of likelihood:

1. Propagation delay. You changed the record 2 minutes ago. Wait for the TTL to expire. Check with dig @8.8.8.8 to see what Google’s DNS returns.

2. Browser cache. Chrome caches DNS aggressively. Try chrome://net-internals/#dns and clear it, or just use a different browser.

3. Wrong record type. You added a CNAME where you needed an A record, or vice versa.

4. Cloudflare proxy interfering. Your service needs direct access but the record is proxied. Switch to DNS-only.

5. Missing trailing dot. In some DNS configurations, mail.example.com and mail.example.com. are different. The dot makes it fully qualified. Most providers handle this for you, but raw zone files need it.

# Quick sanity check script
for record in A AAAA MX TXT CNAME; do
  echo "=== $record ==="
  dig +short $record example.com
done

DNS is not complicated once you understand the record types. The frustrating part is always the caching and propagation delays. Lower your TTLs before making changes, use dig to verify, and be patient.

Let us build a WebSocket server from scratch using only the Node.js standard library.

The Handshake#

WebSocket starts as an HTTP upgrade request. The client sends:

GET / HTTP/1.1
Host: localhost:8080
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Sec-WebSocket-Version: 13

The server responds:

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=

The Sec-WebSocket-Accept value is computed from the client’s key:

const crypto = require('crypto');

function computeAcceptKey(clientKey) {
  const MAGIC = '258EAFA5-E914-47DA-95CA-C5AB0DC85B11';
  return crypto
    .createHash('sha1')
    .update(clientKey + MAGIC)
    .digest('base64');
}

The magic string is literally part of the RFC. Every WebSocket implementation uses it.

The Server#

const http = require('http');
const crypto = require('crypto');

const MAGIC = '258EAFA5-E914-47DA-95CA-C5AB0DC85B11';

const server = http.createServer();
const clients = new Set();

server.on('upgrade', (req, socket) => {
  const clientKey = req.headers['sec-websocket-key'];
  if (!clientKey) {
    socket.destroy();
    return;
  }

  const acceptKey = crypto
    .createHash('sha1')
    .update(clientKey + MAGIC)
    .digest('base64');

  socket.write(
    'HTTP/1.1 101 Switching Protocols\r\n' +
    'Upgrade: websocket\r\n' +
    'Connection: Upgrade\r\n' +
    `Sec-WebSocket-Accept: ${acceptKey}\r\n` +
    '\r\n'
  );

  clients.add(socket);
  console.log(`Client connected (${clients.size} total)`);

  socket.on('data', (buffer) => {
    const message = decodeFrame(buffer);
    if (message === null) return;

    console.log('Received:', message);

    // Echo to all clients
    for (const client of clients) {
      client.write(encodeFrame(message));
    }
  });

  socket.on('close', () => {
    clients.delete(socket);
    console.log(`Client disconnected (${clients.size} total)`);
  });

  socket.on('error', () => {
    clients.delete(socket);
  });
});

server.listen(8080, () => {
  console.log('WebSocket server on ws://localhost:8080');
});

Frame Decoding#

WebSocket messages are wrapped in frames. The frame format:

 0               1               2               3
 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7
+-+-+-+-+-------+-+-------------+-------------------------------+
|F|R|R|R| opcode|M| Payload len |    Extended payload length    |
|I|S|S|S|  (4)  |A|     (7)     |            (16/64)            |
|N|V|V|V|       |S|             |                               |
| |1|2|3|       |K|             |                               |
+-+-+-+-+-------+-+-------------+-------------------------------+
|     Masking key (if MASK set)                                 |
+-------------------------------+-------------------------------+
|     Payload Data                                              |
+---------------------------------------------------------------+

Client-to-server frames are always masked. Server-to-client frames are never masked. Here is the decoder:

function decodeFrame(buffer) {
  const firstByte = buffer[0];
  const opcode = firstByte & 0x0f;

  // 0x1 = text, 0x8 = close, 0x9 = ping, 0xA = pong
  if (opcode === 0x8) return null; // close frame

  const secondByte = buffer[1];
  const isMasked = (secondByte & 0x80) !== 0;
  let payloadLength = secondByte & 0x7f;
  let offset = 2;

  if (payloadLength === 126) {
    payloadLength = buffer.readUInt16BE(2);
    offset = 4;
  } else if (payloadLength === 127) {
    payloadLength = Number(buffer.readBigUInt64BE(2));
    offset = 10;
  }

  let maskingKey;
  if (isMasked) {
    maskingKey = buffer.slice(offset, offset + 4);
    offset += 4;
  }

  const payload = buffer.slice(offset, offset + payloadLength);

  if (isMasked) {
    for (let i = 0; i < payload.length; i++) {
      payload[i] ^= maskingKey[i % 4];
    }
  }

  return payload.toString('utf-8');
}

Frame Encoding#

Sending from server to client is simpler because we do not mask:

function encodeFrame(message) {
  const payload = Buffer.from(message, 'utf-8');
  const length = payload.length;

  let header;
  if (length < 126) {
    header = Buffer.alloc(2);
    header[0] = 0x81; // FIN + text opcode
    header[1] = length;
  } else if (length < 65536) {
    header = Buffer.alloc(4);
    header[0] = 0x81;
    header[1] = 126;
    header.writeUInt16BE(length, 2);
  } else {
    header = Buffer.alloc(10);
    header[0] = 0x81;
    header[1] = 127;
    header.writeBigUInt64BE(BigInt(length), 2);
  }

  return Buffer.concat([header, payload]);
}

Testing It#

Open a browser console:

const ws = new WebSocket('ws://localhost:8080');
ws.onmessage = (e) => console.log('Got:', e.data);
ws.onopen = () => ws.send('hello from browser');

Or use a Node.js client:

const net = require('net');
const crypto = require('crypto');

const key = crypto.randomBytes(16).toString('base64');
const socket = net.createConnection(8080, 'localhost', () => {
  socket.write(
    'GET / HTTP/1.1\r\n' +
    'Host: localhost:8080\r\n' +
    'Upgrade: websocket\r\n' +
    'Connection: Upgrade\r\n' +
    `Sec-WebSocket-Key: ${key}\r\n` +
    'Sec-WebSocket-Version: 13\r\n' +
    '\r\n'
  );
});

What This Does Not Handle#

This implementation is deliberately minimal. A production WebSocket server also needs:

• Ping/pong frames for connection keepalive
• Fragmented messages where large payloads span multiple frames
• Close handshake with proper status codes
• Per-message compression (the permessage-deflate extension)
• Backpressure when clients read slower than you write

That is why libraries like ws exist. But knowing the protocol means you understand what those libraries are doing, and you can debug problems at the frame level instead of guessing.

The entire WebSocket protocol fits in RFC 6455, which is surprisingly readable as RFCs go. The core framing logic is maybe 100 lines of code. Everything else is just handling edge cases.

Pattern 1: Print a Column#

awk splits each line into fields. $1 is the first, $2 is the second, $0 is the whole line:

# Print the first column of ps output
ps aux | awk '{print $1}'

# Print columns 1 and 11 (user and command)
ps aux | awk '{print $1, $11}'

The default field separator is whitespace. Change it with -F:

# Parse /etc/passwd (colon-separated)
awk -F: '{print $1, $7}' /etc/passwd

# Parse CSV
awk -F, '{print $2}' data.csv

Pattern 2: Filter Rows#

Put a condition before the action:

# Lines where column 3 is greater than 100
awk '$3 > 100' data.txt

# Lines containing "ERROR"
awk '/ERROR/' app.log

# Lines where the first field matches a pattern
awk '$1 ~ /^192\.168/' access.log

Combine conditions:

# Memory usage over 50% AND the process is not root
ps aux | awk '$4 > 50.0 && $1 != "root" {print $1, $4, $11}'

Pattern 3: Sum a Column#

# Total bytes transferred (column 10 in access logs)
awk '{sum += $10} END {print sum}' access.log

# Average response time
awk '{sum += $NF; n++} END {print sum/n}' response_times.log

END runs after all lines are processed. NF is the number of fields on the current line, so $NF is the last field.

Pattern 4: Count and Group#

# Count HTTP status codes
awk '{count[$9]++} END {for (c in count) print c, count[c]}' access.log | sort -rn -k2

# Output:
# 200 45231
# 304 12044
# 404 891
# 500 23

This uses an associative array. count[$9]++ increments the count for each unique value in column 9.

Top 10 IP addresses by request count:

awk '{count[$1]++} END {for (ip in count) print count[ip], ip}' access.log | sort -rn | head -10

Pattern 5: Transform Fields#

# Convert bytes to megabytes
awk '{printf "%s %.1fMB\n", $1, $2/1048576}' sizes.txt

# Add a prefix to every line
awk '{print "[INFO]", $0}' app.log

# Swap two columns
awk '{temp=$1; $1=$2; $2=temp; print}' data.txt

Real-World Examples#

Parse Docker ps output#

docker ps --format '{{.Names}}\t{{.Status}}\t{{.Ports}}' | \
  awk -F'\t' '{printf "%-20s %-15s %s\n", $1, $2, $3}'

Find the largest Docker images#

docker images --format '{{.Repository}}:{{.Tag}}\t{{.Size}}' | \
  awk -F'\t' '{
    size = $2
    if (size ~ /GB/) {
      gsub(/GB/, "", size)
      mb = size * 1024
    } else if (size ~ /MB/) {
      gsub(/MB/, "", size)
      mb = size
    } else {
      mb = 0
    }
    printf "%8.0fMB  %s\n", mb, $1
  }' | sort -rn | head -10

Calculate uptime percentage#

Given a log with timestamps and status:

awk '
  /UP/   { up++ }
  /DOWN/ { down++ }
  END {
    total = up + down
    if (total > 0)
      printf "Uptime: %.2f%% (%d/%d checks)\n", (up/total)*100, up, total
  }
' monitoring.log

Extract unique error messages#

awk '/ERROR/ {
  msg = ""
  for (i=4; i<=NF; i++) msg = msg " " $i
  errors[msg]++
}
END {
  for (e in errors) printf "%5d %s\n", errors[e], e
}' app.log | sort -rn | head -20

Built-in Variables#

When to Use awk vs. Other Tools#

• cut: When you just need a column and nothing else. cut -d: -f1 /etc/passwd is simpler than awk for trivial extraction.
• grep: When you just need to filter lines by a pattern.
• sed: When you need to do find-and-replace on text.
• awk: When you need to do math, grouping, conditional logic, or work with multiple fields at once.

The mental model is simple. awk reads lines, splits them into fields, and lets you write rules about what to do with each line. Once that clicks, you start seeing awk solutions everywhere.

Not because I write C. Because Make is a task runner that is already installed on every machine.

The Basics#

.PHONY: build test deploy clean

build:
	docker compose build

test:
	docker compose run --rm app npm test

deploy: build
	rsync -a --delete ./dist/ server:/opt/app/
	ssh server "cd /opt/app && docker compose up -d"

clean:
	docker compose down -v
	rm -rf dist node_modules

Now:

make build
make test
make deploy
make clean

.PHONY tells Make these targets are commands, not files. Without it, if you have a directory called build, Make would say “build is up to date” and do nothing.

Dependencies#

The power of Make is in the dependency graph. deploy depends on build:

deploy: build
	rsync -a --delete ./dist/ server:/opt/app/

When you run make deploy, it runs build first. If build fails, deploy never runs. This is what npm scripts cannot do cleanly.

Variables#

SERVER := myserver
REMOTE_DIR := /opt/app
IMAGE := myapp:latest

build:
	docker build -t $(IMAGE) .

push: build
	docker push $(IMAGE)

deploy: push
	ssh $(SERVER) "cd $(REMOTE_DIR) && docker pull $(IMAGE) && docker compose up -d"

Override from the command line:

make deploy SERVER=staging-server

Multi-Stage Builds With File Targets#

Make tracks file timestamps. This is useful for caching:

node_modules: package.json package-lock.json
	npm ci
	touch node_modules

dist: node_modules $(shell find src -type f)
	npm run build

.PHONY: serve
serve: dist
	python3 -m http.server 8080 -d dist

node_modules only rebuilds if package.json or package-lock.json changed. dist only rebuilds if source files changed. Make compares file timestamps automatically.

Help Target#

Add a self-documenting help target:

.DEFAULT_GOAL := help

help: ## Show this help
	@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | \
		awk 'BEGIN {FS = ":.*?## "}; {printf "  \033[36m%-15s\033[0m %s\n", $$1, $$2}'

build: ## Build the Docker image
	docker compose build

test: ## Run the test suite
	docker compose run --rm app npm test

deploy: ## Deploy to production
	./deploy.sh

clean: ## Remove all build artifacts
	rm -rf dist node_modules

Now make with no arguments prints:

  build           Build the Docker image
  test            Run the test suite
  deploy          Deploy to production
  clean           Remove all build artifacts

Environment-Specific Configs#

ENV ?= development

ifeq ($(ENV),production)
  COMPOSE_FILE := docker-compose.yml -f docker-compose.prod.yml
else
  COMPOSE_FILE := docker-compose.yml
endif

up:
	docker compose $(addprefix -f ,$(COMPOSE_FILE)) up -d

# Usage:
# make up           → development
# make up ENV=production  → production

Parallel Execution#

Make can run independent targets in parallel:

make -j4 lint test typecheck

As long as these targets do not depend on each other, Make runs all four simultaneously. This is free parallelism with no extra scripting.

A Real Project Makefile#

.PHONY: help dev build test lint deploy clean
.DEFAULT_GOAL := help

IMAGE := myapp
TAG := $(shell git rev-parse --short HEAD)

help:
	@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | \
		awk 'BEGIN {FS = ":.*?## "}; {printf "  \033[36m%-15s\033[0m %s\n", $$1, $$2}'

dev: ## Start development server
	docker compose up -d
	docker compose logs -f app

build: ## Build production image
	docker build -t $(IMAGE):$(TAG) -t $(IMAGE):latest .

test: ## Run tests
	docker compose run --rm app npm test

lint: ## Run linter
	docker compose run --rm app npm run lint

deploy: build ## Deploy to production
	docker push $(IMAGE):$(TAG)
	ssh server "docker pull $(IMAGE):$(TAG) && docker compose up -d"
	@echo "Deployed $(TAG)"

clean: ## Clean up everything
	docker compose down -v
	docker image prune -f

The Makefile is not trendy. It does not have a package ecosystem or a plugin system. But it works everywhere, it handles dependencies, it supports parallelism, and it does not require installing anything. For project automation, that is hard to beat.

Quick HTTP Server#

The classic. Serve the current directory:

python3 -m http.server 8080

Bind to a specific interface:

python3 -m http.server 8080 --bind 127.0.0.1

This is genuinely useful for transferring files between machines, testing static sites, or sharing a directory with a coworker.

JSON Pretty Printing#

When you do not have jq installed:

curl -s https://api.example.com/data | python3 -m json.tool

Or with color:

echo '{"a":1,"b":[2,3]}' | python3 -c "
import json, sys
data = json.load(sys.stdin)
print(json.dumps(data, indent=2, sort_keys=True))
"

Parse Timestamps#

Convert epoch timestamps that show up in logs:

python3 -c "
from datetime import datetime
import sys
for line in sys.stdin:
    ts = int(line.strip())
    print(datetime.fromtimestamp(ts).isoformat())
" <<< "1711708800"
# 2024-03-29T12:00:00

Find Large Files#

find works, but if you want more control over the output:

python3 -c "
import os
files = []
for root, dirs, fnames in os.walk('/var/log'):
    for f in fnames:
        path = os.path.join(root, f)
        try:
            size = os.path.getsize(path)
            if size > 100_000_000:  # 100MB
                files.append((size, path))
        except OSError:
            pass
for size, path in sorted(files, reverse=True):
    print(f'{size / 1e9:.1f}G  {path}')
"

Base64 Decode#

Decode those opaque values in Kubernetes secrets or JWTs:

python3 -c "
import base64, json, sys
token = sys.argv[1]
parts = token.split('.')
for part in parts[:2]:
    padded = part + '=' * (4 - len(part) % 4)
    decoded = base64.urlsafe_b64decode(padded)
    print(json.dumps(json.loads(decoded), indent=2))
" "eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxMjM0In0.signature"

Port Scanner#

Quick check which ports are open without installing nmap:

python3 -c "
import socket, sys
host = sys.argv[1]
for port in range(1, 1025):
    s = socket.socket()
    s.settimeout(0.1)
    if s.connect_ex((host, port)) == 0:
        print(f'{port}/tcp open')
    s.close()
" 10.0.0.1

Monitor a Log File With Filtering#

Like tail -f | grep but with more logic:

python3 -c "
import sys, re, time

pattern = re.compile(r'(ERROR|WARN|FATAL)', re.IGNORECASE)
seen = set()

for line in sys.stdin:
    if pattern.search(line):
        key = line.strip()
        if key not in seen:
            seen.add(key)
            print(line, end='')
" < /var/log/syslog

This deduplicates repeated errors so you see each unique message only once.

CSV to JSON#

Convert that spreadsheet export into something usable:

python3 -c "
import csv, json, sys
reader = csv.DictReader(sys.stdin)
print(json.dumps(list(reader), indent=2))
" < data.csv

URL Encode and Decode#

# Encode
python3 -c "from urllib.parse import quote; print(quote('hello world & friends'))"
# hello%20world%20%26%20friends

# Decode
python3 -c "from urllib.parse import unquote; print(unquote('hello%20world%20%26%20friends'))"
# hello world & friends

Generate Random Passwords#

python3 -c "
import secrets, string
alphabet = string.ascii_letters + string.digits + '!@#$%'
password = ''.join(secrets.choice(alphabet) for _ in range(24))
print(password)
"

IP Address Math#

Check if an address is in a subnet:

python3 -c "
from ipaddress import ip_address, ip_network
net = ip_network('10.0.0.0/24')
print(ip_address('10.0.0.42') in net)  # True
print(ip_address('10.14.0.1') in net)    # False
print(f'Network: {net.network_address}')
print(f'Broadcast: {net.broadcast_address}')
print(f'Hosts: {net.num_addresses - 2}')
"

The standard library covers most sysadmin needs without pip. json, csv, http.server, ipaddress, secrets, urllib, base64, socket. It is all there. When the bash one-liner turns into a bash five-liner with nested quotes and escaped dollars signs, switch to Python. Your future self will thank you.

The Minimal Service#

Create /etc/systemd/system/myapp.service:

[Unit]
Description=My Application
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/node /opt/myapp/server.js
Restart=unless-stopped
RestartSec=5
WorkingDirectory=/opt/myapp

[Install]
WantedBy=multi-user.target

Then:

sudo systemctl daemon-reload
sudo systemctl enable myapp
sudo systemctl start myapp

That is it. The app starts on boot, restarts on crash, and logs to journald.

Service Types#

Type=simple is correct for most things. The process you start is the service. But there are others:

# App forks into background (like old-school daemons)
Type=forking
PIDFile=/var/run/myapp.pid

# App signals readiness via sd_notify
Type=notify

# App is a one-shot script, not a long-running process
Type=oneshot
RemainAfterExit=yes

Use simple unless you have a specific reason not to. oneshot is useful for startup scripts that need to run once.

Environment Variables#

Three ways to pass them:

# Inline (good for a few vars)
[Service]
Environment=PORT=3000
Environment=NODE_ENV=production

# From a file (good for secrets)
[Service]
EnvironmentFile=/opt/myapp/.env

# Multiple files
[Service]
EnvironmentFile=/opt/myapp/.env
EnvironmentFile=/opt/myapp/.env.local

The EnvironmentFile approach keeps secrets out of the unit file. The unit file itself might be in version control or readable by any user. The .env file can be locked down with permissions.

Running As a Non-Root User#

[Service]
User=myapp
Group=myapp
ExecStart=/usr/bin/node /opt/myapp/server.js

Create the user first:

sudo useradd -r -s /usr/sbin/nologin -d /opt/myapp myapp
sudo chown -R myapp:myapp /opt/myapp

The -r flag creates a system user. -s /usr/sbin/nologin prevents interactive login.

Resource Limits#

Prevent a runaway process from eating all your RAM or CPU:

[Service]
MemoryMax=512M
CPUQuota=50%
TasksMax=100
LimitNOFILE=65535

MemoryMax kills the process if it exceeds the limit. CPUQuota throttles but does not kill. TasksMax limits the number of threads and child processes. LimitNOFILE sets the open file descriptor limit.

Restart Policies#

# Restart on any exit (crash, signal, clean exit)
Restart=always

# Restart on failure only (non-zero exit, signal)
Restart=on-failure

# Restart unless you explicitly stopped it
Restart=unless-stopped

I use unless-stopped for everything. It restarts on crashes but stays down when I manually stop the service for debugging.

Add rate limiting to prevent restart loops:

RestartSec=5
StartLimitIntervalSec=60
StartLimitBurst=5

This allows 5 restarts within 60 seconds. After that, the service enters a failed state and stops trying. Check what happened with journalctl -u myapp.

Logging#

systemd captures stdout and stderr automatically:

# View logs
journalctl -u myapp

# Follow in real time
journalctl -u myapp -f

# Last 50 lines
journalctl -u myapp -n 50

# Since last boot
journalctl -u myapp -b

# JSON output for parsing
journalctl -u myapp -o json-pretty

No log rotation config needed. journald handles it automatically based on disk space.

Dependencies and Ordering#

[Unit]
Description=My App
After=network.target postgresql.service
Requires=postgresql.service

After controls startup order. Requires means if PostgreSQL stops, myapp stops too. Use Wants instead of Requires for soft dependencies where your app can handle the other service being down.

Pre and Post Commands#

[Service]
ExecStartPre=/opt/myapp/migrate.sh
ExecStart=/usr/bin/node /opt/myapp/server.js
ExecStartPost=/usr/bin/curl -s http://localhost:3000/health
ExecStopPost=/opt/myapp/cleanup.sh

ExecStartPre runs before the main process. Good for database migrations or config validation. If it fails, the service does not start.

Debugging#

# Check service status
systemctl status myapp

# View full logs on failure
journalctl -u myapp -e --no-pager

# Check the actual unit file being used
systemctl cat myapp

# Verify unit file syntax
systemd-analyze verify myapp.service

The systemd-analyze verify command catches syntax errors without starting anything. Use it before reloading.

systemd is verbose and opinionated, but once you get past the initial learning curve, it handles all the edge cases that screen and nohup do not. Crashes, reboots, logging, resource limits, dependencies. Write the unit file once and forget about it.

Git worktrees fix this entirely: they let you check out several branches at once, each in its own directory, all sharing one repository.

What Is a Worktree?#

A worktree is a separate working directory that shares the same .git data. Each worktree can have a different branch checked out simultaneously:

# You are on main in /home/user/project
git worktree add ../project-hotfix hotfix/urgent-fix

Now you have two directories:

/home/user/project          → main branch
/home/user/project-hotfix   → hotfix/urgent-fix branch

Both share the same git history. Commits in one are visible in the other. But they have completely independent working directories.

The Workflow#

My typical day looks like this:

# Main work
cd ~/projects/app

# Someone needs a review - create a worktree
git worktree add ~/reviews/pr-142 origin/feature/new-api

# Review in a completely separate directory
cd ~/reviews/pr-142
npm install  # only needed first time
npm test

# Done reviewing, remove it
cd ~/projects/app
git worktree remove ~/reviews/pr-142

The key advantage: my main working directory is completely untouched. No stashing, no branch switching, no reinstalling dependencies.

Practical Patterns#

Parallel testing#

Run tests on two branches at the same time:

git worktree add /tmp/test-main main
git worktree add /tmp/test-feature feature/new-thing

# In two terminals
cd /tmp/test-main && npm test
cd /tmp/test-feature && npm test

Long-running branches#

If you maintain a staging branch alongside main:

git worktree add ~/staging staging

Keep it around permanently. Push to staging by working in that directory. No branch switching ever.

CI simulation#

Test your CI pipeline locally against a clean checkout:

git worktree add /tmp/ci-test HEAD
cd /tmp/ci-test
npm ci
npm run lint
npm test
npm run build

This catches the classic “works on my machine” issues where you have uncommitted files or stale build artifacts that mask problems.

Managing Worktrees#

# List all worktrees
git worktree list

# Output:
# /home/user/project         abc1234 [main]
# /home/user/project-hotfix  def5678 [hotfix/urgent-fix]
# /tmp/ci-test               abc1234 (detached HEAD)

# Remove a worktree
git worktree remove /home/user/project-hotfix

# Clean up stale worktree references
git worktree prune

Gotchas#

You cannot check out the same branch in two worktrees. Git prevents this because having the same branch in two places would make it ambiguous which working directory a commit applies to.

If you need to work on the same branch in two places, create a local tracking branch:

git worktree add ../experiment experiment-copy
cd ../experiment
git checkout -b experiment-local origin/experiment

The other thing to watch out for is node_modules. Each worktree needs its own node_modules because paths are different. The first npm install in a new worktree takes time. After that, it is fast because the npm cache is shared.

vs. git stash#

Stash is fine for quick interruptions. If someone asks you a question and you need to check something on another branch for thirty seconds, stash works.

But for anything that takes more than a minute, worktrees are better. They do not mess with your working directory, they let you have multiple contexts open in different editor windows, and they do not have the failure mode where you forget to pop the stash and lose track of what you were doing.

I keep a ~/worktrees/ directory for temporary ones and let them accumulate during the week. Friday afternoon I run git worktree list and clean up anything I am done with.

docker exec#

The obvious one. Drop into a shell:

docker exec -it mycontainer sh

Use sh not bash because Alpine images do not have bash. Once inside, you can poke around:

# Check what processes are running
ps aux

# Look at environment variables
env | sort

# Test network connectivity
wget -qO- http://other-service:3000/health

# Check disk usage
df -h

But what if the container has no shell? Minimal images built FROM scratch or distroless images strip everything out.

docker logs#

Before exec-ing into anything, check the logs:

# Last 100 lines
docker logs --tail 100 mycontainer

# Follow in real time
docker logs -f mycontainer

# Logs since a specific time
docker logs --since 2026-03-29T10:00:00 mycontainer

# With timestamps
docker logs -t mycontainer

The --since flag is underrated. When a container has been running for weeks, you do not want to scroll through megabytes of old logs.

docker inspect#

This tells you everything about the container’s configuration:

# Full JSON dump
docker inspect mycontainer

# Just the IP address
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' mycontainer

# Environment variables
docker inspect -f '{{json .Config.Env}}' mycontainer | jq .

# Mount points
docker inspect -f '{{json .Mounts}}' mycontainer | jq .

# Port mappings
docker inspect -f '{{json .NetworkSettings.Ports}}' mycontainer | jq .

# Health check status
docker inspect -f '{{json .State.Health}}' mycontainer | jq .

docker cp#

Copy files in or out of a running container:

# Copy a config file out for inspection
docker cp mycontainer:/etc/nginx/nginx.conf ./nginx.conf

# Copy a fix in
docker cp ./fixed-config.json mycontainer:/app/config.json

This works even if the container has no shell or text editor.

nsenter: The Nuclear Option#

When docker exec is not enough or the container has no shell, nsenter lets you enter the container’s namespaces from the host:

# Get the container's PID
PID=$(docker inspect -f '{{.State.Pid}}' mycontainer)

# Enter its network namespace
sudo nsenter -t $PID -n ss -tlnp

# Enter all namespaces (like exec but from the host)
sudo nsenter -t $PID -m -u -i -n -p -- /bin/sh

This is useful for network debugging. You can run tcpdump, ss, or ip inside the container’s network namespace even if those tools are not installed in the container.

docker diff#

See what files changed since the container started:

docker diff mycontainer

Output looks like:

C /app
A /app/data/cache.db
C /tmp
A /tmp/something-unexpected

A means added, C means changed, D means deleted. This is great for finding containers that are writing to unexpected places.

Debugging Network Issues#

When containers cannot talk to each other:

# Check which networks a container is on
docker inspect mycontainer -f '{{json .NetworkSettings.Networks}}' | jq 'keys'

# Ping from one container to another
docker exec web ping -c 3 api

# Check DNS resolution
docker exec web nslookup api

# List all containers on a network
docker network inspect mynetwork -f '{{range .Containers}}{{.Name}} {{.IPv4Address}}{{end}}'

One-Liner Health Checks#

# Which containers are unhealthy?
docker ps --filter health=unhealthy --format '{{.Names}}: {{.Status}}'

# Which containers restarted recently?
docker ps --format '{{.Names}}: {{.Status}}' | grep -i restarting

# Resource usage snapshot
docker stats --no-stream --format 'table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}'

The instinct to restart a broken container is strong. But nine times out of ten, the information you need to fix the actual problem is sitting right there in the logs, the inspect output, or the filesystem diff. Restarting just resets the evidence.

Automatic TLS#

This is the obvious one. Caddy gets certificates from Let’s Encrypt automatically. But the real power is the DNS challenge with wildcard certs:

{
  acme_dns cloudflare {env.CLOUDFLARE_API_TOKEN}
}

*.example.dev {
  tls {
    dns cloudflare {env.CLOUDFLARE_API_TOKEN}
  }

  @app host app.example.dev
  handle @app {
    reverse_proxy app:3000
  }
}

One wildcard cert covers every subdomain. No per-service cert management. No Certificate Transparency log leaking your internal subdomain names. And the DNS challenge means you do not even need port 80 open.

Host Matching#

Caddy lets you route by hostname inside a single server block:

*.example.dev {
  @git host git.example.dev
  handle @git {
    reverse_proxy gitea:3000
  }

  @chat host chat.example.dev
  handle @chat {
    reverse_proxy element:80
  }

  handle {
    respond "Not found" 404
  }
}

In Nginx, each of these would be a separate server block with its own SSL configuration. In Caddy, it is one block with named matchers.

VPN-Only Routes#

I restrict admin services to my VPN subnet:

@admin host admin.example.dev
handle @admin {
  @blocked not remote_ip 10.0.0.0/24
  respond @blocked "Access Denied" 403
  reverse_proxy admin:3001
}

If the request does not come from the VPN, it gets a 403. Combined with DNS that only resolves through the VPN, this is two layers of protection without any auth middleware.

SPA Fallback With Sub-Paths#

Serving a single-page app under a sub-path is surprisingly tricky. The app lives at /app/ but the client-side router handles routes like /app/settings:

handle_path /app* {
  reverse_proxy spa:3000 {
    @notfound status 404
    handle_response @notfound {
      rewrite * /index.html
      reverse_proxy spa:3000
    }
  }
}

handle_path strips the /app prefix before forwarding. If the upstream returns 404, we rewrite to index.html and try again. The SPA router takes over from there.

Gzip and Security Headers#

*.example.dev {
  encode gzip

  header {
    Strict-Transport-Security "max-age=31536000; includeSubDomains"
    X-Content-Type-Options "nosniff"
    X-Frame-Options "DENY"
    Referrer-Policy "strict-origin-when-cross-origin"
  }
}

These apply to every subdomain automatically. No copying the same headers into fifteen different location blocks.

Static Files With Custom 404#

@landing host example.dev
handle @landing {
  root * /srv/example.dev
  file_server
}

handle_errors {
  @404 expression {err.status_code} == 404
  handle @404 {
    root * /srv
    rewrite * /404.html
    file_server
  }
}

The handle_errors block catches 404s across all routes and serves a custom page. One error handler for the entire server.

WebSocket Proxying#

Caddy handles WebSocket upgrades automatically:

reverse_proxy game:3000

That is it. No proxy_http_version 1.1, no proxy_set_header Upgrade, no proxy_set_header Connection. Caddy detects the upgrade request and does the right thing.

Streaming Responses#

For Server-Sent Events or long-running responses:

reverse_proxy backend:3000 {
  flush_interval -1
  transport http {
    read_timeout 0
    write_timeout 0
  }
}

flush_interval -1 disables response buffering. The timeout overrides prevent Caddy from killing long-lived connections.

The Caddyfile syntax is not perfect. The matcher system takes some getting used to, and the documentation could be better organized. But the amount of boilerplate it eliminates compared to Nginx is staggering. My entire server config is under 120 lines, and it handles TLS, routing, static files, reverse proxying, WebSockets, and error pages for a dozen services.

This means changing your DNS server does nothing. The ISP sees the domain name in the TLS handshake before the connection is encrypted and drops the packet. You need a different approach. Mine has two parts: zapret, which splits up the TLS ClientHello so the DPI system cannot read the SNI field in time, and DNS-over-TLS, so DNS queries stop travelling in plain text.

How DPI Blocking Works#

When you connect to a website over HTTPS, the first thing your browser sends is a TLS ClientHello message. This message is not encrypted. It contains the Server Name Indication field, which tells the server which domain you are trying to reach. This is necessary because multiple domains can share the same IP address, and the server needs to know which certificate to present.

Your ISP inspects this field. If the domain matches their blocklist, they inject a TCP RST packet or simply drop the connection. From your perspective, the site just times out or you get a connection reset.

The Zapret Approach#

There is a tool called zapret that manipulates outgoing packets to confuse DPI systems. It does not encrypt anything or route traffic through a proxy. Instead, it exploits the fact that DPI systems have to make real-time decisions about packets and often take shortcuts.

The basic techniques include fragmenting the ClientHello across multiple TCP segments, adding padding to push the SNI field into a second packet, or modifying TCP flags in ways that confuse stateful inspection engines but are tolerated by the destination server.

I run zapret as a systemd service. It hooks into the outgoing packet flow using nfqueue and modifies specific packets before they leave my machine. The destination server receives the same data, just split up differently, and handles it fine. The ISP DPI system sees fragments that it cannot reassemble quickly enough and lets them through.

DNS-over-TLS Complement#

Zapret handles the TLS handshake problem, but DNS queries can still leak information. Even if the ISP cannot block the connection, they can see which domains you are resolving and log that data.

I force all DNS through DNS-over-TLS using systemd-resolved. The configuration tells resolved to only use TLS-encrypted DNS and to require valid certificates from the upstream resolver. I use Cloudflare and Google as upstream servers since both support DNS-over-TLS.

The resolved config looks straightforward. You set the DNS servers and tell it to require encrypted transport. After restarting the service, all DNS queries from the machine go through TLS on port 853 instead of plaintext on port 53.

The VPN Conflict#

This setup conflicts with split-tunnel VPN configurations. When the VPN is active, DNS queries for private services need to go to the VPN DNS server over the tunnel. But the DPI bypass forces all DNS through TLS, which means queries to the VPN DNS server also try to use TLS.

The VPN DNS server is just dnsmasq. It does not speak TLS. So the queries fail and private domain names do not resolve.

The fix is a PostUp hook in the WireGuard configuration that disables DNS-over-TLS specifically on the VPN interface. The system still uses DoT for all other DNS queries, but queries routed through the VPN go as plain DNS over the encrypted tunnel. Since the tunnel itself is encrypted, this is fine from a privacy perspective.

Getting this right took some trial and error. The symptoms were confusing: public websites worked, VPN connected successfully, but private subdomains just would not resolve. Once I traced it to the DoT setting, the fix was one line.

Performance Impact#

Zapret adds essentially zero latency. The packet manipulation happens in kernel space and takes microseconds. DNS-over-TLS adds a small amount of latency to the first query for each domain, maybe 10 to 20 milliseconds. Subsequent queries are cached by resolved so you do not notice it.

I have been running this setup for months and have never noticed any performance impact in normal usage. Streaming, gaming, and large downloads all work at full speed. The only time I notice it is when I forget to check if the service is running and a blocked site fails to load.

Reliability#

The DPI bypass works about 99 percent of the time. Occasionally the ISP updates their inspection rules and a specific technique stops working. Zapret has multiple strategies and you can configure fallbacks. Usually updating to the latest version fixes any regressions within a day or two.

The DNS-over-TLS setup is rock solid. I have never had it fail. Systemd-resolved handles reconnections and failovers between upstream servers automatically.

Legal and Ethical Notes#

This is not about accessing illegal content. My ISP blocks legitimate websites, often because a previous user of the domain or IP address did something that triggered a broad block. The blocking is opaque, there is no notification, and there is no process for getting a site unblocked.

Bypassing DPI to access legal content that your ISP has incorrectly or broadly blocked is a reasonable thing to do. The tools involved are all open source and widely used. Nobody has ever gotten in trouble for running zapret on their own machine.

If you are in a situation where your ISP does DPI blocking, this setup gives you back control over your own internet connection without the overhead and privacy implications of routing all your traffic through a third-party VPN.

jq is the tool that makes JSON usable from the command line. Once you learn the basics, you will never go back.

Basics#

Pretty-print some JSON:

curl -s https://api.github.com/users/torvalds | jq .

Extract a single field:

echo '{"name": "Linus", "repos": 5}' | jq .name
# "Linus"

Drop the quotes with -r:

echo '{"name": "Linus"}' | jq -r .name
# Linus

Arrays#

Get the first element:

echo '[1, 2, 3]' | jq '.[0]'
# 1

Iterate over all elements:

echo '[{"name": "a"}, {"name": "b"}]' | jq '.[].name'
# "a"
# "b"

Filtering#

This is where it gets powerful. Say you have a list of GitHub repos:

curl -s https://api.github.com/users/torvalds/repos | \
  jq '[.[] | select(.stargazers_count > 1000)] | length'

That filters repos with more than 1000 stars and counts them.

Constructing Objects#

You can reshape JSON on the fly:

curl -s https://api.github.com/users/torvalds/repos | \
  jq '.[] | {name: .name, stars: .stargazers_count, lang: .language}'

This outputs a new object for each repo with only the fields you care about.

Real World: Parsing Docker#

List container names and their memory usage:

docker stats --no-stream --format '{{json .}}' | \
  jq -r '[.Name, .MemUsage] | @tsv'

Find containers using more than 100MB:

docker stats --no-stream --format '{{json .}}' | \
  jq -r 'select(.MemPerc | gsub("%"; "") | tonumber > 1) | .Name'

Working With Nested Data#

Given deeply nested JSON:

{
  "data": {
    "users": [
      {"id": 1, "profile": {"email": "a@b.com"}},
      {"id": 2, "profile": {"email": "c@d.com"}}
    ]
  }
}

Extract all emails:

jq '[.data.users[].profile.email]' data.json
# ["a@b.com", "c@d.com"]

Combining With Other Tools#

jq plays well with the rest of the Unix toolkit:

# Get all unique languages across repos
curl -s https://api.github.com/users/torvalds/repos | \
  jq -r '.[].language // empty' | sort -u

# Convert JSON array to lines for xargs
echo '["file1.txt", "file2.txt"]' | jq -r '.[]' | xargs rm

# Merge multiple JSON files
jq -s 'add' file1.json file2.json > merged.json

The Cheat Sheet#

jq has a learning curve, but it is one of those tools where the investment pays off within a week. Every time you find yourself writing a Python script just to parse some JSON, remember that jq probably does it in one line.

The Basics#

The SSH config lives at ~/.ssh/config. Here is a minimal example:

Host myserver
  HostName 203.0.113.50
  User root
  IdentityFile ~/.ssh/id_ed25519

Now ssh myserver is all you need. But it gets better.

Jump Hosts#

If you need to reach a server through a bastion:

Host bastion
  HostName bastion.example.com
  User admin

Host internal
  HostName 10.0.1.50
  User deploy
  ProxyJump bastion

ssh internal automatically tunnels through the bastion. No manual port forwarding, no two-step login.

Wildcard Hosts#

This is the one most people miss:

Host *.prod
  User deploy
  IdentityFile ~/.ssh/prod_key
  StrictHostKeyChecking accept-new

Host web.prod
  HostName 10.0.1.10

Host api.prod
  HostName 10.0.1.11

Host db.prod
  HostName 10.0.1.12

The wildcard block sets defaults for anything ending in .prod. Individual host blocks just add the hostname. You get consistent configuration without repetition.

Keep Connections Alive#

Add this to your Host * block:

Host *
  ServerAliveInterval 60
  ServerAliveCountMax 3
  AddKeysToAgent yes
  IdentitiesOnly yes

ServerAliveInterval sends a keepalive every 60 seconds. This stops firewalls and NATs from killing idle connections. AddKeysToAgent automatically loads keys into ssh-agent after first use.

Connection Multiplexing#

This is the real speed trick:

Host *
  ControlMaster auto
  ControlPath ~/.ssh/sockets/%r@%h-%p
  ControlPersist 600

The first SSH connection to a host creates a socket. Subsequent connections to the same host reuse it. No new TCP handshake, no new key exchange. The second ssh myserver connects instantly.

ControlPersist 600 keeps the socket alive for 10 minutes after you disconnect. So if you SSH in, run a command, disconnect, and SSH back in within 10 minutes, the second connection is instant.

Create the socket directory first:

mkdir -p ~/.ssh/sockets

Port Forwarding Shortcuts#

Host db-tunnel
  HostName myserver.com
  User admin
  LocalForward 5432 localhost:5432
  RequestTTY no
  ExitOnForwardFailure yes

Now ssh db-tunnel opens a tunnel to the remote Postgres. No remembering the -L syntax. ExitOnForwardFailure means if the port is already in use, SSH exits instead of connecting without the tunnel.

SSHFP for Host Verification#

Tired of the “Are you sure you want to continue connecting?” messages? If you control the DNS, add SSHFP records:

ssh-keygen -r myserver.com

This outputs DNS records you can add to your zone. Then set:

Host *
  VerifyHostKeyDNS yes

SSH verifies the host key against DNS automatically. No more blindly accepting fingerprints.

The SSH config file is probably the highest-leverage dotfile you can invest time in. Five minutes of setup saves thousands of keystrokes.

Server Name Is Permanent#

The most important decision you make is server_name in the Synapse config. This is the domain that appears in every user ID. If you set it to matrix.example.com, your users will be @name:matrix.example.com. If you set it to example.com, they will be @name:example.com.

You want the bare domain. Shorter IDs look better and are easier to share. But here is the catch: once you set the server name and create the first user, you cannot change it. Ever. The signing keys, room IDs, and event hashes are all tied to that name. Changing it means starting over.

I went with the bare domain and use well-known delegation to tell other servers that the actual homeserver lives at a subdomain.

Well-Known Delegation#

When another Matrix server wants to talk to yours, it looks up your server name. If your server name is example.com but Synapse runs on matrix.example.com, the remote server needs a way to discover that.

You serve two JSON endpoints on the bare domain:

The server endpoint returns which host and port handles federation. The client endpoint returns where the client API lives. Both are tiny JSON responses that Caddy can serve inline with the respond directive. No files, no separate service.

The important detail that the docs barely mention: the client well-known response needs a CORS header. Specifically Access-Control-Allow-Origin: *. Without it, Element Web running in a browser cannot discover the homeserver. You will stare at a login screen that says it cannot reach the server and wonder what you did wrong.

Cloudflare and Federation#

If your domain is behind Cloudflare, federation works fine as long as you set up the well-known endpoints correctly. Federation traffic goes through port 443, which Cloudflare proxies without issues.

The one thing to watch: Cloudflare has a 100-second timeout on the free plan. For a small personal server this is irrelevant, but if you ever end up in a large federated room, the initial sync could theoretically hit that limit.

I have been in rooms with several hundred members and never hit the timeout. But it is worth knowing about.

Voice and Video Calls#

Element supports one-to-one and group calls, but they need a TURN server to relay media when direct peer-to-peer connections fail. This happens more often than you would think. Corporate firewalls, symmetric NAT, and mobile carriers all block direct WebRTC connections.

I already had a coturn server running for another project. Adding Matrix support was just a matter of generating a shared secret and putting it in the Synapse config. Synapse generates time-limited TURN credentials for each call using HMAC, which is more secure than static credentials.

The config in Synapse points to the TURN server by IP address, not hostname. This avoids DNS resolution issues. You list both UDP and TCP transport options so clients behind restrictive firewalls can fall back to TCP.

Testing calls is annoying because you need two separate accounts on two separate devices. You cannot call yourself. I created a test account, logged in on my phone, and called from my laptop. The first call had one-way audio because the TURN server was binding relay ports on Docker internal IPs instead of the public IP. Adding --relay-ip and --listening-ip flags to coturn fixed it immediately.

Registration and Invites#

I started with invite-only registration using tokens. Synapse has a built-in token system where you generate a token, give it to someone, and they use it during registration. Each token can be limited to a single use.

Then I realized I was the only one creating accounts, so I just opened registration entirely. Synapse will complain about open registration without verification, but you can silence that with a config flag. For a small server with friends, the spam risk is minimal.

If spam does become a problem, switching back to token-based registration is a one-line config change. No need to over-engineer it upfront.

Password Policy#

Synapse has a built-in password strength checker that rejects most passwords. It is aggressive. Passwords that I considered perfectly fine were being rejected. The policy can be disabled in the config, which I did immediately.

Element Web also has its own client-side password strength check that is separate from the server policy. Even after disabling the server policy, Element still complained. That one I could not disable through config. It is more of an annoyance than a blocker since it warns but does not prevent you from proceeding.

Resource Usage#

Synapse is the heaviest thing on my server. At idle with a handful of users, it uses about 300 megabytes. During federation sync or when someone joins a large room, it can spike to a gigabyte.

The PostgreSQL database backing it uses another 70 megabytes. Element Web is negligible since it is just static files served by nginx in a container.

The config has a caches.global_factor setting that controls memory usage. The default is 0.5. Turning it down to 0.3 reduces memory at the cost of more database queries. For a small server the difference is not noticeable.

Was It Worth the Effort#

Absolutely. My friends and I have a chat platform that we fully control. Voice calls work, screen sharing works, file sharing works. The web client means nobody has to install anything. And because it federates, anyone on any Matrix server can reach us if they know our addresses.

The setup took about two hours. Most of that was the TURN server configuration and testing calls. If you skip voice and video, you could have it running in thirty minutes.

The Problem#

I run several services on my VPS that should not be publicly accessible: a git server, an admin panel, a VPN management UI, and a mail server admin interface. These need to be reachable from my laptop and phone, but nobody else should be able to see them.

The obvious solution is a VPN. But I do not want all my traffic going through a server in Germany when I am watching a video or browsing the web. I want my normal internet connection for everything except the few private services.

Split Tunnel Configuration#

The key is the AllowedIPs setting in the WireGuard client config. Instead of 0.0.0.0/0 which routes everything, you specify only the VPN subnet:

[Interface]
PrivateKey = <client-private-key>
Address = 10.0.0.2/24
DNS = 10.0.0.1

[Peer]
PublicKey = <server-public-key>
Endpoint = your-server-ip:51820
AllowedIPs = 10.0.0.0/24

With this config, only traffic destined for 10.0.0.x addresses goes through the tunnel. Everything else takes the normal route. Your browsing speed stays the same, your public IP does not change, and you still get access to your private services.

DNS Is Where the Magic Happens#

The split tunnel routes packets, but your browser needs to know that admin.example.com should resolve to a VPN IP instead of a public one. This is where a local DNS server comes in.

I run dnsmasq on the VPN server, listening only on the WireGuard interface. It has entries like:

address=/git.example.com/10.0.0.1
address=/admin.example.com/10.0.0.1
address=/vpn.example.com/10.0.0.1

For any domain not in this list, dnsmasq forwards the query to Cloudflare or Google DNS. So private subdomains resolve to the VPN IP, and everything else resolves normally.

The client config sets DNS = 10.0.0.1 which tells the system to use the VPN DNS server. Combined with the split tunnel routing, DNS queries for private domains go through the tunnel to dnsmasq, and all other DNS queries go through the tunnel too but get forwarded to public DNS by dnsmasq.

The DNS-over-TLS Conflict#

This one took me a while to figure out. I use a DPI bypass tool on my local machine that forces all DNS through DNS-over-TLS via systemd-resolved. When the VPN comes up, the DNS setting should override this, but it does not always work cleanly.

The fix was a PostUp script in the WireGuard config:

PostUp = resolvectl dnsovertls %i no

This disables DNS-over-TLS specifically on the VPN interface so queries to 10.0.0.1 go through as plain DNS over the tunnel. Without this, the system tries to establish a TLS connection to the dnsmasq server, which obviously does not support TLS, and private domains fail to resolve.

No Public DNS Leaks#

One nice side effect of this setup: your private subdomains are completely invisible. There are no public DNS records for them. They do not show up in certificate transparency logs because the wildcard cert covers them. Someone scanning your domain will find the public subdomains but have no idea the private ones exist.

The only way to discover them is to be on the VPN, at which point dnsmasq resolves them. And even if someone knows the subdomain name and tries to access it from the public internet, Caddy returns 403 based on the source IP check.

Mobile Access#

WireGuard works great on phones. The official app for both iOS and Android supports split tunneling natively. I have the same config on my phone: only VPN traffic goes through the tunnel. This means I can check my git server from my phone without draining battery or slowing down everything else.

The one downside is that you have to manually toggle the VPN on when you need it. There is no automatic connection based on network or location. But honestly, it takes one tap and the connection establishes in under a second.

Resource Usage#

WireGuard is absurdly lightweight. The server side runs as a single container using about 30MB of RAM. The CPU usage is essentially zero unless you are pushing serious bandwidth. On the client side, the kernel module handles everything so there is no userspace overhead.

Compare this to OpenVPN which needs a full userspace daemon, TLS handshakes, and significantly more CPU per connection. For a personal VPN with one or two clients, WireGuard is comically overpowered.

Final Thoughts#

Split tunnel WireGuard gives you the best of both worlds: fast, private access to your services without compromising your normal internet experience. The setup takes maybe 30 minutes and the result is rock solid. I have had zero downtime in months.

The only complexity is the DNS piece, and even that is just a few lines in a dnsmasq config file. If you are running private services on a VPS, this is the way to expose them.

That is when I started using traps. trap cleanup EXIT runs a cleanup function however the script ends: a normal exit, an error or a Ctrl-C.

The Problem#

Bash scripts are fragile by default. If a command fails, the script keeps going. If you hit Ctrl-C, whatever state you left things in is where they stay. If your SSH connection drops mid-script, same thing.

Most people know about set -e which exits on the first error. But that only covers one failure mode. What about cleanup?

Enter trap#

The trap builtin lets you run a command when the script receives a signal or exits:

#!/bin/bash
set -euo pipefail

TMPDIR=$(mktemp -d)

cleanup() {
  echo "Cleaning up $TMPDIR..."
  rm -rf "$TMPDIR"
}

trap cleanup EXIT

# Your actual work here
cp important-files "$TMPDIR/"
process "$TMPDIR/important-files"
mv "$TMPDIR/important-files" /final/destination/

The EXIT trap fires no matter how the script ends. Normal exit, error, Ctrl-C, even kill. The temp directory always gets cleaned up.

A Real Deploy Script#

Here is the pattern I use for deployments:

#!/bin/bash
set -euo pipefail

SERVER="myserver"
REMOTE_DIR="/opt/app"
BACKUP_DIR="/opt/app-backup-$(date +%s)"
ROLLED_BACK=false

rollback() {
  if [ "$ROLLED_BACK" = true ]; then return; fi
  ROLLED_BACK=true
  echo "Deploy failed. Rolling back..."
  ssh "$SERVER" "rm -rf $REMOTE_DIR && mv $BACKUP_DIR $REMOTE_DIR"
  echo "Rollback complete."
}

trap rollback ERR

# Backup current version
ssh "$SERVER" "cp -r $REMOTE_DIR $BACKUP_DIR"

# Deploy
rsync -a --delete ./dist/ "$SERVER:$REMOTE_DIR/"
ssh "$SERVER" "cd $REMOTE_DIR && docker compose up -d"

# Success - remove backup
ssh "$SERVER" "rm -rf $BACKUP_DIR"
echo "Deploy complete."

If anything after the backup fails, the old version gets restored automatically. No manual intervention, no half-broken state.

set -euo pipefail#

This trio should be at the top of every script you write:

• set -e exits on any command failure
• set -u treats unset variables as errors instead of empty strings
• set -o pipefail makes pipes fail if any command in the chain fails, not just the last one

Without pipefail, this silently succeeds even when curl fails:

curl https://example.com/data | jq .result
echo "Done"

jq gets an empty input, outputs null, and the script continues. With pipefail, the whole pipe fails and set -e kills the script.

Trapping Multiple Signals#

You can trap specific signals separately:

trap "echo Interrupted; cleanup; exit 130" INT
trap "echo Terminated; cleanup; exit 143" TERM
trap cleanup EXIT

But usually just trapping EXIT is enough. It covers everything.

The key insight is that bash does not protect you from yourself. You have to opt into safety. But once you do, scripts become surprisingly reliable.

It sounds like a mess, but it is actually pretty clean. The trick is structure: one compose file, two networks and some naming discipline.

One Compose File, Two Networks#

Every service lives in a single docker-compose.yml. I tried splitting it into multiple compose files early on and it was a nightmare. Services that need to talk to each other end up on different networks, you forget which file defines what, and debugging becomes archaeology.

One file means one place to look. The trade-off is that the file is about 200 lines long, but that is way better than hunting through five separate files.

The networking is simple: two Docker bridge networks. The first one connects Caddy to the internet. The second one connects Caddy to every backend service. Services that need to be reached by Caddy join the backend network. Services that need direct port access to the internet get explicit port mappings.

The WireGuard container is the exception. It runs with host networking because it needs raw access to the network stack for the VPN tunnel. Everything else uses the bridge networks.

Container Naming#

I name every container explicitly with container_name. Docker Compose generates names like wleeaf-caddy-1 by default, which is ugly in logs and hard to type in commands. Explicit names mean I can do docker logs caddy instead of docker logs wleeaf-caddy-1.

This also matters for DNS resolution inside Docker networks. When you give a container a name, other containers on the same network can reach it by that name. So my Caddy config just says reverse_proxy gitea:3000 and Docker handles the resolution.

Resource Reality#

People overestimate how much RAM you need. Here is what my containers actually use:

The reverse proxy sits at 15 megabytes. The git server uses about 110. The mail server is the hungriest at 160. The Matrix homeserver varies between 100 and 500 depending on federation activity. The database backing it uses 70. Everything else is under 50 each.

Total memory usage hovers around 2.5 gigabytes. That leaves five gigs free for spikes, builds, and the occasional agent session eating a gig for a few minutes.

CPU is basically idle. The load average rarely goes above 0.1 unless someone is building a Docker image or running a CI job. Four cores is overkill for this workload.

Secrets Management#

I keep secrets in a .env file next to the compose file. Docker Compose reads it automatically. Database passwords, API tokens, and hashed passwords all live there. The compose file references them with the dollar-sign-curly-brace syntax.

The .env file is not in any git repo. I back it up manually. Some people use Docker secrets or Vault for this, but for a single-server personal setup, a .env file is fine.

Restart Policies#

Every service has restart: unless-stopped. This means containers come back after a reboot or a crash, but stay down if I explicitly stop them. I used to use always but that fights you when you are debugging, because the container keeps restarting while you are trying to read the logs.

The one exception is the CI runner. It has restart: unless-stopped too, but I sometimes need to stop it to free resources during heavy builds. With unless-stopped, it stays down until I bring it back up.

Volume Strategy#

I mount named directories from the host filesystem, not Docker volumes. Every service gets a directory under the main project folder. Caddy gets ./caddy, Gitea gets ./gitea, the mail server gets ./stalwart, and so on.

This makes backups trivial. I can tar the entire project directory and I have everything: configs, data, and the compose file itself. Docker volumes are fine, but they hide the data in a location you have to look up every time.

The Upgrade Dance#

Upgrading a service usually means changing the image tag and running docker compose up -d. For services using latest, I pull first with docker compose pull and then bring them up.

I do not use Watchtower or any automatic update tool. I want to know exactly when things change. An automatic update at 3 AM that breaks the mail server is not a problem I want to have.

The whole setup has been running for months with essentially zero unplanned downtime. The only time services go down is when I am actively working on them. For a personal infrastructure project, that is more than good enough.

The usual advice goes something like this: spam filters will eat your messages, IP reputation is impossible to build, and you will spend your weekends debugging DKIM signatures instead of doing anything useful. After a couple weeks of running Stalwart on a cheap Hetzner VPS, I can say that some of those warnings are legitimate but most of them are outdated.

The Setup#

I went with Stalwart, which is a relatively new mail server written in Rust. It handles SMTP, IMAP, and JMAP out of the box, and the admin UI is surprisingly decent for a self-hosted project. The whole thing runs in a single Docker container with a RocksDB backend, which means no separate database to manage.

The stack looks like this:

• Stalwart for mail handling on ports 25, 465, 587, and 993
• Caddy as the reverse proxy with a wildcard TLS cert
• SnappyMail for webmail access
• Cloudflare for DNS on the free plan

The VPS itself is nothing special. 4 cores, 8 gigs of RAM, running Ubuntu 24.04. Stalwart barely touches the resources and sits at around 150MB of RAM even after a few thousand messages.

DNS Was the Easy Part#

Setting up DNS records is the part that scares most people, but it is genuinely straightforward if you take it step by step. You need an A record for your mail subdomain, an MX record pointing to it, SPF to declare which IPs can send for your domain, DKIM for cryptographic signature verification, and DMARC to set a policy for failed checks.

The Cloudflare API makes all of this scriptable. I wrote a shell script that creates all five records in about 30 seconds. The DKIM key was the only tricky part because Stalwart generates it internally and you have to copy it out of the admin panel and paste it into DNS.

The Reputation Problem#

This is the real challenge. When you spin up a brand new IP and start sending mail, every major provider treats you with suspicion. Gmail sent my first few messages straight to spam even though SPF, DKIM, and DMARC all passed perfectly.

The fix was not technical, it was behavioral. I marked those messages as not spam in my Gmail account, added the addresses to my contacts, and started having normal conversations. Within a few days, Gmail stopped flagging my messages. Google Postmaster Tools confirmed that the domain reputation was climbing.

One thing that helped: my VPS provider has decent IP reputation to begin with. If you are on a provider known for abuse, you will have a much harder time. Also make sure your reverse DNS is set correctly because Gmail checks that the PTR record for your IP matches your mail hostname.

TLS Certificate Sharing#

I did not want to manage separate certificates for the mail server. Caddy already handles TLS for all my subdomains using a wildcard cert from Lets Encrypt via DNS challenge. I just mounted the certificate directory read-only into the Stalwart container.

The only catch is that Caddy stores the cert in a specific directory structure and you need to figure out the exact path. Once you find that, a volume mount in Docker Compose is all you need.

Was It Worth It#

Honestly, yes. The total ongoing cost is zero beyond the VPS I was already paying for. I have full control over my email, the data never touches a third party, and the whole thing took an afternoon to set up.

The one thing I would do differently: I would have registered my domain and set up the mail server on the same day. Domain age matters for reputation, and the sooner you start sending from a domain, the sooner it builds trust with the big providers.

If you are already running a VPS for other things, adding a mail server is a no-brainer. If you are starting from scratch just for email, you might want to think about whether the learning experience is worth the first week of dealing with spam folders.