<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>systemd · wleeaf field notes</title><link>https://articles.wleeaf.dev/tags/systemd/</link><description>Hands-on guides and build logs from the wleeaf studio: self-hosting, Linux, networking, Docker, the command line and reverse engineering.</description><language>en-us</language><lastBuildDate>Wed, 01 Apr 2026 00:00:00 +0000</lastBuildDate><atom:link href="https://articles.wleeaf.dev/tags/systemd/feed.xml" rel="self" type="application/rss+xml"/><item><title>Writing systemd Services That Actually Work</title><link>https://articles.wleeaf.dev/posts/systemd-services-from-scratch/</link><pubDate>Wed, 01 Apr 2026 00:00:00 +0000</pubDate><guid>https://articles.wleeaf.dev/posts/systemd-services-from-scratch/</guid><description>Unit files, restart policies, environment variables, and journalctl. A practical guide to running your own daemons.</description><category>linux</category><category>systemd</category><category>infrastructure</category><content:encoded><![CDATA[<p>At some point you write a script or a server that needs to run forever. You could use screen, tmux, nohup, or a cron <code>@reboot</code> hack. Or you could write a systemd service and get automatic restarts, logging, resource limits, and dependency management for free.</p>
<h2 id="the-minimal-service">The Minimal Service<a class="anchor" href="#the-minimal-service" aria-label="Link to “The Minimal Service”">#</a></h2>
<p>Create <code>/etc/systemd/system/myapp.service</code>:</p>
<div class="code">
<div class="code-head"><span class="code-lang">ini</span><button type="button" class="code-copy" data-copy-code hidden><svg viewBox="0 0 24 24" aria-hidden="true"><rect x="8" y="8" width="12" height="12" rx="2.5"/><path d="M16 8V6.5A2.5 2.5 0 0 0 13.5 4h-7A2.5 2.5 0 0 0 4 6.5v7A2.5 2.5 0 0 0 6.5 16H8"/></svg><span>Copy</span></button></div>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="k">[Unit]</span>
</span></span><span class="line"><span class="cl"><span class="na">Description</span><span class="o">=</span><span class="s">My Application</span>
</span></span><span class="line"><span class="cl"><span class="na">After</span><span class="o">=</span><span class="s">network.target</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">[Service]</span>
</span></span><span class="line"><span class="cl"><span class="na">Type</span><span class="o">=</span><span class="s">simple</span>
</span></span><span class="line"><span class="cl"><span class="na">ExecStart</span><span class="o">=</span><span class="s">/usr/bin/node /opt/myapp/server.js</span>
</span></span><span class="line"><span class="cl"><span class="na">Restart</span><span class="o">=</span><span class="s">unless-stopped</span>
</span></span><span class="line"><span class="cl"><span class="na">RestartSec</span><span class="o">=</span><span class="s">5</span>
</span></span><span class="line"><span class="cl"><span class="na">WorkingDirectory</span><span class="o">=</span><span class="s">/opt/myapp</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">[Install]</span>
</span></span><span class="line"><span class="cl"><span class="na">WantedBy</span><span class="o">=</span><span class="s">multi-user.target</span></span></span></code></pre></div>
</div>
<p>Then:</p>
<div class="code">
<div class="code-head"><span class="code-lang">bash</span><button type="button" class="code-copy" data-copy-code hidden><svg viewBox="0 0 24 24" aria-hidden="true"><rect x="8" y="8" width="12" height="12" rx="2.5"/><path d="M16 8V6.5A2.5 2.5 0 0 0 13.5 4h-7A2.5 2.5 0 0 0 4 6.5v7A2.5 2.5 0 0 0 6.5 16H8"/></svg><span>Copy</span></button></div>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">sudo systemctl daemon-reload
</span></span><span class="line"><span class="cl">sudo systemctl <span class="nb">enable</span> myapp
</span></span><span class="line"><span class="cl">sudo systemctl start myapp</span></span></code></pre></div>
</div>
<p>That is it. The app starts on boot, restarts on crash, and logs to journald.</p>
<h2 id="service-types">Service Types<a class="anchor" href="#service-types" aria-label="Link to “Service Types”">#</a></h2>
<p><code>Type=simple</code> is correct for most things. The process you start is the service. But there are others:</p>
<div class="code">
<div class="code-head"><span class="code-lang">ini</span><button type="button" class="code-copy" data-copy-code hidden><svg viewBox="0 0 24 24" aria-hidden="true"><rect x="8" y="8" width="12" height="12" rx="2.5"/><path d="M16 8V6.5A2.5 2.5 0 0 0 13.5 4h-7A2.5 2.5 0 0 0 4 6.5v7A2.5 2.5 0 0 0 6.5 16H8"/></svg><span>Copy</span></button></div>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="c1"># App forks into background (like old-school daemons)</span>
</span></span><span class="line"><span class="cl"><span class="na">Type</span><span class="o">=</span><span class="s">forking</span>
</span></span><span class="line"><span class="cl"><span class="na">PIDFile</span><span class="o">=</span><span class="s">/var/run/myapp.pid</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># App signals readiness via sd_notify</span>
</span></span><span class="line"><span class="cl"><span class="na">Type</span><span class="o">=</span><span class="s">notify</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># App is a one-shot script, not a long-running process</span>
</span></span><span class="line"><span class="cl"><span class="na">Type</span><span class="o">=</span><span class="s">oneshot</span>
</span></span><span class="line"><span class="cl"><span class="na">RemainAfterExit</span><span class="o">=</span><span class="s">yes</span></span></span></code></pre></div>
</div>
<p>Use <code>simple</code> unless you have a specific reason not to. <code>oneshot</code> is useful for startup scripts that need to run once.</p>
<h2 id="environment-variables">Environment Variables<a class="anchor" href="#environment-variables" aria-label="Link to “Environment Variables”">#</a></h2>
<p>Three ways to pass them:</p>
<div class="code">
<div class="code-head"><span class="code-lang">ini</span><button type="button" class="code-copy" data-copy-code hidden><svg viewBox="0 0 24 24" aria-hidden="true"><rect x="8" y="8" width="12" height="12" rx="2.5"/><path d="M16 8V6.5A2.5 2.5 0 0 0 13.5 4h-7A2.5 2.5 0 0 0 4 6.5v7A2.5 2.5 0 0 0 6.5 16H8"/></svg><span>Copy</span></button></div>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="c1"># Inline (good for a few vars)</span>
</span></span><span class="line"><span class="cl"><span class="k">[Service]</span>
</span></span><span class="line"><span class="cl"><span class="na">Environment</span><span class="o">=</span><span class="s">PORT=3000</span>
</span></span><span class="line"><span class="cl"><span class="na">Environment</span><span class="o">=</span><span class="s">NODE_ENV=production</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># From a file (good for secrets)</span>
</span></span><span class="line"><span class="cl"><span class="k">[Service]</span>
</span></span><span class="line"><span class="cl"><span class="na">EnvironmentFile</span><span class="o">=</span><span class="s">/opt/myapp/.env</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Multiple files</span>
</span></span><span class="line"><span class="cl"><span class="k">[Service]</span>
</span></span><span class="line"><span class="cl"><span class="na">EnvironmentFile</span><span class="o">=</span><span class="s">/opt/myapp/.env</span>
</span></span><span class="line"><span class="cl"><span class="na">EnvironmentFile</span><span class="o">=</span><span class="s">/opt/myapp/.env.local</span></span></span></code></pre></div>
</div>
<p>The <code>EnvironmentFile</code> approach keeps secrets out of the unit file. The unit file itself might be in version control or readable by any user. The <code>.env</code> file can be locked down with permissions.</p>
<h2 id="running-as-a-non-root-user">Running As a Non-Root User<a class="anchor" href="#running-as-a-non-root-user" aria-label="Link to “Running As a Non-Root User”">#</a></h2>
<div class="code">
<div class="code-head"><span class="code-lang">ini</span><button type="button" class="code-copy" data-copy-code hidden><svg viewBox="0 0 24 24" aria-hidden="true"><rect x="8" y="8" width="12" height="12" rx="2.5"/><path d="M16 8V6.5A2.5 2.5 0 0 0 13.5 4h-7A2.5 2.5 0 0 0 4 6.5v7A2.5 2.5 0 0 0 6.5 16H8"/></svg><span>Copy</span></button></div>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="k">[Service]</span>
</span></span><span class="line"><span class="cl"><span class="na">User</span><span class="o">=</span><span class="s">myapp</span>
</span></span><span class="line"><span class="cl"><span class="na">Group</span><span class="o">=</span><span class="s">myapp</span>
</span></span><span class="line"><span class="cl"><span class="na">ExecStart</span><span class="o">=</span><span class="s">/usr/bin/node /opt/myapp/server.js</span></span></span></code></pre></div>
</div>
<p>Create the user first:</p>
<div class="code">
<div class="code-head"><span class="code-lang">bash</span><button type="button" class="code-copy" data-copy-code hidden><svg viewBox="0 0 24 24" aria-hidden="true"><rect x="8" y="8" width="12" height="12" rx="2.5"/><path d="M16 8V6.5A2.5 2.5 0 0 0 13.5 4h-7A2.5 2.5 0 0 0 4 6.5v7A2.5 2.5 0 0 0 6.5 16H8"/></svg><span>Copy</span></button></div>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">sudo useradd -r -s /usr/sbin/nologin -d /opt/myapp myapp
</span></span><span class="line"><span class="cl">sudo chown -R myapp:myapp /opt/myapp</span></span></code></pre></div>
</div>
<p>The <code>-r</code> flag creates a system user. <code>-s /usr/sbin/nologin</code> prevents interactive login.</p>
<h2 id="resource-limits">Resource Limits<a class="anchor" href="#resource-limits" aria-label="Link to “Resource Limits”">#</a></h2>
<p>Prevent a runaway process from eating all your RAM or CPU:</p>
<div class="code">
<div class="code-head"><span class="code-lang">ini</span><button type="button" class="code-copy" data-copy-code hidden><svg viewBox="0 0 24 24" aria-hidden="true"><rect x="8" y="8" width="12" height="12" rx="2.5"/><path d="M16 8V6.5A2.5 2.5 0 0 0 13.5 4h-7A2.5 2.5 0 0 0 4 6.5v7A2.5 2.5 0 0 0 6.5 16H8"/></svg><span>Copy</span></button></div>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="k">[Service]</span>
</span></span><span class="line"><span class="cl"><span class="na">MemoryMax</span><span class="o">=</span><span class="s">512M</span>
</span></span><span class="line"><span class="cl"><span class="na">CPUQuota</span><span class="o">=</span><span class="s">50%</span>
</span></span><span class="line"><span class="cl"><span class="na">TasksMax</span><span class="o">=</span><span class="s">100</span>
</span></span><span class="line"><span class="cl"><span class="na">LimitNOFILE</span><span class="o">=</span><span class="s">65535</span></span></span></code></pre></div>
</div>
<p><code>MemoryMax</code> kills the process if it exceeds the limit. <code>CPUQuota</code> throttles but does not kill. <code>TasksMax</code> limits the number of threads and child processes. <code>LimitNOFILE</code> sets the open file descriptor limit.</p>
<h2 id="restart-policies">Restart Policies<a class="anchor" href="#restart-policies" aria-label="Link to “Restart Policies”">#</a></h2>
<div class="code">
<div class="code-head"><span class="code-lang">ini</span><button type="button" class="code-copy" data-copy-code hidden><svg viewBox="0 0 24 24" aria-hidden="true"><rect x="8" y="8" width="12" height="12" rx="2.5"/><path d="M16 8V6.5A2.5 2.5 0 0 0 13.5 4h-7A2.5 2.5 0 0 0 4 6.5v7A2.5 2.5 0 0 0 6.5 16H8"/></svg><span>Copy</span></button></div>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="c1"># Restart on any exit (crash, signal, clean exit)</span>
</span></span><span class="line"><span class="cl"><span class="na">Restart</span><span class="o">=</span><span class="s">always</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Restart on failure only (non-zero exit, signal)</span>
</span></span><span class="line"><span class="cl"><span class="na">Restart</span><span class="o">=</span><span class="s">on-failure</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Restart unless you explicitly stopped it</span>
</span></span><span class="line"><span class="cl"><span class="na">Restart</span><span class="o">=</span><span class="s">unless-stopped</span></span></span></code></pre></div>
</div>
<p>I use <code>unless-stopped</code> for everything. It restarts on crashes but stays down when I manually stop the service for debugging.</p>
<p>Add rate limiting to prevent restart loops:</p>
<div class="code">
<div class="code-head"><span class="code-lang">ini</span><button type="button" class="code-copy" data-copy-code hidden><svg viewBox="0 0 24 24" aria-hidden="true"><rect x="8" y="8" width="12" height="12" rx="2.5"/><path d="M16 8V6.5A2.5 2.5 0 0 0 13.5 4h-7A2.5 2.5 0 0 0 4 6.5v7A2.5 2.5 0 0 0 6.5 16H8"/></svg><span>Copy</span></button></div>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="na">RestartSec</span><span class="o">=</span><span class="s">5</span>
</span></span><span class="line"><span class="cl"><span class="na">StartLimitIntervalSec</span><span class="o">=</span><span class="s">60</span>
</span></span><span class="line"><span class="cl"><span class="na">StartLimitBurst</span><span class="o">=</span><span class="s">5</span></span></span></code></pre></div>
</div>
<p>This allows 5 restarts within 60 seconds. After that, the service enters a failed state and stops trying. Check what happened with <code>journalctl -u myapp</code>.</p>
<h2 id="logging">Logging<a class="anchor" href="#logging" aria-label="Link to “Logging”">#</a></h2>
<p>systemd captures stdout and stderr automatically:</p>
<div class="code">
<div class="code-head"><span class="code-lang">bash</span><button type="button" class="code-copy" data-copy-code hidden><svg viewBox="0 0 24 24" aria-hidden="true"><rect x="8" y="8" width="12" height="12" rx="2.5"/><path d="M16 8V6.5A2.5 2.5 0 0 0 13.5 4h-7A2.5 2.5 0 0 0 4 6.5v7A2.5 2.5 0 0 0 6.5 16H8"/></svg><span>Copy</span></button></div>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># View logs</span>
</span></span><span class="line"><span class="cl">journalctl -u myapp
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Follow in real time</span>
</span></span><span class="line"><span class="cl">journalctl -u myapp -f
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Last 50 lines</span>
</span></span><span class="line"><span class="cl">journalctl -u myapp -n <span class="m">50</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Since last boot</span>
</span></span><span class="line"><span class="cl">journalctl -u myapp -b
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># JSON output for parsing</span>
</span></span><span class="line"><span class="cl">journalctl -u myapp -o json-pretty</span></span></code></pre></div>
</div>
<p>No log rotation config needed. journald handles it automatically based on disk space.</p>
<h2 id="dependencies-and-ordering">Dependencies and Ordering<a class="anchor" href="#dependencies-and-ordering" aria-label="Link to “Dependencies and Ordering”">#</a></h2>
<div class="code">
<div class="code-head"><span class="code-lang">ini</span><button type="button" class="code-copy" data-copy-code hidden><svg viewBox="0 0 24 24" aria-hidden="true"><rect x="8" y="8" width="12" height="12" rx="2.5"/><path d="M16 8V6.5A2.5 2.5 0 0 0 13.5 4h-7A2.5 2.5 0 0 0 4 6.5v7A2.5 2.5 0 0 0 6.5 16H8"/></svg><span>Copy</span></button></div>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="k">[Unit]</span>
</span></span><span class="line"><span class="cl"><span class="na">Description</span><span class="o">=</span><span class="s">My App</span>
</span></span><span class="line"><span class="cl"><span class="na">After</span><span class="o">=</span><span class="s">network.target postgresql.service</span>
</span></span><span class="line"><span class="cl"><span class="na">Requires</span><span class="o">=</span><span class="s">postgresql.service</span></span></span></code></pre></div>
</div>
<p><code>After</code> controls startup order. <code>Requires</code> means if PostgreSQL stops, myapp stops too. Use <code>Wants</code> instead of <code>Requires</code> for soft dependencies where your app can handle the other service being down.</p>
<h2 id="pre-and-post-commands">Pre and Post Commands<a class="anchor" href="#pre-and-post-commands" aria-label="Link to “Pre and Post Commands”">#</a></h2>
<div class="code">
<div class="code-head"><span class="code-lang">ini</span><button type="button" class="code-copy" data-copy-code hidden><svg viewBox="0 0 24 24" aria-hidden="true"><rect x="8" y="8" width="12" height="12" rx="2.5"/><path d="M16 8V6.5A2.5 2.5 0 0 0 13.5 4h-7A2.5 2.5 0 0 0 4 6.5v7A2.5 2.5 0 0 0 6.5 16H8"/></svg><span>Copy</span></button></div>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="k">[Service]</span>
</span></span><span class="line"><span class="cl"><span class="na">ExecStartPre</span><span class="o">=</span><span class="s">/opt/myapp/migrate.sh</span>
</span></span><span class="line"><span class="cl"><span class="na">ExecStart</span><span class="o">=</span><span class="s">/usr/bin/node /opt/myapp/server.js</span>
</span></span><span class="line"><span class="cl"><span class="na">ExecStartPost</span><span class="o">=</span><span class="s">/usr/bin/curl -s http://localhost:3000/health</span>
</span></span><span class="line"><span class="cl"><span class="na">ExecStopPost</span><span class="o">=</span><span class="s">/opt/myapp/cleanup.sh</span></span></span></code></pre></div>
</div>
<p><code>ExecStartPre</code> runs before the main process. Good for database migrations or config validation. If it fails, the service does not start.</p>
<h2 id="debugging">Debugging<a class="anchor" href="#debugging" aria-label="Link to “Debugging”">#</a></h2>
<div class="code">
<div class="code-head"><span class="code-lang">bash</span><button type="button" class="code-copy" data-copy-code hidden><svg viewBox="0 0 24 24" aria-hidden="true"><rect x="8" y="8" width="12" height="12" rx="2.5"/><path d="M16 8V6.5A2.5 2.5 0 0 0 13.5 4h-7A2.5 2.5 0 0 0 4 6.5v7A2.5 2.5 0 0 0 6.5 16H8"/></svg><span>Copy</span></button></div>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Check service status</span>
</span></span><span class="line"><span class="cl">systemctl status myapp
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># View full logs on failure</span>
</span></span><span class="line"><span class="cl">journalctl -u myapp -e --no-pager
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Check the actual unit file being used</span>
</span></span><span class="line"><span class="cl">systemctl cat myapp
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Verify unit file syntax</span>
</span></span><span class="line"><span class="cl">systemd-analyze verify myapp.service</span></span></code></pre></div>
</div>
<p>The <code>systemd-analyze verify</code> command catches syntax errors without starting anything. Use it before reloading.</p>
<p>systemd is verbose and opinionated, but once you get past the initial learning curve, it handles all the edge cases that screen and nohup do not. Crashes, reboots, logging, resource limits, dependencies. Write the unit file once and forget about it.</p>
]]></content:encoded></item></channel></rss>