cli · wleeaf field notes

awk in 10 Minutes With Practical Examples

Sat, 04 Apr 2026 00:00:00 +0000

awk is one of those tools that looks intimidating until you learn five patterns. Then you use it every day. I am not going to explain the full language. Here are the patterns that actually matter.

Pattern 1: Print a Column#

awk splits each line into fields. $1 is the first, $2 is the second, $0 is the whole line:

bash

# Print the first column of ps output
ps aux | awk '{print $1}'

# Print columns 1 and 11 (user and command)
ps aux | awk '{print $1, $11}'

The default field separator is whitespace. Change it with -F:

bash

# Parse /etc/passwd (colon-separated)
awk -F: '{print $1, $7}' /etc/passwd

# Parse CSV
awk -F, '{print $2}' data.csv

Pattern 2: Filter Rows#

Put a condition before the action:

bash

# Lines where column 3 is greater than 100
awk '$3 > 100' data.txt

# Lines containing "ERROR"
awk '/ERROR/' app.log

# Lines where the first field matches a pattern
awk '$1 ~ /^192\.168/' access.log

Combine conditions:

bash

# Memory usage over 50% AND the process is not root
ps aux | awk '$4 > 50.0 && $1 != "root" {print $1, $4, $11}'

Pattern 3: Sum a Column#

bash

# Total bytes transferred (column 10 in access logs)
awk '{sum += $10} END {print sum}' access.log

# Average response time
awk '{sum += $NF; n++} END {print sum/n}' response_times.log

END runs after all lines are processed. NF is the number of fields on the current line, so $NF is the last field.

Pattern 4: Count and Group#

bash

# Count HTTP status codes
awk '{count[$9]++} END {for (c in count) print c, count[c]}' access.log | sort -rn -k2

# Output:
# 200 45231
# 304 12044
# 404 891
# 500 23

This uses an associative array. count[$9]++ increments the count for each unique value in column 9.

Top 10 IP addresses by request count:

bash

awk '{count[$1]++} END {for (ip in count) print count[ip], ip}' access.log | sort -rn | head -10

Pattern 5: Transform Fields#

bash

# Convert bytes to megabytes
awk '{printf "%s %.1fMB\n", $1, $2/1048576}' sizes.txt

# Add a prefix to every line
awk '{print "[INFO]", $0}' app.log

# Swap two columns
awk '{temp=$1; $1=$2; $2=temp; print}' data.txt

Real-World Examples#

Parse Docker ps output#

bash

docker ps --format '{{.Names}}\t{{.Status}}\t{{.Ports}}' | \
  awk -F'\t' '{printf "%-20s %-15s %s\n", $1, $2, $3}'

Find the largest Docker images#

bash

docker images --format '{{.Repository}}:{{.Tag}}\t{{.Size}}' | \
  awk -F'\t' '{
    size = $2
    if (size ~ /GB/) {
      gsub(/GB/, "", size)
      mb = size * 1024
    } else if (size ~ /MB/) {
      gsub(/MB/, "", size)
      mb = size
    } else {
      mb = 0
    }
    printf "%8.0fMB  %s\n", mb, $1
  }' | sort -rn | head -10

Calculate uptime percentage#

Given a log with timestamps and status:

bash

awk '
  /UP/   { up++ }
  /DOWN/ { down++ }
  END {
    total = up + down
    if (total > 0)
      printf "Uptime: %.2f%% (%d/%d checks)\n", (up/total)*100, up, total
  }
' monitoring.log

Extract unique error messages#

bash

awk '/ERROR/ {
  msg = ""
  for (i=4; i<=NF; i++) msg = msg " " $i
  errors[msg]++
}
END {
  for (e in errors) printf "%5d %s\n", errors[e], e
}' app.log | sort -rn | head -20

Built-in Variables#

Variable	Meaning
`NR`	Current line number
`NF`	Number of fields on current line
`FS`	Field separator
`OFS`	Output field separator
`$0`	Entire line
`$1..$n`	Individual fields
`FILENAME`	Current filename

When to Use awk vs. Other Tools#

cut: When you just need a column and nothing else. cut -d: -f1 /etc/passwd is simpler than awk for trivial extraction.
grep: When you just need to filter lines by a pattern.
sed: When you need to do find-and-replace on text.
awk: When you need to do math, grouping, conditional logic, or work with multiple fields at once.

The mental model is simple. awk reads lines, splits them into fields, and lets you write rules about what to do with each line. Once that clicks, you start seeing awk solutions everywhere.

jq Is the SQL of JSON

Fri, 27 Mar 2026 00:00:00 +0000

Every API you talk to returns JSON. Every config file is JSON. Every log pipeline emits JSON. And yet most people still pipe curl output through grep and pray.

jq is the tool that makes JSON usable from the command line. Once you learn the basics, you will never go back.

Basics#

Pretty-print some JSON:

bash

curl -s https://api.github.com/users/torvalds | jq .

Extract a single field:

bash

echo '{"name": "Linus", "repos": 5}' | jq .name
# "Linus"

Drop the quotes with -r:

bash

echo '{"name": "Linus"}' | jq -r .name
# Linus

Arrays#

Get the first element:

bash

echo '[1, 2, 3]' | jq '.[0]'
# 1

Iterate over all elements:

bash

echo '[{"name": "a"}, {"name": "b"}]' | jq '.[].name'
# "a"
# "b"

Filtering#

This is where it gets powerful. Say you have a list of GitHub repos:

bash

curl -s https://api.github.com/users/torvalds/repos | \
  jq '[.[] | select(.stargazers_count > 1000)] | length'

That filters repos with more than 1000 stars and counts them.

Constructing Objects#

You can reshape JSON on the fly:

bash

curl -s https://api.github.com/users/torvalds/repos | \
  jq '.[] | {name: .name, stars: .stargazers_count, lang: .language}'

This outputs a new object for each repo with only the fields you care about.

Real World: Parsing Docker#

List container names and their memory usage:

bash

docker stats --no-stream --format '{{json .}}' | \
  jq -r '[.Name, .MemUsage] | @tsv'

Find containers using more than 100MB:

bash

docker stats --no-stream --format '{{json .}}' | \
  jq -r 'select(.MemPerc | gsub("%"; "") | tonumber > 1) | .Name'

Working With Nested Data#

Given deeply nested JSON:

json

{
  "data": {
    "users": [
      {"id": 1, "profile": {"email": "a@b.com"}},
      {"id": 2, "profile": {"email": "c@d.com"}}
    ]
  }
}

Extract all emails:

bash

jq '[.data.users[].profile.email]' data.json
# ["a@b.com", "c@d.com"]

Combining With Other Tools#

jq plays well with the rest of the Unix toolkit:

bash

# Get all unique languages across repos
curl -s https://api.github.com/users/torvalds/repos | \
  jq -r '.[].language // empty' | sort -u

# Convert JSON array to lines for xargs
echo '["file1.txt", "file2.txt"]' | jq -r '.[]' | xargs rm

# Merge multiple JSON files
jq -s 'add' file1.json file2.json > merged.json

The Cheat Sheet#

Pattern	What it does
`.field`	Get field
`.[]`	Iterate array
`select(cond)`	Filter
`map(expr)`	Transform array
`length`	Count
`keys`	Object keys
`to_entries`	Object to key-value pairs
`@csv`	Output as CSV
`@tsv`	Output as TSV
`//`	Default value

jq has a learning curve, but it is one of those tools where the investment pays off within a week. Every time you find yourself writing a Python script just to parse some JSON, remember that jq probably does it in one line.