awk in 10 Minutes With Practical Examples

#awk#cli#linux

awk is one of those tools that looks intimidating until you learn five patterns. Then you use it every day. I am not going to explain the full language. Here are the patterns that actually matter.

Pattern 1: Print a Column

awk splits each line into fields. $1 is the first, $2 is the second, $0 is the whole line:

# Print the first column of ps output
ps aux | awk '{print $1}'

# Print columns 1 and 11 (user and command)
ps aux | awk '{print $1, $11}'

The default field separator is whitespace. Change it with -F:

# Parse /etc/passwd (colon-separated)
awk -F: '{print $1, $7}' /etc/passwd

# Parse CSV
awk -F, '{print $2}' data.csv

Pattern 2: Filter Rows

Put a condition before the action:

# Lines where column 3 is greater than 100
awk '$3 > 100' data.txt

# Lines containing "ERROR"
awk '/ERROR/' app.log

# Lines where the first field matches a pattern
awk '$1 ~ /^192\.168/' access.log

Combine conditions:

# Memory usage over 50% AND the process is not root
ps aux | awk '$4 > 50.0 && $1 != "root" {print $1, $4, $11}'

Pattern 3: Sum a Column

# Total bytes transferred (column 10 in access logs)
awk '{sum += $10} END {print sum}' access.log

# Average response time
awk '{sum += $NF; n++} END {print sum/n}' response_times.log

END runs after all lines are processed. NF is the number of fields on the current line, so $NF is the last field.

Pattern 4: Count and Group

# Count HTTP status codes
awk '{count[$9]++} END {for (c in count) print c, count[c]}' access.log | sort -rn -k2

# Output:
# 200 45231
# 304 12044
# 404 891
# 500 23

This uses an associative array. count[$9]++ increments the count for each unique value in column 9.

Top 10 IP addresses by request count:

awk '{count[$1]++} END {for (ip in count) print count[ip], ip}' access.log | sort -rn | head -10

Pattern 5: Transform Fields

# Convert bytes to megabytes
awk '{printf "%s %.1fMB\n", $1, $2/1048576}' sizes.txt

# Add a prefix to every line
awk '{print "[INFO]", $0}' app.log

# Swap two columns
awk '{temp=$1; $1=$2; $2=temp; print}' data.txt

Real-World Examples

Parse Docker ps output

docker ps --format '{{.Names}}\t{{.Status}}\t{{.Ports}}' | \
  awk -F'\t' '{printf "%-20s %-15s %s\n", $1, $2, $3}'

Find the largest Docker images

docker images --format '{{.Repository}}:{{.Tag}}\t{{.Size}}' | \
  awk -F'\t' '{
    size = $2
    if (size ~ /GB/) {
      gsub(/GB/, "", size)
      mb = size * 1024
    } else if (size ~ /MB/) {
      gsub(/MB/, "", size)
      mb = size
    } else {
      mb = 0
    }
    printf "%8.0fMB  %s\n", mb, $1
  }' | sort -rn | head -10

Calculate uptime percentage

Given a log with timestamps and status:

awk '
  /UP/   { up++ }
  /DOWN/ { down++ }
  END {
    total = up + down
    if (total > 0)
      printf "Uptime: %.2f%% (%d/%d checks)\n", (up/total)*100, up, total
  }
' monitoring.log

Extract unique error messages

awk '/ERROR/ {
  msg = ""
  for (i=4; i<=NF; i++) msg = msg " " $i
  errors[msg]++
}
END {
  for (e in errors) printf "%5d %s\n", errors[e], e
}' app.log | sort -rn | head -20

Built-in Variables

VariableMeaning
NRCurrent line number
NFNumber of fields on current line
FSField separator
OFSOutput field separator
$0Entire line
$1..$nIndividual fields
FILENAMECurrent filename

When to Use awk vs. Other Tools

  • cut: When you just need a column and nothing else. cut -d: -f1 /etc/passwd is simpler than awk for trivial extraction.
  • grep: When you just need to filter lines by a pattern.
  • sed: When you need to do find-and-replace on text.
  • awk: When you need to do math, grouping, conditional logic, or work with multiple fields at once.

The mental model is simple. awk reads lines, splits them into fields, and lets you write rules about what to do with each line. Once that clicks, you start seeing awk solutions everywhere.

← all articles wleeaf.dev →