awk + sed Cheatsheet — 80+ One-Liners for Text Processing on Linux and macOS

Replace the FIRST occurrence of `old` on each line with `new`. Default sed substitution: per-line, first match only.

old old old
foo old bar old

new old old
foo new bar old

sed 's/cat/dog/' pets.txt

echo 'hello world' | sed 's/world/sed/'

Global replace — substitute EVERY occurrence on each line, not just the first.

old old old

new new new

⚠ Common pitfall: Forgetting g is the #1 sed surprise. `sed s/,/|/` on a CSV row swaps only the first comma.

sed 's/ /_/g' file.txt

sed -i 's/http:/https:/g' index.html

Replace only the Nth match on each line (here the 2nd). Numeric flag.

a a a a

a new a a

sed 's/,/|/3' csv-row.txt   # change only the 3rd comma

Combine N and g: replace from the Nth match to the end of the line.

x x x x x

x x new new new

sed 's/ /_/2g' file   # keep first space, underscore the rest

Print only line 5. `-n` mutes the default print, `p` prints the address.

a
b
c
d
e
f

e

sed -n '5p' /etc/passwd

sed -n '100,200p' big.log   # range

Print line range 10–20 inclusive. Faster than head + tail.

(file with 30 lines)

(lines 10 through 20)

sed -n '10,20p' big.log

sed -n '/START/,/END/p' file   # regex range

Delete line 5 and print the rest.

a
b
c
d
e

a
b
c
d

sed '1d' file.csv         # strip header row

sed '$d' file           # drop last line

Delete empty lines (zero characters between start ^ and end $).

a

b

c

a
b
c

⚠ Common pitfall: This only catches truly empty lines. For lines with spaces/tabs use `/^[[:space:]]*$/d`.

sed '/^$/d' notes.md

sed '/^[[:space:]]*$/d' file   # also blanks

Delete every line that matches a regex.

ok 1
err 2
ok 3
err 4

ok 1
ok 3

sed '/^#/d' config           # drop comment lines

sed '/DEBUG/d' app.log

Insert a new line BEFORE line 5. The text after i\\ becomes the inserted line.

a
b
c
d
e
f

a
b
c
d
NEW LINE
e
f

sed '1i\
#!/bin/bash' script.sh   # add shebang at top

Append a new line AFTER line 5.

1
2
3
4
5
6

1
2
3
4
5
appended text
6

sed '$a\
EOF marker' file   # append at end

Change (replace entirely) the matching line(s) with new text.

a
b
old line
d

a
b
replacement
d

sed '/^TODO/c\
DONE' tasks.md

Transliterate characters: every `a`→`x`, `b`→`y`, `c`→`z`. Like `tr` but inside sed.

cab cab

zxy zxy

sed 'y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/' file   # upper-case

Extended regex with -E: replace any run of digits with N. -E enables ERE (no backslash before +, ?, |).

order 42 then 1337

order N then N

⚠ Common pitfall: GNU sed accepts -r as an alias for -E; BSD only takes -E. Always write -E for portability.

sed -E 's/(\w+)@(\w+)/REDACTED/g' file

Print the first 5 lines — a sed equivalent of `head -5`. -n mutes default output, the range address picks lines 1 through 5.

a
b
c
d
e
f
g

a
b
c
d
e

sed -n '1,5p' file

sed '5q' file   # quit after line 5, even faster

Print the first 5 lines then quit. Faster than a range on huge files because sed stops reading after line 5.

(file with 1M lines)

(first 5 lines only)

sed '10q' big.log

sed '/READY/q' boot.log   # quit at first READY

GNU step address: print line 2, then every 3rd line after (2, 5, 8 …). The `first~step` form is a GNU extension.

l1
l2
l3
l4
l5
l6
l7
l8

l2
l5
l8

⚠ Common pitfall: The `~` step address is GNU-only. BSD/macOS sed rejects it; use awk `NR%3==2` for portability.

sed -n '1~2p' file   # odd lines

sed -n '0~2p' file   # even lines (GNU)

Delete a line range (2 through 4) and print the rest.

a
b
c
d
e

a
e

sed '1,10d' file   # drop first 10 lines

sed '2,$d' file   # keep only line 1

Print every line once. With -n the default print is off, so `p` restores it — equivalent to `cat`.

a
b

a
b

⚠ Common pitfall: Drop the -n and `sed p` prints every line TWICE — default print plus the explicit p. A classic accident.

sed -n 'p' file   # = cat

sed 'p' file   # every line doubled

Case-INSENSITIVE substitution with the I flag (GNU). Matches Old, OLD, old all the same.

Old OLD old

new new new

⚠ Common pitfall: The I flag is GNU sed. BSD/macOS does not support it — fall back to a character class like `s/[Oo]ld/new/g`.

sed 's/error/ERR/Ig' app.log

Wrap each whole line in brackets. `&` in the replacement means "the entire matched text".

hello
world

[hello]
[world]

sed 's/[0-9]*/<&>/' file   # bracket the leading number

Duplicate the content of each line with a back-reference. `\(.*\)` captures the line, `\1\1` prints it twice.

ab

abab

sed -E 's/(.*)/\1 \1/' file   # space between copies, ERE

Print the line matching /foo/ plus the next 2 lines. The `+N` relative range is a GNU extension.

a
foo
b
c
d

foo
b
c

⚠ Common pitfall: The `addr,+N` form is GNU-only. On BSD use `sed -n "/foo/{N;N;p;}"` to grab a fixed number of following lines.

grep -A2 foo file   # the grep equivalent

Keep only lines matching /foo/ by deleting the rest. The `!` negates the address, so non-matching lines get deleted.

foo 1
bar 2
foo 3

foo 1
foo 3

sed '/^#/!d' config   # keep only comment lines

Strip LEADING whitespace from each line (left-trim). Pairs with the trailing-whitespace trim.

    hi
	bye

hi
bye

sed -E 's/^[[:space:]]+//' file   # POSIX class, portable

Print lines matching /foo/ case-insensitively (GNU I address modifier).

Foo
bar
FOO

Foo
FOO

sed -n '/error/Ip' app.log

Edit the file IN PLACE — overwrites the original. GNU sed syntax.

⚠ Common pitfall: On macOS / BSD this fails. BSD sed needs an extension argument: `sed -i "" "s/a/b/" file` (empty string) or `sed -i.bak`. Portable: `sed -i.bak` works everywhere.

sed -i 's/foo/bar/g' file.txt

sed -i.bak 's/foo/bar/g' file.txt   # portable

BSD/macOS in-place edit. The `""` is the (empty) backup extension and is REQUIRED on macOS.

hello old world

(file rewritten to: hello new world)

sed -i '' 's/http:/https:/g' index.html

Regex address range — print lines from the first /start/ match to the next /end/ match.

a
start
b
c
end
d

start
b
c
end

sed -n '/BEGIN CERTIFICATE/,/END CERTIFICATE/p' chain.pem

When line matches /pattern/, append the next line into pattern space with N, then print both as one chunk.

sed -n '/ERROR/{N;p;}' app.log   # show ERROR plus the next line

Join every pair of lines with a space — read 2 lines into pattern space (N) then replace the embedded newline.

name
Lei
role
dev

name Lei
role dev

sed 'N;s/\n/ /' two-line-records.txt

Delete the BODY of a block, keeping the markers. `//!d` means "delete every line that does NOT match the most recent regex".

sed '/^BEGIN/,/^END/{//!d}' file

Classic sed: reverse the file line by line using hold space (h, G).

1
2
3

3
2
1

sed '1!G;h;$!d' file   # like `tac`

Run multiple substitutions in one pass with -e. Order matters.

sed -e 's/foo/bar/g' -e 's/baz/qux/g' file

Run sed commands from a file (one command per line). Easier to version-control complex pipelines.

sed -f rewrites.sed input.txt > output.txt

Collapse every line of the file into a single comma-separated row (hold-space accumulator pattern).

a
b
c

a, b, c

sed 'H;1h;$!d;x;s/\n/, /g' names.txt

Use | as the delimiter so you do not have to escape slashes in URLs. Back-reference \1 keeps the host.

http://example.com/page

https://example.com/page

⚠ Common pitfall: sed picks the delimiter from whatever follows `s`. Pick anything not in your data: |, #, !, @ are common.

sed 's|/usr/local|/opt|g' paths.conf

Edit every .md in the directory in place, leaving .md.bak backups. Portable across GNU and BSD.

⚠ Common pitfall: If the glob expands to many files, the backup count explodes. Clean them with `rm *.md.bak` after verifying the rewrite.

sed -i.bak 's/2025/2026/g' docs/*.md && rm docs/*.md.bak

Read the whole file into pattern space (loop :a / N / branch ba until last line) then replace all newlines with spaces — collapse to one line.

a
b
c

a b c

sed ':a;N;$!ba;s/\n/,/g' file   # comma-join the whole file

Reformat ISO dates (YYYY-MM-DD) to DD/MM/YYYY using three capture groups and back-references.

2026-05-30

30/05/2026

sed -E 's/(....)-(..)-(..)/\2\/\3\/\1/' dates.txt   # to MM/DD/YYYY

Print lines in unambiguous form: tabs show as \t, line ends as $, non-printing bytes as octal escapes. Great for spotting hidden characters.

(line with a tab)

col1\tcol2$

sed -n 'l' suspicious.txt   # reveal tabs and CR

cat -A file   # similar idea

Strip trailing carriage returns — convert Windows CRLF line endings to Unix LF.

line\r

line

⚠ Common pitfall: BSD sed does not understand `\r`. On macOS use a literal CR: type `sed $'s/\r$//'` or `tr -d '\r'` instead.

sed -i 's/\r$//' file.txt   # dos2unix, GNU

sed 's/$/\r/' file   # unix2dos

For each line matching /foo/, print its line number (=) then the line itself (p) — grep with line numbers, two outputs per hit.

a
foo
b
foo

2
foo
4
foo

sed -n '/ERROR/{=;p;}' app.log

Squeeze consecutive blank lines down to a single blank line. Reads the next line on a blank, deletes if both are blank, loops with D.

a



b

a

b

cat -s file   # the simpler equivalent on GNU coreutils

Delete every other line, keeping the odd ones. `n` prints the current line and loads the next, `d` deletes that next line.

1
2
3
4
5

1
3
5

sed 'n;d' file   # keep odd lines

sed '1d;n;d' file   # keep even lines

Convert camelCase to snake_case-ish by inserting an underscore between a lowercase and an uppercase letter (then lowercase separately).

fooBarBaz

foo_Bar_Baz

sed -E 's/([a-z])([A-Z])/\1_\2/g' file | sed 's/.*/\L&/'   # full snake_case, GNU \L

Split each character onto its own line by inserting a newline after every character (GNU sed honors \n in the replacement).

abc

a
b
c

⚠ Common pitfall: A literal `\n` in the replacement is a GNU extension. On BSD insert a real newline with a backslash-newline in the script.

sed 's/\(.\)/\1\n/g' file   # one char per line, GNU

Delete every even-numbered line (keep odd), using the GNU `first~step` address starting at 0 with step 2.

1
2
3
4

1
3

⚠ Common pitfall: GNU-only address. BSD/macOS needs awk `NR%2` instead.

sed '1~2d' file   # delete odd lines, keep even

Squeeze runs of the SAME character down to one — `(.)` captures a char, `\1+` matches its repeats, replacement keeps one.

aaabbbcccd

abcd

echo 'heelllooo' | sed -E 's/(.)\1+/\1/g'   # helo

Swap the first two lines using hold space: stash line 1 (h;d), then on line 2 append it (H) and swap (x).

one
two
three

two
one
three

sed -e '1{h;d;}' -e '2{H;x;}' file

Drop the last line of a file. `$` is the address "last line".

a
b
c
d

a
b
c

sed '$d' file.log

Count the lines in a file. `=` prints the current line number; `$` limits to the last line.

a
b
c
d

4

sed -n '$=' file

Add line numbers to a file (poor-man's `nl`). The first sed inserts a number line; the second joins.

a
b
c

1	a
2	b
3	c

sed = file | sed 'N;s/\n/\t/'

Like `grep pattern file` — print only matching lines. Useful when sed is already in your pipeline.

sed -n '/ERROR/p' app.log

Print each email header block — from the From: line up to the next blank line.

sed -n '/^From: /,/^$/p' inbox.mbox

Print line NUMBERS of matches (without the content).

a
foo
b
foo bar
c

2
4

sed -n '/TODO/=' notes.md

Indent every line by four spaces (useful when pasting into Markdown code blocks).

a
b

    a
    b

sed 's/^/    /' snippet.txt

Strip trailing whitespace from every line — kill those invisible spaces at EOL.

sed -i 's/[ \t]*$//' src/**/*.py

Reverse the file line order — same idea as `tac`, but works on BSD where `tac` is missing.

sed -n '1!G;h;$p' file

Double-space a file — append an empty line after every line.

a
b

a

b

sed 'G' notes.txt

Print non-empty lines only — short way to skip blank lines.

sed -n '/^$/!p' file

Join ALL lines of the file into one space-separated line (sed loop using label :a and branch ta).

sed -e :a -e '$!N;s/\n/ /;ta' lines.txt

Print just the second line. Swap the number to grab any single line.

a
b
c

b

sed -n '1p' file   # first line

sed -n '3p;7p' file   # lines 3 and 7

Print the last line of a file (a sed `tail -1`). `$` is the last-line address.

a
b
c

c

sed -n '$p' file

Prefix every line with "> " — quote text like an email reply.

hi
there

> hi
> there

sed 's/^/> /' quote.txt

Append a semicolon to the end of every line. `$` here is the end-of-line anchor.

a
b

a;
b;

sed 's/$/,/' col.txt   # turn a column into CSV-ish

Squeeze runs of spaces into a single space — quick whitespace normalizer.

a    b   c

a b c

sed -E 's/ +/ /g' file   # same with ERE

Print only the first and last line of a file in one pass.

a
b
c
d

a
d

sed -n '1p;$p' big.log

Convert tabs to commas — a quick TSV-to-CSV when no field contains a comma.

a	b	c

a,b,c

⚠ Common pitfall: BSD sed does not expand `\t`; on macOS press a literal tab or use `sed $'s/\t/,/g'`.

sed $'s/\t/,/g' in.tsv > out.csv   # portable

Print the line numbers of all blank lines — handy for finding paragraph breaks.

a

b


c

2
4
5

sed -n '/^$/=' essay.txt

After every line matching /foo/, read and insert the entire contents of insert.txt. The `r` command appends a file.

sed '/<!-- INCLUDE -->/r snippet.html' page.html

Print the lines BETWEEN /foo/ and /bar/, excluding the two boundary lines themselves.

x
foo
A
B
bar
y

A
B

sed -n '/BEGIN/,/END/{/BEGIN/d;/END/d;p;}' block.txt

Remove consecutive DUPLICATE lines (a sed `uniq`). Compares each line with the next via a back-reference.

a
a
b
b
b
c

a
b
c

sed '$!N;/^\(.*\)\n\1$/!P;D' file   # = uniq for adjacent dups

Keep everything before the LAST colon on each line (greedy `.*` eats up to the final colon).

a:b:c

a:b

sed 's/:[^:]*$//' file   # same: drop after last colon

GNU `sed -i "…"` works; BSD/macOS `sed -i "…"` requires an extension argument right after -i (use `""` for no backup).

⚠ Common pitfall: A non-portable `sed -i` is the single most common cross-platform script bug. Use `sed -i.bak` everywhere — both GNU and BSD accept it, you can `rm *.bak` after.

sed -i.bak 's/old/new/g' file   # portable

case $(sed --version 2>&1) in *GNU*) SED='sed -i' ;; *) SED="sed -i ''" ;; esac

sed defaults to BRE (basic regex): +, ?, |, (), {} are LITERAL unless escaped. With -E you get ERE where they are metacharacters.

⚠ Common pitfall: `sed s/foo+/bar/` does NOT match `foooo` — it matches the literal string `foo+`. Either escape `\+` (GNU) or use `sed -E s/foo+/bar/`.

sed -E 's/(foo)+/bar/' file   # works on both GNU and BSD

When the pattern or replacement contains `/`, switch the delimiter. sed lets you pick almost any character right after `s`.

⚠ Common pitfall: `sed s/http:\/\/a\.com/b.com/g` is unreadable. `sed s|http://a.com|b.com|g` is the same thing — readable.

sed 's|http://|https://|g' file

sed 's#/var/log#/opt/log#g' config

sed regex is greedy and there is no `?` lazy modifier in BRE/ERE. `.*` will grab the LONGEST run, not the shortest.

⚠ Common pitfall: `sed "s/<.*>//" "<a><b>"` deletes the entire `<a><b>` — not just `<a>`. Use `[^>]*` instead: `s/<[^>]*>//`.

sed 's/<[^>]*>//g' page.html   # strip tags, no greediness trap

In BRE (default sed), capture groups are `\(...\)` and back-references are `\1`. ERE (-E) uses bare `(…)`.

sed 's/\(foo\)\(bar\)/\2\1/' file        # BRE

sed -E 's/(foo)(bar)/\2\1/' file       # ERE, clearer

After N pulls a second line into pattern space, `^` and `$` still match only the very start and end — NOT each embedded line. Use `\n` to target line boundaries inside.

⚠ Common pitfall: People expect `s/^/> /g` after N to prefix both lines. It only prefixes the first. Match `\n` to reach the second: `s/\n/\n> /`.

sed 'N;s/^/> /;s/\n/\n> /' file

If the input file does not end in a newline, sed preserves that — the output also lacks a final newline, which can break later tools that expect one.

⚠ Common pitfall: A no-newline-at-EOF file fed to a loop can silently drop the last record. Normalize with `sed -e '$a\' file` (GNU) or `printf '\n' >> file`.

[ -n "$(tail -c1 file)" ] && echo >> file   # add newline if missing

Inside single quotes the shell passes the sed script verbatim, but you cannot put a single quote in it. Close-quote, escape, reopen: `'\''`.

⚠ Common pitfall: To replace a straight quote you write a four-char dance: `sed 's/x/'\''/g'`. Or switch the script to double quotes and escape `$` and backslashes instead.

sed "s/x/'/g" file   # double-quote the whole script

By default sed processes one line at a time, so a regex cannot span a newline unless you first pull more lines into pattern space with N or a loop.

⚠ Common pitfall: `sed "s/foo\nbar/X/"` never matches by default because foo and bar are on different lines. Slurp the file first: `sed ":a;N;$!ba;s/foo\nbar/X/"`.

sed ':a;N;$!ba;s/foo\nbar/X/g' file   # cross-line match

In the replacement, `&`, `\`, and the delimiter are special. A literal ampersand must be escaped as `\&`, or `&` re-inserts the whole match.

⚠ Common pitfall: `sed "s/price/price & tax/"` becomes "price price tax" — the bare & expands. Escape it: `s/price/price \& tax/`.

sed 's/x/A \& B/' file   # literal ampersand

Print the first field of every line. Fields are split on whitespace by default.

alice 30 dev
bob 28 ops

alice
bob

awk '{ print $1 }' /etc/passwd   # actually need -F:

ls -l | awk '{ print $9 }'

Print the LAST field of every line. NF is the number-of-fields variable.

a b c
1 2 3 4

c
4

ls -l | awk '{ print $NF }'   # filenames

Print line 5 only. NR is the running record number (= line number for default RS).

a
b
c
d
e
f

e

awk 'NR==5'  file

awk 'NR>=10 && NR<=20' file   # range

Count lines (equivalent to `wc -l`) — print the final value of NR in the END block.

a
b
c

3

awk 'END { print NR }' file

Use `,` as the field separator (CSV) and print column 2. -F sets FS at startup.

alice,30,dev
bob,28,ops

30
28

⚠ Common pitfall: -F splits on a LITERAL char by default; -F"\t" needs the quotes. For regex separators use -F"[,;]" (with -E semantics inside awk).

awk -F, '{ print $1, $3 }' data.csv

awk -F: '{ print $1 }' /etc/passwd

Run an action before any input. Useful for headers, constants, or as a stand-alone calculator with no input file.

hi

awk 'BEGIN { print 2+2 }'   # 4

awk 'BEGIN { for(i=1;i<=5;i++) print i }'

Print lines matching a regex (essentially `grep pattern`). The default action is print, so `awk /pattern/` works too.

awk '/ERROR/' app.log

awk '/^200/ { print $7 }' access.log

Range pattern — print every line from the first `/start/` match through the next `/end/` match (inclusive).

awk '/BEGIN/,/END/' file

Formatted output with printf — left-pad column 1 to width 10, right-pad column 2 as a 5-wide integer.

alice 30
bob 285

alice          30
bob           285

awk '{ printf "%s=%d\n", $1, $2 }' kv.txt

Re-emit a CSV with `|` as the separator. The trick: OFS only kicks in when you REASSIGN a field, so `$1=$1` triggers it.

a,b,c
1,2,3

a|b|c
1|2|3

⚠ Common pitfall: Without `$1=$1` you just get the original line back. That trap eats 30 minutes the first time.

awk 'BEGIN{FS=",";OFS="\t"} {$1=$1;print}' a.csv > a.tsv

Pass a shell variable into awk with -v. Cleaner than embedding shell quoting inside the awk program.

awk -v user="$USER" '{ print user, $1 }' file

awk -v d=$(date +%F) 'BEGIN{ print d }'

Print lines with at least one field — i.e. skip blank lines. NF is truthy when > 0.

a

b

c

a
b
c

awk 'NF' messy.txt > clean.txt

Prepend a line number to each line. Cheap `cat -n`.

a
b
c

1 a
2 b
3 c

awk '{ print NR, $0 }' notes.md

Swap the first two columns. Comma in print inserts OFS (a space by default) between them.

alice 30
bob 28

30 alice
28 bob

awk '{print $2, $1}' names.txt

Print lines that have more than 3 fields. Useful to spot malformed rows wider than expected.

a b c
a b c d e

a b c d e

awk -F, 'NF != 5' data.csv   # rows that are not exactly 5 columns

Print rows where field 2 equals the string "dev". String comparison needs the value in double quotes.

alice dev
bob ops
cara dev

alice dev
cara dev

awk '$3 == "200"' access.log   # exact status match

Print rows where field 1 matches a regex (here: starts with `a`). `~` is the match operator, `!~` is no-match.

alice 1
bob 2
amy 3

alice 1
amy 3

awk '$0 !~ /DEBUG/' app.log   # drop DEBUG lines

awk as a calculator: `**` (and `^`) is exponentiation. Prints 1024.

1024

awk 'BEGIN { print sqrt(2), 7%3, int(3.9) }'   # 1.41421 1 3

Print each line with its per-file line number (FNR) and the source file name (FILENAME). Handy across many input files.

awk '{print FILENAME":"FNR": "$0}' *.log   # grep -Hn style

Print line range 10 to 20 using a numeric NR condition. Equivalent to `sed -n "10,20p"`.

(30-line file)

(lines 10 through 20)

awk 'NR>=10 && NR<=20' big.log

Print the last line of the file. In END, `$0` still holds the final record awk read.

a
b
c

c

awk 'END{print $0}' file   # like tail -1

Split on tabs and print column 1 — the right separator for TSV files.

a	b	c

a

awk -F'\t' '{print $2}' data.tsv

Print rows where the numeric value in field 3 exceeds 100. awk compares numerically when both sides look like numbers.

a 1 50
b 2 150
c 3 99

b 2 150

awk '$NF > 1000' sizes.txt

Blank out field 2. Reassigning a field rebuilds $0 with OFS, leaving a double separator where the field was.

a b c

a  c

⚠ Common pitfall: This leaves a double space, not a removed column. To truly drop a field you must shift the rest down or rebuild the record manually.

awk '{$2=$3; $3=""; print}' file   # crude column drop

Read an environment variable through the built-in ENVIRON array — no -v plumbing needed.

awk 'BEGIN { print ENVIRON["USER"], ENVIRON["PWD"] }'

Count occurrences of column 1 — classic group-by. Associative array indexed by the value.

apple
banana
apple
cherry
banana
apple

apple 3
banana 2
cherry 1

awk '{ c[$7]++ } END { for (k in c) print k, c[k] }' access.log   # URL frequency

Two-file join — read map.txt into the array `a`, then enrich each line of data.txt with the matching value. FNR==NR is true only for the first file.

map.txt: alice dev\nbob ops\ndata.txt: alice 30\nbob 28

alice 30 dev
bob 28 ops

awk 'NR==FNR{a[$1]=$2;next} {print $0, a[$1]}' users.tsv logins.tsv

Deduplicate while PRESERVING ORDER (unlike `sort -u`). `seen[$0]++` returns 0 the first time (which is false → negated to true → print).

a
b
a
c
b
d

a
b
c
d

awk '!seen[$0]++' urls.txt

Average column 3 — accumulate then divide by NR. If you want only non-blank rows in the average, use `NF { s+=$3; n++ } END { print s/n }`.

a 1 10
b 2 20
c 3 30

20

awk '{ s+=$3 } END { print s/NR }' scores.txt

Find the row with the MAX value in column 2. Streaming max — single pass, O(1) memory.

a 10
b 50
c 30

b 50

awk 'BEGIN{m=-1} $2>m{m=$2;line=$0} END{print line}' scores.txt

split() a string into an array. Here: take each / -separated path and print the last segment (basename).

/a/b/c.txt
/x/y

c.txt
y

awk '{ n=split($0,p,"/"); print p[n] }' paths.txt

gsub() = global substitution. Replace all digit runs with N. Trailing `1` triggers default print.

order 42 then 1337

order N then N

awk '{ gsub(/secret-[a-z0-9]+/, "[REDACTED]") } 1' file

Truncate every line to 80 chars. substr(string, start, length) — 1-indexed start.

awk '{ print substr($0, 1, 80) }' wide.log

Upper-case the entire line. Companion: `tolower()`.

hello
world

HELLO
WORLD

awk '{ print toupper($1), tolower($2) }' data.txt

Convert TSV to CSV in one line. Tab is the input FS, comma is the output OFS, `$1=$1` forces re-emit.

awk -F'\t' 'BEGIN{OFS=","} {$1=$1;print}' in.tsv > out.csv

Print only lines longer than 80 chars. `length` with no argument returns length of $0.

awk 'length > 120' src/**/*.py   # find long lines

Paragraph mode — set RS to empty string and awk treats blank-line-separated blocks as records. $1 is the first line of each paragraph.

awk 'BEGIN { RS=""; FS="\n" } { print $1 }' essay.txt

List human users only — UID column ≥ 1000. : separator, condition before { action }.

awk -F: '$3 >= 1000 && $7 != "/usr/sbin/nologin" { print $1 }' /etc/passwd

Group-by SUM: total field 2 per distinct field-1 key. The associative-array workhorse for aggregating data.

a 10
b 5
a 20
b 7

a 30
b 12

awk '{rev[$1]+=$3} END{for(k in rev)print k,rev[k]}' sales.txt

One-pass min / max / average of column 1. Seed min high and max low, then fold each value in.

5
2
9
4

min 2 max 9 avg 5

awk '{s+=$1;n++} END{print s/n}' nums.txt   # just the mean

Keep the FIRST row for each distinct value of column 2 (dedupe by a key column, order-preserving).

a x
b x
c y
d y

a x
c y

awk '!seen[$1]++' file   # dedupe by first column

Print total line count and the field count of the LAST line — quick shape check for a table.

a b c
d e f

2 3

awk -F, 'END{print NR" rows, "NF" cols"}' data.csv

Print every line except the last by buffering one line behind. `p` holds the previous record; the last one never gets printed.

a
b
c
d

a
b
c

awk 'NR>1{print prev} {prev=$0}' file   # same idea, named var

Reverse the file by storing every line in an array keyed by NR, then printing from last to first. An awk `tac`.

1
2
3

3
2
1

awk '{a[NR]=$0} END{for(i=NR;i;i--)print a[i]}' file

Transform one CSV column in place: upper-case field 2 and re-emit the whole row. Setting FS=OFS keeps the comma format.

a,b,c
d,e,f

a,B,c
d,E,f

awk 'BEGIN{FS=OFS=","}{$3=$3*1.1}1' prices.csv   # bump column 3 by 10%

Extract the first number on each line. match() sets RSTART and RLENGTH, which substr() then slices out.

order 42 total
item 7

42
7

awk 'match($0,/[A-Z]{3,}/){print substr($0,RSTART,RLENGTH)}' file

Normalize internal whitespace on non-blank lines (the $1=$1 rebuild collapses runs of FS), while leaving blank lines untouched.

a   b   c

x  y

a b c

x y

awk '{$1=$1}1' file   # squeeze whitespace everywhere

Reverse the field order on each line by printing fields from NF down to 1, joining with OFS.

a b c

c b a

awk '{for(i=NF;i>=1;i--)printf "%s%s",$i,(i>1?" ":"\n")}' file

Print a block between markers with a flag variable — more flexible than a range address because you control whether the markers print.

x
start
A
end
y

start
A
end

awk '/BEGIN/{f=1;next} /END/{f=0} f' file   # exclude both markers

Split a file into 100-line chunks by redirecting print to a computed filename per record (an awk `split`).

awk '{print > ("chunk_" int(NR/1000) ".txt")}' big.log

Sum column 1. The canonical "awk over numbers" recipe.

10
20
30

60

awk '{s+=$1} END {print s}' nums.txt

Sum column 3 of a CSV, skipping the header row.

awk -F, 'NR>1 {s+=$3} END {print s}' sales.csv

Reorder columns: print col3, col2, col1.

a b c
1 2 3

c b a
3 2 1

awk '{print $NF, $1}' file   # last column, then first

Pick specific columns (col1 and col3 here). Combined with -F you get cheap cut-with-regex.

awk -F, '{ print $1, $3 }' data.csv

Find processes using >50% CPU and print PID + command. Real pipeline use.

ps aux | awk '$3 > 50 {print $2, $11}'

df -h | awk '$5+0 > 80 {print $0}'   # disks >80% full

Sort lines by length (shortest first). Combine awk with sort.

awk '{print length, $0}' words.txt | sort -n | head

Group-by count + sort by count descending — top-N analysis in one line.

awk '{c[$1]++} END {for(k in c)print c[k],k}' access.log | sort -rn | head

Print even-numbered lines (2, 4, 6 …). Use `% 2 == 1` for odd lines.

a
b
c
d

b
d

awk 'NR%2==1' interleaved.txt   # odd lines

Print the file AND count matches of /pattern/ as you go. Two actions per line, no extra pass.

awk '{print} /ERROR/{c++} END{print c, "errors"}' app.log

Generate 5 random integers in [0, 100). Always call srand() in BEGIN — otherwise awk seeds with a fixed value.

awk 'BEGIN{srand();for(i=0;i<10;i++)print int(rand()*1000)}'

List usernames with their line number — quick `nl` for a colon-separated file.

awk -F: '{print NR": "$1}' /etc/passwd

Count CSV columns by reading the last line's NF. Quick how-wide-is-this-CSV check.

awk -F, 'END {print NF}' wide.csv

Count lines, like `wc -l` but without the leading whitespace some wc versions add.

a
b
c
d

4

awk 'END{print NR}' file

Count total words across the file by summing NF on every line — an awk `wc -w`.

one two
three

3

awk '{w+=NF} END{print w}' essay.txt

Print only the last field of each line — pick out filenames, status codes, or trailing values.

GET /a 200
POST /b 404

200
404

awk '{print $(NF-1)}' file   # second-to-last field

Print just the first line then exit immediately — an efficient `head -1` that stops reading.

a
b
c

a

awk 'NR==1{print;exit}' huge.csv   # grab the header fast

Show the distinct column counts across a CSV — a quick way to detect ragged rows.

awk -F, '{print NF}' data.csv | sort | uniq -c

Count lines matching /foo/ (a `grep -c`). The `+0` forces a numeric 0 instead of an empty string when there are no matches.

foo
bar
foo

2

awk '/ERROR/{c++} END{print c+0}' app.log

Shuffle lines by tagging each with a random number, sorting, then stripping the tag — a portable `shuf`.

awk 'BEGIN{srand()}{print rand(),$0}' file | sort -n | cut -d' ' -f2-

Print the median line by reading the file twice: first pass counts lines, second pass prints the middle one.

awk 'NR==FNR{n++;next} FNR==int((n+1)/2)' file file

Trim leading and trailing spaces from each line in one gsub (alternation of `^ +` and ` +$`), then default-print.

   hi   
  bye

hi
bye

awk '{gsub(/^[ \t]+|[ \t]+$/,"")}1' file   # also tabs

Top values of CSV column 1 by frequency — extract, count, sort descending. The classic log-analysis pipeline.

awk '{print $1}' access.log | sort | uniq -c | sort -rn | head

Generate the numbers 1 to 10 with no input file — an awk `seq`.

1
2
3
4
5
6
7
8
9
10

awk 'BEGIN{for(i=0;i<=100;i+=5)print i}'   # 0,5,10,...,100

Swap the first and last fields of each line using a temp variable, then default-rebuild and print.

a b c
1 2 3

c b a
3 2 1

awk '{t=$1;$1=$NF;$NF=t;print}' file

awk numbers are C doubles. `awk "BEGIN{print 0.1+0.2}"` prints 0.3 because of awk default OFMT — but `printf "%.17f\n", 0.1+0.2` reveals 0.30000000000000004.

⚠ Common pitfall: Never compare floats with ==. Sum, then compare with a tolerance: `if (s > target - 0.001 && s < target + 0.001)`. For exact money math use integers (cents) and divide at the end.

awk 'BEGIN { printf "%.17f\n", 0.1+0.2 }'   # 0.30000000000000004

awk 'BEGIN { OFMT="%.10g"; print 0.1+0.2 }'

Setting OFS does NOTHING unless you reassign at least one field. The output is verbatim until you "touch" a field.

⚠ Common pitfall: Standard fix: `$1=$1` as a no-op reassignment that triggers OFS rewriting. `awk "BEGIN{OFS=\",\"}{$1=$1;print}"`.

awk 'BEGIN{OFS=","}{$1=$1;print}' file

NR keeps counting across files; FNR resets per file. `NR==FNR` is "we are still on the first file" — the classic two-file join idiom.

awk 'NR==FNR{a[$1]=$2;next} {print $0, a[$1]}' map.txt data.txt

awk silently treats field values as either string or number depending on context. To force numeric (e.g. for sorting via comparison), add 0: `$3+0 > 100`.

⚠ Common pitfall: `awk "$3 > 100" data.txt` works if column 3 looks like a number, but `awk "$3 > 100" data-with-units.txt` (rows like "42MB") silently does string comparison and misleads you.

df -h | awk '$5+0 > 80'   # force numeric on "80%"

There are three common awks. gawk (GNU) has the most features (true regex, --csv, networking). mawk is the fastest. BSD awk on macOS is the most limited.

⚠ Common pitfall: `gsub(/regex/, "x", arr[i])` works on gawk but not BSD awk. Scripts that use `length(array)` need gawk. Test with `awk --version` to know which one you have.

awk --version 2>&1 | head -1

brew install gawk   # macOS users wanting gawk

An unset awk variable is both numeric 0 and the empty string, chosen by context. Convenient for counters, but it hides typos in variable names silently.

⚠ Common pitfall: Typo a counter as `coutn[$1]++` and awk happily creates a new array — no error, just a wrong answer. There is no "undefined variable" warning unless you run `gawk --lint`.

gawk --lint '{c[$1]++} END{for(k in c)print k,c[k]}' file

`print` appends ORS (a newline by default); `printf` does NOT add anything — you must include `\n` yourself or output runs together.

⚠ Common pitfall: `awk "{printf \$1}"` glues every field-1 onto one long line with no separators. Add the newline: `printf "%s\n", $1`.

awk '{printf "%s\n", $1}' file   # remember the \n

A single-character -F is taken literally; a multi-character -F value is treated as a regex. So `-F.` splits on ANY char, not on a literal dot.

⚠ Common pitfall: `awk -F. "{print \$1}"` on `a.b.c` prints an empty field because `.` is the regex "any char". Escape it: `-F"\\."` or `-F"[.]"`.

awk -F'[.]' '{print $1}' host.example.com   # split on literal dot

Field values like "010" are treated as decimal 10 in arithmetic, but string-vs-number coercion around leading zeros and bases can surprise you, especially with strtonum().

⚠ Common pitfall: gawk `strtonum("010")` returns 8 (octal) while `"010"+0` returns 10. Mixing the two on the same data gives inconsistent results. Pick one coercion and stick with it.

awk 'BEGIN{print "010"+0}'   # 10

gawk 'BEGIN{print strtonum("010")}'   # 8

Assigning to $0 re-runs field splitting with the current FS, and assigning to any field rebuilds $0 with OFS. The two can clobber each other if you are not careful about order.

⚠ Common pitfall: If you do `$0 = $0 "x"; print $3`, the fields are recomputed from the new $0 — your earlier field edits may vanish. Finish field edits before touching $0 wholesale.

awk '{$0=$0" extra"; print NF}' file   # NF recomputed