Columns and Reports with awk

awk: a small language for columns

awk reads a file line by line and splits each line into fields: $1 is the first, $2 the second, and so on ($0 is the whole line). By default fields are separated by spaces.

awk '{print $3}' server.log
awk '{print $2, $3}' server.log | head -n 3

In the terminal

student@vulntech:~$ awk '{print $3}' server.log
INFO
INFO
WARN
ERROR
INFO
INFO
INFO
error
INFO
WARN
ERROR
student@vulntech:~$ awk '{print $2, $3}' server.log | head -n 3
09:00:01 INFO
09:00:05 INFO
09:02:13 WARN

Combined with the top-list pipeline, that's a log summary in one line:

awk '{print $3}' server.log | sort | uniq -c | sort -rn

In the terminal

student@vulntech:~$ awk '{print $3}' server.log | sort | uniq -c | sort -rn
      6 INFO
      2 WARN
      2 ERROR
      1 error

Other separators: -F

awk -F: '{print $1, $7}' /etc/passwd | head -n 4
awk -F, '{print $1}' scores.csv

In the terminal

student@vulntech:~$ awk -F: '{print $1, $7}' /etc/passwd | head -n 4
root /bin/bash
bin /sbin/nologin
daemon /sbin/nologin
lp /sbin/nologin
student@vulntech:~$ awk -F, '{print $1}' scores.csv
name
Ada
Bo
Cy
Di
Eli
Flo
Gus

Conditions

Put a condition before the braces and awk only handles the lines where it's true. NR is the line number, so NR > 1 skips the header:

awk -F, 'NR > 1 && $3 > 20 {print $1, $3}' scores.csv
awk -F, '$2 == "red"' scores.csv

In the terminal

student@vulntech:~$ awk -F, 'NR > 1 && $3 > 20 {print $1, $3}' scores.csv
Ada 42
Di 108
Eli 23
student@vulntech:~$ awk -F, '$2 == "red"' scores.csv
Ada,red,42
Cy,red,15
Gus,red,15

Totals: END

Code in an END { } block runs once, after the last line, which makes it perfect for totals:

awk -F, 'NR > 1 {total += $3} END {print "Total points:", total}' scores.csv
awk -F, 'NR > 1 {sum[$2] += $3} END {for (team in sum) print team, sum[team]}' scores.csv | sort

In the terminal

student@vulntech:~$ awk -F, 'NR > 1 {total += $3} END {print "Total points:", total}' scores.csv
Total points: 219
student@vulntech:~$ awk -F, 'NR > 1 {sum[$2] += $3} END {for (team in sum) print team, sum[team]}' scores.csv | sort
blue 30
green 117
red 72

Tip: printf in awk lines up columns: awk -F, '{printf "%-6s %5s\n", $1, $3}' scores.csv.

Remember: awk '{print $1}' prints a column, -F, changes the separator, 'condition {action}' filters, and END {} prints totals.