Columns and Sorting: cut, sort and uniq

cut: take out columns

Lots of files are made of columns with a separator. In /etc/passwd it's a colon, in a CSV file it's a comma. cut -d sets the separator (delimiter) and -f picks the fields:

head -n 3 /etc/passwd
cut -d: -f1 /etc/passwd | head -n 3
cut -d: -f1,7 /etc/passwd | head -n 3

In the terminal

student@vulntech:~$ head -n 3 /etc/passwd
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
student@vulntech:~$ cut -d: -f1 /etc/passwd | head -n 3
root
bin
daemon
student@vulntech:~$ cut -d: -f1,7 /etc/passwd | head -n 3
root:/bin/bash
bin:/sbin/nologin
daemon:/sbin/nologin

cat scores.csv
cut -d, -f1,3 scores.csv

In the terminal

student@vulntech:~$ cat scores.csv
name,team,points
Ada,red,42
Bo,blue,7
Cy,red,15
Di,green,108
Eli,blue,23
Flo,green,9
Gus,red,15
student@vulntech:~$ cut -d, -f1,3 scores.csv
name,points
Ada,42
Bo,7
Cy,15
Di,108
Eli,23
Flo,9
Gus,15

-c cuts by character position instead:

cut -c 1-3 scores.csv | head -n 3

In the terminal

student@vulntech:~$ cut -c 1-3 scores.csv | head -n 3
nam
Ada
Bo,

sort: put lines in order

sort fruit.txt
sort -u fruit.txt

In the terminal

student@vulntech:~$ sort fruit.txt
apple
apple
apple
banana
fig
fig
pear
student@vulntech:~$ sort -u fruit.txt
apple
banana
fig
pear

Sort the players by points, highest first. tail -n +2 skips the header line:

tail -n +2 scores.csv | sort -t, -k3 -n
tail -n +2 scores.csv | sort -t, -k3 -nr | head -n 1

In the terminal

student@vulntech:~$ tail -n +2 scores.csv | sort -t, -k3 -n
Bo,blue,7
Flo,green,9
Cy,red,15
Gus,red,15
Eli,blue,23
Ada,red,42
Di,green,108
student@vulntech:~$ tail -n +2 scores.csv | sort -t, -k3 -nr | head -n 1
Di,green,108

uniq: repeated lines

uniq drops a line that's the same as the one just before it, so it only works on sorted input. uniq -c counts each group:

sort fruit.txt | uniq -c

In the terminal

student@vulntech:~$ sort fruit.txt | uniq -c
      3 apple
      1 banana
      2 fig
      1 pear

The top-10 pipeline

This combination is worth learning by heart. It answers what appears most often?:

sort fruit.txt | uniq -c | sort -rn
tail -n +2 scores.csv | cut -d, -f2 | sort | uniq -c | sort -rn

In the terminal

student@vulntech:~$ sort fruit.txt | uniq -c | sort -rn
      3 apple
      2 fig
      1 pear
      1 banana
student@vulntech:~$ tail -n +2 scores.csv | cut -d, -f2 | sort | uniq -c | sort -rn
      3 red
      2 green
      2 blue

Think of it like this: sort | uniq -c | sort -rn | head is like counting votes: put identical ballots in piles, count each pile, then line the piles up from biggest to smallest.

Remember: cut -d: -f1 takes columns, sort -n sorts numbers, uniq -c counts repeats, and sort | uniq -c | sort -rn | head makes a top list.