Unit 3: Searching and pipes · Lesson 5 of 6 · 18 min
Quick reports with awk
Total bytes, count statuses and find the busiest hour with one-line awk programs.
mission
- You are
- You are preparing numbers for the morning report.
- Situation
- Management wants the traffic mix for 14 October and how much data the scanner pulled.
- Goal
- Answer with awk one-liners instead of a spreadsheet.
- Hand in
- The status mix, the busiest hour and the bytes sent to the scanner.
How awk reads a line
awk '$9 == 404 {print $7}'
- 1
awkthe program - 2
$9 == 404pattern: which lines - 3
{print $7}action: what to do
Count the statuses
awk '{print $9}' access.log | sort | uniq -c | sort -rn
1183 200
136 304
59 404
Sum the bytes per client
awk can keep totals in an array and print them at the END.
awk '{bytes[$1]+=$10} END {for (ip in bytes) print bytes[ip], ip}' access.log | sort -rn | head -n 3 926126 198.51.100.138 892486 198.51.100.210 890481 192.0.2.171 awk '$1 == "203.0.113.45" {sum += $10} END {print sum}' access.log 8153
you should see The scanner pulled about 8 KB. Small, but it included the Git config.
Find the busiest hour
Split on colons instead of spaces and field 2 is the hour.
cut -d: -f2 access.log | sort | uniq -c | sort -rn | head -n 3
98 02
64 14
60 22
you should see The busiest hour is 02:00, in the middle of the night. That is when the scanner ran.
Which missing pages were asked for most
awk '$9 == 404 {print $7}' access.log | sort | uniq -c | sort -rn | head -n 5
5 /careers
4 /pricing.html
4 /apple-touch-icon.png
3 /blog/old-post
2 /wp-login.php
defender tip Real customers hitting /careers is a broken link worth fixing. Not every 404 is an attack.
What a defender takes away
- 1awk is pattern plus action per line.
- 2Arrays and END turn awk into a quick report tool.
- 3Always ask what a spike in time lines up with.
Questions
0 of 3 solved
Answer each one from what you find on your lab machine.