Introduction to awk for Data Processing

What is awk?

awk is a programming language designed for text processing and data extraction. It excels at working with structured data like CSV files, log files, and command output.

Basic Syntax

awk 'pattern { action }' file

Field Processing

# Print specific columns (space-delimited)
awk '{print $1, $3}' file.txt

# Custom delimiter
awk -F":" '{print $1, $3}' /etc/passwd

# Print last field
awk '{print $NF}' file.txt

Filtering

# Lines matching pattern
awk '/error/' logfile.log

# Numeric comparison
awk '$3 > 100' data.txt

# Combined conditions
awk '$3 > 100 && $5 == "active"' data.txt

Practical Examples

# Sum a column
awk '{sum += $3} END {print "Total:", sum}' data.txt

# Count lines matching pattern
awk '/ERROR/ {count++} END {print count}' app.log

# Calculate averages
awk '{sum += $2; n++} END {print "Average:", sum/n}' scores.txt

# Top memory-consuming processes
ps aux | awk 'NR>1 {print $4, $11}' | sort -rn | head -10

Built-in Variables

• NR — current line number
• NF — number of fields
• FS — field separator
• $0 — entire line