BASH
Extracting Specific Data from Log Files
Master how to parse and extract specific information like IP addresses, request paths, or status codes from log files using powerful bash commands for quick analysis.
#!/bin/bash
# Configuration
LOG_FILE="/var/log/nginx/access.log"
if [ ! -f "$LOG_FILE" ]; then
echo "Error: Log file not found at ${LOG_FILE}"
exit 1
fi
echo "Top 10 Most Frequent IP Addresses:"
awk '{print $1}' "${LOG_FILE}" | sort | uniq -c | sort -nr | head -n 10
echo "
Top 10 Most Requested URLs:"
awk '{print $7}' "${LOG_FILE}" | sort | uniq -c | sort -nr | head -n 10
echo "
Counts of HTTP Status Codes:"
awk '{print $9}' "${LOG_FILE}" | sort | uniq -c | sort -nr
echo "
Requests with 4xx or 5xx errors (first 5 lines):"
awk '($9 ~ /^[45][0-9][0-9]$/)' "${LOG_FILE}" | head -n 5
How it works: This script demonstrates how to extract and analyze common data points from Nginx-style access logs using `awk`, `sort`, `uniq`, and `head`. It provides examples for identifying the most frequent IP addresses, requested URLs, and counts of HTTP status codes. It also shows how to filter for specific error codes. This is an essential skill for web developers to monitor website traffic, identify popular content, debug performance issues, and spot potential security threats by quickly parsing large log files.