BASH
Processing Log Files and Extracting Data
Learn to efficiently process log files in Bash, using `grep`, `awk`, and `sed` to filter, extract, and transform specific data, invaluable for monitoring and debugging web applications.
#!/bin/bash
LOG_FILE="app.log"
# Create a dummy log file for demonstration
cat << EOF > "$LOG_FILE"
[2023-10-26 10:00:01] INFO User 'alice' logged in from 192.168.1.100
[2023-10-26 10:00:05] ERROR Failed to connect to DB for user 'bob'
[2023-10-26 10:00:10] INFO User 'charlie' registered
[2023-10-26 10:00:15] DEBUG Processing request /api/data
[2023-10-26 10:00:20] ERROR Invalid input for user 'alice'
[2023-10-26 10:00:25] INFO User 'david' logged in from 192.168.1.101
EOF
echo "--- All log entries ---"
cat "$LOG_FILE"
echo ""
echo "--- Filtering ERROR entries ---"
grep "ERROR" "$LOG_FILE"
echo ""
echo "--- Extracting usernames from INFO login entries ---"
grep "INFO User .* logged in" "$LOG_FILE" | awk -F"'" '{print $2}'
echo ""
echo "--- Counting unique usernames who logged in ---"
grep "INFO User .* logged in" "$LOG_FILE" | awk -F"'" '{print $2}' | sort -u | wc -l
echo ""
echo "--- Showing the timestamp and message for ERROR entries (excluding type) ---"
grep "ERROR" "$LOG_FILE" | sed -E 's/\[[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}\] ERROR (.*)/\1/g'
echo ""
echo "--- Extracting all IP addresses from log ---"
grep -oE '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' "$LOG_FILE" | sort -u
echo ""
# Clean up dummy log file
rm "$LOG_FILE"
How it works: This snippet demonstrates powerful log file processing using standard Unix utilities. It creates a dummy `app.log` and then showcases filtering with `grep` (e.g., for "ERROR" entries), extracting specific fields with `awk` (using single quotes as delimiters to get usernames), counting unique entries (`sort -u | wc -l`), and transforming output with `sed` (to remove log level and timestamp prefixes). It also shows how to extract patterns like IP addresses using `grep -oE`, proving invaluable for debugging and analytics.