BASH

Downloading Files Conditionally with curl (If-Modified-Since)

Download files from a URL in Bash using 'curl', conditionally updating only if the remote file is newer, optimizing bandwidth and speeding up deployments.

#!/bin/bash

DOWNLOAD_URL="https://raw.githubusercontent.com/username/repo/main/some_file.txt" # Replace with a real URL
LOCAL_PATH="./downloaded_file.txt"

# Ensure the URL is accessible for testing purposes. For example:
# DOWNLOAD_URL="https://example.com/some_asset.zip"
# If you don't have a specific file, you can use a generic test file:
# DOWNLOAD_URL="https://www.google.com/robots.txt"

# Check if the file exists locally
if [[ -f "$LOCAL_PATH" ]]; then
  echo "Local file '$LOCAL_PATH' exists. Checking for newer version..."
  # Download only if newer than local file
  curl -s -L -R -z "$LOCAL_PATH" -o "$LOCAL_PATH" "$DOWNLOAD_URL"
  if [[ $? -eq 0 ]]; then
    echo "File downloaded/updated (if newer)."
  else
    echo "No update needed or download failed."
  fi
else
  echo "Local file '$LOCAL_PATH' does not exist. Downloading..."
  # Download the file directly
  curl -s -L -R -o "$LOCAL_PATH" "$DOWNLOAD_URL"
  if [[ $? -eq 0 ]]; then
    echo "File downloaded successfully."
  else
    echo "Error: Failed to download file."
    exit 1
  fi
fi

echo "
Contents of $LOCAL_PATH:"
cat "$LOCAL_PATH"

# Clean up example file
rm -f "$LOCAL_PATH"
How it works: This snippet demonstrates how to use `curl` to download a file from a URL, but only if the remote version is newer than a locally existing one. The `-z "$LOCAL_PATH"` option (short for `--time-cond`) sends an `If-Modified-Since` header with the local file's modification time. If the remote file hasn't changed since then, the server will respond with `304 Not Modified`, and `curl` will not re-download the file. This is highly efficient for recurring tasks like fetching assets, build artifacts, or configuration files, saving bandwidth and reducing processing time, especially in automated deployment pipelines. The `-s` for silent, `-L` for follow redirects, and `-R` for remote-time are also commonly useful.

Need help integrating this into your project?

Our team of expert developers can help you build your custom application from scratch.

Hire DigitalCodeLabs