PYTHON
Fetching All Paginated Results from an API
Learn to iterate through pages of a paginated API to retrieve all available data. This Python snippet demonstrates a common pattern for fetching complete datasets.
import requests
def fetch_all_paginated_data(base_url, api_key):
all_data = []
page = 1
while True:
# Assuming pagination uses 'page' and 'per_page' query parameters
# Or, modify for 'next' link in response, etc.
params = {'page': page, 'per_page': 50}
headers = {'Authorization': f'Bearer {api_key}'} # Or 'x-api-key'
try:
response = requests.get(base_url, params=params, headers=headers)
response.raise_for_status() # Raise HTTPError for bad responses (4xx or 5xx)
data = response.json()
# Adjust based on API response structure
# Example: API returns {'items': [...], 'total_pages': X}
items = data.get('items', [])
total_pages = data.get('total_pages', 1)
if not items: # No more items or empty page
break
all_data.extend(items)
if page >= total_pages: # Reached last page
break
page += 1
except requests.exceptions.RequestException as e:
print(f"API request failed: {e}")
break
except ValueError: # JSON decoding failed
print("Failed to decode JSON response.")
break
return all_data
# Example Usage:
# API_ENDPOINT = 'https://api.example.com/v1/resources'
# YOUR_API_KEY = 'your_secret_api_key'
# all_resources = fetch_all_paginated_data(API_ENDPOINT, YOUR_API_KEY)
# print(f"Fetched {len(all_resources)} resources.")
How it works: This Python snippet demonstrates how to fetch all records from a paginated API endpoint. It uses the `requests` library to make HTTP GET requests. The `fetch_all_paginated_data` function iteratively calls the API, incrementing the `page` parameter until no more items are returned or the last page is reached, as indicated by the API's response (e.g., `total_pages`). It collects data from each page into a single list and handles basic error conditions for network issues or invalid JSON responses.