JAVASCRIPT
Fetch All Data from a Paginated REST API
Efficiently retrieve complete datasets from APIs that return paginated results, combining data from multiple pages into a single collection, ideal for large data fetches.
async function fetchAllPaginatedData(url, page = 1, allData = []) {
try {
const response = await fetch(`${url}?page=${page}&limit=100`); // Assuming 'page' and 'limit' query params
if (!response.ok) {
throw new Error(`HTTP error! status: ${response.status}`);
}
const data = await response.json();
// Assuming the API returns an array of items and a way to know if there are more pages
// This example assumes a 'data' array and 'totalPages' or 'nextPage' indicator
const items = data.items || data; // Adjust based on your API structure
allData.push(...items);
// Example 1: API provides total pages
if (data.page < data.totalPages) {
return fetchAllPaginatedData(url, page + 1, allData);
}
// Example 2: API provides a 'next' link or 'hasNextPage' boolean
// if (data.nextPage) {
// return fetchAllPaginatedData(data.nextPage, 1, allData); // If next is a full URL, reset page or use it directly
// }
// if (data.hasNextPage) {
// return fetchAllPaginatedData(url, page + 1, allData);
// }
return allData;
} catch (error) {
console.error("Error fetching paginated data:", error);
throw error;
}
}
// Example usage:
// fetchAllPaginatedData('https://api.example.com/products')
// .then(products => console.log('All products:', products))
// .catch(error => console.error('Failed to get all products:', error));
How it works: This snippet demonstrates how to recursively fetch all data from an API that uses pagination. It makes an initial request, then checks the response to determine if more pages exist (e.g., by comparing current page to total pages, or looking for a `nextPage` indicator). If so, it calls itself with the next page number, aggregating all results into a single array until no more pages are available. This pattern ensures you retrieve a complete dataset without manual iteration.