JAVASCRIPT
Handling Cursor-Based Pagination for API Results
Efficiently fetch large datasets from APIs using cursor-based pagination, navigating through results with `next_cursor` tokens for scalable data retrieval.
async function fetchAllPagesWithCursor(baseUrl, initialParams = {}) {
let allData = [];
let nextCursor = null;
let hasMore = true;
while (hasMore) {
const url = new URL(baseUrl);
// Merge initial parameters with cursor for the current request
const currentParams = { ...initialParams };
if (nextCursor) {
currentParams.cursor = nextCursor;
}
Object.keys(currentParams).forEach(key => url.searchParams.append(key, currentParams[key]));
try {
const response = await fetch(url.toString());
if (!response.ok) {
throw new Error(`HTTP error! status: ${response.status}`);
}
const result = await response.json();
// Assuming API response structure like: { data: [...], metadata: { next_cursor: "abc", has_more: true } }
allData = allData.concat(result.data);
nextCursor = result.metadata ? result.metadata.next_cursor : null;
hasMore = result.metadata ? result.metadata.has_more : false;
if (!nextCursor && hasMore) {
// Edge case: API indicates more data but doesn't provide a cursor
console.warn('API indicates more data but no next_cursor was provided. Stopping pagination.');
hasMore = false;
}
} catch (error) {
console.error('Error fetching paginated data:', error);
hasMore = false; // Stop on error
}
}
return allData;
}
// Usage example:
// fetchAllPagesWithCursor('https://api.example.com/items', { limit: 100 })
// .then(items => console.log(`Fetched ${items.length} items across all pages.`))
// .catch(error => console.error('Failed to fetch all items:', error));
How it works: This JavaScript function demonstrates how to implement cursor-based pagination to retrieve all available data from an API. Unlike offset-based pagination, which can be inefficient for large datasets, cursor-based pagination uses an opaque token (the 'cursor') to mark the last item fetched, making it more robust and scalable. The function iteratively makes API calls, appending the `next_cursor` from the previous response to the subsequent request's parameters until the API indicates there are no more results (`has_more: false`) or no further cursor is provided.