PHP
Fetch Paginated Data from an API using Offset and Limit
Efficiently retrieve large datasets from external APIs by implementing pagination with offset and limit parameters to fetch data in manageable chunks using PHP.
<?php
function fetchPaginatedData($baseUrl, $limit = 10, $maxPages = 5) {
$allData = [];
$offset = 0;
$page = 0;
while ($page < $maxPages) {
$url = "{$baseUrl}?limit={$limit}&offset={$offset}";
echo "Fetching from: " . $url . "
";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
// Add any necessary headers like API keys here
// curl_setopt($ch, CURLOPT_HTTPHEADER, ['Authorization: Bearer YOUR_API_KEY']);
$response = curl_exec($ch);
$httpCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
$error = curl_error($ch);
curl_close($ch);
if ($response === false || $httpCode !== 200) {
echo "Error fetching data: " . ($error ? $error : "HTTP Status: {$httpCode}") . "
";
break; // Stop on error
}
$pageData = json_decode($response, true);
// Assuming the API returns an array of items directly, or under a 'data' key
$items = $pageData['data'] ?? $pageData; // Adjust key based on actual API response structure
if (empty($items)) {
echo "No more items to fetch.
";
break; // No more data
}
$allData = array_merge($allData, $items);
$offset += $limit;
$page++;
// Add a small delay to avoid hitting rate limits if fetching many pages quickly
// usleep(100000); // 100 milliseconds
}
return $allData;
}
// Example Usage:
$apiBaseUrl = 'https://api.restful-api.dev/objects'; // A simple API that supports limit/offset
$allObjects = fetchPaginatedData($apiBaseUrl, 5, 2); // Fetch 2 pages, 5 items per page
echo "Total objects fetched: " . count($allObjects) . "
";
// print_r($allObjects);
?>
How it works: This PHP snippet demonstrates how to fetch data from a paginated API using `offset` and `limit` parameters. The `fetchPaginatedData` function iteratively makes `cURL` requests, incrementing the `offset` for each subsequent page. It collects all fetched items into a single array until no more data is returned, or a maximum number of pages (`maxPages`) is reached. This approach is essential for handling APIs that return large datasets in chunks, preventing single requests from becoming too large or timing out.