PHP
Process Large Datasets Efficiently Using Eloquent Query Chunking
Optimize memory usage when iterating through large result sets in Laravel with Eloquent's `chunk` method, preventing memory exhaustion and improving performance.
<?php
namespace App\Models;
use Illuminate\Database\Eloquent\Model;
class Product extends Model
{
// ... model definition ...
}
// Scenario: Processing millions of products without running out of memory
Product::chunk(1000, function ($products) {
foreach ($products as $product) {
// Process each product
$product->update(['status' => 'processed']);
// Or send to an external service
// MyExternalService::send($product);
}
});
// Using `chunkById` for even better performance with large IDs
Product::chunkById(500, function ($products) {
foreach ($products as $product) {
// Process each product
// MyBatchProcessor::processProduct($product);
}
}, $column = 'id', $alias = null);
How it works: When dealing with very large datasets, fetching all records into memory at once can lead to memory exhaustion. Eloquent's `chunk` method solves this by retrieving a small 'chunk' of records at a time and feeding them to a given callback. This allows you to process large numbers of records without loading the entire dataset into memory simultaneously. The `chunkById` method is an even more efficient variation that internally orders by the primary key to prevent issues with changing offsets during iteration, making it ideal for large tables with frequently updated data.