🐼 Lazy JSON

LazyCollection::fromJson($source, 'data.*.users.*')
    ->map($this->mapToUser(...))
    ->filter($this->filterUser(...))
    ->values()
    ->chunk(1_000)
    ->each($this->storeUsersChunk(...));

Framework-agnostic package to load JSON of any size and from any source into Laravel lazy collections.

Lazy JSON recursively turns any JSON array or object into a lazy collection, consuming only a few KB of memory while parsing JSON of any dimension.

It optionally allows to extract only some sub-trees, instead of the whole JSON, with an easy dot-notation syntax.

Under the hood, 🧩 JSON Parser is used to parse JSONs and extract sub-trees.

Need to lazy load items from paginated JSON APIs? Consider using 🐼 Lazy JSON Pages instead.

📦 Install

Via Composer:

composer require cerbero/lazy-json

👣 Basics

Depending on our coding style, we can call Lazy JSON in 3 different ways:

use Cerbero\LazyJson\LazyJson;
use Illuminate\Support\LazyCollection;

use function Cerbero\LazyJson\lazyJson;

// auto-registered lazy collection macro
$lazyCollection = LazyCollection::fromJson($source);

// static method
$lazyCollection = LazyJson::from($source);

// namespaced helper
$lazyCollection = lazyJson($source);

The variable $source in our examples represents any JSON source. Once we define the source, we can chain any method of the Laravel lazy collection to process the JSON in a memory-efficient way:

LazyCollection::fromJson($source)
    ->values()
    ->map(/* ... */)
    ->where(/* ... */)
    ->each(/* ... */);

💧 Sources

A JSON source is any data point that provides a JSON. A wide range of sources are supported by default:

strings, e.g. {"foo":"bar"}
iterables, i.e. arrays or instances of Traversable
file paths, e.g. /path/to/large.json
resources, e.g. streams
API endpoint URLs, e.g. https://endpoint.json or any instance of Psr\Http\Message\UriInterface
PSR-7 requests, i.e. any instance of Psr\Http\Message\RequestInterface
PSR-7 messages, i.e. any instance of Psr\Http\Message\MessageInterface
PSR-7 streams, i.e. any instance of Psr\Http\Message\StreamInterface
Laravel HTTP client requests, i.e. any instance of Illuminate\Http\Client\Request
Laravel HTTP client responses, i.e. any instance of Illuminate\Http\Client\Response
user-defined sources, i.e. any instance of Cerbero\JsonParser\Sources\Source

For more information about JSON sources, please consult the 🧩 JSON Parser documentation.

🎯 Dots

If we only need a sub-tree of a large JSON, we can use a simple dot-notation syntax to extract the desired path (or dot).

Consider this JSON for example. To extract only the cities and avoid parsing the rest of the JSON, we can set the results.*.location.city dot:

$source = 'https://randomuser.me/api/1.4?seed=json-parser&results=5';

$dot = 'results.*.location.city';

LazyCollection::fromJson($source, $dot)->each(function (string $value, string $key) {
    // 1st iteration: $key === 'city', $value === 'Sontra'
    // 2nd iteration: $key === 'city', $value === 'San Rafael Tlanalapan'
    // 3rd iteration: $key === 'city', $value === 'گرگان'
    // ...
});

The dot-notation syntax is very simple and it can include any of the following 4 elements:

a key of a JSON array, e.g. 0
a key of a JSON object, e.g. results
a dot to indicate the nesting level within a JSON, e.g. results.0
an asterisk to indicate all items within an array, e.g. results.*

If we need to extract several sub-trees, Lazy JSON supports multiple dots:

$dots = ['results.*.gender', 'results.*.email'];

LazyCollection::fromJson($source, $dots)->each(function (string $value, string $key) {
    // 1st iteration: $key === 'gender', $value === 'female'
    // 2nd iteration: $key === 'email', $value === 'sara.meder@example.com'
    // 3rd iteration: $key === 'gender', $value === 'female'
    // 4th iteration: $key === 'email', $value === 'andrea.roque@example.com'
    // ...
});

📆 Change log

Please see CHANGELOG for more information on what has changed recently.

🧪 Testing

composer test

💞 Contributing

Please see CONTRIBUTING and CODE_OF_CONDUCT for details.

🧯 Security

If you discover any security related issues, please email andrea.marco.sartori@gmail.com instead of using the issue tracker.

🏅 Credits

⚖️ License

The MIT License (MIT). Please see License File for more information.

cerbero90/lazy-json