Exception when searching large datasets for common query using paginate

[razvaniacob]

Scout Version

10.8

Scout Driver

Typesense

Laravel Version

10.44.0

PHP Version

8.2.13

Description

When searching for a string that is relatively common throughout a collection of indexed data, the results can not be fetched or displayed because the search with the Scout driver results in the following exception:

Typesense\\Exceptions\\ObjectUnprocessable(code: 0): Only upto 250 hits can be fetched per page

Steps to reproduce

1. Start a fresh Laravel project, install Scout and the Typesense Scout driver according to documentation
2. Add a model and make it searchable
3. Add a large dataset (~ 50000+ records with "lorem ipsum" text)
4. Query the dataset and try to display the results in a paginated way:

$items = Item::search($request-input('query', ''))
    -paginate(10);

Expected Behavior

I expect the search to succeed, even when there are more then 250 hits, because that's why I use pagination on my frontend.

Actual Behavior

Following exception is thrown:

Typesense\\Exceptions\\ObjectUnprocessable(code: 0): Only upto 250 hits can be fetched per page

[karakhanyans]

@razvaniacob working on it. 👌

[driesvints]

@jasonbosco do you maybe know more about this?

[jasonbosco]

@driesvints Typesense has a max of 250 hits per page, and beyond that we'd have to use the page parameter to fetch additional pages. It looks like we're not managing this limit automatically within the Scout driver's pagination mechanism.

@karakhanyans is looking into this.

[karakhanyans]

Hi @razvaniacob

I did setup a fresh project and could not reproduce the issue you are having.

1. I have setup typesense driver
2. Added Searchable to User model
3. Configured schema
4. Imported 10K users with factories
5. And did search with pagination ( User::search('m')->paginate(10);

The results worked fine.

Here is the repo where I did all that. The results are returned with /users endpoint.

https://github.com/karakhanyans/laravel-scout-typesense

Could you please fork the repo, and add the steps that you did and push them so I can reproduce error?

Thanks.

[razvaniacob]

Thanks @karakhanyans,

I've tested with your code and it works.

So then I started experimenting with what I have so when I do something like this:

return ImportedProperty::search('pipera')->paginate(10)->onEachSide(1)->withQueryString()
    ->through(fn ($obj) => [
        'name' => $obj->source_id,
    ]);

It works, but if I do it like this:

return ImportedProperty::search('pipera')
    ->query(fn (Builder $query) => $query->with(['imported_district', 'neighbourhood']))
    ->paginate(10)->onEachSide(1)->withQueryString()
    ->through(fn ($obj) => [
        'name' => $obj->source_id,
        'district' => $obj->imported_district?->name ?? ($obj->neighbourhood?->name ?? ''),
    ]);

It fails with the error

Maybe it has something to do with the
->query(fn (Builder $query) => $query->with(['imported_district', 'neighbourhood']))
line?

Any thoughts?

[karakhanyans]

@razvaniacob have you tried this line ->query(fn (Builder $query) => $query->with(['imported_district', 'neighbourhood'])) like this $query->with(['imported_district', 'neighbourhood']), without putting it inside query, as it's just a ->with.

[razvaniacob]

I followed the documentation found here Laravel Scout Documentation

If I do just ->with... I get this error
Method Laravel\Scout\Builder::with does not exist.