Problem with offset parameter

[nikmich1]

While browsing the USGS Lidar point cloud catalog, I get an "Invalid paging params" error for the offset parameter >150000, although the catalog has many more items. Is there a way to go beyond 150000 somehow?

The issue can be reproduced with

from sciencebasepy import SbSession
sb = SbSession()
catalog_id = "4f70ab64e4b058caae3f8def" # USGS Lidar Point Cloud

sb.find_items({'parentId': catalog_id, 'fields': 'title,spatial', 'max': 10, 'offset': 151000}) # works until offset of 150000

[jllong-usgs]

The 150000 limit is a consequence of the way elasticsearch has been implemented in the application and can not be extended. The best way to get around this is to determine a set of more focused queries that will each return less than the max limit of records but together will return the full set.

…

-- Dell Long Engineering Branch Supervisor Enterprise Technology Office Water Mission Area US Geological Survey ***@***.******@***.***> From: Niklas M. ***@***.***> Date: Friday, March 8, 2024 at 4:04 AM To: DOI-USGS/sciencebasepy ***@***.***> Cc: Subscribed ***@***.***> Subject: [EXTERNAL] [DOI-USGS/sciencebasepy] Problem with offset parameter (Issue #49) This email has been received from outside of DOI - Use caution before clicking on links, opening attachments, or responding. While browsing the USGS Lidar point cloud catalog, I get an "Invalid paging params" error for the offset parameter >150000, although the catalog has many more items. Is there a way to go beyond 150000 somehow? The issue can be reproduced with from sciencebasepy import SbSession sb = SbSession() catalog_id = "4f70ab64e4b058caae3f8def" # USGS Lidar Point Cloud sb.find_items({'parentId': catalog_id, 'fields': 'title,spatial', 'max': 10, 'offset': 151000}) # works until offset of 150000 — Reply to this email directly, view it on GitHub<#49>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AB72TJ7ORV3ZL5HKGTGXQLTYXGLJ5AVCNFSM6AAAAABEMT6NNKVHI2DSMVQWIX3LMV43ASLTON2WKOZSGE3TKOBQGY2DIOA>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>