size > 30 for JumboConnector causes error 500
Closed this issue · 1 comments
Aypac commented
Thanks for the lib, very useful!
It appears that jumbo has limited the size to 30 per page. This leads to this command throwing a 500 error: list(connector.search_all_products(query='Olijfolie'))
. I tried adding the size=30
parameter to the function (against all warnings), but it doesn't seem to work. For now I wrote my own workaround:
def get_all(query, connector):
all = []
scraped_prod_num = 0
all_prod_num = 99
page = 0
while scraped_prod_num < all_prod_num:
res = connector.search_products(query=query, size=30, page=page)
all_prod_num = int(res['products']['total'])
pd = res['products']['data']
all += pd
scraped_prod_num += len(pd)
assert len(all) == scraped_prod_num
if page == 0:
print('')
print(f"\rFound {res['products']['total']:d} products. Retrieved {scraped_prod_num:d}", end="")
page += 1
time.sleep(random.random()*1.2)
print('. Done.')
return all
But I think it'd be nice for it to be fixed in the code ;)
Error: requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://mobileapi.jumbo.com/v15/search?offset=0&limit=1000&q=Olijfolie
and the same for any request with size > 30.
bartmachielsen commented
Hi @Aypac, I just updated the library to set the default pagination to 30, Update to 0.5 to get the new version.