requests-cache/aiohttp-client-cache

HTTP responses larger than 400kB are not handeled by DynamoDBBackend

Closed this issue · 2 comments

domin4815 commented

The problem

DynamoDBBackend raises an exception when the HTTP response size is greater than 400kB. DynamoDB has a size limit of 400kB for an entity stored in the table.

Expected behavior

The cache could skip the HTTP response instead of trying to put it in the DynamoDB table. Another way would be to somehow split the response and put it as multiple entities.

Steps to reproduce the behavior

Try to perform HTTP GET on https://microsoftedge.github.io/Demos/json-dummy-data/5MB.json with DynamoDBBackend.

Workarounds

We can probably define a filter function for it so the heavy responses are not cached.

Environment

  • aiohttp-client-cache version: 0.9.0
  • Python version: 3.11
  • Platform: macOS with moto as DynamoDB mock
JWCook commented

I agree, I think skipping the cache would be better than raising an exception in this case. I'll get that fixed soon.

Also related: requests-cache/requests-cache#579
I probably don't have time to implement a secondary storage backend for large responses right now, but it's definitely something I've considered.

  • 👍1
JWCook commented

These changes are now available in v0.10.