Unzipped size must be smaller than 132586253 bytes

[mebibou]

When using the layer

arn:aws:lambda:us-east-1:552188055668:layer:geolambda-python:3

I cannot deploy to Lambda because I will get this error:

An error occurred: MyLambdaFunction - Unzipped size must be smaller than 132586253 bytes (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: c9e35258-e012-4ca1-a70f-05037bb70aef).

How can I use the layer on python 3 without having a limit problem?

[vincentsarago]

Hi @mebibou, This error means that your own package is too big (when unzipped). From my understanding, the geolambda layer must weight around 120Mb (unzipped) so you need to make sure your package weight less than 130Mb (unzipped) (130Mb + 120Mb == 250Mb which is the limit)

[mebibou]

Ok that makes sense, I guess its because I am using some libraries that import numpy on their own as well and not sure how to make sure I don't double import it. That aside, I managed to go around the limitation by using zip: true in the serverless.yml (which zipped the dependencies), but now I am getting this error when running the function:

[Errno 2] No such file or directory: '/usr/bin/ogr2ogr': '/usr/bin/ogr2ogr'

I did set GDAL_DATA: /opt/share/gdal in my env though.

My full serverless.yml:

service: helloworld

provider:
  name: aws
  runtime: python3.7

plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    dockerizePip: non-linux
    zip: true

package:
  exclude:
    - '**/*'
  include:
    - '*.py'

functions:
  hello:
    handler: handler.handle
    events:
      - http:
          path: /
          method: post
    layers:
      - arn:aws:lambda:us-east-1:552188055668:layer:geolambda-python:3
    environment:
      GDAL_DATA: /opt/share/gdal
      PROJ_LIB: /opt/share/proj

[mebibou]

Oh so I actually found how to change the path where to find the ogr2ogr bin, because I am using pygdaltools I had to set this at the beginning of the handler file:

import gdaltools
gdaltools.Wrapper.BASEPATH = '<path>'

but I've tried with /opt/share and /opt/share/gdal but I still get the error No such file or directory: '/opt/share/ogr2ogr' or No such file or directory: '/opt/share/gdal/ogr2ogr'

Where is the ogr2ogr file in the layer exactly?

[matthewhanson]

Hello @mebibou sorry for the delay.

ogr2ogr is not in the Layer, none of the GDAL binaries are included, just the libraries.

So all you need to do is copy ogr2ogr from the GeoLambda docker image (https://hub.docker.com/repository/docker/developmentseed/geolambda) into your own deployment package, in which case it will be in under /var/task

[mebibou]

Ok so what I did:

docker pull developmentseed/geolambda:latest
docker run --rm --entrypoint /bin/sh developmentseed/geolambda:latest -c "cat /usr/local/bin/ogr2ogr" > ogr2ogr
chmod 755 ogr2ogr

Then in my serverless.yml I included the library ogr2ogr (in the package/include part), modified the code to found the binary in my python file to gdaltools.Wrapper.BASEPATH = '/var/task'.

Now running the function gives a different error:

(127, b'/var/task/ogr2ogr: error while loading shared libraries: libgdal.so: cannot open shared object file: No such file or directory\\n')

Do I need to also pull all the libraries missing? that doesnt sound right

[matthewhanson]

Hi @mebibou sorry this got dropped. Did you figure this out? You should be installing ogr2ogr from within the GeoLambda container, rather than using the one from your local machine which may be using a GDAL compiled differently than the one in the container.