In this chapter, we will work a few of the many different services that aws provides and how it provides us with many different tools to be able to deploy our work.
AWS Lambda is an AWS service that lets you run code without provisioning or managing servers. So itโs serverless. This means that the user is only interacting with the function as an entity, not with anything related to the server they are running. This simplifies some operations and costs as AWS only charges for the time the function uses.
However, there may be some restrictions in the working model of lambda, such as the time the code could be running, memory, and such. These constraints may enable us to use lambda with different solutions.
What can we do if we want to run libraries that do not come with the basic python installation, plus a library on memory that exceeds the limits of the Lambda?
EFS stands for Elastic File System and is a file system located inside a VPC (Virtual private cloud) that allows you to store or access data. Having the libraries in EFS and having the Lambda function we will use use these libraries.
Everything happens in a VPC context. That's why we need to create a VPC (or use a prebuilt VPC) first.
Next, you choose to create a simple VPC with a public network and we can give it a name.
Then we have our newly created VPC.
AWS is the arrangement of many interactions of different components to ensure security. For most of these interactions, we use what is called a "security group", which contains sets of rules that allow an entity to connect to the outside (outbound rules) or allow something to connect to it from outside the entity (inbound rules).
"Create security group" and then add an inbound rule for NFS. For this exercise, weโll arrenged that.
Go to console and look for EFS. Then click in "create file system" and then give a name and choose the VPC that the EFS would be connected to.
Then click a button that says "Manage". Now we will add security group that we created before in addition you can add default security group.
The important part is the directory name at the end. This folder is where we'll have the libraries that the Lambda Function will be able to connect to.
Now we'll test that EFS works like a proper file system and that we can interact with it. Before we create a lambda function, we need to define the privileges of the IAM role we will use.
Again, we will create Lambda over the AWS console. I used Python 3.8 for this work, but you could choose a different version or language.
๐ธ From here we select the VPC area Choose the subnet and add a VPC and Security Group we created earlier here.
Here choose the EFS we created before, add the Access point and the path where we are going to mount the EFS.
Here we will test whether the connection between EFS and Lambda works with a small example.
Go to the "Code" part of your Lambda. Copy the following exapmle code.
import json
import os
import sys
sys.path.append('/mnt/access')
def lambda_handler(event, context):
print('Before: ',os.listdir('/mnt/access'))
with open('/mnt/access/somefile.txt','w') as file:
file.write('hello world')
print('After: ',os.listdir('/mnt/access'))
These codes print the files and folders in /mnt/access before and after the creation of a simple file that says "hello world". Then you have to click on 'Deploy', 'Test' and finally you will get a printout.
Congratulations! We have the EFS already connected with our simple Lambda Function.
Now what we can create our test sample with our EFS through an EC2 instance, this will make easier install files on our /mnt/access folder.
EC2 stands for Elastic cloud computing. It allows you to create a server. Weโll just create a simple and free kind of EC2 instance and then weโll mount the EFS that we created on this instance.
๐ธ I have progressed through Ubuntu 20.04, but you can proceed as you wish, except for the "configure instance" and "configure security group" fields, except this part, other parts are optional.
sudo apt-get update
sudo apt-get install nfs-common
mkdir mnt
After doing that let's go to the EFS we created and click the attach part in the upper right. Copy the NFS client command paste that but without the "efs" at the end.
sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport fs-095de34f07883e4b0.efs.eu-central-1.amazonaws.com:/ mnt
sudo apt-get install python3.8
sudo apt-get install python3-pip
sudo update-alternatives --config python3
pip3 install --upgrade --target mnt/access/ numpy
Letโs go back to the Lambda. Deploy the example code.
import json
import os
import sys
import numpy as np
# connecting to the folder with the libraries
sys.path.append('/mnt/access')
def lambda_handler(event, context):
print(np.ones(3))
Great! We connected our EFS to our lambda function and we are using the libraries that we installed there in our EC2!
๐ Next Lecturer we will do different example with more libraries using Serverless API and API Gateway
To reach : https://github.com/sedaatalay/Deploying-large-python-packages-on-AWS-Lambda-using-EFS
Thanks for your time :)