PyLambdaFlows

PyLambdaFlows lets you run your program on AWS Lambda for a large scale execution. It makes you able to use AWS service without major alteration on your code. Moreover, you can define very complex dependencies between your step and create your own dependencies function ! This project was inspired by PyWren library.

In order to create a lambda gesture, pylambdaFlow use dynamodb.

Documentation : This section contains some ways to get few docs.
Examples : This section will describe to you how you can use pyLambdaFlows.
Installation : This section will explain to you how can you install this library
Features : This section will explain to you shortly all available features on pylambdalflows (computational graph, custom op, and so on).
AWS setup : This section will show you how can you correctly set up your AWS credentials to be ready to go!

Documentation

You can either find out the documentation here at readthedocs or generate your own using the provided makefile or just read through the source code ;-).

Examples

Firstly you have to split your code into kernel file. One file kernel per operation is the common way to work with. In this example, we want to apply a map and a reduce onto an int array. The kernel map can be defined as follow :

from pyLambdaFlows.decorator import kernel
@kernel
def  lambda_handler(inputData):
   return inputData*inputData

And the reduce op can be written like this : from pyLambdaFlows.decorator import kernel

from statistics import mean
from pyLambdaFlows.decorator import kernel
@kernel
def lambda_handler(inputData):
   result = mean(inputData)
   return result

Then, we need to create the call graph on a main file:

import pyLambdaFlows
array = pyLambdaFlows.op.Source()
squared_array = pyLambdaFlows.op.Map(array, "map.py", name="square_op")
mean_squared = pyLambdaFlows.op.Reduce(squared_array, "mean.py", name="mean_op")

Finally, we can call the AWS API for uploading and calling lambda function.

with pyLambdaFlows.Session(credentials_csv="./accessKeys.csv") as sess:
   mean_squared.compile()
   result = mean_squared.eval(feed_dict={array:[1,2,3]})
   print(result)

Output :

4.666666666666667

Installation

Firstly you need to clone this repo on your computer with this git command :

git clone https://github.com/Enderdead/pyLambdaFlows.git

Then it's recommended to install dependencies using pip3 tool :

pip3 install -r requirements.txt

Finally, you can install pylambdaflow with the setup script :

python3 setup.py install --user

Performance

pyLambda is a proof of concept which means that overall implementation isn't as good as it can be. That means pyLambdaFlow can't handle a large scale project, but feel free to improve it yourself. Basic benchmark was done to compare this efficiency against pyrewn.

AWS setup

To use this library, you have to create your AWS credentials. It's pretty simple using the AWS web interface. You can find out some instructions in the pyLambdaFlow doc.