untereiner/scaleway-functions-runtimes

Runtimes for Scaleway Function As A Service Platform

Core Runtime for Scaleway Functions

This repository contains the core runtime used inside Scaleway Functions's platform, built with Golang, as well as language specific sub-runtimes.

This runtime is the Gateway to functions running in Scaleway's Cloud. Its goal is to provide a set of features optimized for Function As A Service technologies, and make it easy for users to run their applications in the cloud.

A runtime on Scaleway Serverless Platform is made of 2 components:

• Core Runtime written in Golang: Starts a HTTP server, handles incoming request, transforms data into event objects, handles authentication (users can deploy private functions), starts a sub-runtime and proxy request to the sub-runtime
• Sub-Runtime (Language specific code), Starts a HTTP server, receives requests from Core-Runtime containing "event", "context" and user's handler informations (handler file and exported function to execute). The sub-runtime is a bridge between the core-runtime and the user's code. We can not dynamically import and execute node.js code in Golang, so we have to create a bridge in node.js to handle this logic.

We decided to split the logic in 2 components because we wanted to handle the common logic (transforming HTTP requests in event/context, handling authentication...) in one place, instead of maintaining 5 different repositories (at this time), all doing the same logic. It makes it easier for us to maintain our runtimes, as sub-runtimes have a minimal task to fulfill: import and invoke user's code.

Features

• Simple HTTP Webserver
• Manages Authentication (On Scaleway Serverless Platform, users can choose to deploy their functions privately and they will have to provide a valid JWT generated by our APIs to trigger the function, see the official documentation
• Handles incoming requests, transforms data into events (and execution context of the function)
• Dynamically import and execute functions handlers, provided by end-users.
• Manages Handler's responses based on trigger-type (for example, HTTP requires function handlers to return an HTTP response while CRON jobs do not require any of it).

At the moment, we only support the following runtimes (see the corresponsing sub-runtimes):

• Node.js version 8, version 10 and version 14
• Python version 2.7 and version 3.7
• Golang version 1.11+

How to use this runtime

You can use this runtime on your own infrastructure, or on your local environment if you wish to contribute to its development.

Introduction

In order to create a new runtime extending Scaleway's Core runtime, for example ruby or dotnet, you will have to respect certain criterias:

• Your runtime should create an HTTP server, listening on given $SCW_UPSTREAM_PORT environment variable, on interface 127.0.0.1 (As both core-runtime and your sub-runtime will run in the same docker container, core-runtime will proxy requests to local network interface).
• Listen for traffic on endpoint POST / (e.g. POST 127.0.0.1:$SCW_UPSTREAM_PORT)
• Manage incoming requests, with the following structure:
  • event: Data for the event that triggered the function execution (in case of an HTTP request, it contains the request body, incoming headers, query parameters...)
  • context: Execution context of your function
  • handlerPath: Path to your Handler file (e.g. for handler located in /home/app/function/handler.js: /home/app/function/handler), you will have to handle dynamic import of your handler file (if dynamic language)
  • handlerName: Name of the exported function to use as a handler (e.g. /home/app/function/handler.js: module.exports.handle = ..., handlerName is handle). Full Example of request body for an function invoked via HTTP Trigger:

    {
      "event": {
          "body": {...request body},
          "headers": {...request headers},
          "httpMethod": "POST",
          "path": "/test",
          "pathParameters": {},
          "queryStringParameters": {
              "query": "value"
          },
          "multiValueHeaders": null,
          "multiValueQueryStringParameters": null,
          "stageVariables": null,
          "isBase64Encoded": false,
          "requestContext": {
              "eventType": "http",
              "httpMethod": "POST",
              "path": "/test",
              "stage": "dev",
          },
          "resource": "http"
      },
      "context": {
          "functionName": "myFunction",
          "memoryInMb": 128
      },
      "handlerPath": "/home/app/function/handler",
      "handlerName": "handle"
    }

• Send an HTTP response with the following structure (for HTTP Triggers):
  • body: Response body
  • statusCode: Status Code for HTTP Response to the client invoking the function
  • headers: Map of headers (key: value) to send in HTTP Response Example of Response from custom runtimes (invoked via HTTP Trigger):

    {
      "statusCode": 200,
      "body": {
          "key": "value"
      },
      "headers": {
          "key": "value"
      }
    }

Requirements

To start using Scaleway's core-runtime, you'll need to go through multiple steps:

• Install Golang (version >= 1.11 as we are using go mod for our dependencies)
• Download this repository (outside of your $GOPATH as we are using go mod).
• Pre-requisites related to your custom runtime (for example, to run node.js runtime, you need to install node.js)

Specific Instructions for Dynamic Languages

In order to create a custom runtime extending this core-runtime with a dynamic language (such as Python, Node.js or Ruby for example), you will have to develop a sub-runtime script, in charge of:

• Starting an HTTP Server on port $SCW_UPSTREAM_PORT (provided via environment variable) on host $SCW_UPSTREAM_HOST (e.g. 127.0.0.1:8081 is the default for core-runtime's sub runtimes), handling POST requests on /.
• Dynamically import function handler from given handlerPath and handlerName (see part How to use this runtime above).
• Handle potential errors (while importing handler), and during handler execution, to format the error response properly (this may be useful, to provide good feedbacks on errors to your users).
• Send Back an HTTP response with handler's return result.

You may find examples of Node and Python custom runtimes (developped and maintained by Scaleway, used on Scaleway Functions platform).

In order to configure core-runtime to use your "Language Specific Runtime", you will have to set different environment variables:

variable name	description
SCW_HANDLER_NAME	Exported function to use as a handler (e.g. handle for a node.js function with a module.exports.handle handler)
SCW_HANDLER_PATH	Absolute path to your handler file (e.g. /home/app/function/handler or /home/app/function/handler.js)
SCW_RUNTIME_BINARY	Absolute path to the binary of the language you wish to use to execute your runtime (e.g. /usr/local/bin/node or /usr/local/bin/python)
SCW_RUNTIME_BRIDGE	Absolute Path to your custom-runtime entrypoint (e.g. /home/app/myruntime.js)

This Core-runtime will take care of executing $SCW_RUNTIME_BINARY $SCW_RUNTIME_BRIDGE (e.g. /usr/local/bin/node /home/app/myruntime.js) to start the sub-runtime HTTP server.

When the Core-runtime receives an HTTP request, it will transform it into a usable event structure, create the context object, and execute your sub-runtime code by sending an HTTP request to its HTTP server.

Example

• I have a Node.js handler (see this example) (located in /home/app/function/handler.js) with our custom runtime defined here (located in /home/app/index.js).
• I configure my environment properly:

  export SCW_RUNTIME_BINARY=/usr/local/bin/node
  export SCW_RUNTIME_BRIDGE=/home/app/index.js
  export SCW_HANDLER_PATH=/home/app/function/handler
  export SCW_HANDLER_NAME=handle
  export SCW_UPSTREAM_HOST=http://127.0.0.1
  export SCW_UPSTREAM_PORT=8081
  # PORT used by core runtime to serve HTTP server
  export PORT=8080
  # For this example, we're setting function's authentication privacy to public
  export SCW_PUBLIC=true

• I can run my core-runtime:

  sudo -s
  go run main.go

• I can trigger my function:

  curl localhost:8080

Python

```bash
export SCW_RUNTIME_BINARY=/bin/sh
export SCW_RUNTIME_BRIDGE=/runtimes/python3/bootup.sh
export SCW_HANDLER_PATH=/home/app/function/handler
export SCW_HANDLER_NAME=handle
export SCW_UPSTREAM_HOST=http://127.0.0.1
export SCW_UPSTREAM_PORT=8081
export PORT=8080
export SCW_PUBLIC=true
export SCW_HANDLER_IS_BINARY=false
```

Specific Instructions for compiled languages

In order to create a custom runtime extending this core-runtime with a compiled language (such as Golang, C# for example), you will most likely have to develop a custom Library that will wrap your function handler's code with the necessary logic described above.

As the Function Handler's code is compiled as a binary, you will have to include the runtime logic (creating server, handling incoming traffic and outgoing responses...) alonside your handler code.

You may find an example of a Golang custom runtime here (developed and maintained by Scaleway and running on Scaleway Serverless platform).

In order to configure core-runtime to use your handler as a Binary (compiled code), you will have to set different environment variables:

• SCW_HANDLER_IS_BINARY=true this is important, as the core-runtime will initialize your runtime by running a command (for example /home/app/function/handler if you compiled your dotnet program into a handler binary).
• SCW_HANDLER_PATH=absolute/path/to/binary (for example /home/app/function/handler if you compiled your program into a handler binary)

Example

In this example, we are using the official Golang sub-runtime for Serverless Scaleway.

• I have a Golang Handler in a file main.go using my custom runtime described in this sample
• I build a handler binary and output it to /home/app/function directory:

  go build -o /home/app/function/handler main.go

• I configure my environment properly:

  export SCW_HANDLER_PATH=/home/app/function/handler
  export SCW_HANDLER_IS_BINARY=true
  export SCW_UPSTREAM_PORT=8081
  # PORT used by core runtime to serve HTTP server
  export PORT=8080
  # For this example, we're setting function's authentication privacy to public
  export SCW_PUBLIC=true

• I can run my core-runtime binary:

  go run main.go

• I can trigger my function:

  curl localhost:8080

Contributing

Everyone is free to contribute to this project by sending PRs or opening issues.

If you wish to integrate a new language to this runtime, feel free to either add the code and documentation in the runtimes directory. If you do not wish to code the runtime yourself, we would be happy to hear from you on either via Issues on this project, on by messages on our Community Slack Platform (channel #srvless-private-beta).

Refer to Issue templates.