Packaging DuckDB for usage in AWS Lambda functions with Node.js, and publishing as public Lambda layers.
You can use the published layers in your own serverless applications by referencing it as outlined in the different framework's docs:
The layers get automatically published to all currently available (in the moment of publishing) AWS regions.
You can have a look at the example repository, which uses this AWS Lambda layer: tobilg/serverless-duckdb. More specifically, the src/functions/query.js should give you a good idea on how it can be used.
The recommendation is to use a bundler such as WebPack for the packaging of your application. This means you can locally use (e.g. for testing) the official duckdb npm package (via npm i --save duckdb), but exclude it in the packaging process, because after being deployed to a Lambda function, the updated DuckDB version from the Lambda layer will be used.
import DuckDB from 'duckdb';
// Instantiate DuckDB
const duckDB = new DuckDB.Database(':memory:');
// Create connection
const connection = duckDB.connect();
// Promisify query method
const query = (query) => {
return new Promise((resolve, reject) => {
connection.all(query, (err, res) => {
if (err) reject(err);
resolve(res);
})
})
}
// Will show DuckDB version
await query(`PRAGMA version;`);This contains a AWS Lambda layer in different flavors, both x86 and arm64 architectures are supported.
The ARNs follow the following logic:
arn:aws:lambda:$REGION:041475135427:layer:duckdb-nodejs-$ARCHITECTURE:$VERSION
where $ARCHITECTURE can have the following values:
x86arm64
The following DuckDB default extensions are enabled and contained in the static build:
parquet: Adds support for reading and writing parquet fileshttpfs: Adds support for reading and writing files over a HTTP(S) connectionjson: Adds support for JSON operationsfts: Adds support for Full-Text Search Indexesicu: Adds support for time zones and collations using the ICU library
Layer version to DuckDB version mapping:
| Layer version | DuckDB version |
|---|---|
| 3 | v0.8.0 |
| 4 | v0.8.1 |
| 5 | v0.9.0 |
| 6 | v0.9.1 |
| 7 | v0.9.2 |
| 8 | v0.10.0 |
| 9 | v0.10.1 |
| 10 | v0.10.2 |
| 11 | v1.0.0 |
| 12 | v1.1.0 |
| 13 | v1.1.1 |
| 14 | v1.1.3 |
The ARNs of the latest x86 version of the DuckDB Node.js Lambda layer are:
| Region | Layer ARN |
|---|---|
| af-south-1 | arn:aws:lambda:af-south-1:041475135427:layer:duckdb-nodejs-x86:14 |
| ap-east-1 | arn:aws:lambda:ap-east-1:041475135427:layer:duckdb-nodejs-x86:14 |
| ap-northeast-1 | arn:aws:lambda:ap-northeast-1:041475135427:layer:duckdb-nodejs-x86:14 |
| ap-northeast-2 | arn:aws:lambda:ap-northeast-2:041475135427:layer:duckdb-nodejs-x86:14 |
| ap-northeast-3 | arn:aws:lambda:ap-northeast-3:041475135427:layer:duckdb-nodejs-x86:14 |
| ap-south-1 | arn:aws:lambda:ap-south-1:041475135427:layer:duckdb-nodejs-x86:14 |
| ap-south-2 | arn:aws:lambda:ap-south-2:041475135427:layer:duckdb-nodejs-x86:14 |
| ap-southeast-1 | arn:aws:lambda:ap-southeast-1:041475135427:layer:duckdb-nodejs-x86:14 |
| ap-southeast-2 | arn:aws:lambda:ap-southeast-2:041475135427:layer:duckdb-nodejs-x86:14 |
| ap-southeast-3 | arn:aws:lambda:ap-southeast-3:041475135427:layer:duckdb-nodejs-x86:14 |
| ap-southeast-4 | arn:aws:lambda:ap-southeast-4:041475135427:layer:duckdb-nodejs-x86:14 |
| ca-central-1 | arn:aws:lambda:ca-central-1:041475135427:layer:duckdb-nodejs-x86:14 |
| eu-central-1 | arn:aws:lambda:eu-central-1:041475135427:layer:duckdb-nodejs-x86:14 |
| eu-central-2 | arn:aws:lambda:eu-central-2:041475135427:layer:duckdb-nodejs-x86:14 |
| eu-north-1 | arn:aws:lambda:eu-north-1:041475135427:layer:duckdb-nodejs-x86:14 |
| eu-south-1 | arn:aws:lambda:eu-south-1:041475135427:layer:duckdb-nodejs-x86:14 |
| eu-south-2 | arn:aws:lambda:eu-south-2:041475135427:layer:duckdb-nodejs-x86:14 |
| eu-west-1 | arn:aws:lambda:eu-west-1:041475135427:layer:duckdb-nodejs-x86:14 |
| eu-west-2 | arn:aws:lambda:eu-west-2:041475135427:layer:duckdb-nodejs-x86:14 |
| eu-west-3 | arn:aws:lambda:eu-west-3:041475135427:layer:duckdb-nodejs-x86:14 |
| me-central-1 | arn:aws:lambda:me-central-1:041475135427:layer:duckdb-nodejs-x86:14 |
| me-south-1 | arn:aws:lambda:me-south-1:041475135427:layer:duckdb-nodejs-x86:14 |
| sa-east-1 | arn:aws:lambda:sa-east-1:041475135427:layer:duckdb-nodejs-x86:14 |
| us-east-1 | arn:aws:lambda:us-east-1:041475135427:layer:duckdb-nodejs-x86:14 |
| us-east-2 | arn:aws:lambda:us-east-2:041475135427:layer:duckdb-nodejs-x86:14 |
| us-west-1 | arn:aws:lambda:us-west-1:041475135427:layer:duckdb-nodejs-x86:14 |
| us-west-2 | arn:aws:lambda:us-west-2:041475135427:layer:duckdb-nodejs-x86:14 |
Layer version to DuckDB version mapping:
| Layer version | DuckDB version |
|---|---|
| 1 | v0.8.0 |
| 2 | v0.8.1 |
| 3 | v0.9.0 |
| 4 | v0.9.1 |
| 5 | v0.9.2 |
| 6 | v0.10.0 |
| 7 | v0.10.1 |
| 8 | v0.10.2 |
| 9 | v1.0.0 |
| 10 | v1.1.0 |
| 11 | v1.1.1 |
| 12 | v1.1.3 |
The ARNs of the latest arm64 version of the DuckDB Node.js Lambda layer are:
| Region | Layer ARN |
|---|---|
| us-east-1 | arn:aws:lambda:us-east-1:041475135427:layer:duckdb-nodejs-arm64:12 |
| us-east-2 | arn:aws:lambda:us-east-2:041475135427:layer:duckdb-nodejs-arm64:12 |
| us-west-1 | arn:aws:lambda:us-west-1:041475135427:layer:duckdb-nodejs-arm64:12 |
| us-west-2 | arn:aws:lambda:us-west-2:041475135427:layer:duckdb-nodejs-arm64:12 |
| af-south-1 | arn:aws:lambda:af-south-1:041475135427:layer:duckdb-nodejs-arm64:12 |
| ap-east-1 | arn:aws:lambda:ap-east-1:041475135427:layer:duckdb-nodejs-arm64:12 |
| ap-southeast-3 | arn:aws:lambda:ap-southeast-3:041475135427:layer:duckdb-nodejs-arm64:12 |
| ap-south-1 | arn:aws:lambda:ap-south-1:041475135427:layer:duckdb-nodejs-arm64:12 |
| ap-northeast-3 | arn:aws:lambda:ap-northeast-3:041475135427:layer:duckdb-nodejs-arm64:12 |
| ap-northeast-2 | arn:aws:lambda:ap-northeast-2:041475135427:layer:duckdb-nodejs-arm64:12 |
| ap-southeast-1 | arn:aws:lambda:ap-southeast-1:041475135427:layer:duckdb-nodejs-arm64:12 |
| ap-southeast-2 | arn:aws:lambda:ap-southeast-2:041475135427:layer:duckdb-nodejs-arm64:12 |
| ap-northeast-1 | arn:aws:lambda:ap-northeast-1:041475135427:layer:duckdb-nodejs-arm64:12 |
| ca-central-1 | arn:aws:lambda:ca-central-1:041475135427:layer:duckdb-nodejs-arm64:12 |
| eu-central-1 | arn:aws:lambda:eu-central-1:041475135427:layer:duckdb-nodejs-arm64:12 |
| eu-west-1 | arn:aws:lambda:eu-west-1:041475135427:layer:duckdb-nodejs-arm64:12 |
| eu-west-2 | arn:aws:lambda:eu-west-2:041475135427:layer:duckdb-nodejs-arm64:12 |
| eu-south-1 | arn:aws:lambda:eu-south-1:041475135427:layer:duckdb-nodejs-arm64:12 |
| eu-west-3 | arn:aws:lambda:eu-west-3:041475135427:layer:duckdb-nodejs-arm64:12 |
| eu-north-1 | arn:aws:lambda:eu-north-1:041475135427:layer:duckdb-nodejs-arm64:12 |
| me-south-1 | arn:aws:lambda:me-south-1:041475135427:layer:duckdb-nodejs-arm64:12 |
| sa-east-1 | arn:aws:lambda:sa-east-1:041475135427:layer:duckdb-nodejs-arm64:12 |
This layer contains the arrow, aws and httpfs extensions. This enables Lambda functions that are in a VPC and have no internet access to download these extensions via HTTP (e.g. from extensions.quacking.cloud) for cost savings.
VPC access is for example necessary if you want your Lambda functions to access S3 Express One Zone. You can have a look at how to deploy such kind of setup in the tobilg/serverless-duckdb repo.
Currently, it's only available in us-east-1.
Layer version to DuckDB version mapping:
| Layer version | DuckDB version |
|---|---|
| 1 | v0.10.0 |
| 2 | v0.10.1 |
| 4 | v0.10.2 |
| 5 | v1.0.0 |
| 6 | v1.1.0 |
The ARNs of the latest x86_64 version of the DuckDB Node.js Lambda layer with extensions are:
| Region | Layer ARN |
|---|---|
| us-east-1 | arn:aws:lambda:us-east-1:041475135427:layer:duckdb-extensions-nodejs-x86:6 |