joe4dev/trigger-bench

Benchmark for Serverless Function Triggers

Serverless TriggerBench

This repository contains the aws (aws-triggers, azure-triggers) and data analysis scripts of the TriggerBench cross-provider serverless benchmark. It also bundles a customized extension of the serverless benchmarker tool to automate and analyze serverless performance experiments.

The dataset (i.e., data directory) is only included in the Zenodo replication package due to its size (>1GB).

TriggerBench

TriggerBench currently supports three triggers on AWS and eight triggers on Microsoft Azure.

Trigger	AWS Service	Azure Service
HTTP	API Gateway	API Management
Queue	SQS	Queue Storage
Storage	S3	Blob Storage
Database	DynomoDB¹	CosmosDB
Event	SNS¹	Event Grid
Stream	Kinesis¹	Event Hubs
Message	EventBridge¹	Service Bus Topic
Timer	CloudWatch Events¹	Timer

¹ Not implemented.

Dataset

The data directory is only included in the Zenodo replication package due to its size (>1GB).

The aws and azure sub-directories contain data from benchmark executions from April 2022. Each execution is a separate directory with a timestamp in the format yyyy-mm-dd-HH-MM-SS (e.g., 2022-04-15_21-58-52) and contains the following files:

• k6_metrics.csv: Load generator HTTP client logs in CSV format (see K6 docs)
• sb_config.yml: serverless benchmarker execution configuration including experiment label.
• trigger.csv: analyzer output CSV per trace.
  • root_trace_id: The trace id created by k6 and adopted by the invoker function
  • child_trace_id: The trace id newly created by the receiver function if trace propagation is not supported (this is the case for most asynchronous triggers)
  • t1-t4: Timestamps following the trace model (see paper)
  • t5-t9: Additional timestamps for measuring timestamping overhead
  • coldstart_f1=True|False: coldstart status for invoker (f1) and receiver (f2) functions
• trace_ids.txt: text file with each pair of root_trace_id and child_trace_id on a new line.
• traces.json: raw trace JSON representation as retrieved from the provider tracing service. For AWS, see X-Ray segment docs. For Azure, see Application Insights telemetry data model.
• workload_options.json: K6 load scenario configuration.

Replicate Data Analysis

Installation

1. Install Python 3.10+
2. Install Python dependencies pip install -r requirements.txt

Create Plots

1. Run python plots.py generates the plots and the statistical summaries presented in the paper.

By default, the plots will be saved into a plots sub-directory. An alternative output directory can be configured through the environment variable PLOTS_PATH.

Hint: For interactive development, we recommend the VSCode Python extension in interactive mode.

Replicate Cloud Experiments

The following experiment plan automate benchmarking experiments with different types workloads (constant and bursty). This generates a new dataset in the same format as described above.

1. Set up a load generator as vantage point following the description in LOADGENERATOR.
2. Choose the PROVIDER (aws or azure) in the constant.py experiment plan
3. Run the constant.py experiment plan
   1. Open tmux
   2. Activate virtualenv source sb-env/bin/activate
   3. Run ./constant.py 2>&1 | tee -a constant.log

Contributors

The initial trigger implementations for AWS and Azure are based on two master thesis projects at Chalmers University of Technology in Sweden supervised by Joel:

• AWS + Azure: Performance Comparison of Function-as- a-Service Triggers: A Cross-Platform Performance Study of Function Triggers in Function-as-a-Service by Marcus Bertilsson and Oskar Grönqvist, 2021.
• Azure Extension: Serverless Function Triggers in Azure: An Analysis of Latency and Reliability by Henrik Lagergren and Henrik Tao, 2022.

Joel contributed many improvements to their original source code as documented in the import commits a00b67a and 6d2f5ef and developed TriggerBench as integrated benchmark suite (see commit history for detailed changelog).