Tools to benchmark sagemaker serverless endpoint configurations and help find the most optimal one
To install the toolkit into your environment, first clone this repo. Then inside of the repo directory run
pip install sm-serverless-benchmarking
In order to run the benchmark, your user profile or execution role would need to have the appropriate IAM Permissions Including:
- CreateModel
- CreateEndpointConfig / DeleteEndpointConfig
- CreateEndpoint / DeleteEndpoint
- CreateProcessingJob (if using SageMaker Runner)
- InvokeEndpoint
- GetMetricStatistics
To run a benchmark locally, provide your sagemaker Model name and a list of example invocation arguments. Each of these arguments will be passed directly to the SageMaker runtime InvokeEndpoint API
from sm_serverless_benchmarking import benchmark
from sm_serverless_benchmarking.utils import convert_invoke_args_to_jsonl
model_name = "<SageMaker Model Name>"
example_invoke_args = [
{'Body': '1,2,3,4,5', "ContentType": "text/csv"},
{'Body': '6,7,8,9,10', "ContentType": "text/csv"}
]
example_args_file = convert_invoke_args_to_jsonl(example_invoke_args,
output_path=".")
r = benchmark.run_serverless_benchmarks(model_name, example_args_file)
Alternativelly, you can run the benchmarks as SageMaker Processing job
from sm_serverless_benchmarking.sagemaker_runner import run_as_sagemaker_job
run_as_sagemaker_job(
role="<execution_role_arn>",
model_name="<model_name>",
invoke_args_examples_file="<invoke_args_examples_file>",
)
A utility function sm_serverless_benchmarking.utils.convert_invoke_args_to_jsonl
is provided to convert a list of invocation argument examples into a JSONLines file. If working with data that cannot be serialized to JSON such as binary data including images, audio, and video, use the sm_serverless_benchmarking.utils.convert_invoke_args_to_pkl
function which will serilize the examples to a pickle file instead.
Refer to the sample_notebooks directory for complete examples
By default two types of benchmarks will be executed
- Stability Benchmark For each memory configuration, and a MaxConcurency of 1, will invoke the endpoint a specified number of times sequentially. The goal of this benchmark is to determine the most cost effective and stable memory configuration
- Concurrency Benchmark Will invoke an endpoint with a simulated number of concurrent clients under different MaxConcurrency configurations
For either of the two approaches described above, you can specify a number of parameters to configure the benchmarking job
cold_start_delay (int, optional): Number of seconds to sleep before starting the benchmark. Helps to induce a cold start on initial invocation. Defaults to 0.
memory_sizes (List[int], optional): List of memory configurations to benchmark Defaults to [1024, 2048, 3072, 4096, 5120, 6144].
stability_benchmark_invocations (int, optional): Total number of invocations for the stability benchmark. Defaults to 1000.
stability_benchmark_error_thresh (int, optional): The allowed number of endpoint invocation errors before the benchmark is terminated for a configuration. Defaults to 3.
include_concurrency_benchmark (bool, optional): Set True to run the concurrency benchmark with the optimal configuration from the stability benchmark. Defaults to True.
concurrency_benchmark_max_conc (List[int], optional): A list of max_concurency settings to benchmark. Defaults to [2, 4, 8].
concurrency_benchmark_invocations (int, optional): Total number of invocations for the concurency benchmark. Defaults to 1000.
concurrency_num_clients_multiplier (List[int], optional): List of multipliers to specify the number of simulated clients which is determined by max_concurency * multiplier. Defaults to [1, 1.5, 1.75, 2].
result_save_path (str, optional): The location to which the output artifacts will be saved. Defaults to ".".