HTTP Continuous benchmarking
This repository provides a continuous benchmarking.
If your project needs benchmark monitoring, it collects performance data by this repository.
Type | Badge/URL |
---|---|
CI | |
Release | |
Go Report Card | |
Coveralls | |
Coverage report for GitHub Pages | Coverage report GitHub Pages |
Docker Registry | http-continuous-benchmarking |
Concept
- Simple and lightweight benchmark tool
- This tool is not recommended for if large-scale performance measurements
- Continuous benchmarking tool
- Always monitor performance impacts for source code changes
- Lightweight to make various requests
- The user does not do complicated things
- Create measured reports and notify warnings
- Create a continuous report and associate it with a commit hash
- Warn and notice when the threshold is reached
How to use GitHub Actions
With input variable.
(※) is required.
With input name | Description | Example |
---|---|---|
target_url (※) | Request destination URL. | http(s)://xxxxxxx.com/api/v1/users |
http_headers (※) | Request HTTP Headers. Json {} format. |
{"Authorization": "Bearer xxx", "Content-Type": "application/json"} |
thread_num (※) | Client thread num. | 5 |
trial_num (※) | Benchmark trial number while 5seconds. If its 5times, the benchmark try 5times * 5seconds. | 5 (Ex: Case of API 100rps, 100(rps) * 5(seconds) * 5(times)) |
req_http_method_ratio (※) | HTTP method percentage of request. Json {} format. |
{"POST": 4, "GET": 6} |
req_body | HTTP Request Body. If you use PUT or PATCH or POST, its required. Json {} format. |
{"email": "xx@gmail.com"} |
enable_alert | Alert. You want to catch alert when over threshold. | true or false |
slack_web_hook_url | Slack web hook url. You can get here | https://hooks.slack.com/services/XXXXXXX/XXXXXXX/XXXXXXXXXXXXXX |
slack_channel | Slack channel. | test-01 |
slack_notify_threshold_latency_millis | Threshold latency milliseconds for notification. This is to check latency average. | 100 |
slack_notify_threshold_rps | Threshold request per seconds for notification. | 20 |
Sample GitHub actions workflow yaml.
on: [push]
jobs:
benchmarking:
runs-on: ubuntu-latest
name: Attack
steps:
- name: Benchmarking
id: benchmarking
uses: tomoyane/http-continuous-benchmarking@1.1.0
with:
target_url: 'https://example.com'
http_headers: '{"Content-Type":"application/json"}'
thread_num: '1'
trial_num: '1'
req_http_method_ratio: '{"GET": 10}'
- name: Completed
run: echo "Completed benchmarking"
With Slack notification workflow yaml.
on: [push]
jobs:
benchmarking:
runs-on: ubuntu-latest
name: Attack
steps:
- name: Benchmarking
id: benchmarking
uses: tomoyane/http-continuous-benchmarking@1.1.0
with:
target_url: 'https://example.com'
http_headers: '{"Content-Type":"application/json"}'
thread_num: '1'
trial_num: '1'
req_http_method_ratio: '{"GET": 10}'
enable_alert: 'true'
slack_web_hook_url: 'https://hooks.slack.com/services/XXXXXXX/XXXXXXX/XXXXXXXXXXXXXX'
slack_channel: 'test-01'
slack_notify_threshold_latency_millis: 20
- name: Completed
run: echo "Completed benchmarking"
If result is over threshold, you will catch bellow message.
How to use simple application
This repository application usage
Application basic usage.
# Set required environment variable before execution
$ git clone https://github.com/tomoyane/http-continuous-benchmarking.git
$ cd http-continuous-benchmarking; go build
$ export INPUT_TARGET_URL='https://example.com' \
INPUT_REQ_HTTP_METHOD_RATIO='{"GET":10}' \
INPUT_HTTP_HEADERS='{"Content-Type":"application/json"}' \
INPUT_THREAD_NUM=2 \
INPUT_TRIAL_NUM=2
$ ./http-continuous-benchmarking
## With Alert to Slack
$ export INPUT_TARGET_URL='https://example.com' \
INPUT_REQ_HTTP_METHOD_RATIO='{"GET":10}' \
INPUT_HTTP_HEADERS='{"Content-Type":"application/json"}' \
INPUT_THREAD_NUM=2 \
INPUT_TRIAL_NUM=2 \
INPUT_ENABLE_ALERT=true \
INPUT_SLACK_WEB_HOOK_URL="https://hooks.slack.com/services/XXXXXXX/XXXXXXX/XXXXXXXXXXXXXX" \
INPUT_SLACK_CHANNEL="test-01" \
INPUT_SLACK_NOTIFY_THRESHOLD_LATENCY_MILLIS=100 # Case of latency
$ ./http-continuous-benchmarking
Docker image usage.
$ docker pull tomohito/http-continuous-benckmarking
$ docker run \
--env INPUT_TARGET_URL='https://example.com' \
--env INPUT_REQ_HTTP_METHOD_RATIO='{"GET":10}' \
--env INPUT_HTTP_HEADERS='{"Content-Type":"application/json"}' \
--env INPUT_THREAD_NUM=2 \
--env INPUT_TRIAL_NUM=2 \
-i tomohito/http-continuous-benckmarking
Example benchmark result
Metrics report HTML
Log
$ ./http-continuous-benchmarking
HTTP request pattern according to the ratio = GET GET GET GET GET GET GET GET GET GET
Start warnmup for duration 5 seconds
End warmup
Start time = 1614653873
(Thread-3): Start attack for duration 5 seconds
(Thread-1): Start attack for duration 5 seconds
(Thread-2): Start attack for duration 5 seconds
(Thread-1): End attack
(Thread-2): End attack
(Thread-3): End attack
GET request stats information
Latency 99 percentile: 190 milliseconds
Latency 95 percentile: 172 milliseconds
Latency avg percentile: 115 milliseconds
Latency max percentile: 210 milliseconds
Latency min percentile: 72 milliseconds
Request per seconds: 30
(Thread-1): Start attack for duration 5 seconds
(Thread-3): Start attack for duration 5 seconds
(Thread-2): Start attack for duration 5 seconds
(Thread-1): End attack
(Thread-2): End attack
(Thread-3): End attack
GET request stats information
Latency 99 percentile: 153 milliseconds
Latency 95 percentile: 129 milliseconds
Latency avg percentile: 94 milliseconds
Latency max percentile: 192 milliseconds
Latency min percentile: 65 milliseconds
Request per seconds: 36
End time = 1614653894