An async Rust application that inserts simulated time-series data into a TimescaleDB instance and reports ingestion rate statistics.
An example of time-series data is a stream of measurements coming from a group of devices. A measurement has a timestamp, a device identifier, and a number of metrics that are generated by that device.
The following command:
$ time ./tsdbperf --workers 8 --devices 10 --measurements 100000 --metrics 10
[2020-10-31T14:33:47Z INFO tsdbperf] Number of workers: 8
[2020-10-31T14:33:47Z INFO tsdbperf] Devices per worker: 10
[2020-10-31T14:33:47Z INFO tsdbperf] Metrics per device: 10
[2020-10-31T14:33:47Z INFO tsdbperf] Measurements per device: 100000
[2020-10-31T14:33:59Z INFO tsdbperf] Wrote 8000000 measurements in 12.59 seconds
[2020-10-31T14:33:59Z INFO tsdbperf] Wrote 635677 measurements per second
[2020-10-31T14:33:59Z INFO tsdbperf] Wrote 6356774 metrics per second
real 0m12.791s
user 0m5.981s
sys 0m1.845sruns 8 parallel workers, for each worker it generates data for 10 devices and for each device it generates 100,000 measurements each having 10 metrics. On my laptop, that has 12 CPUs, it took 12.59 seconds to insert 8,000,000 measurements.
Now that we have generated the data we can run queries to inspect it:
postgres@localhost> select count(distinct(device_id)) from measurement
+---------+
| count |
|---------|
| 80 |
+---------+
postgres@localhost> select count(*) from measurement
+---------+
| count |
|---------|
| 8000000 |
+---------+
postgres@localhost> select min(time), max(time) from measurement
+-------------------------------+-------------------------------+
| min | max |
|-------------------------------+-------------------------------|
| 2020-10-30 10:47:08.315839+00 | 2020-10-31 14:33:47.316317+00 |
+-------------------------------+-------------------------------+The hypertable chunk interval is 1 day, with the data spanning 2 days we should have 2 chunks:
postgres@localhost> select chunk_table, table_size
from chunk_relation_size_pretty('measurement')
+-----------------------------------------+--------------+
| chunk_table | table_size |
|-----------------------------------------+--------------|
| _timescaledb_internal._hyper_5_9_chunk | 458 MB |
| _timescaledb_internal._hyper_5_10_chunk | 504 MB |
+-----------------------------------------+--------------+With 10 metrics per measurement the schema looks like:
postgres@localhost> \d measurement
+-----------+--------------------------+-------------+
| Column | Type | Modifiers |
|-----------+--------------------------+-------------|
| time | timestamp with time zone | not null |
| device_id | oid | |
| m1 | double precision | |
| m2 | double precision | |
| m3 | double precision | |
| m4 | double precision | |
| m5 | double precision | |
| m6 | double precision | |
| m7 | double precision | |
| m8 | double precision | |
| m9 | double precision | |
| m10 | double precision | |
+-----------+--------------------------+-------------+
Indexes:
"measurement_device_id_time_idx" btree (device_id, "time" DESC)
"measurement_time_idx" btree ("time" DESC)
Triggers:
ts_insert_blocker BEFORE INSERT ON measurement FOR EACH ROW EXECUTE FUNC...
Number of child tables: 2 (Use \d+ to list them.)If we run with the --with-jsonb option then all metrics are inserted into
a metrics column as JSONB:
postgres@localhost:postgres> select time, metrics from measurement where device_id=3 limit 5
+-------------------------------+----------------------------------------------------+
| time | metrics |
|-------------------------------+----------------------------------------------------|
| 2020-12-19 14:34:04.116101+00 | {"m1": 995.5610470263719, "m2": 939.2307180345128} |
| 2020-12-19 14:34:05.116101+00 | {"m1": 995.6124798524728, "m2": 938.6678274276368} |
| 2020-12-19 14:34:06.116101+00 | {"m1": 996.0597176913179, "m2": 938.5417961830267} |
| 2020-12-19 14:34:07.116101+00 | {"m1": 995.5232004280007, "m2": 938.7215776266249} |
| 2020-12-19 14:34:08.116101+00 | {"m1": 995.0165418185591, "m2": 939.023750940609} |
+-------------------------------+----------------------------------------------------+Github release action creates binaries for Linux and Mac, they can be downloaded from the Releases page as compressed binaries for both x86-64 and AArch64 architectures.
The binary needs to be made executable after downloading:
$ gunzip tsdbperf-x86_64-apple-darwin.gz
$ chmod +x tsdbperf-x86_64-apple-darwin
$ ./tsdbperf-x86_64-apple-darwin --help
tsdbperf 0.1.8
USAGE:
tsdbperf-x86_64-apple-darwin [FLAGS] [OPTIONS]
FLAGS:
--dry-run Skip DB inserts, report only data generation timings
-h, --help Prints help information
--no-hypertables Run the tests without creating hypertables
-V, --version Prints version information
--with-copy-upserts Run the tests with copy in upserts
--with-jsonb Insert metrics as a JSONB column
--with-upserts Run the tests with upserts
OPTIONS:
--batch-size <batch-size> Number of measurements per insert [default: 10000]
--chunk-interval <chunk-interval> Hypertable chunk interval in seconds [default: 86400]
--db-host <db-host> Database host [default: localhost]
--db-name <db-name> Database name [default: postgres]
--db-password <db-password> Database password [default: postgres]
--db-user <db-user> Database user [default: postgres]
--devices <num-devices> Number of devices per worker [default: 10]
--measurements <num-measurements> Number of measurements per device [default: 100000]
--metrics <num-metrics> Number of metrics per device [default: 10]
--workers <num-workers> Number of parallel workers [default: number of CPUs]If you have docker-compose installed the easiest way to run tsdbperf is to download this compose file, run a local instance of TimescaleDB:
$ docker-compose up -d timescale
Creating network "docker_default" with the default driver
Creating docker_timescale_1 ... donethen run tsdbperf with docker-compose run --rm tsdbperf:
$ docker-compose run --rm tsdbperf
[2020-11-07T20:08:06Z INFO tsdbperf] Number of workers: 12
[2020-11-07T20:08:06Z INFO tsdbperf] Devices per worker: 10
[2020-11-07T20:08:06Z INFO tsdbperf] Metrics per device: 10
[2020-11-07T20:08:06Z INFO tsdbperf] Measurements per device: 100000
[2020-11-07T20:08:27Z INFO tsdbperf] Wrote 12000000 measurements in 19.47 seconds
[2020-11-07T20:08:27Z INFO tsdbperf] Wrote 616428 measurements per second
[2020-11-07T20:08:27Z INFO tsdbperf] Wrote 6164278 metrics per secondIf you want to override any of the default options you can pass them
to the run command, for example to run with 6 workers, 15 devices
per worker, and 5 metrics per device:
$ docker-compose run --rm tsdbperf --workers 6 --devices 15 --metrics 5
[2020-11-07T19:55:12Z INFO tsdbperf] Number of workers: 6
[2020-11-07T19:55:12Z INFO tsdbperf] Devices per worker: 15
[2020-11-07T19:55:12Z INFO tsdbperf] Metrics per device: 5
[2020-11-07T19:55:12Z INFO tsdbperf] Measurements per device: 100000
[2020-11-07T19:55:28Z INFO tsdbperf] Wrote 9000000 measurements in 14.64 seconds
[2020-11-07T19:55:28Z INFO tsdbperf] Wrote 614880 measurements per second
[2020-11-07T19:55:28Z INFO tsdbperf] Wrote 3074400 metrics per secondTo shutdown TimescaleDB and clean up containers run:
$ docker-compose down -v
Stopping docker_timescale_1 ... done
Removing docker_timescale_1 ... done
Removing network docker_defaultIf you have docker installed and would like to run tests against a running TimescaleDB instance you can use the following command:
$ docker run --rm --network host vincev/tsdbperf --db-host localhost
[2020-11-25T22:41:57Z INFO tsdbperf] Number of workers: 12
[2020-11-25T22:41:57Z INFO tsdbperf] Devices per worker: 10
[2020-11-25T22:41:57Z INFO tsdbperf] Metrics per device: 10
[2020-11-25T22:41:57Z INFO tsdbperf] Measurements per device: 100000
[2020-11-25T22:42:16Z INFO tsdbperf] Wrote 12000000 measurements in 18.73 seconds
[2020-11-25T22:42:16Z INFO tsdbperf] Wrote 640752 measurements per second
[2020-11-25T22:42:16Z INFO tsdbperf] Wrote 6407518 metrics per second