Many distributed systems and algorithms rely on tight timing assumptions on communication in the cluster. These assumptions drive failure detectors, leader change mechanisms, or simply help achieve maximum performance by helping systems stay in more optimal "(partially-)synchronous" execution paths. Unfortunately, there is not much open information about communication timing one may expect in various environments.
Our Cloud Latency Tester (CLT) is a tool to benchmark round-trip latency distributions between nodes in a cluster. The tool measures end-to-end latency, as experienced by applications deployed in some servers/VMs. This end-to-end latency includes more than just network latency, as the OS kernel, hypervisor, and the application add some latency overheads.
The tool is a basic echo-type application running over TCP using the sockets. While this approach is not the most sophisticated, it is representative of a typical application or system. Each worker node sends messages of configured size to other nodes, each peer echoes received messages, allowing the sender to record the round-trip time between itself and all nodes in the cluster. CLT records the data and eventually writes it to a CSV file for processing. CLT also provides some tools for the basic processing of data.
To build CLT executables, run make build command.
CLT has three executable files: master, node, and processing. Only the master and node executables are needed to run the experiments.
- master is a utility that starts and stops the testing. The master uses a JSON configuration file that describes the cluster and experimental parameters to connect to all nodes and instructs them to run an experiment.
- node is the actual echo-style application that must be deployed in the desired topology for testing. Nodes also write their raw data locally to disk. Each node has its own unique NodeID expressed in a "zoneId.nodeId" format.
Configuration JSON file specifies the nodes used in the experiment and the experimental parameters. Below is a sample file that expects the CLT to run 3 nodes on the local machine for 20 minutes with 300 messages per second from each node, carrying 1024 bytes of payload in each message:
{
"cluster_membership": {
"private_address": {
"1.1": "tcp://127.0.0.1:1735",
"1.2": "tcp://127.0.0.1:1736",
"1.3": "tcp://127.0.0.1:1737"
},
"public_address": {
"1.1": "tcp://127.0.0.1:1735",
"1.2": "tcp://127.0.0.1:1736",
"1.3": "tcp://127.0.0.1:1737"
}
},
"testing_duration_minutes": 20,
"payload_size":1024,
"testing_rate_s":300,
"self_loop":true,
"csv_prefix": "testing",
"csv_row_output_limit": 20000000,
"mem_row_output_limit": 20000000,
"communication_timeout_ms": 5000
}
-
The cluster_membership field describes the nodes expected to participate in the experiment. Each node can have a private and public address. These addresses can be the same, but can also differ, for instance, to test latency on a connection from a remote region over the public IP interface. As a rule, a node will use the peer's public_address to talk if the peer has a different "zoneId."
The addresses are specified in a map if NodeID -> Address, where address is a string specifying a protocol, IP address, and port number in the following format: "tcp://IP:Port". Currently, only TCP protocol is supported.
-
The testing_duration_minutes specifies testing duration in minutes. The nodes will automatically stop after that time.
-
The payload_size specifies the size of payload in each message, in bytes.
-
The testing_rate_s sets the number of communication rounds each node initiates per second.
-
The self_loop parameter specifies whether the nodes talk to themselves.
-
The csv_prefix is the file name prefix for raw CSV data.
-
The csv_row_output_limit sets maximum size of CSV.
-
The mem_row_output_limit sets maximum number of raw data entries kept in memory before writing to CSV. It is possible to store many hours worth of data in memory before writing out to disk.
-
The communication_timeout_ms is the upper limit on round-trip latency the tool expects. Essentially, this is a timeout for rounds taking too long.
To start a node, place the executable to a desired machine, and start it with the following command:
-
node -id=<NodeID> -port=<portNum>The NodeId must be a valid NodeID from a testing configuration file eventually supplied to master. Similarly, the port number must be the same port number as described in the address string in the configuration file.
To start the experiment, use the master utility with the following command:
-
master -start -config=<configFile.json>The master utility will connect to all nodes specified in the config file, deliver the configuration parameters, such as communication rate and payload size, and struct the nodes to start the experiment.
Coming soon
In March 2023, we deployed CLT in three public clouds - Azure, AWS, and GCP. The deployment topology is shown below:
We focused on testing latency between nodes/processes in the same region, although we performed some measurements across WAN. Nodes 1.1, 1.2, and 1.3 were deployed in the same subnet of the same availability zone. Node 1.4 is in the same AZ as nodes 1.1 - 1.3, but different subnet. Nodes 1.5 and 1.6 are in different AZs of the same region. We also used nodes 2.1 and 3.1 in different regions.
In each cloud we used smaller 2 vCPU VMs with 8 GB of RAM. While these VMs are on a smaller side, they represent more popular VM types in each cloud. We deployed CLT on top of Ubuntu 22.04 LTS.
Figure below shows the summary over a 6-hour run conducted on a weekday:
A more detailed view at latency distribution (top) and latency between individual nodes (bottom) in the same subnet of the AZ (nodes 1.1 - 1.3):
Latency distribution across AZs of the same region:
Latency for different payload sizes:
Latency for quorums:
Cross-region latency:
For more data and details, please see the full report (coming soon)
Terraform scripts for each supported cloud provider can be found at "./environments/$PROVIDER/"
Specialized tools that do not fit into the main purpose of the project are stored under the "tools" directory. Each will contain it's own README explaining what it's purpose is.