This repository contains code for testing and comparing the performance of the KMIP and PKCS#11 protocols when used in a PKI context.
The envisaged scenario is that of a CA application communicating with a HSM over a network. To emulate this, a client and server is implemented for the following:
- KMIP - KMIP specifies the TTLV format as the default wire format, and this implementation uses the PyKMIP project for TTLV encoding/decoding.
- PKCS#11 using gRPC - The PKCS#11 standard defines a C API, and does not specify a canonical way to use PKCS#11 over a network. This implementation uses the gRPC protocol to transmit messages using protocol buffers.
- PKCS#11 using a REST API - In order to evaluate an alternate transport method, this implementation uses a simple JSON-based REST API for singing operations.
All implementations use TLS 1.3 with mutual authentication.
In order to test the performance of the protocols without relying on real HMS or specific cryptographic implementations, the servers use a "mock HSM" which emulates a HSM that performs signatures in constant time and can be tuned for different number of signatures per second.
The experiment runner can be used to run a set of experiments,
specified in JSON format. An example of such an experiment set can be seen in
the experiments.json file. A full JSON-schema is also available in
the experiment runner source code. The runner will perform all experiments from the given file
(by default experiments.json) and write output in CSV format to another file
(by default results.csv). Other locations can be specified using the --experiments-file
and --output-file command line options.
As an example, if the experiments.json file contains the following:
{
"experiments": [
{
"api": "kmip",
"hsm_capacity": 1000,
"num_signatures": 10000,
"kmip_batch_count": 100,
"threaded": false
}
]
}the command python3 experiment_runner.py will perform all experiments specified in
the "experiments" array. In this case this will be a single experiment where a KMIP
server will be started, using a mock HSM capable of performing 1000 signatures per
second. The time taken for a KMIP client to perform 10000 signatures using this server
will be timed, and the client will batch 100 signing requests in each message to the server.
The experiment runner stores relevant information about each experiment in CSV format. The fields in this format is in the following order:
- API used
- HSM capacity in (signatures per second)
- Number of signatures performed
- KMIP batch count (only present when using the KMIP API, otherwise left empty)
- Time taken (in seconds)
- Boolean indicating if threaded mode was used
For the example above, the output to the results.csv file might be something like:
kmip,1000,10000,100,36.754375431999506,False
When running the experiments on a Linux system, it is possible to use the
netem module to emulate
specific network conditions. This can be used to measure how the different APIs
perform depending on network latency. For example, to emulate a network latency
between client and server of 1 ms, running the command
sudo tc qdisc add dev lo root netem delay 1mswill add a delay of 1ms to all traffic sent on the loopback interface,
affecting the traffic between client and server on the local network.
Note that this requires sudo privileges.
In order to remove this emulated delay once the experiments have been run, the corresponding command is
sudo tc qdisc delete dev lo root netem