A tool to parse and transform Neo4j query logs. Allows generating JMeter test plans to replay queries.
usage: query-log-parser.sh
-i,--input <query.log> path of the query.log file to process. For Aura, the connection URL (ex: neo4j+s://ffffffff.databases.neo4j.io)
-o,--output [json|jmeter|standard] output type
-start <YYYY-MM-DD HH:MM:SS> Start timestamp, UTC time. (Optional. Defaults to 5min ago)
-end <YYYY-MM-DD HH:MM:SS> End timestamp, UTC time. (Optional. Defaults to now)
-q,--query-limit <x> Limit to the number of queries to consider (Optional. default:1000)
-p,--aura-project <gcp_project> GCP project hosting the Aura logs (Optional. only used if -i points to a Aura URL)
-s,--jmeter-speed <x> Multiplier applied to all time intervals to speed up/slow down the JMeter replay (Optional. default:1)
-t,--jmeter-max-threads <x> Limit to the number of JMeter thread groups to create (Optional. default:100)
-snl,--jmeter-sampler-name-length <x> Max length of Bolt Sampler (Optional. default:25)
-sto,--jmeter-sampler-tx-timeout <x> Transaction timeout in seconds (Optional. default:60)
-sam,--jmeter-sampler-access-mode [READ|WRITE] Access mode for tests against a cluster (Optional. default=WRITE)
-srr,--jmeter-sampler-record-results [true|false] Whether the Bolt response should be recorded by the Sampler [true|false] (Optional. default:true)
query.log can be in standard or json format (dbms.logs.query.format=json in neo4j 4.3). Format is auto-detected.
ex:
./query-log-parser.sh -i /path/to/query.log -o jmeter
./query-log-parser.sh -i /path/to/query.log -o json
./query-log-parser.sh -i "neo4j+s://12345.databases.neo4j.io" -o standard -p "aws-aura-customer"
outputs
In the same directory as the input file, with an extra suffix :
- json : query.log in json format ("inputfile.json")
- standard : query.log in standard format ("inputfile.std")
- jmeter : a JMeter test plan file to replay the query log ("inputfile.jmx")
For Aura exports, the file will be generated in a local subdirectory named "output".
The feature aims to generate a valid test plan xml file, with :
- a Bolt Connection Element
- Thread Groups for each client identified in the source field of the query.log, configured to run in parallel.
- within each thread group, a Bolt Sampler for each query (sent via bolt), scheduled according to the query timestamps with Timers.
- speedFactor : multiplier applied to all time intervals to speed up/slow down the replay (default=1)
- maxThreads : limit to the number of thread groups to create, and to the number of query sources to consider (default=100). Queries beyond that limit will be discarded.
- maxQueries : limit to the number of queries to consider (default=1000).
- samplerNameMaxLength : max length of Bolt Sampler "test name" (default=25). The Bolt Sampler is named with the truncated cypher query. That helps distinguish queries in JMeter listeners.
- samplerTxTimeout : transaction timeout in seconds, used by the JMeter's Neo4j driver (default=60). Applies to all samplers.
- samplerAccessMode READ|WRITE : Access mode for tests against a cluster (default=WRITE). Applies to all samplers.
- samplerRecordQueryResults true|false : whether the Bolt response should be recorded by the Sampler (default=true). Slows tests down, but helps debug. Applies to all samplers.
- [TODO] filters : filters on various fields (database, user, timestamp, duration, protocol, client, driver, metadata, regex against cypher) to select/exclude specific queries