move
Part of the senzing-tools suite.
This command can run stand-alone or as part of senzing-tools.
Stand-alone syntax:
move --input-url "https://public-read-access.s3.amazonaws.com/TestDataSets/SenzingTruthSet/truth-set-3.0.0.jsonl" --output-url "amqp://guest:guest@192.168.6.96:5672?exchange=senzing-rabbitmq-exchange&queue-name=senzing-rabbitmq-queue&routing-key=senzing.records"as part of senzing-tools it is a sub-command:
senzing-tools move --input-url "https://public-read-access.s3.amazonaws.com/TestDataSets/SenzingTruthSet/truth-set-3.0.0.jsonl" --output-url "amqp://guest:guest@192.168.6.96:5672?exchange=senzing-rabbitmq-exchange&queue-name=senzing-rabbitmq-queue&routing-key=senzing.records"go run . --input-url "file:///home/roncewind/data/truth-set-3.0.0.jsonl" --output-url "amqp://user:bitnami@senzing-rabbitmq:5672?exchange=senzing-rabbitmq-exchange&queue-name=senzing-rabbitmq-queue&routing-key=senzing.records"
comparison of different buffer and worker settings
| read chan buf | workers | time to move 1M (s) |
|---|---|---|
| 10 | 20 | 14 |
| 100 | 20 | 14 |
| 10 | 10 | 15 |
| 50 | 100 | 15 |
| 0 | 20 | 18 |
| 100 | 1000 | 19 |
| 10 | 4 | 21 |