move

If you are beginning your journey with Senzing, please start with Senzing Quick Start guides.

You are in the Senzing Garage where projects are "tinkered" on. Although this GitHub repository may help you understand an approach to using Senzing, it's not considered to be "production ready" and is not considered to be part of the Senzing product. Heck, it may not even be appropriate for your application of Senzing!

⚠️ WARNING: move is still in development ⚠️ _

At the moment, this is "work-in-progress" with Semantic Versions of 0.n.x. Although it can be reviewed and commented on, the recommendation is not to use it yet.

Synopsis

move is a command in the senzing-tools suite of tools. This command moves records between files and queues.

Overview

move moves records from a file to a queue or another file. In other words, it is a general tool for moving records around. When it does this it validates each record to ensure that it is valid JSON and contains two necessary key-value pairs: RECORD_ID and DATA_SOURCE.

A file is given to move with the command-line parameter input-url or as the environment variable SENZING_TOOLS_INPUT_URL. Note this is a URL so local files will need file:// and remote files http:// or https://. If the given file has the .gz extension, it will be treated as a compressed file JSONL file. If the file has a .jsonl extension it will be treated accordingly. If the file has another extension it will be rejected, unless the input-file-type or SENZING_TOOLS_INPUT_FILE_TYPE is set to JSONL. For example, if you have a JSONL formatted file with the URL file:///tmp/data.json, it will be rejected unless --input-file-type=JSONL parameter or the equivalent environment variable is set.

Using the command-line parameter output-url or the environment variable SENZING_TOOLS_OUTPUT_URL a queue can also be specified to put records into. URLs starting with amqp:// are interpreted as RabbitMQ queues. URLs starting with sqs:// are interpreted as SQS queue look-ups. URLs starting with https:// are interpreted as SQS queue URLs. See Parameters for additional information on SQS. Files can also be specified as an output URL.

Install

The move command is installed with the senzing-tools suite of tools. See senzing-tools install.

Use

senzing-tools move [flags]

For options and flags:
1. Online documentation
2. Runtime documentation:
```
senzing-tools move --help
```
In addition to the following simple usage examples, there are additional Examples.

Using command line options

✏️ Specify file URL using command line option. Example:

senzing-tools move \
    --input-url "https://public-read-access.s3.amazonaws.com/TestDataSets/SenzingTruthSet/truth-set-3.0.0.jsonl" \
    --output-url "amqp://guest:guest@192.168.6.128:5672"

See Parameters for additional parameters.

Using environment variables

✏️ Specify file URL using environment variable. Example:

export SENZING_TOOLS_INPUT_URL=https://public-read-access.s3.amazonaws.com/TestDataSets/SenzingTruthSet/truth-set-3.0.0.jsonl
export SENZING_TOOLS_OUTPUT_URL=amqp://guest:guest@192.168.6.128:5672
senzing-tools move

See Parameters for additional parameters.

Using Docker

This usage shows how to move a file with a Docker container.

✏️ Run senzing/senzing-tools. Example:

docker run \
    --env SENZING_TOOLS_COMMAND=move \
    --env SENZING_TOOLS_INPUT_URL=https://public-read-access.s3.amazonaws.com/TestDataSets/SenzingTruthSet/truth-set-3.0.0.jsonl \
    --rm \
    senzing/senzing-tools

See Parameters for additional parameters.

Parameters

SENZING_TOOLS_INPUT_FILE_TYPE
SENZING_TOOLS_INPUT_URL
SENZING_TOOLS_JSON_OUTPUT
SENZING_TOOLS_LOG_LEVEL
SENZING_TOOLS_OUTPUT_URL
Notes about the output-url:
- At present SQS URLs can be formed in two ways 1. If your environment is set up such that SQS queue names can be queried, then an sqs:// URL can be used when formed as such sqs://lookup?queue-name=myqueue 1. Alternatively, use the direct HTTPS URL of the SQS queue, which is found in the AWS SQS console.

senzing-garage/move