/redoer

Perform Senzing Redo operations.

Primary LanguagePythonApache License 2.0Apache-2.0

redoer

If you are beginning your journey with Senzing, please start with Senzing Quick Start guides.

You are in the Senzing Garage where projects are "tinkered" on. Although this GitHub repository may help you understand an approach to using Senzing, it's not considered to be "production ready" and is not considered to be part of the Senzing product. Heck, it may not even be appropriate for your application of Senzing!

Synopsis

Process Senzing redo records.

Overview

The redoer.py python script processes Senzing "redo" records. The senzing/redoer docker image is a wrapper for use in docker formations (e.g. docker-compose, kubernetes).

To see all of the subcommands, run:

$ ./redoer.py

usage: redoer.py [-h]
                 {redo,redo-withinfo-kafka,redo-withinfo-rabbitmq,redo-withinfo-sqs,read-from-kafka,read-from-kafka-withinfo,read-from-rabbitmq,read-from-rabbitmq-withinfo,read-from-sqs,read-from-sqs-withinfo,write-to-kafka,write-to-rabbitmq,write-to-sqs,sleep,version,docker-acceptance-test}
                 ...

Process Senzing redo records. For more information, see https://github.com/senzing-garage/redoer

positional arguments:
  {redo,redo-withinfo-kafka,redo-withinfo-rabbitmq,redo-withinfo-sqs,read-from-kafka,read-from-kafka-withinfo,read-from-rabbitmq,read-from-rabbitmq-withinfo,read-from-sqs,read-from-sqs-withinfo,write-to-kafka,write-to-rabbitmq,write-to-sqs,sleep,version,docker-acceptance-test}

                                 Subcommands (SENZING_SUBCOMMAND):
    redo                         Read Senzing redo records from Senzing SDK and send to G2Engine.process()
    redo-withinfo-kafka          Read Senzing redo records from Senzing SDK, send toG2Engine.processWithInfo(), results sent to Kafka.
    redo-withinfo-rabbitmq       Read Senzing redo records from Senzing SDK, send to G2Engine.processWithInfo(), results sent to RabbitMQ.
    redo-withinfo-sqs            Read Senzing redo records from Senzing SDK, send to G2Engine.processWithInfo(), results sent to AWS SQS.
    read-from-kafka              Read Senzing redo records from Kafka and send to G2Engine.process()
    read-from-kafka-withinfo     Read Senzing redo records from Kafka and send to G2Engine.processWithInfo()
    read-from-rabbitmq           Read Senzing redo records from RabbitMQ and send to G2Engine.process()
    read-from-rabbitmq-withinfo  Read Senzing redo records from RabbitMQ and send to G2Engine.processWithInfo()
    read-from-sqs                Read Senzing redo records from AWS SQS and send to G2Engine.process()
    read-from-sqs-withinfo       Read Senzing redo records from AWS SQS and send to G2Engine.processWithInfo()
    write-to-kafka               Read Senzing redo records from Senzing SDK and send to Kafka.
    write-to-rabbitmq            Read Senzing redo records from Senzing SDK and send to RabbitMQ.
    write-to-sqs                 Read Senzing redo records from Senzing SDK and send to AWS SQS.
    sleep                        Do nothing but sleep. For Docker testing.
    version                      Print version of program.
    docker-acceptance-test       For Docker acceptance testing.

optional arguments:
  -h, --help            show this help message and exit

To see the options for a subcommand, run commands like:

./redoer redo --help

Contents

  1. Preamble
    1. Legend
  2. Expectations
  3. Demonstrate using Command Line Interface
    1. Prerequisites for CLI
    2. Download
    3. Environment variables for CLI
    4. Run command
  4. Demonstrate using Docker
    1. Prerequisites for Docker
    2. Database support
    3. External database
    4. Run Docker container
  5. Configuration
  6. References
  7. License

Preamble

At Senzing, we strive to create GitHub documentation in a "don't make me think" style. For the most part, instructions are copy and paste. Whenever thinking is needed, it's marked with a "thinking" icon 🤔. Whenever customization is needed, it's marked with a "pencil" icon ✏️. If the instructions are not clear, please let us know by opening a new Documentation issue describing where we can improve. Now on with the show...

Legend

  1. 🤔 - A "thinker" icon means that a little extra thinking may be required. Perhaps there are some choices to be made. Perhaps it's an optional step.
  2. ✏️ - A "pencil" icon means that the instructions may need modification before performing.
  3. ⚠️ - A "warning" icon means that something tricky is happening, so pay attention.

Expectations

  • Space: This repository and demonstration require 6 GB free disk space.
  • Time: Budget 40 minutes to get the demonstration up-and-running, depending on CPU and network speeds.
  • Background knowledge: This repository assumes a working knowledge of:

Demonstrate using Command Line Interface

Prerequisites for CLI

🤔 The following tasks need to be complete before proceeding. These are "one-time tasks" which may already have been completed.

  1. Install system dependencies:
    1. Use apt based installation for Debian, Ubuntu and others
      1. See apt-packages.txt for list
    2. Use yum based installation for Red Hat, CentOS, openSuse and others.
      1. See yum-packages.txt for list
  2. Install Python dependencies:
    1. See requirements.txt for list
      1. Installation hints
  3. 🤔 Optional: Some databases need additional support. For other databases, this step may be skipped.
    1. Db2: See Support Db2.
    2. MS SQL: See Support MS SQL.
  4. Configure Senzing database

Download

  1. Get a local copy of template-python.py. Example:

    1. ✏️ Specify where to download file. Example:

      export SENZING_DOWNLOAD_FILE=~/redoer.py

    2. Download file. Example:

      curl -X GET \
  --output ${SENZING_DOWNLOAD_FILE} \
  https://raw.githubusercontent.com/Senzing/redoer/main/redoer.py

    3. Make file executable. Example:

      chmod +x ${SENZING_DOWNLOAD_FILE}

  2. 🤔 Alternative: The entire git repository can be downloaded by following instructions at Clone repository

Environment variables for CLI

  1. ✏️ Identify the Senzing g2 directory. Example:

    export SENZING_G2_DIR=/opt/senzing/g2

    1. Here's a simple test to see if SENZING_G2_DIR is correct. The following command should return file contents. Example:

      cat ${SENZING_G2_DIR}/g2BuildVersion.json

  2. Set common environment variables Example:

    export PYTHONPATH=${SENZING_G2_DIR}/sdk/python

  3. 🤔 Set operating system specific environment variables. Choose one of the options.

    1. Option #1: For Debian, Ubuntu, and others. Example:

      export LD_LIBRARY_PATH=${SENZING_G2_DIR}/lib:${SENZING_G2_DIR}/lib/debian:$LD_LIBRARY_PATH

    2. Option #2 For Red Hat, CentOS, openSuse and others. Example:

      export LD_LIBRARY_PATH=${SENZING_G2_DIR}/lib:$LD_LIBRARY_PATH

Run command

  1. Run the command. Example:

    ${SENZING_DOWNLOAD_FILE} --help

Demonstrate using Docker

Prerequisites for Docker

🤔 The following tasks need to be complete before proceeding. These are "one-time tasks" which may already have been completed.

  1. The following software programs need to be installed:
    1. docker

Database support

🤔 Optional: Some databases need additional support. For other databases, these steps may be skipped.

  1. Db2: See Support Db2 instructions to set SENZING_OPT_IBM_DIR_PARAMETER.
  2. MS SQL: See Support MS SQL instructions to set SENZING_OPT_MICROSOFT_DIR_PARAMETER.

External database

🤔 Optional: Use if storing data in an external database. If not specified, the internal SQLite database will be used.

  1. ✏️ Specify database. Example:

    export DATABASE_PROTOCOL=postgresql
export DATABASE_USERNAME=postgres
export DATABASE_PASSWORD=postgres
export DATABASE_HOST=senzing-postgresql
export DATABASE_PORT=5432
export DATABASE_DATABASE=G2

  2. Construct Database URL. Example:

    export SENZING_DATABASE_URL="${DATABASE_PROTOCOL}://${DATABASE_USERNAME}:${DATABASE_PASSWORD}@${DATABASE_HOST}:${DATABASE_PORT}/${DATABASE_DATABASE}"

  3. Construct parameter for docker run. Example:

    export SENZING_DATABASE_URL_PARAMETER="--env SENZING_DATABASE_URL=${SENZING_DATABASE_URL}"

Run Docker container

Although the Docker run command looks complex, it accounts for all of the optional variations described above. Unset *_PARAMETER environment variables have no effect on the docker run command and may be removed or remain.

  1. ✏️ Set environment variables. Example:

    export SENZING_SUBCOMMAND=redo

  2. Run Docker container. Example:

    sudo docker run \
  --env SENZING_SUBCOMMAND \
  --interactive \
  --rm \
  --tty \
  ${SENZING_DATABASE_URL_PARAMETER} \
  ${SENZING_OPT_IBM_DIR_PARAMETER} \
  ${SENZING_OPT_MICROSOFT_DIR_PARAMETER} \
  senzing/redoer

Configuration

Configuration values specified by environment variable or command line parameter.

License

View license information for the software container in this Docker image. Note that this license does not permit further distribution.

This Docker image may also contain software from the Senzing GitHub community under the Apache License 2.0.

Further, as with all Docker images, this likely also contains other software which may be under other licenses (such as Bash, etc. from the base distribution, along with any direct or indirect dependencies of the primary software being contained).

As for any pre-built image usage, it is the image user's responsibility to ensure that any use of this image complies with any relevant licenses for all software contained within.

References

  1. Development
  2. Errors
  3. Examples
  4. Related artifacts:
    1. DockerHub
    2. Helm Chart