/sqsmv

Moves messages from one sqs queue into another queue e.g. replay dlq to non-dlq

Primary LanguageRustMIT LicenseMIT

sqsmv

STATUS: alpha CI

Table of Contents

Description

Command-line tool that moves messages from one SQS queue into another queue e.g. replay dlq to non-dlq

It does so by:

  1. reading the "from" sqs queue in a batch of 1-10 messages
  2. writes to the target "to" queue in same batches ^
  3. deletes from the origin "from" queue messages of the batches ^
  4. repeats 1-3 until the origin queue is empty, an error occurs, or the max iterations (1m) is exceeded

Installation Options

Download the binary

  1. Download the v0.1.0-alpha binary for MacOx or Linux
  2. unzip it
  3. you may need to chmod +x sqsmv to give execute prives
  4. run it!
  5. (Optionally) add it to your PATH somewhere

Or install with Cargo

On Linux (prerequisite):apt-get install libssl-dev

On Mac & Linux:

  1. if you don't have cargo: install it
  2. cargo install sqsmv

Usage

Assuming you have AWS Named Profiles set up and both queues are under the same subaccount, this is how to run sqsmv from the command line ( also see sqsmv --help).

AWS_PROFILE=my-profile \
  sqsmv \
  --from-q <FROM_SQS_QUEUE> \
  --to-q <TO_SQS_QUEUE>

# e.g. it should look something like this:
AWS_PROFILE=profile11111111 \
  sqsmv \
  -f'https://sqs.us-east-1.amazonaws.com/11111111/my-queue-dlq' \
  -t'https://sqs.us-east-1.amazonaws.com/11111111/my-queue'

Errors

This program is very conservative with errors right now.

  1. If any READs on the SQS fail, the program will exit, reporting the READ failure
  2. If any WRITEs fail, the program will exit, reporting WRITE failures
  3. If any DELETEs fail, the program will exit, reporting DELETE failures that user can then use to do any cleanup necessary

Known Issues

AWS profiles configured with role_arn/source_profile can not be resolved at this time. Only profiles configured simply with valid aws_access_key_id/aws_secret_access_key are supported at this time.

Development

AWS_PROFILE=my-profile \
  cargo run -- \
  -f'https://sqs.us-east-1.amazonaws.com/<my-acct#>/my-queue-dlq' \
  -t'https://sqs.us-east-1.amazonaws.com/<my-acct#>/my-queue'