The OMAR SCDF S3 source application is a Spring Cloud Data Flow service involved in the ingest of imagery. It operates through monitoring an AWS SQS, downloading the message information, and forwarding the message down the ingest pipeline.
- The file to be downloaded is stored in a S3 Bucket.
- The S3 Bucket has a SNS Topic associated with it that sends a notification when a new file is added to the bucket.
- The SQS queue is registered to that SNS Topic.
The message sent by this app sends the following info:
- filename - The name of the file
- fileFullPath - The full path, which includes the filename, to where the file was downloaded to
Omar SCDF S3 Source app has the following properties which can be overwritten during deployment.
- spring.cloud.stream.bindings.output.destination
- The message output desination. (String, default: files-extracted)
- sqs.queue
- AWS SQS to listen to. (String, default: aws-sqs-queue)
- local.dir
- Where the file is downloaded to. (String, default: /tmp)
- filter.file.type
- Used to tell what type of files to download. (String, default: nitf,ntf)
The Amazon S3 Source (as all other Amazon AWS applications) is based on the Spring Cloud AWS project as a foundation, and its auto-configuration classes are used automatically by Spring Boot. Consult their documentation regarding required and useful auto-configuration properties.
Some of them are about AWS credentials:
- cloud.aws.credentials.accessKey
- cloud.aws.credentials.secretKey
- cloud.aws.credentials.instanceProfile
- cloud.aws.credentials.profileName
- cloud.aws.credentials.profilePath
Other are for AWS Region definition:
- cloud.aws.region.auto
- cloud.aws.region.static
And for AWS Stack:
- cloud.aws.stack.auto
- cloud.aws.stack.name