This project is an example of using Spring Boot with Spring integration and SFTP.
An inbound channel adapter will connect to a SFTP server configured in application.properties.
It will download all remote files and in our example it expects the files to contain a list of numbers separated by a line break.
Each number line will be processed and check if the number is a prime.
If success the line will be logged if not an error will be thrown and the channel adapter poller will process the file again after sftp-poller-delay defined in application.properties.
Some design notes:
- SFTP connection is done via caching session, and it will maintain permanent open connections. Testing of stale sessions is enabled.
- SFTP connection security is done via private key and known hosts file.
- Remote file synchronisation is done only once/file via SftpPersistentAcceptOnceFileListFilter. In case of failure the file is removed automatically from the filter's memory so a new retry will be possible on the next polling.
- A remote file will be copied into a local file as a processing fail-over strategy.
- Local file processing is done only once/file via AcceptOnceFileListFilter, and it will be deleted in case of success by Pseudo Transaction Manager configured inside SftpAdapter.java. In case of failure the local file will be processed again on the next polling.
- Testing is done via an embedded Apache Mina sshd server.
- Docker project for SFTP server is provided to simulate a Production environment
- Automatically reconnect and retry in case of network failure.
- Recovery from catastrophic or processing failure via local file storage and Transaction Manager.
- Send any errors from errorChannel to the provided health-check via Spring Actuator.
- Health-check is secured via Spring Security.
Requires JDK 17
To simulate a production SFTP server (if you don't have one already configured) the project provides a docker setup inside the ./docker folder.
Upload file(s): docker/sftp1/upload/numbers.txt
To run the project:
cd springboot-integration-sftp
docker-compose -f docker/docker-compose.yml build && docker-compose -f docker/docker-compose.yml up
./gradlew bootJar && java -jar build/libs/springboot-integration-sftp-1.0.jar
To run the tests use the following gradle task:
./gradlew clean test
Note
Testing is done with the provided embedded Apache Mina sshd server.
Since both Apache Mina and the docker SFTP provided above use the same SFTP port (2222) you might want to stop the docker container when running the test task
Tip
Health-check can be accessed via: http://localhost:8080/actuator/health/application
In case any Load Balancer will need to check if the application si alive this endpoint will return an HTTP 200 OK no matter the file processing status: http://localhost:8080/actuator/info
Tip
I case you have older JDK version than JDK 17 and you are using IntelliJIDEA as IDE make sure Gradle JVM is set to JDK 17 in:
File | Settings | Build, Execution, Deployment | Build Tools | Gradle -> Gradle JVM
