Unclear dependency requirements for the S3 code examples
LukeDowell opened this issue · 2 comments
Apologies in advance if this is not the preferred way of doing this / if I just missed something obvious and confused myself.
I have a small application that I've been trying to wire up with Spring Integration and S3. After pulling in the integration and aws starters from the initializr, I copy-pasted the example inbound s3 streaming code in the README and also referred to the code in the test. I was getting some whack compilation errors where everything would appear fine but when I'd dig into any files from spring-integration-aws, none of their superclasses could be found. The issue ended up being that I needed spring-integration-file in addition to the normal spring-integration-starter dependency.
I've been working with Spring for a while so I was able to figure it out, but it kind of feels like something that would be very difficult to figure out for a newer engineer or even someone experienced but who is new to Spring. Perhaps this isn't even a problem and more just a question of what is / is not included in a given starter.
The README lists spring-cloud, spring-cloud-aws and spring-integration as required dependencies, should spring-integration-file be added as well?
Below I've added my poms and a snippet of the problem code, please let me know if I'm not being clear:
pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.7.4</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>dev.dowell</groupId>
<artifactId>s3-spring-integration-demo</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>s3-spring-integration-demo</name>
<description>s3-spring-integration-demo</description>
<properties>
<java.version>11</java.version>
<spring-cloud.version>2020.0.6</spring-cloud.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-integration</artifactId>
</dependency>
<dependency> <! -- !!!!!!!! This is the dependency that breaks everything if you remove it !!!!!!! ->
<groupId>org.springframework.integration</groupId>
<artifactId>spring-integration-file</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-devtools</artifactId>
<scope>runtime</scope>
<optional>true</optional>
</dependency>
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.integration</groupId>
<artifactId>spring-integration-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.integration</groupId>
<artifactId>spring-integration-aws</artifactId>
<version>2.5.2</version>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-aws</artifactId>
<version>2.2.6.RELEASE</version>
<type>pom</type>
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>${spring-cloud.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<excludes>
<exclude>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
</exclude>
</excludes>
</configuration>
</plugin>
</plugins>
</build>
</project>
CsvIngestConfig.java
@Configuration
public class CsvIngestConfig {
@Autowired
private AmazonS3 amazonS3;
@Bean
public S3InboundFileSynchronizer s3InboundFileSynchronizer() {
S3InboundFileSynchronizer synchronizer = new S3InboundFileSynchronizer(amazonS3);
synchronizer.setDeleteRemoteFiles(true);
synchronizer.setPreserveTimestamp(true);
synchronizer.setRemoteDirectory("some-remote-s3-bucket");
synchronizer.setFilter(new S3RegexPatternFileListFilter(".*\\.test$"));
// Expression expression = PARSER.parseExpression("#this.toUpperCase() + '.a'");
// synchronizer.setLocalFilenameGeneratorExpression(expression);
return synchronizer;
}
@Bean
@InboundChannelAdapter(value = "s3FilesChannel", poller = @Poller(fixedDelay = "100"))
public S3InboundFileSynchronizingMessageSource s3InboundFileSynchronizingMessageSource() {
S3InboundFileSynchronizingMessageSource messageSource =
new S3InboundFileSynchronizingMessageSource(s3InboundFileSynchronizer());
messageSource.setAutoCreateLocalDirectory(true);
messageSource.setLocalDirectory(new File("./"));
messageSource.setLocalFilter(new AcceptOnceFileListFilter<File>());
return messageSource;
}
@Bean
public PollableChannel s3FilesChannel() {
return new QueueChannel();
}
@Bean
public IntegrationFlow csvFlow(
CsvFileToPojoTransformer csvFileToPojoTransformer,
InvoiceHandler invoiceHandler) {
return IntegrationFlows.from(s3InboundFileSynchronizingMessageSource())
.transform(csvFileToPojoTransformer)
.split()
.handle(invoiceHandler)
.get();
}
}
Well, the dependency is there: https://repo1.maven.org/maven2/org/springframework/integration/spring-integration-aws/2.5.2/spring-integration-aws-2.5.2.pom, but yeah, it is optional.
And README even says that in this section: https://github.com/spring-projects/spring-integration-aws#dependency-management:
org.springframework.integration:spring-integration-file - for S3 channel adapters
We cannot have all the modules as hard dependencies in this project since target applications might not use all the AWS services.
For example we have Spring Cloud Stream Binder for AWS Kinesis project, which really uses only Kinesis and a little bit of DynamoDB, but it definitely fully is not interested in S3, or SQS, or SNS.
Hence many of dependencies in this project are optional.
Ah there it is, my mistake for missing the relevant README section. Thanks for your quick response!