ArrayList payloads lose their elements
carolmorneau opened this issue · 7 comments
Describe the issue
Message payloads of type ArrayList
lose their elements as they go through the Spring Cloud Stream framework.
For instance, an application configured with:
spring:
cloud:
function:
definition: myFunction
stream:
default:
producer:
use-native-encoding: true
and a function defined as:
@Bean
public Function<Message<?>, Message<?>> myFunction() {
return msg -> msg;
}
the output destination receives the Message with an ArrayList payload, however, all elements from the ArrayList have vanished.
It appears that this issue was introduced as part of the following changes in Spring Cloud Function. When I undo the changes from that PR, the issue goes away.
To Reproduce
I have uploaded a minimal application reproducing the issue here.
Run the unit test called testSpringCloudFunctionIssueWhereArrayListPayloadsLoseTheirElements
to reproduce the issue.
Version of the framework
Using version 4.1.0
of Spring Cloud Stream and Spring Cloud Function
[INFO] +- org.springframework.cloud:spring-cloud-stream:jar:4.1.0:compile
[ ... TRUNCATED ]
[INFO] | +- org.springframework.cloud:spring-cloud-function-context:jar:4.1.0:compile
[INFO] | | +- net.jodah:typetools:jar:0.6.2:compile
[INFO] | | +- org.springframework.boot:spring-boot-autoconfigure:jar:3.2.2:compile
[INFO] | | +- org.springframework.cloud:spring-cloud-function-core:jar:4.1.0:compile
Expected behavior
With the above function, I would expect to receive the message with complete arrayList payload on the output destination.
@carolmorneau When specifying collection types (batch mode), you must use a concrete type in the function signature. You cannot use generics or wildcard types when running in batch mode. If you change your function signature to the following, it will work.
@Bean
public Function<Message<List<String>, Message<List<String>> myFunction() {
return msg -> msg;
}
You must also enable batch mode on the consumer binding when running with a binder that supports the batch mode (for example, the Kafka binder). You can enable batch-mode using the property - spring.cloud.stream.bindings.myFunction-in-0.consumer.batch-mode
and set this to true
. You don't need to set this, however, for your unit test.
Thank you @sobychacko for looking into the issue.
The issue is not related to Batch Mode, we are processing messages individually.
In our use case, the Spring Message may have different payload types which is why we use Message<?>
.
The issue occurs specifically when the Spring Message has a payload of type ArrayList. For instance:
List<String> payload = List.of("aSimpleStringElement");
Message<List<String>> msgWithArrayListPayload = MessageBuilder.withPayload(payload).build();
The above aSimpleStringElement
will be lost during this payload conversion.
I was able to pinpoint when the issue was introduced. It appears to be with this change
Ok, that appears to be a regression issue in Spring Cloud Function. Transferring the issue over there. cc @olegz
FYI, since this issue was opened, we found two workarounds in order to avoid loosing payload elements:
- Configure consumer's
user-native-decoding
and producer'suse-native-encoding
totrue
. However, this currently only works when function composition isn't used. This essentially does not attempt payload conversion and avoids the issue. - Provide a user-defined message converter capable of handling payloads of type
ArrayList
. This ensures that at least one MessageConverter can handle ArrayList payloads and avoids loosing payloads.
Also, I created another reproduction sample app based on the function-composition-rabbit sample to showcase the issue.
Reproduction steps:
- Run the sample app with the
issue1112
profile - Send some data to the RabbitMQ exchange (at least 3 messages since batch-size=3)
- Notice that payloads elements are lost
As a side note, for our specific use-case, #1144 is deemed more important than this issue.
I confirm on my end that it was fixed with #1145, but awaiting for your confirmation
The fix on 4.1.2-SNAPSHOT
looks good to me.
Thank you @olegz for the fix