Difficult handling of QueryDSL classes with the JPMS
Closed this issue · 8 comments
When trying to use the JPMS with a Spring Boot application that uses QueryDSL classes, certain issues may arise. When QueryDSL classes are required for classes that are defined outside of the working module, a split-package error will occur, as the package of the QueryDSL class will already exist in the referenced module.
When trying to generate the classes into a "subpackage" (which will then reside in the current working module) the EntityPathResolver does not appear to work properly - as the base Implementation SimpleEntityPathResolver by default only attempts to load the Q-class from the same package as the entity resides in - despite already providing a field "querySuffix" to be able to handle such a field.
Overriding of course works, but as it is no bean the override is required at a level, where the EntityPathResolver is referenced. As that is not a Bean however, the override is required to go up (several) levels to properly work (cf. https://stackoverflow.com/questions/47771175/how-to-register-custom-querydsl-entitypathresolver-with-spring-data-mongoreposit)
From a user perspective this seems a rather laborious approach. A preferred approach for better JPMS support to me would be the availability of a Bean to be overriden for the EntityPathResolver or even simpler a property to be able to configure the querySuffix. This would suffice for me - however I could imagine a future requirement for other configurations of QueryDSL class generation such as the APT options listed in http://querydsl.com/static/querydsl/latest/reference/html/ch03s03.html -> 3.2 APT options
An example project outlining the issue can be found at https://github.com/CybAtax/qdsl-subpackage-demo
Once a desired solution was discussed, I will gladly help with implementation
When QueryDSL classes are required for classes that are defined outside of the working module, a split-package error will occur, as the package of the QueryDSL class will already exist in the referenced module.
I am not sure I understand this explanation and its cause. What causes the split-package, more specifically, creation of a new module? And isn't that the cause for all sorts of workarounds?
I am not familiar with using Java modules and annotation processors (and the code that stems from annotation processing) so I am missing quite a bit of background.
When trying to generate the classes into a "subpackage" (which will then reside in the current working module) the EntityPathResolver does not appear to work properly
It works properly, we assume that Q classes reside in the same package as the actual entity. We favor an approach in which entity classes can be reduced to package-private visibility if they shall not be exported. Such an assumption requires co-location of auxiliary types in the same package.
Hi, thanks for the quick follow-up!
The cause for the split-package are generated QueryDSL classes for classes of other libraries. Lets say Entity A of my module has an element b of type B from a library. When a QClass for A is generated, it is also generated for B. That would cause an issue as now QB is declared in my module, but the package for QB (and B) already exists in the library I imported B from.
And youre right - to work around that I wanted to generated the querydsl classes in a subpackage - as that would not be a duplicate to a dependency there should be no split package.
As far as I understand it with modules similar issues exist as with the usage of OSGi - where one package of the same classpath may only be declared in a singular module.
For reproducing I have pushed a branch demo/split-package to the repo mentioned above
And yeah it does work properly - I misphrased. What I meant was, that the SimpleEntityPathResolver only looks in the same package. Despite having the field for overriding the package lookup, there does not (yet) appear a way to configure it for that instance. And that itself was the cause for opening this issue.
FTR, the error messages are:
package exists in another module: org.mongodb.bson:1:1-23
- [error] cannot find symbol symbol: class ObjectId:16:41-8
- [error] cannot find symbol symbol: class QObjectId location: class org.bson.types.QObjectId:20:25-9
Cannot find symbol symbol: class QObjectId location: package org.bson.types
Also, when using parameters:
'-Aquerydsl.excludedPackages=org.bson,org.bson.types,org.mongodb.bson',
'-Aquerydsl.excludedClasses=org.bson.types.QObjectId',
'-Aquerydsl.unknownAsEmbeddable=true'
the build still fails with:
[error] cannot find symbol symbol: class QObjectId location: package org.bson.types:25:32-10
qdsl-subpackage-demo-demo-split-package/build/generated/sources/annotationProcessor/java/main/demo/qdsl/package_suffix/QEntity.java:25: error: cannot find symbol
public final org.bson.types.QObjectId id;
^
symbol: class QObjectId
location: package org.bson.types
Generally speaking, the Querydsl behavior is causing the problem in the first place. Typically, ObjectId and other driver types are simple types and should be considered primitive types that do not allow for further introspection and that shortcoming causes all sorts of downstream effects.
In any case, the example code is much appreciated and super helpful to understand the origin of this issue.
With the recent major revision, we've introduced an extension SPI (based on RepositoryFragmentsContributor) that lives in each module, for MongoDB, that's MongoRepositoryFragmentsContributor respective ReactiveMongoRepositoryFragmentsContributor.
Mongo's QuerydslContributor encapsulates QuerydslMongoPredicateExecutor creation and applications can utilize e.g. a BeanPostProcessor to configure fragmentsContributor:
new BeanPostProcessor() {
@Override
public Object postProcessBeforeInitialization(Object bean, String beanName) throws BeansException {
if (bean instanceof MongoRepositoryFactoryBean<?, ?, ?> mrfb) {
mrfb.setRepositoryFragmentsContributor(new MyMongoRepositoryFragmentsContributor());
}
return bean;
}
};
private static class MyMongoRepositoryFragmentsContributor implements MongoRepositoryFragmentsContributor {
@Override
public RepositoryComposition.RepositoryFragments contribute(RepositoryMetadata metadata,
MongoEntityInformation<?, ?> entityInformation, MongoOperations operations) {
if (isQuerydslRepository(metadata)) {
QuerydslMongoPredicateExecutor<?> executor = new QuerydslMongoPredicateExecutor<>(entityInformation, operations,
new SimpleEntityPathResolver(…));
return RepositoryComposition.RepositoryFragments
.of(RepositoryFragment.implemented(QuerydslPredicateExecutor.class, executor));
}
return RepositoryComposition.RepositoryFragments.empty();
}
private boolean isQuerydslRepository(RepositoryMetadata metadata) {
return QuerydslPredicateExecutor.class.isAssignableFrom(metadata.getRepositoryInterface());
}
@Override
public RepositoryComposition.RepositoryFragments describe(RepositoryMetadata metadata) {
return MongoRepositoryFragmentsContributor.DEFAULT.describe(metadata);
}
}Going forward, we could consider adding another attribute to EnableMongoRepositories if this customization gains some traction.
The other part to this approach is refining SimpleEntityPathResolver and providing for example an abstract base class for those who would like to come up with their own naming or layout strategy.
Can you give the example above a try and see whether it helps your arrangement and that we can learn how to improve?
Hey, Ive tried the approach in the linked commit, viewable at this branch: https://github.com/CybAtax/qdsl-subpackage-demo/tree/demo/fragment
However, I could not get the setup (using plugin org.springframework.boot, version 4.0.0-M2) to work properly. No matter which way I've tried, eventually I would run into an error where the method MongoCursor#available would not exist (see stacktrace below). From what I understand, this is due to the com.querydsl:querydsl-mongodb:5.1.0 (cf. https://mvnrepository.com/artifact/com.querydsl/querydsl-mongodb/5.1.0) dependency which uses the no longer maintained org.mongodb:mongo-java-driver:3.12.11 (cf. https://mvnrepository.com/artifact/org.mongodb/mongo-java-driver/3.12.11, moved to https://mvnrepository.com/artifact/org.mongodb/mongodb-driver-sync/5.5.1). As documented in the build.gradle of the demo project, I have added the direct dependency to the newer driver-sync dependency - which did unfortunately not work out. Even excluding the mongo-java-driver module from querydsl did not work (due to other issues).
I even tried the setup reverting the added module-info file and reverting to generating querydsl classes in the same package as their entity, but that would yield the error, that the class could not be found, when executing the example Test (after fixing imports, of course.). I suppose that the Path resolution for querydsl classes is just performed earlier.
I am unsure on how to continue testing your nicely documented approach. If you have an idea, feel free to provide more details and I will test that out ASAP.
While generally I like the approach of the bean customization as you've described above, for long term use, I would prefer a simplified version by defining a custom bean / modifying the annotation as that feels more straightforward and simpler to understand.
Thank you so much for taking your time looking into this!
Stacktrace mentioned above (put here to not obstruct the other parts of the comment):
java.lang.NoSuchMethodError: 'int com.mongodb.client.MongoCursor.available()'
java.lang.RuntimeException: java.lang.NoSuchMethodError: 'int com.mongodb.client.MongoCursor.available()'
at org.springframework.data.repository.core.support.RepositoryMethodInvoker$RepositoryFragmentMethodInvoker.lambda$new$0(RepositoryMethodInvoker.java:282)
at org.springframework.data.repository.core.support.RepositoryMethodInvoker.doInvoke(RepositoryMethodInvoker.java:169)
at org.springframework.data.repository.core.support.RepositoryMethodInvoker.invoke(RepositoryMethodInvoker.java:158)
at org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments.invoke(RepositoryComposition.java:545)
at org.springframework.data.repository.core.support.RepositoryComposition.invoke(RepositoryComposition.java:290)
at org.springframework.data.repository.core.support.RepositoryFactorySupport$ImplementationMethodExecutionInterceptor.invoke(RepositoryFactorySupport.java:708)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.doInvoke(QueryExecutorMethodInterceptor.java:171)
at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.invoke(QueryExecutorMethodInterceptor.java:146)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:69)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.data.mongodb.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:159)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:222)
at jdk.proxy3/jdk.proxy3.$Proxy59.findAll(Unknown Source)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:359)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:158)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:135)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:222)
at jdk.proxy3/jdk.proxy3.$Proxy59.findAll(Unknown Source)
at demo.qdsl.package_suffix.QueryTest.queryTest(QueryTest.java:26)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
Caused by: java.lang.NoSuchMethodError: 'int com.mongodb.client.MongoCursor.available()'
at org.springframework.data.mongodb.core.MongoTemplate.executeFindMultiInternal(MongoTemplate.java:3075)
at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:2776)
at org.springframework.data.mongodb.core.ExecutableFindOperationSupport$ExecutableFindSupport.doFind(ExecutableFindOperationSupport.java:197)
at org.springframework.data.mongodb.core.ExecutableFindOperationSupport$ExecutableFindSupport.all(ExecutableFindOperationSupport.java:154)
at org.springframework.data.mongodb.repository.support.SpringDataMongodbQuery.fetch(SpringDataMongodbQuery.java:164)
at org.springframework.data.mongodb.repository.support.QuerydslMongoPredicateExecutor.findAll(QuerydslMongoPredicateExecutor.java:110)
at org.springframework.data.mongodb.repository.support.QuerydslMongoPredicateExecutor.findAll(QuerydslMongoPredicateExecutor.java:59)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:359)
at org.springframework.data.repository.core.support.RepositoryMethodInvoker$RepositoryFragmentMethodInvoker.lambda$new$0(RepositoryMethodInvoker.java:278)
... 28 more
'int com.mongodb.client.MongoCursor.available()'
java.lang.NoSuchMethodError: 'int com.mongodb.client.MongoCursor.available()'
at org.springframework.data.mongodb.core.MongoTemplate.executeFindMultiInternal(MongoTemplate.java:3075)
at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:2776)
at org.springframework.data.mongodb.core.ExecutableFindOperationSupport$ExecutableFindSupport.doFind(ExecutableFindOperationSupport.java:197)
at org.springframework.data.mongodb.core.ExecutableFindOperationSupport$ExecutableFindSupport.all(ExecutableFindOperationSupport.java:154)
at org.springframework.data.mongodb.repository.support.SpringDataMongodbQuery.fetch(SpringDataMongodbQuery.java:164)
at org.springframework.data.mongodb.repository.support.QuerydslMongoPredicateExecutor.findAll(QuerydslMongoPredicateExecutor.java:110)
at org.springframework.data.mongodb.repository.support.QuerydslMongoPredicateExecutor.findAll(QuerydslMongoPredicateExecutor.java:59)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:359)
at org.springframework.data.repository.core.support.RepositoryMethodInvoker$RepositoryFragmentMethodInvoker.lambda$new$0(RepositoryMethodInvoker.java:278)
at org.springframework.data.repository.core.support.RepositoryMethodInvoker.doInvoke(RepositoryMethodInvoker.java:169)
at org.springframework.data.repository.core.support.RepositoryMethodInvoker.invoke(RepositoryMethodInvoker.java:158)
at org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments.invoke(RepositoryComposition.java:545)
at org.springframework.data.repository.core.support.RepositoryComposition.invoke(RepositoryComposition.java:290)
at org.springframework.data.repository.core.support.RepositoryFactorySupport$ImplementationMethodExecutionInterceptor.invoke(RepositoryFactorySupport.java:708)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.doInvoke(QueryExecutorMethodInterceptor.java:171)
at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.invoke(QueryExecutorMethodInterceptor.java:146)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:69)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.data.mongodb.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:159)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:222)
at jdk.proxy3/jdk.proxy3.$Proxy59.findAll(Unknown Source)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:359)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:158)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:135)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:222)
at jdk.proxy3/jdk.proxy3.$Proxy59.findAll(Unknown Source)
at demo.qdsl.package_suffix.QueryTest.queryTest(QueryTest.java:26)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
Using the new generation is a bit of a nightmare right now because we're right in the middle. Using a vanilla start.spring.io app with Boot 4.0 M2 should do.
In your case, you can stay at Boot 3.4.x and apply the following customizations in the Gradle file:
implementation 'org.springframework.data:spring-data-mongodb:5.0.0-M5'
implementation 'org.springframework.data:spring-data-commons:4.0.0-M5'
implementation 'org.mongodb:mongodb-driver-sync:5.5.1'
implementation 'org.mongodb:mongodb-driver-core:5.5.1'
implementation 'org.mongodb:bson:5.5.1'
While generally I like the approach of the bean customization as you've described above, for long term use, I would prefer a simplified version by defining a custom bean / modifying the annotation as that feels more straightforward and simpler to understand.
Querydsl is rather a niche and customizing Querydsl even more so. We don't have plans for extending SPIs for customization. We could introduce a better support for RepositoryFactoryCustomizer to avoid factory casting and the BeanFactoryPostProcessor.
Widening annotations has its limits, at some point an annotation turns into everything and nothing that is difficult to use.
Thanks so much for the setup fix. As seen with the linked commit, I've pushed the fixed setup with a few other changes for better testing and that appears to work. Later I will try to reproduce the setup in the actual application setup and verify it works there properly with a properly set up database. I am really excited about the new major now and being able to use java modules in another application of ours.
I understand that the desire with QueryDSL customization is rather niche. Still, I am happy a solution could be reached with your help.
After testing in our actual application setup I can confirm that the setup works there as well - without any issues thus far. As shown in the linked commit a dependency to org.jspecify:j specify:1.0.0 was required but after that smooth sailing. Thanks so much for all your support and the great product you are providing. Excited for November :)
If you would like any further feedback let me know!
Thanks a lot for letting us know. I've been exploring variants of further customization but each approach ends with either too much coupling, conflicting approaches or a huge API surface that rather takes away from dev experience instead of helping to make things easier.
We have definitively an eye on customization needs and once we have a good idea to approach these, we will give it a try. Other than that, closing the ticket as it is no longer actionable.