GraalVM native images support
Opened this issue · 42 comments
This is more a feature request/question.
Is there any plan to support GraalVM native images out of the box in Flyway? At this moment different frameworks like Micronaut and Quarkus already support it but doing some "tricks and hacks". It would be good if there were support directly in the library so anyone that uses it can benefit.
Would you be able to explain more what GraalVM is and what problems it tries to solve? Also what 'Flyway support' would look like in your eyes.
GraalVM is a new polyglot JVM from Oracle based on OpenJDK that replaces the JIT compiler with another one written in Java (the GraalVM JIT compiler).
But in this case I'm talking about another subproject inside GraalVM, the native images: GraalVM can compile Java bytecode into native images to achieve faster startup and smaller footprint for your applications..
Every day more frameworks and libraries adapt their code to be able to generate native images in applications that use GraalVM native images. As I said, both Micronaut and Quarkus supports native-images for different kind of applications, including Flyway. The Spring team is also working on adding support to GraalVM native images.
There are different limitations https://www.graalvm.org/reference-manual/native-image/Limitations/, so framework and libraries need to be adapted to work with GraalVM. That can include different things to avoid the limitations mentioned. Sometimes it's a matter of initialize some classes at build-time or runtime. Others, as with our Micronaut Flyway integration is replacing some classes to "hack" how the migrations are discovered, because the default approach that Flyway uses (and that works with a regular JVM), doesn't work in a native-image.
For me, what I would expect when I think about "Flyway support for GraalVM native-images" is that when I use Flyway in any application and I try to convert it to a native-image, it just works. I shouldn't need to do anything, or anything apart maybe from configure something. I mean, as a user I should be able to convert my app to a native image and the migrations are discovered and applied when the app starts.
As I mentioned, this doesn't happen and this moment and we needed to do some tricks in Micronaut to make it compatible. This puts on us (any framework maintainer) the "burden" of maintaining that. This is not a problem per-se, but different projects and frameworks need to do something like this. Having support in the library itself will make everything compatible without the need of having custom (and probably duplicated) code in those frameworks.
Another problem is keeping that code updated, for example, Flyway 6.5.5 broke our integration and we need to adapt it to make it work again with GraalVM native-images. If the library includes everything needed we wouldn't worry about this anymore.
As a final comment, different libraries already include special configuration to support GraalVM, for example Netty.
Thank you for the detailed response. I'll add a task for us to investigate how much would need changing to get Flyway compatible.
Awesome! Looking forward to see progress on this 😃
Our current focus is releasing flyway V7, but once that is complete we can look into things like this.
Link to micronaut Flyway integration for reference: https://github.com/micronaut-projects/micronaut-flyway/tree/69fafb73ad9b9bb8309fbcfc5dbd76a16d6fece0/flyway/src/main/java/io/micronaut/flyway/graalvm
Any updates on this?
Looking forward for this nice feature update, We need this feature on Urgent bases.
Actually we use Flyway db on our projects, & want to convert it's to Android 9, 10 & 11 using gluonfx-maven-plugin that's is using Graalvm native image. Without Flyway we are in loss...
What is the current progress should it's will be solved urgent. I think the main problem yet is Flyway is not picking java baised migration files when running with Graalvm native.
I have created a sample project to test Flyway with HSQLDb on GraalVM native-image. Here: https://github.com/ctoabidmaqbool/Flyway-On-GraalVM-NativeImage-Test
When flyway try to looks for migration files it fails in GraalVM native-image. error:
WARN DbValidate - No migrations found. Are your locations set up correctly?
Complete log file: client-debug0.log
Note: I have used graalvm native-image argument --allow-incomplete-classpath for quick compilation otherwise it's looks for every single optional library too.
At least firstly Flyway should resolve migration files scaning in graalvm native-image running, so GraalVM users too can use it. I am not using any frameworks so I am in loss without native support in official library.
+1, supporting GraalVM native images OOTB would be a great feature
+1. But also tip for anyone using Spring Native for this. Add a @ResourceHint(patterns = "org/flywaydb/core/internal/version.txt") to your SpringApplication. The version.txt is what is causing the nullpointer. I guess graalvm provides a way to tell this resource hint manually as well.
+1 for this feature. Much needed one
+1!
Compile with native image runs fine using 8.5.7, but when launchin the compiled executable flyway can't find the files in db/migration.
WARNING org.flywaydb.core.internal.logging.slf4j.Slf4jLog warn Unable to scan location: c:\intellij\test\db\migration (unsupported protocol: resource)
maybe something similar like the protocol: war problem?
Runs fine without native image and happens after native-image build, and all other problems like version.txt are already working out of the box after running native agent
Edit:
micronaut and Quarkus did some changes here for FlyWay in native mode, I guess scaning for classes is not supported in native mode, leaving only the other locations like filesystem:, but thats no option for client applications since it's a security issue :(
Possible solution: if scaner is not usable, add another location option to filename and classpath. since the files can be accessed directly from resources, why not using List of those files, iterating through them.
.locations("list/array/config:configfile_with_contents_of_all_Vx__Script.sql"
+1
Why flyway team is not accounting this feature request into there todo list? Is it is very hard to implement or un-useful feature or what are the reasons for not implementing such a feature for a long time ago.
As now some of the java projects are totally made with grallvm native image. E.g. GluonVM now uses grallvm native images supports. When make android exe it use GrallVM native image and now flyway is no more of use...
Hi, I'm currently working on Flyway support for Spring Boot on native-image and wanted to chime in. We're currently having the same situation like Micronaut and Quarkus where Flyway support relies on a bunch of tricks to work. We really want to avoid that.
The are two parts to this puzzle: Flyway uses reflection, and reflection needs some metadata to work correctly on native-image. I've already contributed the metadata needed in the graalvm-reachability-metadata. It would still be nice to have the metadata right next to the code, this way if something changes in Flyway users will not be broken until the graalvm-reachability-metadata has been updated.
The other thing is that Flyway uses classpath enumeration to find the migration files. This doesn't work like it does on the JVM in a native-image. I've found a technique how to get (at least) resource enumeration to work in a native image, see this comment on the GraalVM issue. Maybe this could be incorporated into the Flyway codebase so that the migration files can be found?
Given that, is there any chance that GraalVM will be supported by Flyway in the foreseeable future?
I've implemented a small PoC to show that it's technically possible to get Flyway to run in a native-image by implementing a custom ResourceProvider: https://github.com/mhalbritter/flyway-native-image
Would be nice if the default resource provider could support the resource: protocol out of the box.
It's over a year or more, what is the progress! Flyway community?
Native images are basically the future of Java and microservices, so making this issue a priority would be a great idea.
As spring v3 has out of the box support for graalvm and all of the other major players do, it's really important to get graal support I guess.
For us we love FlyWay, but we need the native-image support.
As it's over an year or more! Flyway is support system is deep sleeping. Lol!
Flyway is very good db migration tool and is liked by so many Spring developers. Now that Spring native is official it so bad that we won't be able to use flyway with it.
Flyway Team,
It's high time to take some action on this very important feature
For reference: Flyway works with Spring Boot 3.0, because we added support for it in Boot directly. Nonetheless it would be great if Flyway itself would be native-image compatible.
Any progress or response in this side? Flyway is making new and new update with features. Why not taking some attention on this side?
@mhalbritter Any hints how we might use NativeImageResourceProviderCustomizer in out spring boot (3.1.0) app so we can configure flyway? Maybe you have an example that is easy to follow?
Hey @CriPstian, Flyway works with Spring Boot out of the box in a native-image. You don't have to do anything.
You are right @mhalbritter It was a misconfiguration on my side that was overriding the default behavior.
Yeah! It's very very good, if Flyway is working nicely in Spring Boot project. But still it's too very very bad if it's not working natively in Flyway without Spring Boot framework! LOL!
According to the first link, flyway does work with GraalNative if you use the second link:
- https://www.graalvm.org/native-image/libraries-and-frameworks/
- https://github.com/oracle/graalvm-reachability-metadata
Note that I've never tried yet.
Could be interesting to get some feedback from someone who did it and manage to make things work 🙂
See also #2927 (comment)
Flyway itself works in a plain native-image, but the automatic discovery of the migration scripts from src/main/resources doesn't.
@mhalbritter Wouldn't it be possible for you maybe to push a PR to Flyway to fix this issue based on your work here? 🤔
Sure it would, but given the 0 reaction on this issue here I'm not convinced that this is not a waste of my time.
Hi! what is the status of the issue, FlaywayDB on GraalVM native image support?
Recently I have tried myself using very latest depedencies yet, still not get luck from Flywaydb!
here is working sample project: https://github.com/ctoabidmaqbool/Flyway-On-GraalVM-NativeImage-Test
to run project in desktop:
%comspec% /k "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvars64.bat"
set GRAALVM_HOME=D:\Programs\graalvm-svm-java17-windows-gluon-22.1.0.1-Final
set JAVA_HOME=%GRAALVM_HOME%
set PATH=%JAVA_HOME%\bin;%PATH%
set MAVEN_HOME=D:\Programs\apache-maven-3.8.1-bin
set PATH=%MAVEN_HOME%\bin;%PATH%
set GRADLE_HOME=D:\Programs\gradle-7.3.3-bin
set PATH=%GRADLE_HOME%\bin;%PATH%
set GRADLE_USER_HOME=D:\.my-gradle
mvn clean
mvn gluonfx:run
mvn gluonfx:runagent
-------------------------
mvn clean -Pdesktop gluonfx:compile
mvn -Pdesktop gluonfx:link
mvn clean -Pdesktop gluonfx:build
mvn -Pdesktop gluonfx:nativerun
mvn clean -Pdesktop gluonfx:build gluonfx:nativerun
note mvn gluonfx:run and mvn gluonfx:runagent works fine without any issues, but mvn gluonfx:nativerun cause the issue
atm the only workaround i found to get it up and running is copy the sql files from resources to a TEMPDIR on startup before running Flyware, then add this to Flyway.configure().locations("filesystem:" TEMPDIR ), there Flyway can read them from filesystem, and delete them afterwards.
dirty, yes, and not secure. but at least it works until flyway hopefully supports it. Use uuid for tempdir to make it less predictable.
@Khithar I will take a look at your suggested solution, latter!
Normaly for my FlywayDb migration I prefere Java instead of just SQL files, Because by using java I can use the benefit of both languages e.g. Java and SQL. In my practicale case I have 100 of migrations which are in .Java converting them in SQL will also takes much time, and I am sure by just .sql everything is not possible!
Also by using Java baised migration is more secure, fast and better way then just .SQL baised migrations!
Another problem I am facing is that, Let for SQL baised migrations and in GrallVM native-image I can't get the reference / ralative path from res folder
e.g. I have src/main/resources/db/migrations, so how can i told FlywayDb to use this path at runtime Flyway.configure().locations("filesystem:" TEMPDIR ), as it takes absoleute path e.g. C:/somepath
This is the root of your directory where your application is started
File file = new File(Main.class.getProtectionDomain().getCodeSource().getLocation().toURI().getPath());
String basePath = file.getParent();
@Khithar Still this method is not working in GrallVM native image
Even I have tried standerd method, but still not luck,
String path = Main.class.getClassLoader().getResource("db/migrations").getPath();
Flyway flyway = Flyway.configure()
.baselineOnMigrate(true)
.dataSource(dataSource)
.locations("filesystem:" + path)
.sqlMigrationPrefix("V")
.load();
By using above method, when I run, mvn -Pdesktop gluonfx:run or mvn -Pdesktop gluonfx:runagent it's working fine,
But in native-image run e.g. mvn -Pdesktop gluonfx:nativerun it's showing, migration not showing, because it's looking for migration to system path from root folder not in my application resources folder!
SEVERE: Skipping filesystem location: \db\migrations (not found)
INFO: Successfully validated 0 migrations (execution time 00:00.004s)
I have my SQL migration files under src/main/resources/db/migrations
V1_0_1__Create_Table.sql, V1_0_2__Insert_Data.sql
Note: First of all, Flayway Java baised migration should be supported in GraalVM native-image support.
Another one, migration should be used from resources directory or classpath
e.g. src/main/
reading migration from external directory then app internal is a security risk. What is someone add another migrion to migrations directory?
Note: First of all, Flayway Java baised migration should be supported in GraalVM native-image support.
Another one, migration should be used from resources directory or classpath
e.g. src/main/
reading migration from external directory then app internal is a security risk. What is someone add another migrion to migrations directory?
absolutely true, as i said, it's a dirty walkaround from filesystem, and in my case not that evil since the user has database access anyway.
@ctoabidmaqbool1
it will not work from resources, you have to copy the files to your filesystem and read them from there. to make it harder to guess i create a temp dir with random uuid (just to make it harder to place anything into that directory for at least some ssecurity), copy the files from resources to that directory, there flyway reads them, and after that i delete it.
kinda like this, where tempDir is the random UUID dir on your filesystem, for example application dir or system temp
private static void createSqlFiles() {
List<String> listFiles = new ArrayList<>();
listFiles.add("V2__Database_Update.sql");
listFiles.add("V2.1__Add_Pdfsettings.sql");
listFiles.add("V2.2__Add_Ewtext.sql");
for (String file : listFiles) {
copyFile(file);
}
}
private static void copyFile(String fileName) {
File dest = new File(tempDir + fileName);
try (InputStream inputStream = YouClassName.class.getResourceAsStream("migration/" + fileName)) {
FileUtils.copyInputStreamToFile(Objects.requireNonNull(inputStream), dest);
} catch (IOException ex) {
log.error(ex.getMessage());
}
}
Falyway team, still this is not on the roadmap?
It's much appreciated some development in this side have been done, e.g. File baised migrations from file system works in GraalVM native-image.
But still this is not 100% usable, as what is someone is instrusted in using Java baised migrations in graalvm native-image. and not through just file system, but from source code...
I have practically 200 hundred or more of the migrations it's a lot of tuff work to be using Falyway in garralvm native image in this case!