During 'mvn package' the 'migrate' goal under 'generate-sources' fails as DB is not available
Closed this issue · 23 comments
Do we need the 'migrate' goal during the 'generate-sources' phase?
This causes the application to acquire database connection during 'mvn package' which causes the packaging to fail, as DB is not around and is available only in the environment where the jar is run (not where it is packaged).
As the 'migrate' should auto-trigger when the Spring Boot App starts, shouldn't this be left out of packaging?
steve is not a spring boot app. generate-sources needs db access, because jooq's maven plugin connects to database to generate java stubs of the tables. these stubs enable to write sql code in a type safe way. please refer to jooq website and documentation for more info about this. and before jooq, flyway executes the database migrations.
all of this (flyway + jooq) is part of the build process for a safer and more user-friendly deployment of steve.
... in the environment where the jar is run (not where it is packaged).
we encourage to build steve there, where it will be run. every other approach is very error-prone (for example java version differences).
As the 'migrate' should auto-trigger when the Spring Boot App starts ...
there are various ways to execute db migrations: manually, during start-up, during build. i personally like build-time execution, because the earlier the problems are catched the better. if you can build your application, you should be confident enough that it will run. this is the reason, why migration execution during start-up is not favoured by us. i hope i don't need to go on about the manual execution. i will give just one anti-argument: people forget to execute migrations.
Usually an artifact is created once in a Build Phase and then it is promoted to different stages/environments eg. Development, QA or Test etc. then finally to Prod. The hash/checksum of the artifact (jar or docker image) post build is expected to remain same for authorized workload (security and verification) while promoting the artifact to higher stages. With the current situation it seems that the Build needs to happen every time when a promotion is needed which is not how modern CI-CD and Kubernetes (or Docker) workloads operate. The dependency to any stage/environment related connectivity is thus not an automation friendly design approach. Also, since apart from DB the app might depend on other systems like SMTP/IMAP, but these are not required during the Build, so why should the DB?
i am aware of modern approaches of CI-CD, promotions with multiple stages, etc. we are using them at my current work place. steve targets another audience than the one you are talking about.
are you aware of the use cases of steve? and its user base? i guess not. steve does not target a developer audience. its audience consists of mostly non tech-savvy hobbyists or small- to middle-sized companies. people just want to be able to run the application without issues. therefore, the less setup steps they have the better.
if i would follow your line of thinking, i wouldn't even design steve the way it is right now. it's a monolith. should we split it into 100 micro-services for the sake of cloud-friendliness and scalability? no. because the use case is different.
I was not aware that the use case of Steve was not to be usable with modern CI-CD approaches.
But if the build time dependency to DB can be shifted to trigger time, it will, and it will still work for the audience you mentioned.
There is not much difference catching any problems with DB during trigger time, to build time. On the other hand building it every time before running, is quite redundant.
Nobody says that to make any application cloud friendly, it needs splitting into 100 microservices. But the current approach doesn't make it workable even with a pipeline for a monolithic deployment.
There is not much difference catching any problems with DB during trigger time, to build time.
there is a difference: in teams using modern CI-CD approaches, developers develop, build, deploy, monitor and react (if anything goes wrong). and while doing that they are using, possibly internal, tools to abstract away the complexity of underlying infrastructure. there are established processes.
i, as a developer and maintainer of steve, am not in the business of charging stations. i do not manage charging stations and therefore run steve in production. i do not deploy steve. if i can pull as much as complexity into dev and build phases, the likelihood of problems the users might have will be less. when i make a commit and release a new version, i literally push it into the wild. i have no idea who uses it, how they use it, what kind of infrastructure they use for it. therefore, i'd like to have the safety net of "if it builds, it will run". by shifting db stuff to trigger time, i will have this confidence no more. and we probably will see more issues with people having problems.
if i had a tighter grip on the pipeline that i mentioned at the beginning, i would do a lot things differently.
You still have to run the application after any changes during development, for tests. It is not like if the Build went fine there is no need to run any further tests by running the app! Then when the app runs, the effects of DB changes (at trigger time) can still be confirmed by logs and/or success or failure of application initialization.
Also, You can use the Travis CI to run the database related checks by moving them to test cases, and triggering the app with relevant flags for it during the CI run.
I am not sure what can I say more to make you realize that making the Build/Package free of any external dependency outside the jar, will considerably improve the utility of your own project.
You still have to run the application after any changes during development, for tests. It is not like if the Build went fine there is no need to run any further tests by running the app! Then when the app runs, the effects of DB changes (at trigger time) can still be confirmed by logs and/or success or failure of application initialization.
Also, You can use the Travis CI to run the database related checks by moving them to test cases, and triggering the app with relevant flags for it during the CI run.
of course. we have tests, as well, where we test actual behaviour and logic, and where we do not abuse them to verify whether migrations are correctly applied, a column exists, or whatever.
jooq generates sources from db, that are required for compiling and building. generated sources should not be part of the /main/src folder. i don't want to put them into /main/src and manually execute the plugin to update them whenever schema changes.
i am not sure how i can explain "one less thing to worry about" and "being on the safer side" better.
I am not sure what can I say more to make you realize that making the Build/Package free of any external dependency outside the jar, will considerably improve the utility of your own project.
are you looking at this project for mental gymnastics or do you have a use for it? sorry to fail to please you fully. maybe you should realise that this is an open source project: with your know-how and background you can fork it, modify it, adapt it to your use cases. this is what we are trying to communicate every time: steve can only be a basis for your specific use cases. if it does not fit you 100% but only 75%, so be it. we are still happy to be useful.
where we do not abuse them to verify whether migrations are correctly applied, a column exists, or whatever.
Doing this during tests is an abuse of Tests but doing this during build is not an abuse of Build!
"Verification" of migrations running correctly is more suited to Tests than Build.
i am not sure how i can explain "one less thing to worry about" and "being on the safer side" better.
There obviously are better approaches around, it is not at all about 'being on the safer side'. The project can still work sturdily along with automation friendly approaches.
are you looking at this project for mental gymnastics or do you have a use for it? sorry to fail to please you fully.
Sooner or later, more people like me will realize and bring this up, but if you will keep dealing with everyone with such mindset, it will look like you only consider what you do as always the 'right way', which is a pretty self-righteous attitude to have.
with your know-how and background you can fork it, modify it, adapt it to your use cases.
Guess I will have no other option than to do it that way. But having it implemented here will create better usability for everyone, as this is the main stream.
Sooner or later, more people like me will realize and bring this up, but if you will keep dealing with everyone with such mindset, it will look like you only consider what you do as always the 'right way', which is a pretty self-righteous attitude to have.
please... no need for a hostile tone.
i would be happy to change the approach in future if it really becomes a problem and is mentioned by more people. i am not fixated on the current solution. i am just not convinced that your proposal is better for our context and user base. for a better understanding, please take a look at the closed issues and raised questions.
sometimes, it is hard to maintain a balance for the project, if everyone tries to pull it to their side to satisfy their needs. we have to keep the big picture in mind and keep the project neutral by not covering every use case, but hopefully most.
Guess I will have no other option than to do it that way. But having it implemented here will create better usability for everyone, as this is the main stream.
Neither me nor @goekay would argue that it can be done better if you'd position steve as cloud native application. But cloud native is far from being "main stream".
In reality, most people run SteVe to drive 1-2 charge points as an appliance on a raspberry pi directly attached with database & co all local (a few more use a single VM/vServer). For that, the way it's build is quiet convenient.
please... no need for a hostile tone.
Gleichfalls.
For that, the way it's build is quiet convenient.
If you will not improve it to be usable on cloud, you will never have people who could try it for more than the cases on Raspberry pi.
But it is fine. Have it your way.
You missunderstood. It is usable on the cloud. I'm using it myself on EC2 + RDS to host a few charging stations.
So, do you have to Build it still with an RDS connectivity present while doing that? Or is it possible to build it in a pipeline and put just the jar in the container and deploy it to a cluster where it could fire up and make connection with the DB there?
Not using any automatic pipeline.
Do you understand the point then? Or do you think using it on a VM on Cloud in the same way it is used on a Local Machine or Raspberry Pi, is enough to call it 'Cloud Usable'?
That's absolutely cloud useable, yes.
Please don't mix up staging, orchestration, ci/cd, server less, etc with cloud.
You can absolutely use specific aspects of modern software engineering individually.
Please don't mix up staging, orchestration, ci/cd, server less, etc with cloud.
Please explain why you made it to work with RDS then?
You could have make it run with MySQL on the same or different VM on AWS.
That would still be Cloud Usable. No?
Okay i thought you would get it by now.
All these technologies and paradigms exist to solve problems.
Why do i use EC2? because it dont want to have it hosted on premise and deal with hardware. Why RDS? to lower maintenance (no dealing with MySQL upgrades, easy backup and restore)
I pay money to save effort. Problem solved.
Do you want to know whats not useful for SteVe?
Continuous delivery. Continuous delivery enables short intervals between releases. What if i tell you SteVe gets less the four releases per year? Building a convoluted pipeline isnt meaningful - i can just execute 3 commands quiet often before i break even on time.
Also not useful: Serverless architecture - it allows automatic and dynamic scaling on demand. Well load/demand isnt dynamic for steve so thats it.
You can make a case for containerization - personally i'm not using it but i see uses. That's why we support it.
Just using the same thing in a VM inside Cloud is not what Cloud is all about.
Similar to not having to deal with the hardware, Cloud provides a lot of benefits of Automation and as you mentioned 'Containerization'.
Otherwise even without RDS, the whole project can run in a VM as it runs anywhere else.
What if i tell you SteVe gets less the four releases per year?
The pipeline is not required just for SteVe's releases. It helps automate and integrate SteVe with other services and workloads complementing its functionality.
Building a convoluted pipeline isnt meaningful
It doesn't even work with a simple one!
You can make a case for containerization
Even the Dockerfile as of now has the DB sitting inside it!
personally i'm not using it but i see uses.
So, Containerization is ok, but not Serverless!
Can just the jar file be left alone to move around freely?
No. Because you need the DB, even to Build it!
Can just the jar file be left alone to move around freely?
let me tell you the following then: we stopped building fat-jars months ago. just to prevent the artifact from being moved around freely.
;)
let me tell you the following then: we stopped building fat-jars months ago. just to prevent the artifact from being moved around freely.
Wow!
Seems this is quite a disruptive way of doing things, very different from where the rest of the world is headed.
:)
I give up.
I will try to look for other alternatives or just take the code and modify it for interim use.
All these technologies and paradigms exist to solve problems.
i just wanted to emphasise this part. solutions are relative and depend on the context: problem, use case, tooling, manpower, time to invest, etc.
i find it interesting, that someone of your caliber ("it architect" according to your profile) sees only one side of things and misses all other ones, whereas you should have the calm wisdom of seeing the relative nature of why things are the way they are.
not every tech solution has to be cloud-this or cloud-that.
not every tech solution has to jump on the hype bandwagon and follow where the world is headed, in a world where the heading changes every few years anyways (what is the latest javascript framework that everyone should use now? ... which will be forgotten in a few months and change?).
some tech solutions prefer stability and aim to focus on actual business problems (for example: if you want to implement ocpp 2.0 in steve, you are more than welcome. it would be hundred times more important and valuable than the topic of this discussion).
i look forward to your future issues, where you tell us, for example, why we should use hibernate and utilise single page application architecture with ${this weeks flavour of javascript frameworks} instead of old-school jsp, because the world is headed that way.
i am closing this issue and locking the conversation, because both sides made their arguments and i want to prevent unnecessary escalation of discussion. sorry for that.