Streamline Deployment with CloudRun/AppEngine + GitHub Actions
Closed this issue · 43 comments
I believe we could maybe make the deployment process a bit streamlined with either using CloudRun or AppEngine with GitHub Actions driving the deployment. I'd work on this in the coming weeks if there's no objection. The plan is, when doing a tagged release, Github Actions will deploy the new version to either AppEngine or CloudRun.
If there's any ideas or suggestions, please feel free to let me know. 😄
@SudharakaP : do you mean replacing the current VM with GCE by GAE ?
@pascalgrimaud : Correct. That's the plan. Either use GAE or Cloud Run, but I think I like GAE better. 🙃
Although I don't know GAE at all, I'm trusting you and I'm fine with this, but don't hesitate to discuss about it in our mailing list before
/subscribe :-) Scream if you need help on GAE.
@ludoch : We have decided to use CloudRun instead; but thanks a lot. 😄
I've submitted the PR for this. This changes the deployment from the Custom VM to CloudRun. We will use GitHub Actions along with GitHub Secrets to replace the secret values at deployment. So the deployment only consist of creating a release tag and everything else is taken care of. 😄
A sample deployment pipeline execution can be found at my fork; https://github.com/SudharakaP/jhipster-online/runs/743469075?check_suite_focus=true
along with the sample app running on CloudRun (this currently runs on my personal account);
https://jhipster-online-edmea7cs5q-uc.a.run.app/
I haven't configured the, GitHub, GitLab and email secrets so those components will not work; but when I configure this on our GCP account I will do that.
@pascalgrimaud @PierreBesson : Can you give me admin access to this repo? Then I can configure the whole thing on our GCP account. We can test it out and if everyone agrees we could migrate the custom domain to the CloudRun service and then finally shutdown the VM. 😄
cc: @jhipster/developers
@SudharakaP : just done !
@pascalgrimaud @PierreBesson : Thanks much. I've configured the secrets and created the initial CloudRun service for deployment. The only things to do are,
-
Review this PR and merge it.
-
Create a release tag and see if it deploys correctly.
-
Test the deployment (in CloudRun) and if everything is good, hook up the custom domain with CloudRun instead of the VM and shutdown the VM.
Let me know if we are good to go ahead with this. 😄
@SudharakaP : I think it would be better to use a private project for deployment + secrets. I can create one. Are you ok ?
@pascalgrimaud : Sure I am fine with it. You mean an exact copy of this repository which is private? 🤔
@SudharakaP : not a copy. A private repo which contains the deployment part and secrets. Nothing else. As it's private, there won't be any pull requests. Then, we'll need to have some webhook from this project to trigger build in the private project
@pascalgrimaud : Sure. Please go ahead. If you need help let me know. 😄 I don't exactly understand the purpose of a private repo though. Is it to have extra security for the secrets? 🤔
@SudharakaP : secrets should be available only for main branches of the repo, but just in case and for security, I think it's better to have a separated project for that.
@PierreBesson : do you agree with this ? If I remember, it was your suggestion :p
@pascalgrimaud : I think @PierreBesson can verify; but my understanding was that he was talking about secrets in the context of my suggestion to use an encrypted file (with the secrets) which in the deployment process is decrypted.
I don't use that method here, but instead use GitHub Secrets only. I might be wrong, but I am doubtful if we are doing anything really meaningful by just putting them in a private repo since the secrets are already encrypted. 🤔 Anyways upto you guys; let me know. 😄
@ludoch : Hey, I do not know where to exactly post this; but posting it here so that you might know where to redirect. While setting up this task with CloudRun, I noticed the following issues; they are not major but I think might be useful to look at.
-
The integration of CloudRun with Java as described here seems not working I think or need more information; https://cloud.google.com/sql/docs/mysql/connect-run#connecting_to. I spent some time trying to do that but ultimately resolved to Spring Cloud GCP. Maybe there's a way to get it working without the use of Spring Cloud GCP and only using Hikari as described there; but it's not clear. Also the line;
config.setJdbcUrl(String.format("jdbc:mysql:///%s", DB_NAME));has a problem. I think it should be,config.setJdbcUrl(String.format("jdbc:mysql://google/%s", DB_NAME));. -
There doesn't seem to be a way of configuring the name of the docker file. For yaml based builds this is possible; https://cloud.google.com/sdk/gcloud/reference/builds/submit#--config. But if the build is done through a docker file, it seems like the filename should be
Dockerfile.
I'm not a Cloud Run expert, but for our own usage this makes perfect sense.
However, we need to be extra careful: many people are using forks of JHipster Online internally inside their companies, and they need to be able to use it easily. This is the main reason why I used a VM and MySQL initially: they are not the best solution for us, but they will work everywhere.
So I'm really happy if we move to Cloud Run, but I want to be sure JHipster Online will still work easily on our users' VMs.
@jdubois : Thanks a lot for the information. The only thing we did here is to add support for CloudSQL. The VM deployment can still be done if needed so I think users can use it if preferred. Could you double check if it's adequately secure in terms of the setup in the relevant pull request; #174 which was the doubt @PierreBesson raised I think.
If the question is only on GitHub secrets, it should be fine: just make sure you don't do stuff like "echo $SECRET" in the logs... which would be stupid anyway.
Otherwise, only admins can edit those secrets, but they can't see them directly with the UI.
If the question is only on GitHub secrets, it should be fine: just make sure you don't do stuff like "echo $SECRET" in the logs... which would be stupid anyway.
Otherwise, only admins can edit those secrets, but they can't see them directly with the UI.
@jdubois : Wonderful thanks a bunch. I just finished the v2.6.2 deployment. I'll merge this one soon and hopefully from next deployment we can migrate to CloudRun. The SQL db remains the same and the VM deployment can also be done so therefore people who use those will have no problems; but in case if you do see anything feel free to ping me; we can easily migrate back to the VM if necessary without any code changes. 😄
Awesome work, thank you @SudharakaP !
I'm very interested to learn more about the price/performance/maintenance gains of switching to Cloud Run.
Awesome work, thank you @SudharakaP !
I'm very interested to learn more about the price/performance/maintenance gains of switching to Cloud Run.
@jdubois : Thanks. I think the price will be less since CloudRun scales down to zero instances when there's no traffic. As for the performance I anticipate it will be roughly the same. But we'll see and I'll keep you in the loop on how it goes. 👍
Scaling down to zero doesn't look like a good idea to me: the app takes some time to start up, so it means some users might have a very bad experience. And anyway, this app is used also to get our stats, so it's being used nearly all the time, and I don't believe we will gain a lot of money from doing this.
I'd rather have the app running all the time, it's better for our users (and $$ is definitely not an issue)
@jdubois : Yes, I totally forgot about the stats. I agree that scaling down to zero might not be a good idea, but we can keep it running all the time.
The main reason I did this, is to make the deployment process easier and more automated such that anyone can do it without much manual intervention and also thought moving to docker makes more sense. However if you feel like the VM is sufficient for our use, we could keep the current configuration and I am okay with that as well. 🙂
Oh yes, moving to Docker totally makes sense and is a huge +1. It will definitely ease our upgrades, and make everyone's life better (including for people forking the project).
Then, with the stats we should never scale to zero, and again the process will take too much time to wake up:
- We need to have a least one running instance -> can you set this up? And have a look at the price, as it might be a premium feature
- I'm not sure if we should scale up above 1, as we have a non-distributed cache. It's possible, but we need to study this first, and maybe disable the cache or use another caching mechanism. Anyway we had already way too much processing power on the VM, we are far from needing any scalability at the moment.
@jdubois : I looked into these, the max number of instances isn't a problem we can set it to 1. But the minimum number of instances (such that it doesn't fall to 0) cannot be done directly at this moment with Cloud Run managed instances. I think they are going to support it in the future but right now under the default setting it will fall to 0 after some time when no more requests are coming through. There's three approaches we could take,
-
Do a cron scheduled job (can be done within GCP) to ping the service every minute so that CloudRun doesn't scale down to 0.
-
Use CloudRun Anthos which does support setting minimum number of instances.
-
Do not use any of the above and keep the current VM as is.
The first approach is kinda like a workaround, but I think it will be a workable solution and will work for our needs.
My gut feeling is that the second approach of using Cloud Run Anthos might be more costly and perhaps an overkill as it uses Google Kubernetes clusters. I haven't used Cloud Run Anthos before so can research a bit more on this.
WDYT? 🤔
Can we chat abt this next week? :)
Can we chat abt this next week? :)
@saturnism : Yes good idea, I am all in for it. 😄
Can we chat abt this next week?
Thanks @saturnism, can I join you guys ?
Can we chat abt this next week?
Thanks @saturnism, can I join you guys ?
@PierreBesson : I've sent you the invite that we planned next week. Did you receive it? Hope the time is okay; if not let me know we can arrange an alternate time. 😄
No I have not yet received it.
Just to keep everyone on the loop on this; after our meeting with @saturnism it seems that CloudRun isn't the best choice moving forward as it suspends after handing requests. Given we have @scheduled and @async tasks this will create problems. Therefore we will probably look at migrating to Google App Engine Flexible with a custom runtime. There's another meeting scheduled next week and hopefully we'll figure something out (@PierreBesson : sent you the invite as well if you are able to join).
another option is to keep using the GCE instance, but start to use container deployment this way:
https://cloud.google.com/compute/docs/containers/configuring-options-to-run-containers
This would pretty much be similar to the existing setup, except that the instance will automatically run the jhipster online container.
another option is to keep using the GCE instance, but start to use container deployment this way:
https://cloud.google.com/compute/docs/containers/configuring-options-to-run-containersThis would pretty much be similar to the existing setup, except that the instance will automatically run the jhipster online container.
Thanks @saturnism, I like this approach even better, as I think my previous deployment configuration that was done for CloudRun could be easily changed so that we deploy a docker container within the VM.
@PierreBesson @saturnism : So as discussed today, I was trying to deploy to GAE Flexible with a custom runtime. All seems to work well, except there is a 10 minute build timeout that we seem to be hitting on Cloud Build; the issue seems to be mentioned here; https://stackoverflow.com/questions/50046545/google-app-engine-build-timed-out-during-deployment. I've tried to set the timeout to a higher level using gcloud config set app/cloud_build_timeout command but it doesn't seem to take effect for some reason. Any help on this will be appreciated. 😄
@saturnism : I'll send some more info through DM. 😉
@saturnism @PierreBesson : I've configured GAE Flexible and it seems to work fine. 😄 I've sent you the link through DM and if you see any issues let me know. If everything works only thing left to do is to hook up our domain to GAE.
I confirm that it is working fine. Well done !
@PierreBesson : Thanks. Now we have one click deployments lol 😸
@PierreBesson : If this is all good; shall we merge this one? Although we will need to connect in order to do the domain name mapping to app engine as I don't have access to the domain control panel. 😄
Yes I think we can go with it ! We need the help from @jdubois for the DNS change at Gandi !
The DNS is managed by a Cloudflare, and I don’t understand well how this works with Gandi.
I’ll be back from holidays in one week and I can have a look then, is it urgent?
@jdubois No it's not urgent, certainly not urgent to bother you in your vacation !!!
I will see what I can do with @SudharakaP on Cloudflare as he already has access. I think we should be able to manage things from here. We will probably have to change the A record to the VM IP into a CNAME record to the appengine url.
@jdubois No it's not urgent, certainly not urgent to bother you in your vacation !!!
I will see what I can do with @SudharakaP on Cloudflare as he already has access. I think we should be able to manage things from here. We will probably have to change the A record to the VM IP into a CNAME record to the appengine url.
@PierreBesson : Hmm.... 🤔 I don't think I have access to Cloudflare unfortunately. I think if I am not mistaken, from my previous conversation with @jdubois it needs two factor authentication so only he has access. 😄