Performance issue
pascalgrimaud opened this issue · 25 comments
Since the recent change, I find the page is slow to be displayed.
The API is really slow or is it just me ?
cc @jdubois @PierreBesson @SudharakaP
@pascalgrimaud @jdubois : Overall, I don't find it to be slow (maybe because I didn't notice much the load times for our previous VM deployment), but when loading the Audit log I find that it takes a couple more seconds. We are currently using the following configuration;
Lines 1 to 6 in f1a87ff
Previously we were using n1-standard-1 (1 vCPU, 3.75 GB memory). So maybe we could increase the memory to match the previous amount? 😄
Look at my screen. The API to /config took 2,41s, which is too much IMO.
I remember having response faster
Look at my screen. The API to /config took 2,41s, which is too much IMO.
I remember having response faster
@pascalgrimaud : Let us increase the memory to mimic the previous deployment and see if that improves performance. I'll create a PR. 😄
@pascalgrimaud @jdubois : I've done a deployment with these changes. It will be up in around 10 mins. Feel free to let me know if you see further issues. 😄
I don't see improvement.
As you are admin, can you add me the ROLE_ADMIN as well plz ? My login is: pascalgrimaud
So I can check the metrics too :)
I don't see improvement.
As you are admin, can you add me the ROLE_ADMIN as well plz ? My login is: pascalgrimaud
So I can check the metrics too :)
Yes, I don't see any change either. I've given you admin access; let me know if there's any other thing I could help with. 😄
Will look into this more. 🤔
@saturnism : Do you by any chance have any ideas on what could be done here? 🤔
Have you considered running a CPU profiler?
Have you considered running a CPU profiler?
@gmarziou : Great idea. Let me see what could be done within the flex environment to see the CPU stats. 🤔
I was rather thinking to JProfiler for which we are given free licences.
https://www.jhipster.tech/contributing-companies/#companies-helping-to-develop-jhipster
Oh they gave me licenses years ago... Not sure we can still count on them, I should update that page.
I can ask both for JProfiler and Yourkit, which one do you prefer?
I've had a look at the GCP console and I can't find our instance type... I saw that "F1" was the default, and is has a very slow CPU (600 Mhz), so that could be the explanation.
@SudharakaP do you know which instance type we have? And how much does this costs?
I can ask both for JProfiler and Yourkit, which one do you prefer?
Both are good, at work I'm more used to Yourkit.
I've had a look at the GCP console and I can't find our instance type... I saw that "F1" was the default, and is has a very slow CPU (600 Mhz), so that could be the explanation.
@SudharakaP do you know which instance type we have? And how much does this costs?
@jdubois : For app engine flexible as far as I know the instance type is assigned through how we allocate resources; https://cloud.google.com/appengine/docs/flexible/custom-runtimes/configuring-your-app-with-app-yaml#resource-settings. Under the hood these are Compute Engine VMs (just like the one we had before). So in our case given we have 4GB memory and 1 cpu so my guess is it assigns a General-purpose (N1) machine according to https://cloud.google.com/compute/docs/machine-types#machine_type_comparison. Not sure of the exact cost (as I don't know the exact machine type that's been assigned by the app engine), my thinking is that the cost would be very similar to what we had before as we had a n1-standard-1 type.
F1, F2 kinda instance types applies for app engine standard and we are using app engine flexible which does not have those types. Hope this clarifies some details.
Interestingly if I access the direct url; https://jhipster-online.uc.r.appspot.com/ then only the /config endpoint is slow, which I think is understandable as it talks to the github/gitlab api. So I think this is something to do with the CloudFlair proxy perhaps. 🤔
EDIT: but the Audit log loading is still slow. 🤔
I believe the root cause of this problem is that we have our MySQL database in europe-west region whereas the app engine location is set to us-central region. App engine documentation recommends keeping them in the same region; https://cloud.google.com/appengine/docs/locations.
However I don't know an easy way to change the region of the app engine except for creating a new project. It seems app engine doesn't support changing regions as mentioned here; https://issuetracker.google.com/issues/36878762?pli=1.
@saturnism : Is there any other way? Since creating a new project and moving stuff might be a bit cumbersome with the database and stuff probably needed to be moved as well. 😄
@jdubois @PierreBesson : So as discussed with @saturnism I am going to do some more tests by creating a database in us-central1 and see if that solves the latency problem. If that is the case I am going to shut down app engine and then move the database to us-central1. Would that be okay? Or is there a specific reason to keep the app location in europe-west1 ? If we want to keep in in europe-west1 we have to create a new project which I am okay with doing as well. Please let me know. 😄
Hi, it was just in europe-west1 because that's where we are located. Also, I believe we have a bit more users in Europe.
Then, JHipster is fully global, so that's not important, as long as the app server and the database are in the same region.
I have completed the testing. I confirm that the problem is definitely the fact that we use different regions for database and app engine. I'll migrate the database to us-central1 tomorrow. There will be around 15 mins downtime I think. I'll post the exact time I will do this on Twitter. 🤔
Happy to know that I was not crazy when I said there is some slow API :p
Awesome work, thank you @SudharakaP
@jdubois @pascalgrimaud : You are welcome. Please let me know if you see any further issues. 👍🏼
The migration is complete. Hopefully the load times are now reduced. 😄
looks better ! thanks a lot @SudharakaP
looks better ! thanks a lot @SudharakaP
You are welcome 👍🏼