V2 API server still seems unstable
sxa opened this issue · 21 comments
Related: #35
We're still seeing quite a bit of instability (404 errors) when downloading via the API. We need this to be stable to avoid impacting customers.
I had a couple of cases where the website wasn't rendering the download parts, and our ansible playbook testing has been hitting 404s as well (e.g. https://ci.adoptopenjdk.net/view/work%20in%20progress/job/VagrantPlaybookCheck/OS=Win2012,label=vagrant/247/console)
TASK [Java_install : Download Java13] ******************************************
fatal: [172.28.128.120]: FAILED! => {"changed": false, "dest": "C:\\temp\\jdk-13.zip", "msg": "Error downloading 'https://api.adoptopenjdk.net/v2/binary/releases/openjdk13?openjdk_impl=hotspot&os=windows&arch=x64&release=latest&type=jdk&heap_size=normal' to 'C:\\temp\\jdk-13.zip': The remote server returned an error: (404) Not Found.", "status_code": 404, "url": "https://api.adoptopenjdk.net/v2/binary/releases/openjdk13?openjdk_impl=hotspot&os=windows&arch=x64&release=latest&type=jdk&heap_size=normal", "win_get_url": {"dest": "C:\\temp\\jdk-13.zip", "url": "https://api.adoptopenjdk.net/v2/binary/releases/openjdk13?openjdk_impl=hotspot&os=windows&arch=x64&release=latest&type=jdk&heap_size=normal"}}
PLAY RECAP *********************************************************************
I believe @johnoliver increased the resources available to the API recently but that does not appear to have resolved the issues.
Is it production ready now? The front of the api website documents V2 and says V3 is being worked on, but the README in the APi repo directs you to use V3
Will test switching it over - just had another 404 from V2 at https://ci.adoptopenjdk.net/job/VagrantPlaybookCheck/OS=CentOS6,label=vagrant/263/console
TASK [adoptopenjdk_install : Install latest release if one not already installed (Linux/x64)] ***
FAILED - RETRYING: Install latest release if one not already installed (Linux/x64) (3 retries left).
FAILED - RETRYING: Install latest release if one not already installed (Linux/x64) (2 retries left).
FAILED - RETRYING: Install latest release if one not already installed (Linux/x64) (1 retries left).
fatal: [172.28.128.103]: FAILED! => {"attempts": 3, "changed": false, "msg": "Failure downloading https://api.adoptopenjdk.net/v2/binary/releases/openjdk13?openjdk_impl=hotspot&os=linux&arch=x64&heap_size=normal&release=latest&type=jdk, HTTP Error 404: Not Found"}
And https://ci.adoptopenjdk.net/job/VagrantPlaybookCheck/OS=CentOS7,label=vagrant/263/console
TASK [adoptopenjdk_install : Install latest release if one not already installed (Linux/x64)] ***
FAILED - RETRYING: Install latest release if one not already installed (Linux/x64) (3 retries left).
FAILED - RETRYING: Install latest release if one not already installed (Linux/x64) (2 retries left).
FAILED - RETRYING: Install latest release if one not already installed (Linux/x64) (1 retries left).
fatal: [172.28.128.103]: FAILED! => {"attempts": 3, "changed": false, "msg": "Failure downloading https://api.adoptopenjdk.net/v2/binary/releases/openjdk13?openjdk_impl=hotspot&os=linux&arch=x64&heap_size=normal&release=latest&type=jdk, HTTP Error 404: Not Found"}
FYI @Willsparker since I suspect you've seen this a few times too :-)
Is it production ready now? The front of the api website documents V2 and says V3 is being worked on, but the README in the APi repo directs you to use V3
I'd say v3 is ready to use yeah, especially for internal usage.
When I get a spare minute I'll test a branch that changes the adoptopenjdk_install URL to v3, on the vagrantPlaybookCheck job and link the results :-)
Actually, scratch that, just saw @sxa555 has a PR in
Yeah v3 is probably good enough for internal, given now we have a staging server setup so we can properly test the api before deployment. We also fixed an issue where due to a number of programs using the same github api tokens, they were draining the quota and causing some of these issues, this should mean that v2 will be more stable in the future.
As discussed on the call, this is likely a rate-limiting issue and we can use the V3 API trick of backing off and using last modified. Let's see if we can implement that patch here in V2 to stabilize it.
Has v2 been removed?
$> curl -L 'https://api.adoptopenjdk.net/v2/info/releases/openjdk8'
Not found
Using nodejs client to send the request, from other location (to check if it's networking issue), I also got a 404 error.
UPDATE:
It gets back now after 10 min. I have one question, when will v2 be out of service? Is there any exact deadline, if we don't have plan to migrate to v3 now.
Great question @Eskibear!
We do not have a specific date at the moment - it's unlikely to be removed in the next few months though as there is still a significant amount of use of the V2 API and we're only just in the processes of announcing V3 properly (although many have already moved to it as it's been stable, and we've been using it ourselves for the past few months internally on the project in order to ensure it's quality)
Everyone is encouraged to move to V3 ASAP though. The 404 errors are caused by instability of the V2 API at the moment (hence this issue!) and absolutely not as a result of starting to decomission the V2 API.
Hope that helps.
At least today, v3 is also unstable. Getting a lot of timeouts and now even 503s.
@ibauersachs No idea why there would be instability for v3, there were a few restarts on friday due to updates in relation to the release but should not have been for significant amounts of time, do you know roughly what time you were getting these timeouts?
That was around the time I left the above comment, so roughly between 20:00 and 22:00 UTC on April 17.
By release I assume you mean the jdk8u252/jdk11.0.7 etc. updates? I would definitely not expect any downtime on the API for this.
We had some issues with incorrect classification of the new ea releases of openj9 arch that caused us to have to do a redeploy of the api, that will have caused some downtime
@johnoliver - I think the v2 API has been stable for quite some time, can we close this now?
@karianna I just tried, v2 API is even more unstable than before, especially for openjdk11. Can anyone take a look?
# 11 -> (404)
$> curl -L "https://api.adoptopenjdk.net/v2/info/releases/openjdk11"
Not found
# 11 -> (404)
$> curl -L "https://api.adoptopenjdk.net/v2/info/releases/openjdk11?openjdk_impl=hotspot&arch=x64&os=mac&type=jdk&release=latest"
Not found
# 8 -> (404) NOTE: unstable, later when I tried the same command, it's ok.
$> curl -L 'https://api.adoptopenjdk.net/v2/info/releases/openjdk8'
Not found
# 8 -> (ok)
$> curl -L "https://api.adoptopenjdk.net/v2/info/releases/openjdk8?openjdk_impl=hotspot&arch=x64&os=mac&type=jdk&release=latest"
{
"release_name": "jdk8u275-b01",
"release_link": "https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/tag/jdk8u275-b01",
"timestamp": "2020-11-11T15:56:17Z",
"release": true,
"binaries": [
{
"os": "mac",
"architecture": "x64",
"binary_type": "jdk",
"openjdk_impl": "hotspot",
"binary_name": "OpenJDK8U-jdk_x64_mac_hotspot_8u275b01.tar.gz",
"binary_link": "https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u275-b01/OpenJDK8U-jdk_x64_mac_hotspot_8u275b01.tar.gz",
"binary_size": 101776105,
"checksum_link": "https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u275-b01/OpenJDK8U-jdk_x64_mac_hotspot_8u275b01.tar.gz.sha256.txt",
"installer_name": "OpenJDK8U-jdk_x64_mac_hotspot_8u275b01.pkg",
"installer_link": "https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u275-b01/OpenJDK8U-jdk_x64_mac_hotspot_8u275b01.pkg",
"installer_size": 105453786,
"installer_checksum_link": "https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u275-b01/OpenJDK8U-jdk_x64_mac_hotspot_8u275b01.pkg.sha256.txt",
"installer_download_count": 2380,
"version": "8",
"version_data": {
"openjdk_version": "8u275-b01",
"semver": "8.0.275+1"
},
"heap_size": "normal",
"download_count": 373,
"updated_at": "2020-11-11T15:57:01Z"
}
],
"download_count": 15873
}
# 15 -> (ok)
$> curl -L "https://api.adoptopenjdk.net/v2/info/releases/openjdk15?openjdk_impl=hotspot&arch=x64&os=mac&type=jdk&release=latest"
{
"release_name": "jdk-15.0.1+9",
"release_link": "https://github.com/AdoptOpenJDK/openjdk15-binaries/releases/tag/jdk-15.0.1%2B9",
"timestamp": "2020-10-23T20:46:22Z",
"release": true,
"binaries": [
{
"os": "mac",
"architecture": "x64",
"binary_type": "jdk",
"openjdk_impl": "hotspot",
"binary_name": "OpenJDK15U-jdk_x64_mac_hotspot_15.0.1_9.tar.gz",
"binary_link": "https://github.com/AdoptOpenJDK/openjdk15-binaries/releases/download/jdk-15.0.1%2B9/OpenJDK15U-jdk_x64_mac_hotspot_15.0.1_9.tar.gz",
"binary_size": 195773522,
"checksum_link": "https://github.com/AdoptOpenJDK/openjdk15-binaries/releases/download/jdk-15.0.1%2B9/OpenJDK15U-jdk_x64_mac_hotspot_15.0.1_9.tar.gz.sha256.txt",
"installer_name": "OpenJDK15U-jdk_x64_mac_hotspot_15.0.1_9.pkg",
"installer_link": "https://github.com/AdoptOpenJDK/openjdk15-binaries/releases/download/jdk-15.0.1%2B9/OpenJDK15U-jdk_x64_mac_hotspot_15.0.1_9.pkg",
"installer_size": 201928998,
"installer_checksum_link": "https://github.com/AdoptOpenJDK/openjdk15-binaries/releases/download/jdk-15.0.1%2B9/OpenJDK15U-jdk_x64_mac_hotspot_15.0.1_9.pkg.sha256.txt",
"installer_download_count": 4518,
"version": "15",
"version_data": {
"openjdk_version": "15.0.1+9",
"semver": "15.0.1+9"
},
"heap_size": "normal",
"download_count": 2664,
"updated_at": "2020-10-23T20:48:09Z"
}
],
"download_count": 54866
}
Have landed a PR to only refresh every 12 hours - but this will need a proper fix on Monday.
OK so last night's PR did the trick to stabilize, it's roughly an 8-sec outage every 15 mins for the refresh cycle now as opposed to 90 sec. But we now only pull back the last 50 days of results. More work will be done this week to try and get it to point to the V3 backend cache.
Again please do upgrade to V3 as soon as you can.
This should this resolved now that we have no dependency on the Octokit API data anymore