Download terrible slow
donhuvy opened this issue · 31 comments
Dear friends,
I try to download https://github.com/PostgresApp/PostgresApp/releases/download/9.6.0/Postgres-9.6.0.zip
Download speed is terrible slow (about many hours), Please consider using mirror hosting, looking for sponsors or support torrent sharing (like LibreOffice)!
Where are you downloading from? I think that Github hosts binaries on S3, not sure why that is so slow...
I am at my home, in Hanoi, Vietnam. ( I can view YouTube 4K video)
It is
Could you tell me where you are downloading from? In the past I've heard that people from China had problems downloading it.
Does anyone have suggestions where to host Postgres.app? Would Amazon CloudFront work better where you are from?
It's convenient to host it on Github, since we don't have to pay for it. (Around 12.000 people download Postgres.app per month, so we use more than 500GB of bandwidth per month.)
This is my position: (...)
My IP: 14.177.xxx.xxx
ISP: Vietnam Post and Telecom Corporation
Ok, that was more detail than I wanted to know... just country would have been fine :)
I'll investigate what options we have for improving download speeds.
Hosting on Amazon Cloudfront might be an option to increase download speeds. However, that would probably cost several hundred euros per month. It would also make release management more complicated, since we would need to move away from Github's infrastructure.
How many people are having problems with download speed? If this prevents people from using Postgres.app, we need to fix it.
Does anybody else have an idea where to host binaries?
No, this is the wrong product. We don't need a service to archive data, we need a service that will let users download Postgres.app quickly. We need a CDN.
you true. "more than 500GB of bandwidth per month" --> Seem this is cheaper: https://cloud.google.com/storage/pricing#storage-pricing $0.026 * 500 (about $14/month)
We need around 500GB transfer, not storage. (But with Postgres.app version 2.0, that will probably double to more than a TB)
Are you still experiencing slow download speeds?
Is anybody else having trouble with downloading from S3?
just now, at Hanoi, Vietnam
I cannot downlaod v2.0.3 from Belgium.
I tried with wget, Firefox and Chrome, and it always fails at some point.
Auto update from within the app fails too.
I've been trying for several days, but it always fails.
I've a reasonalby good Internet connection here (~ 40Mbps according to fast.com)
wget 'https://github.com/PostgresApp/PostgresApp/releases/download/v2.0.3/Postgres-2.0.3.dmg'
--2017-06-12 11:38:06-- https://github.com/PostgresApp/PostgresApp/releases/download/v2.0.3/Postgres-2.0.3.dmg
Resolving github.com... 192.30.253.112, 192.30.253.113
Connecting to github.com|192.30.253.112|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/3946572/2d4e61b8-35b0-11e7-886d-49eb4a529f0b?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20170612%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20170612T093806Z&X-Amz-Expires=300&X-Amz-Signature=8fb76efc4bc3751376787fc5f4c21077b80fcdb260fa77dca70add4b5089e449&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3DPostgres-2.0.3.dmg&response-content-type=application%2Foctet-stream [following]
--2017-06-12 11:38:06-- https://github-production-release-asset-2e65be.s3.amazonaws.com/3946572/2d4e61b8-35b0-11e7-886d-49eb4a529f0b?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20170612%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20170612T093806Z&X-Amz-Expires=300&X-Amz-Signature=8fb76efc4bc3751376787fc5f4c21077b80fcdb260fa77dca70add4b5089e449&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3DPostgres-2.0.3.dmg&response-content-type=application%2Foctet-stream
Resolving github-production-release-asset-2e65be.s3.amazonaws.com... 52.216.0.152
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com|52.216.0.152|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 90757811 (87M) [application/octet-stream]
Saving to: ‘Postgres-2.0.3.dmg’
Postgres-2.0.3.dmg 5%[====> ] 4.34M 2.10MB/s
Thank you. Chrome with parallel download as suggested worked. Firefox downloads literally at 14k modem speed and gets stuck at 16.9MB, twice. This is from SF bayarea.
Spoke too soon. Chrome stopped at 37MB. Tried again with Chrono Downloader addon, after < 100MB, it gave "SERVER_FORBIDDEN" error. Tried again, same error after < 50 MB. Yes, "a known issue with Github" may be right, but this issue shouldn't be closed until there's a solution for the fix. Have not tried aria2c above, but seems to much setting to deal with and don't think it will fix the "SERVER_FORBIDDEN" error.
Edit: Tried third browser - Safari - stopped at 84MB.
Edit2: It really needs either either a mirror or official torrent.
Edit3: Chrome not able to resume download as it errored in "Failed - Forbidden"
Pretty much running out of options at this point. A mirror or some kind of alternative is really needed.
Hi! I'm currently investigating CDNs to host Postgres.app downloads.
If you could execute the following commands and post the output that would be really sweet:
curl -O https://postgresapp-b082.kxcdn.com/Postgres-2.1.3.dmg
curl -O https://postgresapp.r.worldssl.net/Postgres-2.1.3.dmg
curl -OL https://github.com/PostgresApp/PostgresApp/releases/download/v2.1.3/Postgres-2.1.3.dmg
For example, here's what I get on a 150MBit cable line in Linz, Austria
$ curl -O https://postgresapp-b082.kxcdn.com/Postgres-2.1.3.dmg
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 148M 100 148M 0 0 17.3M 0 0:00:08 0:00:08 --:--:-- 17.8M
$ curl -O https://postgresapp.r.worldssl.net/Postgres-2.1.3.dmg
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 148M 100 148M 0 0 9.8M 0 0:00:15 0:00:15 --:--:-- 17.7M
$ curl -OL https://github.com/PostgresApp/PostgresApp/releases/download/v2.1.3/Postgres-2.1.3.dmg
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 607 0 607 0 0 1150 0 --:--:-- --:--:-- --:--:-- 1151
100 148M 100 148M 0 0 5703k 0 0:00:26 0:00:26 --:--:-- 6922k
@jakob : did you ever contact GitHub support on the speed issue? I had issues with a big GIT repository that was hosted on GitHub but responding quite slow and giving timeouts when accessed via SVN. Their support was quite helpful and moved that repository to a faster server. Since then, the issues were gone. Maybe they can do a similar thing with the release-downloads of this repo?
@tbussmann Contacting Github is a good idea. I'll do that. Can you test the speed from Switzerland?
I'm collecting results in this Google Sheet. Feel free to add your results:
https://docs.google.com/spreadsheets/d/1dsNGhxlOM9j0CuxOgAMRzpdU_FQ38DhLNFzQiZmSIlM/edit?usp=sharing
Download from Switzerland:
$ speedtest --simple
Ping: 8.125 ms
Download: 508.44 Mbit/s
Upload: 378.34 Mbit/s
$ curl -O https://postgresapp-b082.kxcdn.com/Postgres-2.1.3.dmg
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 148M 100 148M 0 0 1825k 0 0:01:23 0:01:23 --:--:-- 1974k
$ curl -O https://postgresapp.r.worldssl.net/Postgres-2.1.3.dmg
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 148M 100 148M 0 0 28.1M 0 0:00:05 0:00:05 --:--:-- 34.8M
$ curl -OL -w "%{url_effective}\n" https://github.com/PostgresApp/PostgresApp/releases/download/v2.1.3/Postgres-2.1.3.dmg
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 607 0 607 0 0 1135 0 --:--:-- --:--:-- --:--:-- 1134
100 148M 100 148M 0 0 1267k 0 0:02:00 0:02:00 --:--:-- 1398k
https://github-production-release-asset-2e65be.s3.amazonaws.com/3946572/aa7af970-1d50-11e8-8805-6cecbf74c740?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20180301%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180301T155517Z&X-Amz-Expires=300&X-Amz-Signature=f49ae385fd96641a610f883cdcd586b1589f200cd83145b7bba2228869005e11&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3DPostgres-2.1.3.dmg&response-content-type=application%2Foctet-stream
as you see, I've changed the last command to show the source from where in GitHub's CDN it actually downloads.
Download from Germany:
Ping: 14.748 ms
Download: 77.69 Mbit/s
Upload: 6.05 Mbit/s
# curl -O https://postgresapp-b082.kxcdn.com/Postgres-2.1.3.dmg
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 148M 100 148M 0 0 1922k 0 0:01:19 0:01:19 --:--:-- 2056k
curl: (16) Error in the HTTP2 framing layer
# ls -al Postgres-2.1.3.dmg
-rw-r--r-- 1 karsten staff 156215679 Mar 1 21:57 Postgres-2.1.3.dmg
# curl -O https://postgresapp.r.worldssl.net/Postgres-2.1.3.dmg
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 148M 100 148M 0 0 6991k 0 0:00:21 0:00:21 --:--:-- 9.9M
curl: (16) Error in the HTTP2 framing layer
curl -OL -w "%{url_effective}\n" https://github.com/PostgresApp/PostgresApp/releases/download/v2.1.3/Postgres-2.1.3.dmg
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 607 0 607 0 0 994 0 --:--:-- --:--:-- --:--:-- 993
100 148M 100 148M 0 0 1054k 0 0:02:24 0:02:24 --:--:-- 1451k
https://github-production-release-asset-2e65be.s3.amazonaws.com/3946572/aa7af970-1d50-11e8-8805-6cecbf74c740?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20180301%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180301T210256Z&X-Amz-Expires=300&X-Amz-Signature=7a4ef3fb54c74a0eb5fea47de22656b120029ab6d9df9d7e983a4d2c243cd666&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3DPostgres-2.1.3.dmg&response-content-type=application%2Foctet-stream
OK, from those tests it looks like option 2 is fast both in Europe and North America. What about the other continents? Can someone from Asia test this?
Also, I've heard back from Github. They said they can't do anything to make our specific downloads faster, but if someone suffers from slow download speeds, they should contact Github support directly, so they can work out the issue.
@jakob Can we just simply mirror the release downloads to a public Google Drive folder? It's so frustratedly slow/impossible to download.
@teefan Please, feel free to do so. Does Google allow hosting public downloads in Drive, or is there some kind of bandwidth limit?
Not a Postgres user but see this thread on a related issue. I ran the tests in western China, where the network is typically slower than that in eastern China.
Unfortunately the first two wouldn't even start 😞
@emiapwil Thanks for trying the links, the first two links are no longer active (it was just a one month trial of the CDN)
It seems that Github release downloads have poor performance in Asia and Australia, and using a CDN would be a good idea.
I've done some more research, and cdn.net looks like they have a service with POPs in Asia and Australia. Maybe I can host postgres.app downloads there in the future?
CDN is definitely an option. Also I would agree with the OP that torrent and mirrors are also worth looking into.
I'd appreciate help! If someone wants to mirror Postgres.app, or seed torrents of the releases, please get in touch!
I’m the founder of another open-source project (Gladys Assistant), and I’ve had the same issues with GitHub downloads being very slow for large files.
To resolve this issue, I’ve been building my own CDN, Elephant CDN, dedicated to large file download for software releases.
I’ve hosted PostgresApp latest release on our CDN so you can try it:
If people in this thread can try downloading the file and telling me their location + download speed, it would be amazing :)