x/website/internal/dl: downloading go1.25.4.linux-amd64.tar.gz with "Accept-Encoding: gzip" header serves a smaller file whose SHA 256 checksum does not match

Question

x/website/internal/dl: downloading go1.25.4.linux-amd64.tar.gz with "Accept-Encoding: gzip" header serves a smaller file whose SHA 256 checksum does not match

Opened this issue 10 hours ago · 14 comments

Go version

go1.25.4.linux-amd64.tar.gz

Output of `go env` in your module/workspace:

AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOAMD64='v1'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/home/dave/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/home/dave/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build4046053950=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/dev/null'
GOMODCACHE='/home/dave/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/dave/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/home/dave/.local/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/dave/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/home/dave/.local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.25.3'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

trying to download, verify, and install go1.25.4.linux-amd64.tar.gz from the go download page
the sha256sum listed on the page does not match the sum calculated from the downloaded file

What did you see happen?

output from sha256sum of downloaded file:
a8ae67a43f46994ebefa22aa83f52f415125bedfd0455eba5a08a3cc82160044 go1.25.4.linux-amd64.tar.gz
does not match
9fa5ffeda4170de60f67f3aa0f824e426421ba724c21e133c1e35d6159ca1bec

What did you expect to see?

sums should match

Answer 1 · 2025-11-08T05:43:35.000Z

I get the 9fa* sum when I download it.
Make sure you got the whole thing. 59766394 bytes.

Answer 2 · 2025-11-08T05:54:50.000Z

Hmmm, downloaded it directly with wget and got the correct results. Maybe a bug with Firefox 144.0.2 Linux??

…

On Sat, Nov 8, 2025, 12:43 AM Keith Randall ***@***.***> wrote: *randall77* left a comment (golang/go#76228) <#76228 (comment)> I get the 9fa* sum when I download it. Make sure you got the whole thing. 59766394 bytes. — Reply to this email directly, view it on GitHub <#76228 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC63OG652BVQCUNIFZ5KBXT33V7J5AVCNFSM6AAAAACLQIOTP2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTKMBWGAYTOMZRGM> . You are receiving this because you authored the thread.Message ID: ***@***.***>

Answer 3 · 2025-11-08T06:09:00.000Z

Related Issues

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

Answer 4 · 2025-11-08T12:11:23.000Z

Met the same issue today, using curl solved this problem. But I found out that if I add -H "Accept-Encoding:gzip, deflate, br, zstd" to download request(I found this in request header from Firefox), this issue appears again.

// checksum 9fa5ffeda4170de60f67f3aa0f824e426421ba724c21e133c1e35d6159ca1bec
curl -L -O  https://go.dev/dl/go1.25.4.linux-amd64.tar.gz

// checksum a8ae67a43f46994ebefa22aa83f52f415125bedfd0455eba5a08a3cc82160044
curl -L -H "Accept-Encoding:gzip, deflate, br, zstd" -O  https://go.dev/dl/go1.25.4.linux-amd64.tar.gz

Does this count as a bug?

Answer 5 · 2025-11-08T14:54:58.000Z

I ran into the same problem and I think there's something wrong with the tarballs.

The files, specifically go1.25.3.linux-amd64.tar.gz and go1.25.4.linux-amd64.tar.gz, appear to have been compressed twice, meaning they should be named *.tar.gz.gz.

Using a tool like gunzip to decompress it just once gives me the correct SHA-256 sum and finally allows me to decompress the contents.

jkunde@fedora-joku:~/test$ wget https://go.dev/dl/go1.25.4.linux-amd64.tar.gz
HTTP response 302  [https://go.dev/dl/go1.25.4.linux-amd64.tar.gz]
Adding URL: https://dl.google.com/go/go1.25.4.linux-amd64.tar.gz
Adding URL: https://dl.google.com/go/go1.25.4.linux-amd64.tar.gz
Saving 'go1.25.4.linux-amd64.tar.gz'
HTTP response 200  [https://dl.google.com/go/go1.25.4.linux-amd64.tar.gz]
go1.25.4.linux-amd64 100% [=========================================================================================>]   56.57M   38.16MB/s
                          [Files: 1  Bytes: 56.57M [27.20MB/s] Redirects: 1  Todo: 0  Errors: 0                      ]
jkunde@fedora-joku:~/test$ sha256sum go1.25.4.linux-amd64.tar.gz 
a8ae67a43f46994ebefa22aa83f52f415125bedfd0455eba5a08a3cc82160044  go1.25.4.linux-amd64.tar.gz
jkunde@fedora-joku:~/test$ gunzip go1.25.4.linux-amd64.tar.gz 
jkunde@fedora-joku:~/test$ sha256sum go1.25.4.linux-amd64.tar
9fa5ffeda4170de60f67f3aa0f824e426421ba724c21e133c1e35d6159ca1bec  go1.25.4.linux-amd64.tar
jkunde@fedora-joku:~/test$ tar -xzf go1.25.4.linux-amd64.tar
jkunde@fedora-joku:~/test$

See also:
https://lore.kernel.org/all/2b875d1c-50ef-4679-bb41-3e1cf89791b3@gmail.com/

Answer 6 · 2025-11-08T15:59:10.000Z

@golang/release

Answer 7 · 2025-11-08T17:37:51.000Z

9fa5ffeda4170de60f67f3aa0f824e426421ba724c21e133c1e35d6159ca1bec is the right sha256 checksum for the go1.25.4.linux-amd64.tar.gz file. It can be reproduced with distpack by building Go 1.25.4 from source:

$ cd $(mktemp -d)
$ git clone https://go.googlesource.com/go         
Cloning into 'go'...
$ cd go                                   
$ git checkout go1.25.4
$ cd src
$ GOOS=linux GOARCH=amd64 ./make.bash -distpack
[...]
Packaging archives for linux/amd64.
distpack: 160043b7f17b6d60 go1.25.4.src.tar.gz
distpack: 9fa5ffeda4170de6 go1.25.4.linux-amd64.tar.gz
distpack: f4fa35e13952bae9 v0.0.1-go1.25.4.linux-amd64.zip
distpack: 58528cce1848ddf4 v0.0.1-go1.25.4.linux-amd64.mod
distpack: f8858a0102ad4cfa v0.0.1-go1.25.4.linux-amd64.info
$ shasum -a 256 ../pkg/distpack/go1.25.4.linux-amd64.tar.gz 
9fa5ffeda4170de60f67f3aa0f824e426421ba724c21e133c1e35d6159ca1bec  ../pkg/distpack/go1.25.4.linux-amd64.tar.gz

It matches what curl -L gives:

$ curl -L 'https://go.dev/dl/go1.25.4.linux-amd64.tar.gz' | shasum -a 256 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    75  100    75    0     0    554      0 --:--:-- --:--:-- --:--:--   555
100 56.9M  100 56.9M    0     0  34.7M      0  0:00:01  0:00:01 --:--:-- 37.8M
9fa5ffeda4170de60f67f3aa0f824e426421ba724c21e133c1e35d6159ca1bec  -

As #76228 (comment) points out, it seems the a8ae67a43f46994ebefa22aa83f52f415125bedfd0455eba5a08a3cc82160044 checksum happens when issuing a GET request with "Accept-Encoding: gzip" header, and the download server applies gzip encoding to the go1.25.4.linux-amd64.tar.gz file and reduces the transfer size (very slightly, since the original file is already a compressed tarball):

$ curl -L -H "Accept-Encoding: gzip" 'https://go.dev/dl/go1.25.4.linux-amd64.tar.gz' | shasum -a 256
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    75  100    75    0     0    596      0 --:--:-- --:--:-- --:--:--   600
100 56.5M  100 56.5M    0     0   9.8M      0  0:00:05  0:00:05 --:--:-- 10.6M
a8ae67a43f46994ebefa22aa83f52f415125bedfd0455eba5a08a3cc82160044  -

Note that other than being compressed slightly more, its content is identical to that of the original file. Only the top-top level checksum doesn't match (since it's compressed to be smaller).

$ diff <(curl -s -L                            'https://go.dev/dl/go1.25.4.linux-amd64.tar.gz') \
       <(curl -s -L -H "Accept-Encoding: gzip" 'https://go.dev/dl/go1.25.4.linux-amd64.tar.gz')
Binary files /dev/fd/11 and /dev/fd/12 differ

$ diff <(curl -s -L                            'https://go.dev/dl/go1.25.4.linux-amd64.tar.gz') \
       <(curl -s -L -H "Accept-Encoding: gzip" 'https://go.dev/dl/go1.25.4.linux-amd64.tar.gz' | gunzip --stdout)
$ echo $?
0

This might not be specific to Go 1.25.4, though, but rather to the dl.google.com server. Something similar can be reproduced with another non-Go file too, like:

$ curl -L --silent 'https://dl.google.com/android/repository/platform-tools-latest-windows.zip' | shasum -a 256
12c2841f354e92a0eb2fd7bf6f0f9bf8538abce7bd6b060ac8349d6f6a61107c  -
$ curl -L -H "Accept-Encoding: gzip" --silent 'https://dl.google.com/android/repository/platform-tools-latest-windows.zip' | shasum -a 256
0948a0217c7ff08455cb500f933d34d01a4e6bc0585f07d716d8c9ce487e909c  -

Answer 8 · 2025-11-08T20:32:46.000Z

Please note that injecting the Accept-Encoding header into a cURL invocation expressly instructs cURL not to interpret the compressed bytes:

# the client reports it is unable to interpret any compression
$ curl -fsSL https://go.dev/dl/go1.25.4.linux-amd64.tar.gz | sha256sum
9fa5ffeda4170de60f67f3aa0f824e426421ba724c21e133c1e35d6159ca1bec  -

# this assumes, the server will set a `Content-Encoding: gzip`
# (the server is allowed not to honour the client's capabilities, in which case this invocation would fail
# (as it would try to decompress presumably uncompressed data))
$ curl -fsSL https://go.dev/dl/go1.25.4.linux-amd64.tar.gz -H 'Accept-Encoding: gzip' | gzip -d | sha256sum
9fa5ffeda4170de60f67f3aa0f824e426421ba724c21e133c1e35d6159ca1bec  -

# cURL can be instructed to try and gain network performance from the HTTP compressed representation feature
$ curl -fsSL https://go.dev/dl/go1.25.4.linux-amd64.tar.gz --compressed | sha256sum
9fa5ffeda4170de60f67f3aa0f824e426421ba724c21e133c1e35d6159ca1bec  -

Answer 9 · 2025-11-08T20:46:52.000Z

The Google server correctly sets Content-Encoding: gzip (as irrespective of representation, the resource is a .gzip file) and sets Content-Type: application/x-gzip when appropriate:

$ curl -v -fsSL https://go.dev/dl/go1.25.4.linux-amd64.tar.gz -o/dev/null
(...)
< HTTP/2 200
< accept-ranges: bytes
< content-disposition: attachment
< content-security-policy: default-src 'none'
< server: downloads
< x-content-type-options: nosniff
< x-frame-options: SAMEORIGIN
< x-xss-protection: 0
< date: Sat, 08 Nov 2025 20:38:16 GMT
< cache-control: no-transform,public,max-age=86400
< last-modified: Wed, 05 Nov 2025 19:03:09 GMT
< etag: "50ad87b"
< content-type: application/x-gzip
< vary: Accept-Encoding
< content-length: 59766394
< age: 0
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

$ curl -v -fsSL https://go.dev/dl/go1.25.4.linux-amd64.tar.gz --compressed -o/dev/null
(...)
< HTTP/2 200
< accept-ranges: bytes
< content-disposition: attachment
< content-encoding: gzip
< content-security-policy: default-src 'none'
< server: downloads
< x-content-type-options: nosniff
< x-frame-options: SAMEORIGIN
< x-xss-protection: 0
< date: Sat, 08 Nov 2025 20:39:13 GMT
< cache-control: no-transform,public,max-age=86400
< last-modified: Wed, 05 Nov 2025 19:03:09 GMT
< etag: "50ad87b-6aa1e4d0"
< content-type: application/x-gzip
< vary: Accept-Encoding
< content-length: 59327961
< age: 0
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

Furthermore unsurprisingly, the compression savings aren't spectacular (somewhat surprisingly, compression saves 0.73% bandwidth, ie 428.16KiB):

$ curl -fsSL https://go.dev/dl/go1.25.4.linux-amd64.tar.gz -H 'Accept-Encoding: gzip' | wc -c
59327961
$ curl -fsSL https://go.dev/dl/go1.25.4.linux-amd64.tar.gz | wc -c
59766394

Answer 10 · 2025-11-08T21:33:07.000Z

I understand the comments made about this issue, but what puzzles me is why it only seems to be affecting that one file when I try to download through Firefox. For example, I can navigate to the download page and download the arm64.tar.gz file instead and the sha256sum of the resulting download matches exactly what is listed on the page. In addition, I can do the same thing with the 1.25.3 amd64.tar.gz and no problems there either

…

On Sat, Nov 8, 2025, 3:47 PM Jonathan Frech ***@***.***> wrote: *jfrech* left a comment (golang/go#76228) <#76228 (comment)> The Google server correctly sets Content-Encoding: gzip (as irrespective of representation, the resource is a .gzip file) and sets Content-Type: application/x-gzip when appropriate: $ curl -v -fsSL https://go.dev/dl/go1.25.4.linux-amd64.tar.gz -o/dev/null (...) < HTTP/2 200 < accept-ranges: bytes < content-disposition: attachment < content-security-policy: default-src 'none' < server: downloads < x-content-type-options: nosniff < x-frame-options: SAMEORIGIN < x-xss-protection: 0 < date: Sat, 08 Nov 2025 20:38:16 GMT < cache-control: no-transform,public,max-age=86400 < last-modified: Wed, 05 Nov 2025 19:03:09 GMT < etag: "50ad87b" < content-type: application/x-gzip < vary: Accept-Encoding < content-length: 59766394 < age: 0 < alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 $ curl -v -fsSL https://go.dev/dl/go1.25.4.linux-amd64.tar.gz --compressed -o/dev/null (...) < HTTP/2 200 < accept-ranges: bytes < content-disposition: attachment < content-encoding: gzip < content-security-policy: default-src 'none' < server: downloads < x-content-type-options: nosniff < x-frame-options: SAMEORIGIN < x-xss-protection: 0 < date: Sat, 08 Nov 2025 20:39:13 GMT < cache-control: no-transform,public,max-age=86400 < last-modified: Wed, 05 Nov 2025 19:03:09 GMT < etag: "50ad87b-6aa1e4d0" < content-type: application/x-gzip < vary: Accept-Encoding < content-length: 59327961 < age: 0 < alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 Furthermore unsurprisingly, the compression savings aren't spectacular (somewhat surprisingly, compression *saves* 0.73% bandwidth, ie 428.16KiB): $ curl -fsSL https://go.dev/dl/go1.25.4.linux-amd64.tar.gz -H 'Accept-Encoding: gzip' | wc -c 59327961 $ curl -fsSL https://go.dev/dl/go1.25.4.linux-amd64.tar.gz | wc -c 59766394 — Reply to this email directly, view it on GitHub <#76228 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC63OGZYPFVR7K4C6BPJBEL33ZJFHAVCNFSM6AAAAACLQIOTP2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTKMBWHA4DIOBRGU> . You are receiving this because you authored the thread.Message ID: ***@***.***>

Answer 11 · 2025-11-08T22:02:01.000Z

@dnelsonmc Does your Firefox download the file as being named "go1.25.4.linux-amd64.tar.gz" or as being named "go1.25.4.linux-amd64.tar"? My cursory Bugzilla searching has lead to a slew of bug reports over the last two decades talking about Firefox unexpectedly decompressing files:

cf. https://stackoverflow.com/a/71688998 [2025-11-08]
cf. https://bugzilla.mozilla.org/show_bug.cgi?id=121001 [2025-11-08]
cf. https://bugzilla.mozilla.org/show_bug.cgi?id=235859 [2025-11-08]
cf. https://bugzilla.mozilla.org/show_bug.cgi?id=233047 [2025-11-08]
cf. https://bugzilla.mozilla.org/show_bug.cgi?id=1470011 [2025-11-08]
cf. https://bugzilla.mozilla.org/show_bug.cgi?id=610679 [2025-11-08]

Answer 12 · 2025-11-08T22:07:57.000Z

The Google download server makes differing decisions about whether or not to serve a compressed representation (I surmise because "go1.25.4.linux-arm64.tar.gz" benefits (slightly!; see my previous comment) from transport compression and "go1.25.4.linux-amd64.tar.gz" doesn't):

$ curl -fsSL https://go.dev/dl/go1.25.4.linux-arm64.tar.gz --compressed -v -o/dev/null
(...)
< HTTP/2 200
< accept-ranges: bytes
< cache-control: no-transform
< cache-control: public
< cache-control: max-age=86400
< content-disposition: attachment
< content-length: 57268847
< content-security-policy: default-src 'none'
< content-type: application/x-gzip
< etag: "50ad923"
< last-modified: Wed, 05 Nov 2025 19:03:39 GMT
< server: downloads
< vary: Accept-Encoding
< x-content-type-options: nosniff
< x-frame-options: SAMEORIGIN
< x-xss-protection: 0
< date: Sat, 08 Nov 2025 22:03:04 GMT
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

$ curl -fsSL https://go.dev/dl/go1.25.4.linux-amd64.tar.gz --compressed -v -o/dev/null
< HTTP/2 200
< accept-ranges: bytes
< content-disposition: attachment
< content-encoding: gzip
< content-security-policy: default-src 'none'
< server: downloads
< x-content-type-options: nosniff
< x-frame-options: SAMEORIGIN
< x-xss-protection: 0
< date: Sat, 08 Nov 2025 22:03:20 GMT
< cache-control: no-transform,public,max-age=86400
< last-modified: Wed, 05 Nov 2025 19:03:09 GMT
< etag: "50ad87b-6aa1e4d0"
< content-type: application/x-gzip
< vary: Accept-Encoding
< content-length: 59327961
< age: 0
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

Answer 13 · 2025-11-08T22:27:26.000Z

I can confirm that Firefox 144.0.2 (64-bit) (Mozilla Firefox Snap for Ubuntu; canonical-002 1.0) incorrectly saves to a file named "go1.25.4.linux-amd64.tar.gz" (which is served with transport compression; see my comment above) the compressed representation but does correctly download "go1.25.4.linux-arm64.tar.gz" (which is served without transport compression; see my comment above):

# ((downloading via browser UI))
$ ls
go1.25.4.linux-amd64.tar.gz  go1.25.4.linux-arm64.tar.gz

$ sha256sum *
a8ae67a43f46994ebefa22aa83f52f415125bedfd0455eba5a08a3cc82160044  go1.25.4.linux-amd64.tar.gz
a68e86d4b72c2c2fecf7dfed667680b6c2a071221bbdb6913cf83ce3f80d9ff0  go1.25.4.linux-arm64.tar.gz

$ file *
go1.25.4.linux-amd64.tar.gz: gzip compressed data, original size modulo 2^32 59766394
go1.25.4.linux-arm64.tar.gz: gzip compressed data, max compression, original size modulo 2^32 214344192 gzip compressed data, reserved method, ASCII, has CRC, was "", from FAT filesystem (MS-DOS, OS/2, NT), original size modulo 2^32 214344192

$ gzip -d <go1.25.4.linux-amd64.tar.gz | sha256sum
9fa5ffeda4170de60f67f3aa0f824e426421ba724c21e133c1e35d6159ca1bec  -

Answer 14 · 2025-11-08T22:35:25.000Z

Suggest to change the issue's title to:

x/website/internal/dl: unpredictably applied HTTP transport compression for .tar.gz resources results in negligible bandwidth savings while confusing contemporary versions of Firefox

Go version

Output of go env in your module/workspace:

What did you do?

What did you see happen?

What did you expect to see?

Output of `go env` in your module/workspace: