open-sw/pkg-cacher

pkg-cacher hangs for a very long time for 404's

jkugler opened this issue · 4 comments

jkugler commented

We are experimenting with pkg-cacher for our purposes. Functionally, it is exactly what we've been looking for. However, we've hit an odd bug.

Config options different than default:

daemon_port=9000
path_map = e2fsprogs-master http://build.whamcloud.com/job/e2fsprogs-master/ ; e2fsprogs-reviews http://build.whamcloud.com/job/toolkit-reviews/ ; toolkit http://build.whamcloud.com/job/toolkit/ ; toolkit-reviews http://build.whamcloud.com/job/toolkit/

It seems to often hang on trying to download this URL:

http://localhost:9000/toolkit/toolkit/arch=x86_64,distro=el6/lastSuccessfulBuild/artifact/_topdir/RPMS/noarch/mpich-doc-3.0.4-3.wc1.noarch.rpm

Then after a while, it will finally fail with:

HTTP request sent, awaiting response... Read error (Connection timed out) in headers.

Looking more closely at that URL, I realized it was invalid ("toolkit" was duplicated), and thus sometimes it would produce the 404, other times it would produce the hang, and then read error.

It's been a long time since I've dug into Perl, so I thought maybe you'd have a quick idea on how to solve this.

BTW, I was using wget to test this.

jkugler commented

Possibly related from the read errors/404s

Mon Mar 17 16:32:13 2014|error [24046]: --- /usr/sbin/pkg-cacher: Fatal: /var/cache/pkg-cacher/packages/toolkit/arch=x86_64,distro=sles11sp3/lastSuccessfulBuild/artifact/_topdir/RPMS/x86_64/posix-1.0-wc1.x86_64.rpm is the wrong size, expected 0, got 3407707
Mon Mar 17 16:42:47 2014|error [25194]: --- /usr/sbin/pkg-cacher: Fatal: /var/cache/pkg-cacher/packages/toolkit/toolkit/arch=x86_64,distro=el6/lastSuccessfulBuild/artifact/_topdir/RPMS/noarch/mpich-doc-3.0.4-3.wc1.noarch.rpm is the wrong size, expected 0, got 3900
Mon Mar 17 16:57:39 2014|error [26662]: --- /usr/sbin/pkg-cacher: Fatal: /var/cache/pkg-cacher/packages/toolkit/toolkit/arch=x86_64,distro=el6/lastSuccessfulBuild/artifact/_topdir/RPMS/noarch/mpich-doc-3.0.4-3.wc1.noarch.rpm is the wrong size, expected 0, got 3924
Mon Mar 17 16:57:40 2014|warn [26661]: Use of uninitialized value $filesize in concatenation (.) or string at /usr/sbin/pkg-cacher line 234.
Mon Mar 17 17:12:42 2014|warn [28344]: Use of uninitialized value $filesize in concatenation (.) or string at /usr/sbin/pkg-cacher line 234.
Mon Mar 17 19:51:45 2014|error [12468]: --- /usr/sbin/pkg-cacher: Fatal: /var/cache/pkg-cacher/packages/toolkit/toolkit/arch=x86_64,distro=el6/lastSuccessfulBuild/artifact/_topdir/RPMS/noarch/mpich-doc-3.0.4-3.wc1.noarch.rpm is the wrong size, expected 0, got 3900
Mon Mar 17 20:06:37 2014|warn [14144]: Use of uninitialized value $filesize in concatenation (.) or string at /usr/sbin/pkg-cacher line 234.
Mon Mar 17 20:08:16 2014|error [14332]: --- /usr/sbin/pkg-cacher: Fatal: /var/cache/pkg-cacher/packages/toolkit/toolkit/arch=x86_64,distro=el6/lastSuccessfulBuild/artifact/_topdir/RPMS/noarch/mpich-doc-3.0.4-3.wc1.noarch.rpm is the wrong size, expected 0, got 4425
Mon Mar 17 20:09:11 2014|warn [14454]: Use of uninitialized value $filesize in concatenation (.) or string at /usr/sbin/pkg-cacher line 234.
Robert-Nelson commented

Thanks for the info, I think I know the problem, I'll send you a new file to try shortly.

----- Original Message -----

| From: "Joshua Kugler" notifications@github.com
| To: "open-sw/pkg-cacher" pkg-cacher@noreply.github.com
| Sent: Monday, March 17, 2014 9:41:31 PM
| Subject: Re: [pkg-cacher] pkg-cacher hangs for a very long time for 404's
| (#9)

| Possibly related from the read errors/404s
| Mon Mar 17 16:32:13 2014|error [24046]: --- /usr/sbin/pkg-cacher: Fatal:
| /var/cache/pkg-cacher/packages/toolkit/arch=x86_64,distro=sles11sp3/lastSuccessfulBuild/artifact/_topdir/RPMS/x86_64/posix-1.0-wc1.x86_64.rpm
| is the wrong size, expected 0, got 3407707
| Mon Mar 17 16:42:47 2014|error [25194]: --- /usr/sbin/pkg-cacher: Fatal:
| /var/cache/pkg-cacher/packages/toolkit/toolkit/arch=x86_64,distro=el6/lastSuccessfulBuild/artifact/_topdir/RPMS/noarch/mpich-doc-3.0.4-3.wc1.noarch.rpm
| is the wrong size, expected 0, got 3900
| Mon Mar 17 16:57:39 2014|error [26662]: --- /usr/sbin/pkg-cacher: Fatal:
| /var/cache/pkg-cacher/packages/toolkit/toolkit/arch=x86_64,distro=el6/lastSuccessfulBuild/artifact/_topdir/RPMS/noarch/mpich-doc-3.0.4-3.wc1.noarch.rpm
| is the wrong size, expected 0, got 3924
| Mon Mar 17 16:57:40 2014|warn [26661]: Use of uninitialized value $filesize
| in concatenation (.) or string at /usr/sbin/pkg-cacher line 234.
| Mon Mar 17 17:12:42 2014|warn [28344]: Use of uninitialized value $filesize
| in concatenation (.) or string at /usr/sbin/pkg-cacher line 234.
| Mon Mar 17 19:51:45 2014|error [12468]: --- /usr/sbin/pkg-cacher: Fatal:
| /var/cache/pkg-cacher/packages/toolkit/toolkit/arch=x86_64,distro=el6/lastSuccessfulBuild/artifact/_topdir/RPMS/noarch/mpich-doc-3.0.4-3.wc1.noarch.rpm
| is the wrong size, expected 0, got 3900
| Mon Mar 17 20:06:37 2014|warn [14144]: Use of uninitialized value $filesize
| in concatenation (.) or string at /usr/sbin/pkg-cacher line 234.
| Mon Mar 17 20:08:16 2014|error [14332]: --- /usr/sbin/pkg-cacher: Fatal:
| /var/cache/pkg-cacher/packages/toolkit/toolkit/arch=x86_64,distro=el6/lastSuccessfulBuild/artifact/_topdir/RPMS/noarch/mpich-doc-3.0.4-3.wc1.noarch.rpm
| is the wrong size, expected 0, got 4425
| Mon Mar 17 20:09:11 2014|warn [14454]: Use of uninitialized value $filesize
| in concatenation (.) or string at /usr/sbin/pkg-cacher line 234.

| —
| Reply to this email directly or view it on GitHub .

jkugler commented

This seems to have fixed the issue, even though the downloads had nothing to do with nginx, nor (I don't think) chunked transfers. But the fact that it doesn't hang on 404's any more is excellent! Thank you!

Robert-Nelson commented

I'm glad its working now, I just uploaded the official version 1.0.2 files.

Btw, looking at the errors in your earlier mails I think the server you're accessing is using chunked transfers :-)

This message happens when there is no Content-Length header which occurs when doing chunked transfers:
Mon Mar 17 16:32:13 2014|error [24046]: --- /usr/sbin/pkg-cacher: Fatal: /var/cache/pkg-cacher/packages/toolkit/arch=x86_64,distro=sles11sp3/lastSuccessfulBuild/artifact/_topdir/RPMS/x86_64/posix-1.0-wc1.x86_64.rpm is the wrong size, expected 0, got 3407707

The hang on 404s was probably caused by a HEAD request with a body which the script didn't expect and it caused a fatal error.

----- Original Message -----

| From: "Joshua Kugler" notifications@github.com
| To: "open-sw/pkg-cacher" pkg-cacher@noreply.github.com
| Cc: "Robert Nelson" robertn@the-nelsons.org
| Sent: Monday, March 24, 2014 11:32:09 AM
| Subject: Re: [pkg-cacher] pkg-cacher hangs for a very long time for 404's
| (#9)

| This seems to have fixed the issue, even though the downloads had nothing to
| do with nginx, nor (I don't think) chunked transfers. But the fact that it
| doesn't hang on 404's any more is excellent! Thank you!

| —
| Reply to this email directly or view it on GitHub .