panic on Writer.Close, runtime error: index out of range

Question

panic on Writer.Close, runtime error: index out of range

toffaletti opened this issue 9 years ago · 10 comments

Unfortunately I don't have a test to reproduce this, but over hundreds of thousands of calls we started seeing a few of these panics:

runtime error: index out of range
goroutine 148112966 [running]:
net/http.func·011()
        /usr/lib/go/src/net/http/server.go:1130 +0xbb
github.com/klauspost/compress/flate.(*compressor).deflateNoSkip(0xc20e431600)
        _vendor/src/github.com/klauspost/compress/flate/deflate.go:594 +0xc68
github.com/klauspost/compress/flate.(*compressor).close(0xc20e431600, 0x0, 0x0)
        _vendor/src/github.com/klauspost/compress/flate/deflate.go:773 +0x49
github.com/klauspost/compress/flate.(*Writer).Close(0xc20e431600, 0x0, 0x0)
        _vendor/src/github.com/klauspost/compress/flate/deflate.go:854

We never saw this before I updated our snapshot on Nov 2nd. The previous update was Sep 8th.

Answer 1 · 2015-11-17T08:00:14.000Z

Thanks for the report - I will investigate at once!

Answer 2 · 2015-11-17T08:33:32.000Z

Can you confirm that deflate.go:594 is also https://github.com/klauspost/compress/blob/master/flate/deflate.go#L594 ?

Answer 3 · 2015-11-17T09:17:31.000Z

Yes, and unfortunately there are two indexes used on that line. I'm not sure which is out of range.

Answer 4 · 2015-11-17T10:04:25.000Z

@maruel has found this bug too https://code.google.com/p/chromium/issues/detail?id=552697#c4 .
The unittest in our code is a bit large but can reproduce this.

Answer 5 · 2015-11-17T14:08:14.000Z

From the other report it is clear that it is d.tokens.n that somehow gets too high. I am trying to recreate a scenario where that happens. Unfortunately my 500k files "webserver" test does not reproduce it.

I am of course continuing work on this.

Answer 6 · 2015-11-17T14:16:12.000Z

OK. Are you checking returned error values?

If you call Write after a write has already failed (in case of the chrominum bug) I can see this error occurring.

Close after a failed write will do the same.

Both are of course bugs, and will be fixed - I just want to be sure I haven't overlooked anything.

Answer 7 · 2015-11-17T14:40:56.000Z

We reproduce in this case:

Compress, writing to an io.Pipe().
Using io.Copy() to write the input data to the compressor.
Length of data is >= 64kb + 1.
The read pipe from io.Pipe() is closed in the middle of the stream.

I was planning try to write a regression test before even bothering you about this but too late, eh. :)

Answer 8 · 2015-11-17T14:54:15.000Z

@toffaletti (shameless) plug for panicparse to make your stack traces shorter.

Answer 9 · 2015-11-17T18:14:52.000Z

Thanks for helping identify this issue. I have merged the fix after testing.

Answer 10 · 2015-11-17T21:39:10.000Z

In our code I don't control whether Write is called again after an error because our write is essentially json.NewEncoder(gzipWriter).Encode(). I will open a new issue if this fix didn't clear up the panic. Thanks so much for the prompt response despite so few details.