archive/zip: io.Reader like archive/tar

[krolaw]

I stream read zips in java and have a couple of Go projects that would benefit from this. Although zips have a footer that allows random access, files are saved sequentially, which allows for streaming as well.

[minux]

What exactly do you want? Are you suggesting that NewReader (or a variant of that) should take an io.Reader instead of an io.ReaderAt? That's impossible to do, as to read a zip file, you have to read the directory at the end, without that information, you don't even know whether each file is compressed or not.

[krolaw]

Impossible? Here's a java snippet (of the critical code) from an installer I wrote years ago, which reads large zip files, and displays images from near the start of the archive while the rest of it downloads.

URLConnection connection = new URL(updateZipURL).openConnection();
BufferedInputStream in = new BufferedInputStream(new MyStream(connection.getInputStream()));
zipStream = new ZipInputStream(in);
ZipEntry entry;
while ((entry = zipStream.getNextEntry()) != null) {
// If image, add to display queue, otherwise unpack to file.
}

API for ZipInputStream here: https://docs.oracle.com/javase/7/docs/api/java/util/zip/ZipInputStream.html

To answer your first question, I want to port the installer to go. I also have another project that would benefit from this library.

[minux]

Of course it won't work in the general case.

consider this:
zip a.zip malicious_stuff
zip b.zip good_stuff
cat a.zip b.zip > c.zip
zip -A c.zip

zipinfo c.zip or unzip c.zip will only show the good_stuff, however, a streaming
uncompressor will definitely see the malicious_stuff. It may or may not see
the good_stuff.

In general, zip file can have arbitrary prefixed data at the front, how could a
streaming uncompressor deal with that?

[krolaw]

I like your example. It shows the care and thoughtfulness that goes into the standard library. The javadoc makes no mention of how false positives are caught (probably aren't) or should be handled.

To deal with arbitrary prefixed data, one could return errors from the Next function:
PreviousFilesInvalidError:
where a directory record ends, but there is still more data in the stream. This would indicate two (or more) zip files concatenated, and that all previous files were false and that more files may be coming. This error may also be used in the case where the stream is closed without a directory record, or could fire an InvalidFileFormatError or similar.

FirstNFilesInvalidError: could be used to catch cases where files were prepended without directory record. Technically this error could/should be used for the above case as well.

Alternatively/Additionally, one could have explanations/warnings of streaming zips in the docs.

Thanks.

[krolaw]

Hi Minux,

I think, given the concerns I'll write my own third party library and copy across bits from archive/zip.

Thanks for your time.

[minux]

Yeah, the feature does make sense for certain applications where you can trust the content to be the "normal" zip files, so implementing it as a go-gettable package would be great (I think you can reuse a lot of the code in the package). But for the standard archive/zip package, I think we'd better not introduce non-general interfaces that might have security issues if used improperly. Thanks. PS: if your installer unzip file directly from http/https, you use use one of the range request adapters (search godoc or golang-nuts) that turns any idempotent http response into seekable stream and then use the standard archive/zip. As long as you extract strictly according to the order in the zip file, I don't think the performance will be that bad (basically it will first seek to the end, read the central directories and then go back to the beginning and extract files.)

[krolaw]

Finally got around to putting something together:
https://github.com/krolaw/zipstream