Problems with Zip Files after upgrade to 7.0.0
Closed this issue · 155 comments
Steps to reproduce
- Login und go to files
- select a couple of files download the generated zip file
- try to decompress with Total Commander
Expected behaviour
Files should be extracted
Actual behaviour
Total Commander gives CRC warnings
Server configuration
Operating system:
debian 7
Web server:
apache 2.2
Database:
postgres 9.1
PHP version:
5.4
ownCloud version: (see ownCloud admin page)
7.0.0
Updated from an older ownCloud or fresh install:
From 6.0.4
I got an hint from the Total Commander forum: http://ghisler.ch/board/viewtopic.php?t=40766 (in german)
The linked forum post says that the generated zip has two issues:
- the content is not compressed
- the file entries in the zip file don't provide valid crc data (and thats why total commander complains).
OSX also says that the downloaded zip file is not a valid zip file.
See also: http://forum.owncloud.org/viewtopic.php?f=29&t=22524
@McNetic can you please have a look at this? This is related to the zip streamer component
Moved to 7.0.2 for investigation.
Ok. I had a look at the issue.
First of all, this is two separate problems, one with TotalCommander, one with OSX finder.
Regarding the TotalCommander issue:
- The content is not compressed. This is totally compliant with the zip standard, and is a known issue already discussed elsewhere (as php does provide no known solution to do chunk-wize deflate compression).
- Totalcommander 8.51a complains about crc errors. The linked forum post suggests that crc is missing from the zip file. AFAICS: That is not true. The crc is in place and correct. What indeed is "missing" is the compressed/uncompressed file size information in the header (in fact, it is set to -1, as the zip standard demands for stream compression). There was an issue with Totalcommander in past versions (see McNetic/PHPZipStreamer#8). Back then, I tested with v8.50 and had not problems. I don't have v8.50 any more, and in more recent versions, the zip handling was changed according to the changelog, so I suspect there went something wrong. I'll try to sort this out with the Totalcommander author.
Regarding the OSX finder issue:
I'm afraid OSX finder is not able to handle this kind of (standards-compliant) zip files correctly. This is a known bug in OSX and Zip64 extension - search google for "osx zip cpgz"). Other software on OSX works fine (see McNetic/PHPZipStreamer#4). I'd be very pleased if someone had an idea how to make OSX finder happy.
Hi, i started the thread on the totalcommander-board, i would like to cross-refrence this one to bring it to the attention of the author of tc.
The ZIP-Files produced by OC6 didn't cause this error in TC - and if i remember correct, they were compressed.
As i've mentioned in the tc-thread, other tools handle the archives just well.
I have the same issue on FreeBSD (as noted in the duplicate ticket). Is there any troubleshooting I need to do for this?
Is there a way to stop the zip creation and maybe just download them all separately.
In my opinion, it's not a Finder-related issue, the problem exists with the command line too:
$ which unzip
/usr/bin/unzip
$ unzip download.zip
Archive: download.zip
warning [download.zip]: zipfile claims to be last disk of a multi-part archive;
attempting to process anyway, assuming all parts have been concatenated
together in order. Expect "errors" and warnings...true multi-part support
doesn't exist yet (coming soon).
error [download.zip]: missing 8469400417 bytes in zipfile
(attempting to process anyway)
error [download.zip]: attempt to seek before beginning of zipfile
(please check that you have transferred or created the zipfile in the
appropriate BINARY mode and that you have compiled UnZip properly)
$ unzip -v
UnZip 5.52 of 28 February 2005, by Info-ZIP. Maintained by C. Spieler. Send
bug reports using http://www.info-zip.org/zip-bug.html; see README for details.
Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip/ ;
see ftp://ftp.info-zip.org/pub/infozip/UnZip.html for other sites.
Compiled with gcc 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68) for Unix on Aug 24 2013.
UnZip special compilation options:
COPYRIGHT_CLEAN (PKZIP 0.9x unreducing method not supported)
SET_DIR_ATTRIB
TIMESTAMP
USE_EF_UT_TIME
USE_UNSHRINK (PKZIP/Zip 1.x unshrinking method supported)
USE_DEFLATE64 (PKZIP 4.x Deflate64(tm) supported)
VMS_TEXT_CONV
[decryption, version 2.9 of 05 May 2000]
UnZip and ZipInfo environment options:
UNZIP: [none]
UNZIPOPT: [none]
ZIPINFO: [none]
ZIPINFOOPT: [none]
@McNetic ^
Afaics, your unzip version is too old. Obviously, Apple is packaging 9 year old software to handle a file format whose spec is a moving target.
nico@haven:~/programming/ZipStreamer$ unzip -v
UnZip 6.00 of 20 April 2009, by Debian. Original by Info-ZIP.
Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip/ ;
see ftp://ftp.info-zip.org/pub/infozip/UnZip.html for other sites.
Compiled with gcc 4.7.2 for Unix (Linux ELF) on Nov 28 2012.
UnZip special compilation options:
ACORN_FTYPE_NFS
COPYRIGHT_CLEAN (PKZIP 0.9x unreducing method not supported)
SET_DIR_ATTRIB
SYMLINKS (symbolic links supported, if RTL and file system permit)
TIMESTAMP
UNIXBACKUP
USE_EF_UT_TIME
USE_UNSHRINK (PKZIP/Zip 1.x unshrinking method supported)
USE_DEFLATE64 (PKZIP 4.x Deflate64(tm) supported)
UNICODE_SUPPORT [wide-chars, char coding: UTF-8] (handle UTF-8 paths)
LARGE_FILE_SUPPORT (large files over 2 GiB supported)
ZIP64_SUPPORT (archives using Zip64 for large files supported)
USE_BZIP2 (PKZIP 4.6+, using bzip2 lib version 1.0.6, 6-Sept-2010)
VMS_TEXT_CONV
WILD_STOP_AT_DIR
[decryption, version 2.11 of 05 Jan 2007]
UnZip and ZipInfo environment options:
UNZIP: [none]
UNZIPOPT: [none]
ZIPINFO: [none]
ZIPINFOOPT: [none]
There's obviously missing LARGE_FILE_SUPPORT (may not be needed in this special case), but in any case ZIP64_SUPPORT and also (unrelated to the issue, but also problematic) UNICODE_SUPPORT.
Ok, I'd like to open a discussion on this one: Obviously, the problem can not be really solved. On the one hand, we need Zip64 support to be able to deliver zip files larger than 4GB. On the other hand, there's prominent software (especially MAC OSX) that can still not handle the Zip64 extension. The latter problem can be overcome by third party products (not all, and I don't remember which, but afair it's mentioned in one of the ZipStreamer issues).
- I think staying with large file support is the way to go (as the default).
- we could add an option to disable Zip64 support manually by the user (though I'm afraid most users won't understand when and why to enable this, and what the consequences may be).
- I could make the Zip64 support in ZipStreamer optional, so Owncloud could summarize the file sizes and disable it when not required, so the problem only occurs for large files.
Which option shall we pick? Do you see other options?
@DeepDiver1975 could you notify whomever takes part in this decision?
@McNetic - while I agree that it is frustrating that the default zip utility in OSX is out of date, I also think that we need to be sure that the burden is not on our OwnCloud end-users to have to worry about using alternate desktop zip utilities. My thought is that we would want the priority to be on end-user compatabilty over support for niche use cases. The need to support zip generation above 4GB is likely a rare one and can even be a server resource issue.
- I think the best of all evils is sticking with the broadest zip client support possible to ensure maximum end-user compatability as a default.
- Allow enabling of Zip64 (and thus 4GB+) as a config setting for specific installations that require it...i.e. corporate deployments where administrators know and can control their desktop environment.
Dropbox even limits the download as zip feature to 1GB and presents users with a nice message saying the contents are too large to download as zip: https://www.dropbox.com/en/help/49
The bad news
This is form OS X Yosemite Beta (10.10):
$ unzip -v
UnZip 5.52 of 28 February 2005, by Info-ZIP. Maintained by C. Spieler. Send
bug reports using http://www.info-zip.org/zip-bug.html; see README for details.
Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip/ ;
see ftp://ftp.info-zip.org/pub/infozip/UnZip.html for other sites.
Compiled with gcc 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.38) for Unix on Jun 28 2014.
UnZip special compilation options:
COPYRIGHT_CLEAN (PKZIP 0.9x unreducing method not supported)
SET_DIR_ATTRIB
TIMESTAMP
USE_EF_UT_TIME
USE_UNSHRINK (PKZIP/Zip 1.x unshrinking method supported)
USE_DEFLATE64 (PKZIP 4.x Deflate64(tm) supported)
VMS_TEXT_CONV
[decryption, version 2.9 of 05 May 2000]
UnZip and ZipInfo environment options:
UNZIP: [none]
UNZIPOPT: [none]
ZIPINFO: [none]
ZIPINFOOPT: [none]
And yes of course own cloud should try to support as many clients as possible. But I'm not sure if it's a good idea to switch between Zip64 support at the settings for the whole server.
Better would be something that let the user of the download decide like:
- compatibility mode
- advanced mode (support for > 4GB, ...) - not working everywhere
This would be especially fine for link-shared folders, where you can have much more users (and clients) then in a small personal setup.
Maybe this decision can be preselected by the user agent information.
Ok, I'd like to open a discussion on this one: Obviously, the problem can not be really solved. On the one hand, we need Zip64 support to be able to deliver zip files larger than 4GB. On the other hand, there's prominent software (especially MAC OSX) that can still not handle the Zip64 extension. The latter problem can be overcome by third party products (not all, and I don't remember which, but afair it's mentioned in one of the ZipStreamer issues).
- I think staying with large file support is the way to go (as the default).
we could add an option to disable Zip64 support manually by the user (though I'm afraid most users won't understand when and why to enable this, and what the consequences may be).- I could make the Zip64 support in ZipStreamer optional, so Owncloud could summarize the file sizes and disable it when not required, so the problem only occurs for large files.
Which option shall we pick? Do you see other options?
@karlitschek This is really an annoying issue to me and Mac users in general. I'd advise to go with the second approach (use zip64 if the overall filesize is >4GB, otherwise use the regular ZIP format that is understood by any zip version).
Users can live with the fact that large archives (>4GB) does not work with OS X. But broken downloads are just really annoying. Your decision please.
also @MTRichards and @craigpg
Agreed. Can you provide a Patch?
@LukasReschke Agreed with you and @karlitschek . I like the second option as well. It provides most users with an easy path to making it all work with no intervention.
@McNetic Could you make zip64 support in ZipStreamer optional? (e.g. enabled per default but can be disabled via the constructor) - I can then take care of the needed changes in ownCloud. Thanks a lot!
I was also hit by this issue under OSX and I also have to say that it is pretty much appreciated if it could be resolved. As for how to fix this and disabling zip64 support by default to support OSX clients I think it might also be a possibility to make zip64 support depending on the operating system a browser reports. So wouldn't it be simply possible to query the browser for the operating system being used and then disable zip64 support for downloads with OSX while other platforms will keep it per default? Would that be an option?
- I'll change the implementation of ZipStreamer to make Zip64 support optional, but I cannot promise a fixed date (I hope only a couple of days).
- It should not at all be a performance issue on the server side. Afaics, the load of ZipStreamer does not change with bigger archives, as it works with fixed size chunks and sends them to the client immediately.
- I like the option of detecting OSX and (perhaps) disabling zip64 support by default - or provide a small hint that explains the issue so power users can override and normal users understand what happens and why.
- By the way, I have a patch quite ready that will enable (optional) deflate compression support. I'm still lacking unit tests for this; and I'm not really happy about the current unit tests (that compare output to some pre-generated zip files; as soon as zip options are changed, the output will change as well). I'd be more than happy if someone had a better idea how to implement the unit tests.
I'll change the implementation of ZipStreamer to make Zip64 support optional, but I cannot promise a fixed date (I hope only a couple of days).
Great. Did you have time to look into this or is there something that we can do to help you? :)
I like the option of detecting OSX and (perhaps) disabling zip64 support by default - or provide a small hint that explains the issue so power users can override and normal users understand what happens and why.
From my understanding we should do the following checks and if one of them is positive we need to use ZIP64 and show a warning to users with an OS X UA:
- Check if the files are >4GB
- Check if there are more than 65’535 files
Or is there any downside if we use regular ZIP instead of ZIP64 as default?
By the way, I have a patch quite ready that will enable (optional) deflate compression support. I'm still lacking unit tests for this; and I'm not really happy about the current unit tests (that compare output to some pre-generated zip files; as soon as zip options are changed, the output will change as well). I'd be more than happy if someone had a better idea how to implement the unit tests.
Where can we see this changes? Which branch/PR?
@guruz Maybe I was wrong there. but didn't we also upgrade to have utf8-filenames? or is that independent from the 64 bit zip file format?
hi,
I was setting up own cloud as a gui front end to send ftp links to clients.
in os x 10.7.5 downloading multiple files (~256mb) resulted in a zip file that requires a 3rd party app to unzip - which is deal breaker.
If we could set it to download multiple files without creating a zip, that would be helpful.
Did this get implemented? Not sure if this is resolved from the above.
No. This still requires an upstream fix.
Is there any news on this issue?
As a semi permanent workaround for OSX users is to install and use the Free Unarchiver utility from the Mac app store. Its better for all uncompress work anyway. Been using it for years and it works like a dream. https://itunes.apple.com/au/app/the-unarchiver/id425424353
This is something that should really be fixed from Apples side - they should not be bundling zip versions from 2005 that does not properly support large files or zip64 encoding.
(can i just reply to the web post via email?)
I have done this option for myself, but when sending links to files to
clients, this doesnt active my goal of simplifying sending files hosted
on our ftp site.
I was trying to avoid asking clients to download a specific software to
download files - either a ftp client or an unarchiving tool.
Just about all of my clients are on a mac.
On 2014-09-11 10:12 am, trentster wrote:
As a semi permanent workaround for OSX users is to install and use the
Free Unarchiver utility from the Mac app store. Its better for all
uncompress work anyway. Been using it for years and it works like a
dream. https://itunes.apple.com/au/app/the-unarchiver/id425424353 [1]This is something that should really be fixed from Apples side - the
should not be bundling zip versions from 2005 that does not support
large files or zip64Reply to this email directly or view it on GitHub [2].
Links:
[1] https://itunes.apple.com/au/app/the-unarchiver/id425424353
[2] #10001 (comment)
I think it has been discussed enough before that using an alternative unarchiver or waiting until Apple has fixed their zip support is no alternative. Thus, we need a short-term solution for the issue discussed here. So any progress on an improved Zip support in owncloud?
Agreed, asking end-users to install anything special is never a solution.
I don't see any issue with cutting off support for outdated OS versions, but this is not the case.
Unfortunatly, Apple's laziness is something that needs to be worked around :-)
On Thu, Sep 11, 2014 at 10:23 AM, christianmatts notifications@github.com
wrote:
(can i just reply to the web post via email?)
I have done this option for myself, but when sending links to files to
clients, this doesnt active my goal of simplifying sending files hosted
on our ftp site.
I was trying to avoid asking clients to download a specific software to
download files - either a ftp client or an unarchiving tool.
Just about all of my clients are on a mac.
On 2014-09-11 10:12 am, trentster wrote:As a semi permanent workaround for OSX users is to install and use the
Free Unarchiver utility from the Mac app store. Its better for all
uncompress work anyway. Been using it for years and it works like a
dream. https://itunes.apple.com/au/app/the-unarchiver/id425424353 [1]This is something that should really be fixed from Apples side - the
should not be bundling zip versions from 2005 that does not support
large files or zip64Reply to this email directly or view it on GitHub [2].
Links:
[1] https://itunes.apple.com/au/app/the-unarchiver/id425424353
[2] #10001 (comment)
Reply to this email directly or view it on GitHub:
#10001 (comment)
I'm on osx 10.7 … is that already outdate?
If so myself/my office and most of my clients are outdated.
And I'm using the own cloud free product so i can't complain,
but this isn't the first time support for products advertising os x
support basically says the solution my problem is to use windows --
which isn't actually helpful.
On 2014-09-11 10:29 am, Cagex wrote:
Agreed, asking end-users to install anything special is never a
solution.I don't see any issue with cutting off support for outdated OS
versions, but this is not the case.Unfortunatly, Apple's laziness is something that needs to be worked
around :-)On Thu, Sep 11, 2014 at 10:23 AM, christianmatts
notifications@github.com
wrote:(can i just reply to the web post via email?)
I have done this option for myself, but when sending links to files to
clients, this doesnt active my goal of simplifying sending files
hosted
on our ftp site.
I was trying to avoid asking clients to download a specific software
to
download files - either a ftp client or an unarchiving tool.
Just about all of my clients are on a mac.
On 2014-09-11 10:12 am, trentster wrote:As a semi permanent workaround for OSX users is to install and use
the
Free Unarchiver utility from the Mac app store. Its better for all
uncompress work anyway. Been using it for years and it works like a
dream. https://itunes.apple.com/au/app/the-unarchiver/id425424353 [1]This is something that should really be fixed from Apples side - the
should not be bundling zip versions from 2005 that does not support
large files or zip64Reply to this email directly or view it on GitHub [2].
Links:
[1] https://itunes.apple.com/au/app/the-unarchiver/id425424353
[2]#10001 (comment)
Reply to this email directly or view it on GitHub:
#10001 (comment)Reply to this email directly or view it on GitHub [1].
Links:
[1] #10001 (comment)
Just a small update: Life got in my way, but I'm on to the issue.
Thanks! I can hardly wait to checkout your commit.
Just a small update: Life got in my way, but I'm on to the issue.
Thanks a lot @McNetic
hello, @McNetic did you find a solution ? is this for next version of the software as @MTRichards said ?
hello, nothing new about the zip problem on Mac ?
Really nothing new? Its really annoying since here on my server there are around 50 users using OSX....
There is a work in progress version by @McNetic - I'm limiting this issue to collaborators since we all know that the issue is annoying but pinging every few days doesn't make it resolved faster. In fact, it notifies everybody in this issue which is a huge waste of time.
Thanks.
The relevant issue in ZipStreamer is McNetic/PHPZipStreamer#12.
@DeepDiver1975 was this put into OC 7 or master ?
@PVince81 This is stable7, but unfortunately the regular ZIP-mode added has still problems with OS X. See McNetic/PHPZipStreamer#16
question is how long we are willing to wait ... or shall we dive into this on our own?
I already tried. - It's a world of pain and undocumented OS X behaviour :(
Just a small update: the remaining issues with OSX finder support are handled here: McNetic/PHPZipStreamer#16. Thanks to @LukasReschke, we got the idea that maybe it could work with with deflate compression enabled. I have a working implementation, but that needs to be merged and unit tests adapted. It depends on pecl_http, however. Is having that as a dependency in owncloud acceptable (ZipStreamer will still work without it, but will of course fall back to store instead of deflate)?
I'll have a look into it as soon as my pneumonia is better.
It depends on pecl_http, however. Is having that as a dependency in owncloud acceptable (ZipStreamer will still work without it, but will of course fall back to store instead of deflate)?
Generally speaking it would be way better to avoid additional dependencies as they are not in every case installable and with distributions such as RHEL the fun gets even bigger. - Is there any way that we can avoid that?
Is having that as a dependency in owncloud acceptable (ZipStreamer will still work without it, but will of course fall back to store instead of deflate)
Well - if there is no way around that: It has to be. But before that I guess we should be certain that there is no way around that. Let us know if we can help you somehow. (man power etc…)
Yes an additional dependency is a serious problem.
Well, as far as I can see, standard php has no way to do stream compression. It has at least 3 ways to deflate, but none of them can work on chunks. The only way I found (after quite a search), is in pecl_http.
It's no voodoo, it's just that php is obviously badly designed, developed and maintained in some ways.
An alternative might be to implement deflate in php itself - but I can't say anything on performance there. It might be fast, perhaps at least for minimum compression, or it might be totally slow. But let's wait and see if it solves the problem at all.
PS: I don't know if anyone already did, but I reported the problem with apply bug reporter, also. It's really a shame they are successful in telling people they build the best technology.
Upstream project is still working on this issue -> OC8.1
Update: There is now a deflate branch in PHPZipStreamer (https://github.com/McNetic/PHPZipStreamer/tree/deflate) that implements deflate compression when pecl_http (should now also work with v >= 2.0) is available. It could now be tested if MacOS is happy with Zip files created this way. I'm still investigating how to make it work without pecl_http.
@LukasReschke @georgehrke my dear mac users - can you please test this branch? THX
Another update: Looking at the deflate rfc, I noticed deflate can fall back to uncompressed blocks, which is totally simple to implement and imposes nearly no overhead compared to not using deflate at all, so I implemented an option to select deflate compression (COMPR::DEFLATE), but set compression level to none (COMPR::NONE). This should produce valid zip files (zip -t does not complain) with compression formally enabled, but without the need for pecl_http. Said branch now contains this addition.
awesome @McNetic THX
Will test later.
As I'm a big fan of digging in binary formats (not) I had the past few hours a look at the implementation and the ZIP specification (and realised that with > 10.8 even the store method might work and used compress => COMPR::STORE and level => COMPR::NORMAL, though maybe not for directories) but there seem to be some problems left:
- Extracting files not within a subdirectory fails because the general purpose purpose bit ADD flag (
0x0008) seems to cause problems with OS X here. When setting the NONE flag (0x0000) those files can be extracted, so this case finally works. - Extracting directories does not work at all and causes the same error when using DEFLATE and when not as well. Interestingly, a simple
zipcommand (zip --copy in.zip --out out.zip) is able to repair the ZIP file. But for me it just looks like they added the CRC? (which we can't do to my understanding)
Files are available at: https://s3.owncloud.com/owncloud/public.php?service=files&t=c8d0394f050872efc8f32bd15d0a4d84
What is interesting is that http://pablotron.org/software/zipstream-php/ works just fine (after changing the version to 0x000A because of some Finder madness). Can we maybe take a look at their implementation and see what makes the difference? (they seem to use gzdeflate and no ZIP64 but that is irrelevant for now)
@McNetic Do you happen to see on a first glance what coul dmake the difference there?
I also compared some 7ZIP and ZipStreamer file using STORE, the output looks pretty much the same except the CRC (excerpt - more was not really relevant):
| ZipStreamer | Working directory ZIP created with 7zip | |
|---|---|---|
| Local file header signature | 50 4B 03 04 | 50 4B 03 04 |
| Version needed to extract | 14 00 | 14 00 |
| General purpose bit flag | 00 00 | 00 00 |
| Compression method | 00 00 | 00 00 |
| File last modification time | BA 74 | BA 74 |
| File last modification date | 4D 46 | 4D 46 |
| CRC-32 | 00 00 00 00 | 00 00 00 00 |
| Compressed size | 00 00 00 00 | 00 00 00 00 |
| Uncompressed size | 00 00 00 00 | 00 00 00 00 |
| File name length (n) | 05 00 | 05 00 |
| Extra field length (m) | 00 00 | 00 00 |
| File name | 74 65 73 74 2F | 74 65 73 74 2F |
| Local file header signature | 50 4B 03 04 | 50 4B 03 04 |
| Version needed to extract (minimum) | 0A 00 | 0A 00 |
| General purpose bit flag | 00 00 | 00 00 |
| Compression method | 00 00 | 00 00 |
| File last modification time | 9C 5B | 9C 5B |
| File last modification date | 4D 46 | 4D 46 |
| CRC-32 | 00 00 00 00 | 5E AC 80 ED |
| Compressed size | 00 00 00 00 | A3 00 00 00 |
| Uncompressed size | 00 00 00 00 | A3 00 00 00 |
| File name length (n) | 10 00 | 10 00 |
| Extra field length (m) (maybe also filename - not sure) | 00 00 | 00 00 |
| File name | 74 65 73 74 2F 77 65 6C 63 6F 6D 65 2E 74 78 74 | 74 65 73 74 2F 77 65 6C 63 6F 6D 65 2E 74 78 74 |
| Data | 57 65 6C 63 6F 6D 65 20 74 6F 20 79 6F 75 72 20 6F 77 6E 43 6C 6F 75 64 20 61 63 63 6F 75 6E 74 21 0A 0A 54 68 69 73 20 69 73 20 6A 75 73 74 20 61 6E 20 65 78 61 6D 70 6C 65 20 66 69 6C 65 20 66 6F 72 20 64 65 76 65 6C 6F 70 65 72 73 20 61 6E 64 20 67 69 74 20 75 73 65 72 73 2E 20 0A 54 68 65 20 70 61 63 6B 61 67 65 64 20 61 6E 64 20 72 65 6C 65 61 73 65 64 20 76 65 72 73 69 6F 6E 73 20 77 69 6C 6C 20 63 6F 6D 65 20 77 69 74 68 20 62 65 74 74 65 72 20 65 78 61 6D 70 6C 65 73 2E 0A 0A | 57 65 6C 63 6F 6D 65 20 74 6F 20 79 6F 75 72 20 6F 77 6E 43 6C 6F 75 64 20 61 63 63 6F 75 6E 74 21 0A 0A 54 68 69 73 20 69 73 20 6A 75 73 74 20 61 6E 20 65 78 61 6D 70 6C 65 20 66 69 6C 65 20 66 6F 72 20 64 65 76 65 6C 6F 70 65 72 73 20 61 6E 64 20 67 69 74 20 75 73 65 72 73 2E 20 0A 54 68 65 20 70 61 63 6B 61 67 65 64 20 61 6E 64 20 72 65 6C 65 61 73 65 64 20 76 65 72 73 69 6F 6E 73 20 77 69 6C 6C 20 63 6F 6D 65 20 77 69 74 68 20 62 65 74 74 65 72 20 65 78 61 6D 70 6C 65 73 2E 0A 0A |
| ??? | 5E AC 80 ED A3 00 00 00 A3 00 00 00 | |
| Central directory file header signature | 50 4B 01 02 | 50 4B 01 02 |
| Version made by | 3F 00 | |
| Version needed to extract (minimum) | 14 00 | 14 00 |
| General purpose bit flag | 00 00 | 00 00 |
| Compression method | 00 00 | 00 00 |
| File last modification time | BA 74 | BA 74 |
| File last modification date | 4D 46 | 4D 46 |
| CRC-32 | 00 00 00 00 | 00 00 00 00 |
| Compressed size | 00 00 00 00 | 00 00 00 00 |
| Uncompressed size | 00 00 00 00 | 00 00 00 00 |
| File name length (n) | 05 00 | 05 00 |
| Extra field length (m) | 00 00 | 24 00 |
| File comment length (k) | 00 00 | 00 00 |
| Disk number where file starts | 00 00 | 00 00 |
| Internal file attributes | 00 00 | 00 00 |
First of all, thank you for your effort.
- For compress == COMPR::STORE, level has no meaning. And we only had COMPR::STORE up until now, so I thought the idea was to use COMPR::DEFLATE (with level = COMPR::NONE if you don't have pecl_http available) to see if MacOS would like that.
- I'm not sure I understand the part about files not in subdirectories. If I remember correctly, owncloud will always put all files in the zip into one top-level directory? Or did you test manually, without owncloud? The problem about the ADD flag is, that I have to provide the CRC and compressed size in the file header, so I would have to make two passes over the files, or hold them in memory. I also wonder that unsetting the ADD flag works at all. If I do this, unzip -t complains about "invalid compressed data to inflate". Maybe I misunderstood what you did?
- I had a look at that pablotron implementation (not sure I found this one when looking for a solution to the problem about two years ago), and for a moment, I wondered about my sanity when seeing the use of gzdeflate(). I'm still puzzled how they could not have noticed the problem with gzdeflate(), as they even have a special handling of "large" files. I suspect you only tried small files with their implementation. The problem arises, when the file is split in chunks to process, so the file must be larger than the chunk size (~1MB for ZipStreamer, configurable but default 20MB for pablotrons ZipStream) to trigger the problem. Every output of gzdeflate() contains a "final" block defined in the deflate specification, which makes concatenating the output impossible. unzip -t complains about bad CRC for files generated in that way (I checked with pablotrons ZipStream to confirm still it really does ;-)).
Aside from that, they in fact do two passes over all files (and hold "small" files in memory), so they can add the CRC/compressed sizes to the header.
So, what I'm not sure you did, but I'd like to know: do zip files work with one top-level directory everything is in, and compress == COMPR::DEFLATE and variying settings of level?
First of all, thank you for your effort.
Nah, nothing to thank. You are the one doing all the hard work here. I'm just doing some monkey testing 😃
- For compress == COMPR::STORE, level has no meaning. And we only had COMPR::STORE up until now, so I thought the idea was to use COMPR::DEFLATE (with level = COMPR::NONE if you don't have pecl_http available) to see if MacOS would like that.
Yep. Unfortunately it doesn't always like that it seems. Apple is doing great things…
- I'm not sure I understand the part about files not in subdirectories. If I remember correctly, owncloud will always put all files in the zip into one top-level directory? Or did you test manually, without owncloud? The problem about the ADD flag is, that I have to provide the CRC and compressed size in the file header, so I would have to make two passes over the files, or hold them in memory. I also wonder that unsetting the ADD flag works at all. If I do this, unzip -t complains about "invalid compressed data to inflate". Maybe I misunderstood what you did?
There is a way to get files not in a top-level directory. To do that create two text files in the top directory and select them both with the checkbox. In the upper right of the table there will be a "Download" button then. In that case without the ADD flag it worked fine. However, that seems more like Finder accepting some probably not correct values but failing with correct ones 🙈
- I had a look at that pablotron implementation (not sure I found this one when looking for a solution to the problem about two years ago), and for a moment, I wondered about my sanity when seeing the use of gzdeflate(). I'm still puzzled how they could not have noticed the problem with gzdeflate(), as they even have a special handling of "large" files. I suspect you only tried small files with their implementation. The problem arises, when the file is split in chunks to process, so the file must be larger than the chunk size (~1MB for ZipStreamer, configurable but default 20MB for pablotrons ZipStream) to trigger the problem. Every output of gzdeflate() contains a "final" block defined in the deflate specification, which makes concatenating the output impossible. unzip -t complains about bad CRC for files generated in that way (I checked with pablotrons ZipStream to confirm still it really does ;-)).
Aside from that, they in fact do two passes over all files (and hold "small" files in memory), so they can add the CRC/compressed sizes to the header.
:insert_sighing_meme_about_all_this_madness_caused_by_apple_doing_some_things_terribly_wrong_since_ages:
So, what I'm not sure you did, but I'd like to know: do zip files work with one top-level directory everything is in, and compress == COMPR::DEFLATE and variying settings of level?
Let me post some test combinations here using COMPR::DEFLATE and ZIP64 support disabled:
- zipping folder
/test/which contains no files belowCOMPR::NONE✅COMPR::NORMAL✅COMPR::MAXIMUM✅COMPR::SUPERFAST✅
- zipping single files
/welcome.txtand/welcome1.txtCOMPR::NONE❌COMPR::NORMAL❌COMPR::MAXIMUM❌COMPR::SUPERFAST❌
- zipping folder
/test/which contains files belowCOMPR::NONE❌COMPR::NORMAL❌COMPR::MAXIMUM❌COMPR::SUPERFAST❌
The confusing thing is the error thrown in the console Error -60005 creating authorization which on OS X usually indicates that a process has not the correct permissions to create something which is why I first suspected that this has to do with some path values, however when looking at the working and the not working ZIP (as compared above) this makes no sense at all… 😞
@McNetic Btw. I just got access from @mmayer to a OS X VM (accessible via VNC etc…) which we can use for unit testing… I guess he won't have anything against if I grant you access to it so that you could do some live testing and ZIP support might work again?
If you're interested you know my mail address (or can look it up on my profile) ;-)
One of our enterprise customer having this issue, can you advise me do you have fix/Patch for this issue ? If “yes”...Can we have our internal QA test fix; we can provide Mac access if he/she needed.
Here is detail from case
“We are having a new issue since migrating to OC EE 7.0.5 - Specifically on Mac OS clients, people are reporting when multiple files or directories are downloaded from ownCloud, it results in a zip file that can not be opened on Mac OS system.
I'm wondering if there is a workaround we can implement server side until we can get to OC EE 8.2, and/or what your official recommendation is?
We have a lot of users which have Macs, and multi-file downloads are effectively broken for them right now.”
Thank you for help
Ilyias, we do not have a patch at the moment that works on OS X reliably. For the moment users can use another unzipper such as Keka or StuffIt.
(as also stated in the discussion above)
@iliyasbasir There is nothing more I can add.
Ok, I found some time to have a look at the OSX problem with the VM provided by @mmayer, myself. I also found other sources complaining about the OSX incompatibility with some zip features. So my conclusion is that the OSX finder cannot handle zip general purpose flag bit 3 set (aka streaming zip files, part of official zip standard since 1993). I can not handle this problem in ZipStreamer itself, as it does also work on networked streams, which don't support fseek(). Only potential upside: maybe Finder does in fact support zip64 - I haven't tried yet.
Now, to solve the problem for owncloud, I could add an option to provide file size and crc32 to ZipStreamer from owncloud directly. There are still some questions how this whole thing should be designed:
- should owncloud try to detect osx and fall back to the proposed solution while retaining the current behaviour otherwise?
- could owncloud calculate and cache the crc32 (and size) for each file on upload (when missing, it wil still be needd to calculate on first time it is downloaded via zip file)?
- for crc32 calculation on download: is (and will be) the stream provided by Filesystem class always seek()able?
I just noticed: Technically, this issue was about a small bug that was fixed in Zipstreamer months ago. The OSX issues are not even mentioned in the issue report. There is also at least one separate issue about the OSX problems (#13669). I'm not quite sure what to do about that.
One could implement the current Zipstreamer version (I'd do a release if necessary, it should not be a step backward related to the OSX issues), and close this issue, and follow up in #13669.
Please apologize, @McNetic, that I get back to you that late. This simply slipped through on my huge backlog :-/
Now, to solve the problem for owncloud, I could add an option to provide file size and crc32 to ZipStreamer from owncloud directly. There are still some questions how this whole thing should be designed:
should owncloud try to detect osx and fall back to the proposed solution while retaining the current behaviour otherwise?
Personally, I'd like to avoid a fallback as much as possible. This leads to confusing behaviour as soon as people start mailing / sharing the generated ZIP files. Unlikely, but can happen. – Yet, I assume that this would require quite more resources and we should do this though. Requires some measuring on my end.
could owncloud calculate and cache the crc32 (and size) for each file on upload (when missing, it wil still be needd to calculate on first time it is downloaded via zip file)?
I think we should avoid this. Files can also be modified with other means than through ownCloud (i.e. a mounted Windows Network Drive) and going this route is probably opening a can of worms as we need to ensure that all backends behave properly. 💥
(and there are actually some cases of custom backends :-()
for crc32 calculation on download: is (and will be) the stream provided by Filesystem class always seek()able?
I'm not completely sure about that. @icewind1991 @DeepDiver1975 can one of you folks please comment on this? THX. – Also input on the other questions is appreciated.
I just noticed: Technically, this issue was about a small bug that was fixed in Zipstreamer months ago. The OSX issues are not even mentioned in the issue report. There is also at least one separate issue about the OSX problems (#13669). I'm not quite sure what to do about that.
One could implement the current Zipstreamer version (I'd do a release if necessary, it should not be a step backward related to the OSX issues), and close this issue, and follow up in #13669.
We can do that if that would help. – Tagging a new release would also be appreciated. (we don't ship master when we can avoid it)
@maddox-DX @LukasReschke Any news?
According to the issue tracker: 
I have asked questions in #10001 (comment) which are unanswered and unless those are answered we can't proceed.
@cdamken this issue is scheduled for 8.2 - there can be no progress ...
@DeepDiver1975 Ok, thanks
@MorrisJobke
00002808
00003426
00003439
After all this, can we fix this for 8.2? Do we know what we want to do?
@cmonteroluque while this is tagged, it is in?
After all this, can we fix this for 8.2? Do we know what we want to do?
we rely on an upstream project which is fighting this issue for a long time. The problem at the end is:
as soon as we fix it for one platform - it will fall a part on another.
Long story short: zip seems to be a nightmare with respect to it's specification.
If I remember correctly this issue was meanwhile reported to Apple as well - so maybe they fix it? 🙊
Well @McNetic has asked for questions which still are unanswered at #10001 (comment) 🙊
If I remember correctly this issue was meanwhile reported to Apple as well - so maybe they fix it? 🙊
Yeah. Maybe in unicorn land 🙊
While it does not solve all of the problems can we at least address the one use case (multiple file downloads) with a small change in behavior i.e. Serial downloads on multi file select rather than zip them up? There are still a number of other impacted use cases (like downloading a folder) but perhaps we could provide some relief on one front?
@bboule serial download is not possible because browsers only support downloading one file at a time. This is the reason why zip is used. Or are you aware of any new tech that could allow that ?
@bboule To my knowledge only FF supports multipart HTTP bodies allowing multiple downloads in one request. Perhaps there's a way for javascript to be used so the browser sends multiple separate requests. Could be security implications though with accidental (or deliberate) DOS.
So perhaps, offering both methods (make zip file and send it or stream) and allowing admins to choose via config.php which method is used under which circumstances (for selections of files, folders, selections over a certain size, etc) would be best. When a user base is Mac heavy, use the old way and just deal with needing the extra space to store the zips while they are downloaded. When there are few to no Mac users, take advantage of streaming.
If hell ever freezes over and Apple decides to invest 10 minutes of engineering effort into something that doesn't have flashy animations then we can leave the old method forever. But an end user doesn't care that it's Apple's fault. To them, it's broken and they want it fixed. Installing an alternate unarchive tool is just a band aid solution to them. Giving admins more power to handle the unique nature of each installation is better than simply waiting on Apple.
we might want to have a look at : https://github.com/barracudanetworks/ArchiveStream-php
@DeepDiver1975 "Only the Zip64 (version 4.5 of the Zip specification) format is supported.". Wasn't that the actual problem with Apple ?
Wasn't that the actual problem with Apple ?
yes - but there will be tar for mac then
@PVince81 Yes indeed but I was hoping we could control it via javascript magic (sounds like my hopes are dashed ;) )
@bboule I read somewhere that you can trigger downloads by appending hidden iframes. Problem is that some browsers will pop up the "save as" dialog. Imagine this if the user selected 100 files...
I see your point not ideal :)
@MTRichards @bboule
I would like to request a backport to OC7 if possible.
I would like to request a backport to OC7 if possible.
We don't even have a patch for master. Even an idea how to fix this properly is very hard :(
Why not generate different kind of zip files based on the client browser OS?
The usual use case is probably downloading a directory and this would enable that.
Why not generate different kind of zip files based on the client browser OS?
The usual use case is probably downloading a directory and this would enable that.
Can we do that? This would be ideal, and allow for future unknown OS support problems with certain file types.
Can we do that? This would be ideal, and allow for future unknown OS support problems with certain file types.
afaik not because MacOS does not support streamed zip archives - but we require zip streaming to have a low memory foodprint
I tried using the deflate branch and toyed with constructor options
$zip = new ZipStreamer([
'zip64' => false,
'compress'=>\ZipStreamer\COMPR::DEFLATE,
'level'=>\ZipStreamer\COMPR::NORMAL
]);
$zip = new ZipStreamer([
'zip64' => false,
'compress'=>\ZipStreamer\COMPR::NONE
]);
$zip = new ZipStreamer([
'zip64' => false,
'compress'=>\ZipStreamer\COMPR::DEFLATE,
'level'=>\ZipStreamer\COMPR::NONE
]);In all cases downloadad zips fail to open with the Archive Utility but allow me to use unzip (5.52 of 28 February 2005):
$ unzip download-11.zip
Archive: download-11.zip
inflating: Neue Textdatei.txt
inflating: üöäß⊂∫∀∃ℕℝ∂.txt
inflating: welcome.txt
$ zipinfo download-11.zip
Archive: download-11.zip 1350 bytes 3 files
-rw-r--r-- 4.5 unx 1487 bl defN 4-Aug-15 11:30 Neue Textdatei.txt
-rw-r--r-- 4.5 unx 0 bl defN 4-Aug-15 11:30 üöäß⊂∫∀∃ℕℝ∂.txt
-rw-r--r-- 4.5 unx 163 bl defN 4-Aug-15 11:30 welcome.txt
3 files, 1650 bytes uncompressed, 940 bytes compressed: 43.0%
$ zipinfo download-12.zip
Archive: download-12.zip 2231 bytes 3 files
-rw-r--r-- 4.5 unx 1658 bl stor 4-Aug-15 11:38 Neue Textdatei.txt
-rw-r--r-- 4.5 unx 0 bl stor 4-Aug-15 11:38 üöäß⊂∫∀∃ℕℝ∂.txt
-rw-r--r-- 4.5 unx 163 bl stor 4-Aug-15 11:38 welcome.txt
3 files, 1821 bytes uncompressed, 1821 bytes compressed: 0.0%
$ zipinfo download-13.zip
Archive: download-13.zip 2256 bytes 3 files
-rw-r--r-- 4.5 unx 1658 bl defN 4-Aug-15 11:41 Neue Textdatei.txt
-rw-r--r-- 4.5 unx 0 bl defN 4-Aug-15 11:41 üöäß⊂∫∀∃ℕℝ∂.txt
-rw-r--r-- 4.5 unx 163 bl defN 4-Aug-15 11:41 welcome.txt
3 files, 1821 bytes uncompressed, 1846 bytes compressed: -1.4%
I was using the "Neue Textdatei.txt" for copy paste exchange ... thats why the size changed.
Setting 'zip64' => false makes the zip file use 0x0a as the min version required, and I fail to understand what the remaining problem with the Archiver Utility is ... so ... I don't know ... progress?
The native OSX Tools (unzip + archive utility) both don't support the zip64 extension, which is part of zip standard since at least 2006. As a bonus, the archive tool also does not support streamed zip files, which is part of zip standard since 1993. Zipstreamer does always use streamed zip - as the name suggests). I think the best way is to detect OSX clients, and disable large zip downloads (with a manual override stating that third party tools are required to unpack).
@DeepDiver1975 I guess detecting OSX and then disabling 64bit support in zipstreamer is the first step. If we can detect if the selected files require using 64bit we can threw en exception. That should solve this for the majority of cases.
I guess detecting OSX and then disabling 64bit support in zipstreamer is the first step. If we can detect if the selected files require using 64bit we can threw en exception. That should solve this for the majority of cases.
It's not that easy. It's also about streamed ZIPs which are unsupported as well. We do need a completely different solution here thus.
OK, can we agree on OSX being broken beyond sanity? Good. Now, the only thing we can do for end users is providing a non zip64 download when the file size allows. That would allow them to use unzip. That's it. No more (archive utility still does not work), no less (makes downloading collections minimally usable).
That being said, IIRC someone mentioned using tar(.gz) for OSX? That approach would still require detecting osx ...
.tar stuff: #17930
That said: Yes, sounds feasible. - Maybe also a notification on the download page or so about this.
Excellent! Do we have a plan? Sounds like it...
Not sure what the plan is. Switching to tar makes things not easier. There are a ton of different incompatible tar dialects as I have discovered with our software download files. We use zip files now because it has better compatibility across systems.
Not sure what the plan is. Switching to tar makes things not easier. There are a ton of different incompatible tar dialects as I have discovered with our software download files.
We would need to test the created tar on Mac and see if it works on this platform.
Tar is no understood by Windows - but we dont care since tar will be a OSX only solution