how to extract a zip when it has no folder structure
hydra3333 opened this issue · 10 comments
hydra3333 commented
As a starting point for iterative testing, if I had something like this, however the .zip file did not have a folder structure, it seems that the files get extracted into the folder above 'ffmpeg_python36_libs'.
What would be the correct variable usage to extract them directly into that folder ?
'ffmpeg_vs_python36_libs' : {
'repo_type' : 'archive',
'url' : 'https://www.python.org/ftp/python/3.6.5/python-3.6.5-embed-amd64.zip',
'folder_name' : 'ffmpeg_python36_libs',
'needs_configure' : False,
'needs_make' : False,
'needs_make_install' : False,
'is_cmake' : False,
'run_post_patch' : [
'{mingw_binpath}/gendef python36.dll',
'DLLTOOL={mingw_binpath}/{cross_prefix_bare}dlltool --output-delaylib libpython3.6m.a --input-def python36.def',
'cp -fv libpython3.6m.a {target_prefix}/lib/libpython3.6m.a',
],
'_info' : { 'version' : '3.6.5', 'fancy_name' : 'fmpeg_python36_libs' },
},
hydra3333 commented
On second thought, it wouldn't hurt to know how to do something like the aforementioned extract from a .zip which does not have a folder structure. I feel I'm assuming something wrong.
DeadSix27 commented
Not sure what you mean by this question though... can you post an example?
hydra3333 commented
Sure,
https://www.python.org/ftp/python/3.6.5/python-3.6.5-embed-amd64.zip
contains no folder structure.
Unless I did something wrong (entirely possible) the build script extracts the content to x86_64 rather than a nominated folder.
DeadSix27 commented
@hydra3333 Can you test:
ae5b5af
DeadSix27 commented
@hydra3333 PS, the folder has to be defined with: 'folder_name' : 'python-3.6.5-embed-amd64',
hydra3333 commented
ok will try when I get a free moment later today :)
thank you.
hydra3333 commented
I can't figure out what I've done wrong.
'ff_vs_python365_libs' : {
'repo_type' : 'archive',
'url' : 'https://www.python.org/ftp/python/3.6.5/python-3.6.5-embed-amd64.zip', # this doesn't extract to the correct folder :(
'folder_name' : 'ff_vs_python365_libs',
'needs_configure' : False,
'needs_make' : False,
'needs_make_install' : False,
'is_cmake' : False,
'run_post_patch' : [
'{mingw_binpath}/gendef python36.dll',
'{mingw_binpath}/{cross_prefix_bare}dlltool --output-delaylib libpython3.6m.a --input-def python36.def',
'cp -fv libpython3.6m.a {target_prefix}/lib/libpython3.6m.a',
],
'_info' : { 'version' : '3.6.5', 'fancy_name' : 'fmpeg_python36_libs' },
},
[22:26:04][INFO] Building DEPENDENCY 'ff_vs_python365_libs'
[22:26:04][DEBUG] Reset CFLAGS to: -ggdb -O3
Changing dir from /home/u/Desktop/workdir to x86_64
[22:26:04][WARNING] Package has the old URL format, please update it.
[22:26:04][INFO] Downloading python-3.6.5-embed-amd64.zip (https://www.python.org/ftp/python/3.6.5/python-3.6.5-embed-amd64.zip)
[22:26:04][INFO] Requesting : https://www.python.org/ftp/python/3.6.5/python-3.6.5-embed-amd64.zip - 6.9MiB
Downloading: python-3.6.5-embed-amd64. N/A% [] 0.0 B/ 6.9 MiB | 0.0 s/B | ETDownloading: python-3.6.5-embed-amd64. 2% [] 178.0 KiB/ 6.9 MiB | 2.3 MiB/s Downloading: python-3.6.5-embed-amd64. 8% [] 623.0 KiB/ 6.9 MiB | 4.6 MiB/s Downloading: python-3.6.5-embed-amd64. 17% [] 1.2 MiB/ 6.9 MiB | 6.6 MiB/s Downloading: python-3.6.5-embed-amd64. 25% [] 1.7 MiB/ 6.9 MiB | 7.4 MiB/s Downloading: python-3.6.5-embed-amd64. 34% [] 2.3 MiB/ 6.9 MiB | 8.2 MiB/s Downloading: python-3.6.5-embed-amd64. 43% [] 3.0 MiB/ 6.9 MiB | 8.5 MiB/s Downloading: python-3.6.5-embed-amd64. 46% [] 3.2 MiB/ 6.9 MiB | 7.0 MiB/s Downloading: python-3.6.5-embed-amd64. 53% [] 3.6 MiB/ 6.9 MiB | 7.2 MiB/s Downloading: python-3.6.5-embed-amd64. 68% [] 4.7 MiB/ 6.9 MiB | 8.4 MiB/s Downloading: python-3.6.5-embed-amd64. 81% [] 5.6 MiB/ 6.9 MiB | 9.0 MiB/s Downloading: python-3.6.5-embed-amd64. 89% [] 6.2 MiB/ 6.9 MiB | 9.2 MiB/s Downloading: python-3.6.5-embed-amd64. 98% [] 6.8 MiB/ 6.9 MiB | 9.3 MiB/s Downloading: python-3.6.5-embed-amd64. 100% [] 6.9 MiB/ 6.9 MiB | 9.3 MiB/s | Time: 0:00:00
[22:26:05][INFO] Unpacking python-3.6.5-embed-amd64.zip
[22:26:05][DEBUG] Running 'unzip python-3.6.5-embed-amd64.zip"-d "ff_vs_python365_libs" "' in '/home/u/Desktop/workdir/x86_64'
unzip: cannot find or open python-3.6.5-embed-amd64.zip-d ff_vs_python365_libs , python-3.6.5-embed-amd64.zip-d ff_vs_python365_libs .zip or python-3.6.5-embed-amd64.zip-d ff_vs_python365_libs .ZIP.
[22:26:05][ERROR] Error [9] running process: 'unzip python-3.6.5-embed-amd64.zip"-d "ff_vs_python365_libs" "' in '/home/u/Desktop/workdir/x86_64'
[22:26:05][ERROR] You can try deleting the product/dependency folder: '/home/u/Desktop/workdir/x86_64' and re-run the script
self.logger.info("Unpacking {0}".format( fileName ))
tars = (".gz",".bz2",".xz",".bz",".tgz") # i really need a better system for this.. but in reality, those are probably the only formats we will ever encounter.
customFolderTarArg = ""
if fileName.endswith(tars):
if customFolder:
os.makedirs(folderName)
customFolderTarArg = ' -C "' + folderName + '" --strip-components 1'
self.run_process('tar -xf "{0}"{1}'.format( fileName, customFolderTarArg ))
else:
if customFolder:
os.makedirs(folderName)
customFolderTarArg = '-d "{0}" '.format(folderName)
self.run_process('unzip {1}"{0}"'.format( customFolderTarArg, fileName ))
self.touch(os.path.join(folderName,"unpacked.successfully"))
os.remove(fileName)
return folderName
else:
self.logger.debug("{0} already downloaded".format( folderName ))
return folderName
#
hydra3333 commented
Well, this worked with the aforementioned python zip file.
exist_ok=True seemed to be needed when re-running.
if fileName.endswith(tars):
if customFolder:
os.makedirs(folderName, exist_ok=True)
customFolderTarArg = ' -C "' + folderName + '" --strip-components 1'
self.run_process('tar -xf "{0}"{1}'.format( fileName, customFolderTarArg ))
else:
if customFolder:
os.makedirs(folderName, exist_ok=True)
customFolderTarArg = '-d "{0}" '.format(folderName)
self.run_process('unzip {0}"{1}"'.format( customFolderTarArg, fileName ))
hydra3333 commented
Hmm, this test "failed" at one level but not at another.
The test folder "sdl2_archive_test" was created OK, however since the zip already contained a top level folder it just extracted that into "sdl2_archive_test " and the source was pushed down a level.
Not the end of the world, as long as you know the extract process acts like that.
Thanks !
'sdl2_archive' : {
'repo_type' : 'archive',
'url' : 'https://www.libsdl.org/release/SDL2-2.0.7.zip',
'folder_name' : 'sdl2_archive_test',
# 'patches' : (
# ('https://raw.githubusercontent.com/DeadSix27/python_cross_compile_script/master/patches/sdl2/0001-SDL2-2.0.5.xinput.diff', '-p0'),
# ),
'custom_cflag' : '-DDECLSPEC=', # avoid SDL trac tickets 939 and 282, and not worried about optimizing yet...
"run_post_install": (
# 'sed -i.bak "s/-mwindows/-ldinput8 -ldxguid -ldxerr8 -luser32 -lgdi32 -lwinmm -limm32 -lole32 -loleaut32 -lshell32 -lversion -luuid/" "{pkg_config_path}/sdl2.pc"', # allow ffmpeg to output anything to console :|
# 'sed -i.bak "s/-mwindows/-ldinput8 -ldxguid -ldxerr8 -luser32 -lgdi32 -lwinmm -limm32 -lole32 -loleaut32 -lshell32 -lversion -luuid/" "{target_prefix}/bin/sdl2-config"', # update this one too for good measure, FFmpeg can use either, not sure which one it defaults to...
'cp -v "{target_prefix}/bin/sdl2-config" "{cross_prefix_full}sdl2-config"',
),
'configure_options': '--prefix={target_prefix} --host={target_host} --disable-shared --enable-static',
'_info' : { 'version' : '2.0.7', 'fancy_name' : 'SDL2' },
},
extracting: sdl2_archive_test/SDL2-2.0.7/Xcode-iOS/Template/SDL iOS Application/Icon.png
inflating: sdl2_archive_test/SDL2-2.0.7/Xcode-iOS/Template/SDL iOS Application/Info.plist
inflating: sdl2_archive_test/SDL2-2.0.7/Xcode-iOS/Template/SDL iOS Application/main.c
creating: sdl2_archive_test/SDL2-2.0.7/Xcode-iOS/Test/
inflating: sdl2_archive_test/SDL2-2.0.7/Xcode-iOS/Test/Info.plist
inflating: sdl2_archive_test/SDL2-2.0.7/Xcode-iOS/Test/README
creating: sdl2_archive_test/SDL2-2.0.7/Xcode-iOS/Test/TestiPhoneOS.xcodeproj/
inflating: sdl2_archive_test/SDL2-2.0.7/Xcode-iOS/Test/TestiPhoneOS.xcodeproj/project.pbxproj
Changing dir from /home/u/Desktop/workdir/x86_64 to sdl2_archive_test
[22:41:14][DEBUG] Setting CFLAGS to '-DDECLSPEC='
[22:41:14][INFO] Configuring 'sdl2_archive' with: --prefix=/home/u/Desktop/workdir/toolchain/x86_64-w64-mingw32/x86_64-w64-mingw32 --host=x86_64-w64-mingw32 --disable-shared --enable-static
[22:41:14][DEBUG] Running './configure --prefix=/home/u/Desktop/workdir/toolchain/x86_64-w64-mingw32/x86_64-w64-mingw32 --host=x86_64-w64-mingw32 --disable-shared --enable-static' in '/home/u/Desktop/workdir/x86_64/sdl2_archive_test'
/bin/sh: 1: ./configure: not found
[22:41:14][ERROR] Error [127] running process: './configure --prefix=/home/u/Desktop/workdir/toolchain/x86_64-w64-mingw32/x86_64-w64-mingw32 --host=x86_64-w64-mingw32 --disable-shared --enable-static' in '/home/u/Desktop/workdir/x86_64/sdl2_archive_test'
[22:41:14][ERROR] You can try deleting the product/dependency folder: '/home/u/Desktop/workdir/x86_64/sdl2_archive_test' and re-run the script
A similar .7z extraction could behave the same (see a prior revoked pull, using package p7zip-full to unzip .7z files eg for "vapoursynth portable" where the .7z files is similar in folder structure to the aforementioned python zip)
hydra3333 commented
I settled on this for my modded copy
tars = (".gz",".bz2",".xz",".bz",".tgz") # i really need a better system for this.. but in reality, those are probably the only formats we will ever encounter.
customFolderTarArg = ""
if fileName.endswith(tars):
if customFolder:
os.makedirs(folderName, exist_ok=True)
customFolderTarArg = ' -C "' + folderName + '" --strip-components 1'
self.logger.info('tar -xf "{0}"{1}'.format( fileName, customFolderTarArg ))
self.run_process('tar -xf "{0}"{1}'.format( fileName, customFolderTarArg ))
elif fileName.endswith(".7z"): # 2018.05.16 new dependency on package "p7zip-full", to unzip .7z files
if customFolder:
os.makedirs(folderName, exist_ok=True)
customFolderTarArg = '-o{0} '.format(folderName)
self.logger.info('7za x -bb3 -r -aoa -y {0} "{1}"'.format( customFolderTarArg, fileName ))
self.run_process('7za x -bb3 -r -aoa -y {0} "{1}"'.format( customFolderTarArg, fileName ))
else: # 2018.05.16 to get here it must be a .zip file
if customFolder:
os.makedirs(folderName, exist_ok=True)
customFolderTarArg = '-d "{0}" '.format(folderName)
self.logger.info('unzip {0} "{1}"'.format( customFolderTarArg, fileName ))
self.run_process('unzip {0} "{1}"'.format( customFolderTarArg, fileName ))
self.touch(os.path.join(folderName,"unpacked.successfully"))