Install failure on CI

whitebox version:
Python version: 3.11.6
Operating System: Linux

Description

I am trying to test a function that uses whitebox with pytest. Tests pass locally but when I run the tests on a CI pipeline (bitbucket) they fail. It appears to be an error with the download and setup. I have tried with different combinations of musl and reset but all fail. CI pipeline is run on linux.

whitebox.download_wbt(
    linux_musl=False,
    reset=False,
    verbose=True,
)

What I Did

See below for snip from CI output.

Downloading WhiteboxTools pre-compiled binary for first time use ...
Downloading WhiteboxTools binary from https://www.whiteboxgeo.com/WBT_Linux/WhiteboxTools_linux_amd64.zip
Decompressing WhiteboxTools_linux_amd64.zip ...
Unexpected error: <class 'EOFError'>

.venv/lib/python3.11/site-packages/_pytest/runner.py:341: in from_call
    result: Optional[TResult] = func()
.venv/lib/python3.11/site-packages/_pytest/runner.py:372: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
.venv/lib/python3.11/site-packages/_pytest/python.py:531: in collect
    self._inject_setup_module_fixture()
.venv/lib/python3.11/site-packages/_pytest/python.py:545: in _inject_setup_module_fixture
    self.obj, ("setUpModule", "setup_module")
.venv/lib/python3.11/site-packages/_pytest/python.py:310: in obj
    self._obj = obj = self._getobj()
.venv/lib/python3.11/site-packages/_pytest/python.py:528: in _getobj
    return self._importtestmodule()
.venv/lib/python3.11/site-packages/_pytest/python.py:617: in _importtestmodule
    mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
.venv/lib/python3.11/site-packages/_pytest/pathlib.py:567: in import_path
    importlib.import_module(module_name)
/usr/local/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1204: in _gcd_import
    ???
<frozen importlib._bootstrap>:1176: in _find_and_load
    ???
<frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:690: in _load_unlocked
    ???

.venv/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:186: in exec_module
    exec(co, module.__dict__)

    whitebox.download_wbt(
.venv/lib/python3.11/site-packages/whitebox/whitebox_tools.py:138: in download_wbt
    with zipfile.ZipFile(zip_name, "r") as zip_ref:
/usr/local/lib/python3.11/zipfile.py:1302: in __init__
    self._RealGetContents()
/usr/local/lib/python3.11/zipfile.py:1369: in _RealGetContents
    raise BadZipFile("File is not a zip file")
E   zipfile.BadZipFile: File is not a zip file

Yes, this is a known issue. Downloaing WBT binaries automatically from whiteboxgeo.com sometimes gets a timeout error. That's whey I placed some backup links in download_wbt just to pass the CI tests.

whitebox-python/whitebox/download_wbt.py

Line 58 in 117f08e

backup_links = {

It's not a timeout error though... it happens here when unzipping the download.

I see that line 108 checks if zip_ext == ".zip": # Decompress Windows/Mac OS zip - but all of the links and backup links are .zip?

Also line 87 sets the zip_name from the url, then the url is potentially updated in line 99. I'm not sure what impact this has (if any) but why not set the zip_name after line 101?

It works fine on Colab. The issue is probably caused by the corrupted file downloaded from whiteboxgeo.com when the download is not complete. It is not a valid zip file, so it throws errors when trying to unzip it.

Ok, so it sounds like you are aware of the issue and know what's causing it. It's not clear what your proposed solution is though. Since it's a timeout issue, does the timeout parameter in the urllib request need to be increased? or do I need to change something with my CI pipeline?

The timeout vlaue is already set to 500 seconds, which should be long enough. The issue is probably because whiteboxgeo.com rejects automated download request from time to time. I don't know why.

Here is a previous discussion on this issue: giswqs/whitebox-bin#1

I don't have a solution to this. Suggestions are welcome.

Finally got to the bottom of this, and error is not what I originally thought.

I am running pytest on the CI pipeline, and it was configured with '-n logical' to run tests in parallel. This appears to cause failure because multiple tests are trying to download and setup the whitebox binary at once. Changing to '-n 1' so the tests run in sequence has fixed this issue for me.

Glad to hear that you sorted things out.