Error on OS X
rodighiero opened this issue · 8 comments
rodighiero commented
Hello everyone,
I carefully followed the instructions, but I have the same error with both my dataset and the one provided. May you help me?
Thanks,
Dario
(3.7) dario@Leviathan Surprise-Machines-2 % pixplot --images "datasets/oslomini/images/*"
Using TensorFlow backend.
/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:30: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
method='lar', copy_X=True, eps=np.finfo(np.float).eps,
/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:167: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
method='lar', copy_X=True, eps=np.finfo(np.float).eps,
/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:284: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
eps=np.finfo(np.float).eps, copy_Gram=True, verbose=0,
/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:862: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
eps=np.finfo(np.float).eps, copy_X=True, fit_path=True,
/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:1101: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
eps=np.finfo(np.float).eps, copy_X=True, fit_path=True,
/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:1127: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
eps=np.finfo(np.float).eps, positive=False):
/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:1362: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
max_n_alphas=1000, n_jobs=None, eps=np.finfo(np.float).eps,
/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:1602: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
max_n_alphas=1000, n_jobs=None, eps=np.finfo(np.float).eps,
/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:1738: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
eps=np.finfo(np.float).eps, copy_X=True, positive=False):
/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/sklearn/decomposition/online_lda.py:29: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
EPS = np.finfo(np.float).eps
Traceback (most recent call last):
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/numba/core/errors.py", line 744, in new_error_context
yield
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/numba/core/lowering.py", line 230, in lower_block
self.lower_inst(inst)
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/numba/core/lowering.py", line 328, in lower_inst
self.storevar(val, inst.target.name)
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/numba/core/lowering.py", line 1278, in storevar
raise AssertionError(msg)
AssertionError: Storing i64 to ptr of i32 ('dim'). FE type int32
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/Caskroom/miniconda/base/envs/3.7/bin/pixplot", line 5, in <module>
from pixplot import parse
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/pixplot/__init__.py", line 2, in <module>
from pixplot.pixplot import process_images, parse
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/pixplot/pixplot.py", line 25, in <module>
from umap import UMAP
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/umap/__init__.py", line 1, in <module>
from .umap_ import UMAP
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/umap/umap_.py", line 53, in <module>
from umap.layouts import (
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/umap/layouts.py", line 36, in <module>
"dim": numba.types.int32,
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/numba/core/decorators.py", line 221, in wrapper
disp.compile(sig)
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/numba/core/dispatcher.py", line 909, in compile
cres = self._compiler.compile(args, return_type)
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/numba/core/dispatcher.py", line 79, in compile
status, retval = self._compile_cached(args, return_type)
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/numba/core/dispatcher.py", line 93, in _compile_cached
retval = self._compile_core(args, return_type)
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/numba/core/dispatcher.py", line 111, in _compile_core
pipeline_class=self.pipeline_class)
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/numba/core/compiler.py", line 606, in compile_extra
return pipeline.compile_extra(func)
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/numba/core/compiler.py", line 353, in compile_extra
return self._compile_bytecode()
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/numba/core/compiler.py", line 415, in _compile_bytecode
return self._compile_core()
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/numba/core/compiler.py", line 395, in _compile_core
raise e
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/numba/core/compiler.py", line 386, in _compile_core
pm.run(self.state)
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/numba/core/compiler_machinery.py", line 339, in run
raise patched_exception
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/numba/core/compiler_machinery.py", line 330, in run
self._runPass(idx, pass_inst, state)
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock
return func(*args, **kwargs)
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/numba/core/compiler_machinery.py", line 289, in _runPass
mutated |= check(pss.run_pass, internal_state)
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/numba/core/compiler_machinery.py", line 262, in check
mangled = func(compiler_state)
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/numba/core/typed_passes.py", line 463, in run_pass
NativeLowering().run_pass(state)
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/numba/core/typed_passes.py", line 384, in run_pass
lower.lower()
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/numba/core/lowering.py", line 136, in lower
self.lower_normal_function(self.fndesc)
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/numba/core/lowering.py", line 190, in lower_normal_function
entry_block_tail = self.lower_function_body()
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/numba/core/lowering.py", line 216, in lower_function_body
self.lower_block(block)
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/numba/core/lowering.py", line 230, in lower_block
self.lower_inst(inst)
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/contextlib.py", line 130, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/numba/core/errors.py", line 751, in new_error_context
raise newerr.with_traceback(tb)
numba.core.errors.LoweringError: Failed in nopython mode pipeline (step: nopython mode backend)
Storing i64 to ptr of i32 ('dim'). FE type int32
File "../../../../usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/umap/layouts.py", line 52:
def rdist(x, y):
<source elided>
result = 0.0
dim = x.shape[0]
^
During: lowering "dim = static_getitem(value=$8load_attr.2, index=0, index_var=$const10.3, fn=<built-in function getitem>)" at /usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/umap/layouts.py (52)
duhaime commented
Ah sorry about that @rodighiero, and thanks for this helpful error message!
It looks like the most recent version of the numba package will raise this error. Could you please reactivate your conda environment (conda activate 3.7) and then downgrade numba with pip install numba==0.51.2, then try to make a plot again?
If you try those steps I'd be grateful if you could let us know how it goes!
rodighiero commented
Thank you for your quick answer, @duhaime. Now it works properly with the dataset on the readme file.
However, now I am testing a pretty large dataset from Harvard Art Museums composed of around 200,000 images, and I got this output after a while:
Traceback (most recent call last):
File "/usr/local/Caskroom/miniconda/base/envs/3.7/bin/pixplot", line 8, in <module>
sys.exit(parse())
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/pixplot/pixplot.py", line 1354, in parse
process_images(**config)
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/pixplot/pixplot.py", line 124, in process_images
kwargs['image_paths'], kwargs['metadata'] = filter_images(**kwargs)
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/pixplot/pixplot.py", line 187, in filter_images
resized = i.resize_to_max(kwargs['lod_cell_height'])
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/pixplot/pixplot.py", line 1292, in resize_to_max
return img_to_array(self.original.resize(size))
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/PIL/Image.py", line 1916, in resize
self.load()
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/PIL/ImageFile.py", line 260, in load
"image file is truncated "
OSError: image file is truncated (53 bytes not processed)
I tried to find the suspect ted truncated file using ImageMagick, using the identity function, but everything seems fine.
Do you have further suggestions?
Thanks a lot,
Dario
duhaime commented
Thanks for your follow up @rodighiero ! This is interesting. It'd be great if we could pinpoint the file that caused this because I'm a little curious as to exactly what's going on and I can't yet replicate the error on my side.
That said, I just pushed a commit that will attempt to fix this issue. You should be able to:
pip uninstall pixplot
pip install https://github.com/yaledhlab/pix-plot/archive/master.zip
and then reprocess your collection. If you try that out I'd be grateful if you could let us know how it goes!
rodighiero commented
Dear @duhaime, your commit fixed my problem and I was able to display 200,000. This software is really impressive, no doubt!
I also pushed myself further as I like the idea to import images and metadata using IIIF, but I encountered another problem. This is the last lines of the terminal:
* loading manifest from url https://iiif.harvardartmuseums.org/manifests/object/223192
* saving manifest to iiif-downloads/manifests/https:||||iiif.harvardartmuseums.org||manifests||object||223192
* loading image from url https://ids.lib.harvard.edu/ids/iiif/18744541/full/709,/0/default.jpg
* saving iiif-downloads/images/18744541.png
* loading manifest from url https://iiif.harvardartmuseums.org/manifests/object/223232
* saving manifest to iiif-downloads/manifests/https:||||iiif.harvardartmuseums.org||manifests||object||223232
* loading image from url https://ids.lib.harvard.edu/ids/iiif/18745051/full/1024,/0/default.jpg
* saving iiif-downloads/images/18745051.png
* loading manifest from url https://iiif.harvardartmuseums.org/manifests/object/223283
* saving manifest to iiif-downloads/manifests/https:||||iiif.harvardartmuseums.org||manifests||object||223283
* loading manifest from url https://iiif.harvardartmuseums.org/manifests/object/223284
* saving manifest to iiif-downloads/manifests/https:||||iiif.harvardartmuseums.org||manifests||object||223284
* loading image from url https://ids.lib.harvard.edu/ids/iiif/18744714/full/664,/0/default.jpg
* saving iiif-downloads/images/18744714.png
* loading manifest from url https://iiif.harvardartmuseums.org/manifests/object/223304
* saving manifest to iiif-downloads/manifests/https:||||iiif.harvardartmuseums.org||manifests||object||223304
Traceback (most recent call last):
File "/usr/local/Caskroom/miniconda/base/envs/3.7/bin/pixplot", line 8, in <module>
sys.exit(parse())
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/pixplot/pixplot.py", line 1354, in parse
process_images(**config)
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/pixplot/pixplot.py", line 124, in process_images
kwargs['image_paths'], kwargs['metadata'] = filter_images(**kwargs)
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/pixplot/pixplot.py", line 156, in filter_images
for i in stream_images(image_paths=get_image_paths(**kwargs)):
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/pixplot/pixplot.py", line 240, in get_image_paths
for i in f: Manifest(url=i).save_images(limit=1)
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/iiif_downloader/__init__.py", line 57, in save_images
meta = requests.get(meta_url).json()
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/site-packages/requests/models.py", line 900, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/json/__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/Caskroom/miniconda/base/envs/3.7/lib/python3.7/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
duhaime commented
@rodighiero thank you for your follow up again! We should start paying you royalties for your QA work!
I tried requesting the data at that last URL but got a 403 response. I'm not sure if one must be on a Harvard VPN to access that data, but I just pushed a commit that should simply carry on in the event that certain urls cannot be parsed for one reason or another. (Errors will be reported in the terminal, but the process shouldn't halt.)
If you try that out with pip uninstall pixplot && pip install https://github.com/yaledhlab/pix-plot/archive/master.zip I'd be grateful if you could follow up again with your findings!
If your 200k image plot is available online, I'd be grateful if you could send a link as well. We haven't built plots that large in a long time, and several features have been added to the scene that might slow those massive plots down a bit. In any event, if your plot is accessible, it'd be great to take a look at some of the performance data...
rodighiero commented
@duhaime thanks again, the script seems to work properly now. I think that record has been deleted, I suppose it can happen. I will send you another message when the computation will be terminated. Happy to keep stressing your library, it is exactly what I needed. I also sent you a private message on Gmail, with further information about the use for which is intended to be used.
Last but not least, I am not sure how to share with you the outcome, which is beautiful. Right now I am looking if someone at Harvard can help me to find a web server with 24GB of free space 😲 in the meanwhile, I can share with you a couple of screenshots.
You have to be round of it! 👏🏻
duhaime commented
Thanks very much for following up @rodighiero! This all sounds great!
Is it agreeable if we close out this issue for the nonce and pick up a new issue if (perhaps I should say when 😳) you hit another bump?
rodighiero commented
Sounds great, thank you for your support!