Mac users - 'tqdm' object has no attribute 'disable’
Opened this issue · 2 comments
If useful to Mac users, I met and solved this error when running segment_files_parallel()
function,
Error was related to tqdm: 'tqdm' object has no attribute 'disable’ , probably linked to parallel processing.
I found comments in this repo:
In the official tqdm repo, they suggest to bypass the issue with a wrapper:
tqdm/tqdm#487 (comment)
conda install tqdm
Conda said it was correctly installed, but (mysteriously) the issue was solved.
Retrieving notices: ...working... done
Collecting package metadata (current_repodata.json): done
Solving environment: done
# All requested packages already installed.
My 2 cents : I suspect to have previously installed tqdm via pip. It may be useful to ensure packages are installed via conda (not sure about this...)
---Update
Previous comment about conda install was NOT a solution.
Found again the same issue when running:
dataset.cluster_ids()
Now I have full trace-stack (below).
If I just re-run the function:
dataset.cluster_ids()
I get no error.
However, keeping in mind that:
-
warning:
If segmentation fails for a vocalisation it will be dropped from the database so that it doesn’t cause trouble downstream.
in https://nilomr.github.io/pykanto/_build/html/_autosummary/pykanto.dataset.html?highlight=fail -
I had this issue when segmenting data:
dataset.segment_into_units()
I wonder if there is a dataloss somewhere.
I don't know how to check for this, but for the sake of reproducibility and check of this error, this is the clustering I've got after running through all of the steps for Storm-petrel sample dataset (I kept parameters as their were in the tutorials):
https://nilomr.github.io/pykanto/_build/html/contents/interactive-app.html
(p.s. fabulous work!)
(_reduce_and_cluster_r pid=3295) Exception ignored in: <function tqdm.__del__ at 0x115653e50>
(_reduce_and_cluster_r pid=3295) Traceback (most recent call last):
(_reduce_and_cluster_r pid=3295) File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/site-packages/tqdm/std.py", line 1162, in __del__
(_reduce_and_cluster_r pid=3295) self.close()
(_reduce_and_cluster_r pid=3295) File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/site-packages/tqdm/std.py", line 1281, in close
(_reduce_and_cluster_r pid=3295) if self.disable:
(_reduce_and_cluster_r pid=3295) AttributeError: 'tqdm_asyncio' object has no attribute 'disable'
---------------------------------------------------------------------------
RayTaskError Traceback (most recent call last)
Cell In[44], line 1
----> 1 dataset.cluster_ids()
File ~/Sites/pykanto/pykanto/utils/compute.py:107, in timing.<locals>.wrap(*args, **kwargs)
104 @wraps(f)
105 def wrap(*args, **kwargs):
106 start = time()
--> 107 output = f(*args, **kwargs)
108 end = time()
109 from pykanto.dataset import KantoData
File ~/Sites/pykanto/pykanto/dataset.py:768, in KantoData.cluster_ids(self, min_sample)
751 @timing
752 def cluster_ids(self, min_sample: int = 10) -> None:
753 """
754 Dimensionality reduction using UMAP + unsupervised clustering using
755 HDBSCAN. This will fail if the sample size for an ID (grouping factor,
(...)
766
767 """
--> 768 df = reduce_and_cluster_parallel(
769 self, min_sample=min_sample, num_cpus=self.parameters.num_cpus
770 )
772 if self.parameters.song_level:
773 self.data = self.data.combine_first(df)
File ~/Sites/pykanto/pykanto/signal/cluster.py:290, in reduce_and_cluster_parallel(dataset, min_sample, num_cpus)
280 obj_ids = [
281 _reduce_and_cluster_r.remote(
282 dataset_ref, i, song_level=song_level, min_sample=min_sample
283 )
284 for i in chunks
285 ]
286 pbar = {
287 "desc": "Projecting and clustering vocalisations",
288 "total": n_chunks,
289 }
--> 290 dfls = [obj_id for obj_id in with_pbar(to_iterator(obj_ids), **pbar)]
292 # Minimally check output
293 try:
File ~/Sites/pykanto/pykanto/signal/cluster.py:290, in <listcomp>(.0)
280 obj_ids = [
281 _reduce_and_cluster_r.remote(
282 dataset_ref, i, song_level=song_level, min_sample=min_sample
283 )
284 for i in chunks
285 ]
286 pbar = {
287 "desc": "Projecting and clustering vocalisations",
288 "total": n_chunks,
289 }
--> 290 dfls = [obj_id for obj_id in with_pbar(to_iterator(obj_ids), **pbar)]
292 # Minimally check output
293 try:
File ~/Sites/miniconda3/lib/python3.9/site-packages/tqdm/notebook.py:259, in tqdm_notebook.__iter__(self)
257 try:
258 it = super(tqdm_notebook, self).__iter__()
--> 259 for obj in it:
260 # return super(tqdm...) will not catch exception
261 yield obj
262 # NB: except ... [ as ...] breaks IPython async KeyboardInterrupt
File ~/Sites/miniconda3/lib/python3.9/site-packages/tqdm/std.py:1195, in tqdm.__iter__(self)
1192 time = self._time
1194 try:
-> 1195 for obj in iterable:
1196 yield obj
1197 # Update and possibly print the progressbar.
1198 # Note: does not call self.update(1) for speed optimisation.
File ~/Sites/pykanto/pykanto/utils/compute.py:39, in to_iterator(obj_ids, breaks)
37 done, obj_ids = ray.wait(obj_ids)
38 if breaks:
---> 39 yield ray.get(done[0])
40 else:
41 try:
File ~/Sites/miniconda3/lib/python3.9/site-packages/ray/_private/client_mode_hook.py:105, in client_mode_hook.<locals>.wrapper(*args, **kwargs)
103 if func.__name__ != "init" or is_client_mode_enabled_by_default:
104 return getattr(ray, func.__name__)(*args, **kwargs)
--> 105 return func(*args, **kwargs)
File ~/Sites/miniconda3/lib/python3.9/site-packages/ray/_private/worker.py:2380, in get(object_refs, timeout)
2378 worker.core_worker.dump_object_store_memory_usage()
2379 if isinstance(value, RayTaskError):
-> 2380 raise value.as_instanceof_cause()
2381 else:
2382 raise value
RayTaskError: ray::_reduce_and_cluster_r() (pid=3295, ip=127.0.0.1)
File "/Users/[MYPATH]/Sites/pykanto/pykanto/signal/cluster.py", line 273, in _reduce_and_cluster_r
return [
File "/Users/[MYPATH]/Sites/pykanto/pykanto/signal/cluster.py", line 274, in <listcomp>
reduce_and_cluster(
File "/Users/[MYPATH]/Sites/pykanto/pykanto/signal/cluster.py", line 211, in reduce_and_cluster
embedding, _ = umap_reduce(
File "/Users/[MYPATH]/Sites/pykanto/pykanto/signal/cluster.py", line 94, in umap_reduce
embedding = reducer.fit_transform(data)
File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/site-packages/umap/umap_.py", line 2772, in fit_transform
self.fit(X, y)
File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/site-packages/umap/umap_.py", line 2684, in fit
self.embedding_, aux_data = self._fit_embed_data(
File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/site-packages/umap/umap_.py", line 2717, in _fit_embed_data
return simplicial_set_embedding(
File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/site-packages/umap/umap_.py", line 1156, in simplicial_set_embedding
embedding = optimize_layout_euclidean(
File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/site-packages/umap/layouts.py", line 338, in optimize_layout_euclidean
for n in tqdm(range(n_epochs), **tqdm_kwds):
File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/site-packages/tqdm/std.py", line 562, in __new__
with cls.get_lock(): # also constructs lock if non-existent
File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/site-packages/tqdm/std.py", line 661, in get_lock
cls._lock = TqdmDefaultWriteLock()
File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/site-packages/tqdm/std.py", line 99, in __init__
cls.create_mp_lock()
File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/site-packages/tqdm/std.py", line 123, in create_mp_lock
cls.mp_lock = RLock()
File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/multiprocessing/context.py", line 73, in RLock
return RLock(ctx=self.get_context())
File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/multiprocessing/synchronize.py", line 187, in __init__
SemLock.__init__(self, RECURSIVE_MUTEX, 1, 1, ctx=ctx)
File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/multiprocessing/synchronize.py", line 80, in __init__
register(self._semlock.name, "semaphore")
File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/multiprocessing/resource_tracker.py", line 147, in register
self._send('REGISTER', name, rtype)
File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/multiprocessing/resource_tracker.py", line 154, in _send
self.ensure_running()
File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/multiprocessing/resource_tracker.py", line 124, in ensure_running
signal.pthread_sigmask(signal.SIG_UNBLOCK, _IGNORED_SIGNALS)
File "/Users/[MYPATH]/Sites/miniconda3/lib/python3.9/signal.py", line 69, in pthread_sigmask
sigs_set = _signal.pthread_sigmask(how, mask)
ray.exceptions.TaskCancelledError: Task: TaskID(c7528efcb2fd36edffffffffffffffffffffffff01000000) was cancelled