Error Running example data in templates/2_Pixie_Cluster_Pixels.ipynb
mikemcka opened this issue · 15 comments
Please refer to our FAQ and look at our known issues before opening a bug report.
Describe the bug
A clear and concise description of what the bug is.
Hi, I have been trying to run template notebook 2_Pixie_Cluster_Pixels.ipynb using the example dataset loaded during the notebook, I get the error running Cell 3.2: Assign pixel SOM clusters:
ValueError Traceback (most recent call last)
Cell In[15], line 2
1 # use pixel SOM weights to assign pixel clusters
----> 2 pixel_som_clustering.cluster_pixels(
3 fovs,
4 channels,
5 base_dir,
6 pixel_pysom,
7 data_dir=pixel_data_dir,
8 multiprocess=multiprocess,
9 batch_size=batch_size
10 )
12 # generate the SOM cluster summary files
13 pixel_som_clustering.generate_som_avg_files(
14 fovs,
15 channels,
(...)
19 pc_chan_avg_som_cluster_name=pc_chan_avg_som_cluster_name
20 )
File /vast/imaging/MikeyM/ark_env/lib/python3.11/site-packages/ark/phenotyping/pixel_som_clustering.py:208, in cluster_pixels(fovs, channels, base_dir, pixel_pysom, data_dir, multiprocess, batch_size, num_parallel_pixels, overwrite)
203 cols_to_drop.append(col)
205 sample_fov = sample_fov.drop(
206 columns=cols_to_drop
207 )
--> 208 misc_utils.verify_same_elements(
209 enforce_order=True,
210 norm_vals_columns=pixel_pysom.norm_data.columns.values,
211 pixel_data_columns=sample_fov.columns.values
212 )
214 # ensure the SOM weights columns are valid indexes
215 misc_utils.verify_same_elements(
216 enforce_order=True,
217 pixel_som_weights_columns=pixel_pysom.weights.columns.values,
218 pixel_data_columns=sample_fov.columns.values
219 )
File /vast/imaging/MikeyM/ark_env/lib/python3.11/site-packages/alpineer/misc_utils.py:236, in verify_same_elements(enforce_order, warn, **kwargs)
225 warnings.warn(
226 err_str
227 % (
(...)
233 )
234 )
235 else:
--> 236 raise ValueError(
237 err_str
238 % (
239 list_one_name,
240 list_two_name,
241 list_one_cast[first_bad_index],
242 list_two_cast[first_bad_index],
243 first_bad_index,
244 )
245 )
246 return True
ValueError: Lists norm vals columns and pixel data columns ordered differently: values CD14 and CD3 do not match at index 0
I have not modified the code in any way prior to running the notebook,
I have attempted to get around the error so I can give the interactive GUI a try by downloading the example_pixel_output_dir.zip
from the data repository (https://huggingface.co/datasets/angelolab/ark_example/tree/main/data/pixie) on hugging face and uploading it to my ark-analysis data file in place of the one you generate during the notebook. the following cells to 3.2 do run however the GUI is a blank object in section 4 onwards.
I also had some trouble using the install commands on github: git clone -b v0.7.0 https://github.com/angelolab/ark-analysis.git
cd ark-analysis
conda env create -f environment.yml
Running these commands throws an error and causes the build to fail when installing scikit image==0.19.3. I managed to install scikit-image seperately using conda and then using conda update with the .yaml file on the environment
I have attached a list of all the packages I have installed in my environment called requirements.txt.
Expected behavior
A clear and concise description of what you expected to happen.
To Reproduce
Please either copy/paste or screenshot all of the code you ran which produced the error. In addition, copy the full error message.
Hi @mikemcka! We have seen this issue before (#1120) and it should have been addressed in this pull request: #1114. In that same PR, we also updated the scikit-image version to be <0.19.3 because 0.19.3 was causing issues, which makes me think you may be using an older version of the codebase. In the requirements.txt file that you shared, it says "ark-analysis @ file:///vast/imaging/MikeyM/ark-analysis", so it's hard to confirm which version of ark you are using.
Can you try creating a completely new conda environment with python version 3.11 (conda create -n new_ark_env python=3.11). Then install ark-analysis using pip install ark-analysis (this should install the latest version, 0.7.1) and try running the notebook again. Let me know if this works!
Looks like there's an issue installing scikit-image (as you noted before). In that same environment, can you try pip install scikit-image==0.19.2 (the version here is important, .19.3 was causing issues), and then try pip install ark-analysis again?
@mikemcka Hmm this seems to be an issue with installing scikit-image (not ark-analysis, though ark-analysis depends on scikit-image). From a quick google search (Sanster/IOPaint#281, scikit-image/scikit-image#6765), it could be related to the python version - however, on my Windows machine and with python version 3.11, I was able to install scikit-image with no problems. I haven't seen this exact error before. I would suggest posting in the scikit-image issues page (https://github.com/scikit-image/scikit-image/issues) - perhaps they could help you there.
Oh actually, I see in your original issue that you were able to install scikit-image 0.19.3 using conda. Perhaps you could try that for scikit-image 0.19.2 as well, and then try installing ark-analysis.
Unfortunately, we've only tested ark-analysis with scikit-image<0.19.3, so we can't guarantee that a newer version would work. However, ark-analysis should work with python 3.10, so you could try making a new environment with python 3.10 and try installing scikit-image 0.19.2 there.
Are you using jupyter lab or jupyter notebook? We recommend using jupyter lab. You can just type jupyter lab in your console - functionally it's similar to jupyter notebook. Let me know if you can see the heatmap in jupyter lab.
Hi, I was perviously running it in jupyter notebook but I have now run the notebook in jupyter lab and the heatmap is still empty.
I have been opening notebooks using both jupyter notebook and jupyter lab through a GUI which creates slurm jobs so you can run the notebooks on compute nodes. Is the version of jupyter lab important?
Hm I personally haven't tested different versions of jupyter lab, so not sure if that is the issue. The project toml for ark does specify ipython>=8.4,<9 and jupyterlab>=3.4.3,<4, so I would try to make sure those versions are used. If you think the compute nodes/slurm might be the issue, maybe you can try running it locally first, just to make sure you can get it working there. Most laptops should have enough RAM to run the example dataset.
Good news, its finally working, I installed it locally and created an enviroment in anaconda navigator following the same process I mentioned above, opened it in jupyter lab and ran all the code, I noticed there was two printouts of loading widget and I intially thought it wasnt working so I tabbed out and came back about 10 minutes later and it was there, it must have taken a while to load. Thankyou so much for all the help. time to get to work!
Great! Good luck!