Running Dictys makefile static.mk

Question

Running Dictys makefile static.mk

Closed this issue a year ago · 21 comments

sureshsatpati commented a year ago

Checks before submitting the issue

I installed the latest Dictys version successfully
I downloaded the whole folder for the tutorial
I ran the tutorial cells and notebooks (if more than one) in the correct order and only once
The preceeding cells and notebooks finished successfully (i.e. the reported error is the first encountered)

Describe the error

How did you install Dictys (which option)?
conda environment
Which tutorial?
https://github.com/pinellolab/dictys/blob/master/doc/tutorials/full-multiome/notebooks/3-static-inference.ipynb
What and where was the error?
make: *** No rule to make target 'tmp_static/Subset1/net_weight.tsv.gz', needed by 'tmp_static/Subset1/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset1/net_meanvar.tsv.gz', needed by 'tmp_static/Subset1/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset1/net_covfactor.tsv.gz', needed by 'tmp_static/Subset1/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset10/net_weight.tsv.gz', needed by 'tmp_static/Subset10/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset10/net_meanvar.tsv.gz', needed by 'tmp_static/Subset10/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset10/net_covfactor.tsv.gz', needed by 'tmp_static/Subset10/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset11/net_weight.tsv.gz', needed by 'tmp_static/Subset11/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset11/net_meanvar.tsv.gz', needed by 'tmp_static/Subset11/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset11/net_covfactor.tsv.gz', needed by 'tmp_static/Subset11/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset12/net_weight.tsv.gz', needed by 'tmp_static/Subset12/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset12/net_meanvar.tsv.gz', needed by 'tmp_static/Subset12/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset12/net_covfactor.tsv.gz', needed by 'tmp_static/Subset12/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset2/net_weight.tsv.gz', needed by 'tmp_static/Subset2/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset2/net_meanvar.tsv.gz', needed by 'tmp_static/Subset2/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset2/net_covfactor.tsv.gz', needed by 'tmp_static/Subset2/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset3/net_weight.tsv.gz', needed by 'tmp_static/Subset3/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset3/net_meanvar.tsv.gz', needed by 'tmp_static/Subset3/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset3/net_covfactor.tsv.gz', needed by 'tmp_static/Subset3/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset4/net_weight.tsv.gz', needed by 'tmp_static/Subset4/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset4/net_meanvar.tsv.gz', needed by 'tmp_static/Subset4/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset4/net_covfactor.tsv.gz', needed by 'tmp_static/Subset4/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset5/net_weight.tsv.gz', needed by 'tmp_static/Subset5/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset5/net_meanvar.tsv.gz', needed by 'tmp_static/Subset5/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset5/net_covfactor.tsv.gz', needed by 'tmp_static/Subset5/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset6/net_weight.tsv.gz', needed by 'tmp_static/Subset6/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset6/net_meanvar.tsv.gz', needed by 'tmp_static/Subset6/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset6/net_covfactor.tsv.gz', needed by 'tmp_static/Subset6/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset7/net_weight.tsv.gz', needed by 'tmp_static/Subset7/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset7/net_meanvar.tsv.gz', needed by 'tmp_static/Subset7/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset7/net_covfactor.tsv.gz', needed by 'tmp_static/Subset7/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset8/net_weight.tsv.gz', needed by 'tmp_static/Subset8/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset8/net_meanvar.tsv.gz', needed by 'tmp_static/Subset8/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset8/net_covfactor.tsv.gz', needed by 'tmp_static/Subset8/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset9/net_weight.tsv.gz', needed by 'tmp_static/Subset9/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset9/net_meanvar.tsv.gz', needed by 'tmp_static/Subset9/net_nweight.tsv.gz'.
make: *** No rule to make target 'tmp_static/Subset9/net_covfactor.tsv.gz', needed by 'tmp_static/Subset9/net_nweight.tsv.gz'.
Traceback (most recent call last):
File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/main.py", line 13, in
docstringrunner(package)
File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/docstring2argparse/init.py", line 340, in docstringrunner
run_args(pkgname,funcs,args)
File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/docstring2argparse/init.py", line 330, in run_args
return func(*a,**ka)
File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/preproc.py", line 140, in qc_reads
raise RuntimeError('All genes removed in QC.')
RuntimeError: All genes removed in QC.
make: *** [makefiles/common.mk:124: tmp_static/Subset12/expression.tsv.gz] Error 1
make: Target 'cpu' not remade because of errors.
Did you deviate from any of the tutorial instructions?

Optional steps (may accelerate troubleshooting)

Attach the error notebook and all preceeding notebooks with run output

Answer 1 · 2023-07-28T03:06:14.000Z

Hi sureshsatpati,

Thank you for reporting this! I understand these messages look daunting. However they are actually fine and do not indicate any problem on their own.

The "make: *** No rule to make target..." error arises because the GPU step was not yet performed. The majority of them should be gone in CPU part 2 as long as no other failures happen for the particular cell subset, e.g. from QC below. I will suppress these messages in the next release because they do cause confusion.

The "RuntimeError: All genes removed in QC." error does occur for certain cell subsets if they are too small so no gene pass QC. It does not mean all cell subsets failed.

If the notebook did stop running because of any error, or did not produce the final network file upon finishing, please do post any error you may received. The above ones look benign to me.

Lingfei

Answer 2 · 2023-07-28T20:12:29.000Z

Hi Lingfei, thank you for the prompt response!

I have another question about the input files for the next step, "python3 -m dictys preproc selects_rna -h" from the tutorial.

Looking at the positional arguments, it looks like I needs 1 "reads" file and 1 "names" file as input. What files are those? Could you please possibly share demo files for those files? Or point me to where I should find them if they were produced in previous steps?

Could you also please provide the same information for the input files for the following steps?

Thank you,
Dr. Suresh Satpati

Answer 3 · 2023-07-29T01:27:50.000Z

Hi Suresh,

Sure. The GRN inference pipeline is based on make, so you can see the exact commands executed there. You can simply search for python3 -m dictys preproc selects_rna in the tutorial notebook above to check its parameters and look into the files. To understand how these files were produced, you can further search for these file names upwards in the same notebook. You can also inspect the relevant .mk Makefiles to understand each rule that created these files.

I hope that answers your question.

Lingfei

Answer 4 · 2023-07-31T18:28:36.000Z

Thanks Lingfei!
When I ran GPU part for network reconstruct I see below errors. Can you please suggest what could be the possible reason for this.

AssertionError: Torch not compiled with CUDA enabled
make: *** [makefiles/common.mk:169: tmp_static/Subset2/net_weight.tsv.gz] Error 1
WARNING:31373:2023-07-31 13:01:51,898:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/utils/parallel.py:63:Environmental variables have different values.
Traceback (most recent call last):
File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/main.py", line 13, in
docstringrunner(package)
File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/docstring2argparse/init.py", line 340, in docstringrunner
run_args(pkgname,funcs,args)
File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/docstring2argparse/init.py", line 330, in run_args
return func(*a,**ka)
File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/network.py", line 787, in reconstruct
model=model('pyromodel',np.array(np.nonzero(mask)),mask.shape,npc,dc,floattype=dtype,device=device,scale_lyapunov=scale_lyapunov)
File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/network.py", line 626, in init
super().init(name,npc,dcs,**ka)
File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/network.py", line 499, in init
self.dcs=self.tensor(dcs.T)
File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/network.py", line 211, in tensor
return torch.tensor(v,device=device,dtype=dtype,**ka)
File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/torch/cuda/init.py", line 239, in _lazy_init
raise AssertionError("Torch not compiled with CUDA enabled")

Answer 5 · 2023-07-31T19:33:49.000Z

Sure. Here: https://github.com/pinellolab/dictys#faq

Answer 6 · 2023-08-03T07:39:58.000Z

Hi Suresh,

We just updated our support for pytorch 2. You may try a fresh reinstall to see if the problem goes away.

Lingfei

Answer 7 · 2023-08-25T14:32:07.000Z

Hi Lingfei,
Thanks for the update and congratulations for the paper published in nature methods.

As you suggested I reinstalled Dictys and I started running from the aggregate network command once again where I gets the following error:

WARNING:74140:2023-07-31 12:03:51,345:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset1 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:74140:2023-07-31 12:03:51,346:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset1 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:74140:2023-07-31 12:03:51,346:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset10 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:74140:2023-07-31 12:03:51,347:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset10 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:74140:2023-07-31 12:03:51,347:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset11 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:74140:2023-07-31 12:03:51,347:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset11 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:74140:2023-07-31 12:03:51,348:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset12 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:74140:2023-07-31 12:03:51,348:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset12 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:74140:2023-07-31 12:03:51,349:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset2 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:74140:2023-07-31 12:03:51,349:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset2 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:74140:2023-07-31 12:03:51,349:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset3 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:74140:2023-07-31 12:03:51,350:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset3 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:74140:2023-07-31 12:03:51,350:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset4 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:74140:2023-07-31 12:03:51,351:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset4 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:74140:2023-07-31 12:03:51,351:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset5 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:74140:2023-07-31 12:03:51,351:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset5 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:74140:2023-07-31 12:03:51,352:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset6 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:74140:2023-07-31 12:03:51,352:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset6 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:74140:2023-07-31 12:03:51,353:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset7 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:74140:2023-07-31 12:03:51,355:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset7 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:74140:2023-07-31 12:03:51,355:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset8 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:74140:2023-07-31 12:03:51,356:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset8 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:74140:2023-07-31 12:03:51,356:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset9 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:74140:2023-07-31 12:03:51,356:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset9 due to error: FileNotFoundError(2, 'No such file or directory')
Traceback (most recent call last):
File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/main.py", line 13, in
docstringrunner(package)
File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/docstring2argparse/init.py", line 340, in docstringrunner
run_args(pkgname,funcs,args)
File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/docstring2argparse/init.py", line 330, in run_args
return func(*a,**ka)
File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/network.py", line 1071, in tofile
n=network.from_folders(diri_data,diri_work,fi_subsets,dynamic=dynamic,nettype=nettype,optional=optional,fi_c=fi_c)
File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py", line 398, in from_folders
raise RuntimeError('No network found.')
RuntimeError: No network found.
make: *** [makefiles/common.mk:175: output/static.h5] Error 1

Please suggest how to proceed?

Thanks,
Suresh

Answer 8 · 2023-08-26T04:10:22.000Z

Hi Suresh,

That appears a consequence of an earlier error. Can you confirm all four checkboxes on the top of the issue? If so, could you upload the full notebook?

Also, if you simply want to run Dictys successfully, I recommend the short-multiome tutorial as a start.

Thanks,
Lingfei

Answer 9 · 2023-08-29T16:09:59.000Z

Hi Lingfei,
Yes, I installed dictys using conda environment in my HPCC account and also another installation in the same HPCC with the help of an IT guy. I still see the same error in both of them. As you suggested I am doing the short-multiome tutorial and will come back to you If I see the same issue. How can I upload my full notebook? Did you mean uploading the steps I followed till now in the HPCC?

Thanks,
Suresh

Answer 10 · 2023-08-29T19:20:23.000Z

Hi Linfei,
When I run the GRN inference with scRNA-seq on GPU with below commands I get below error. I assume its with the GPU installation with CUDA.

module purge
module load miniconda3
eval "$(/risapps/rhel7/miniconda3/38_4.9.2/bin/conda shell.bash hook)"
. activate dictys-new

set -eo pipefail

#GRN inference with scRNA-seq on GPU
make -f makefiles/static.mk -j 2 -k gpu || true
make: *** [makefiles/common.mk:169: tmp_static/Subset7/net_weight.tsv.gz] Error 1
make: *** [makefiles/common.mk:169: tmp_static/Subset6/net_weight.tsv.gz] Error 1
Traceback (most recent call last):
File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/site-packages/dictys/main.py", line 13, in
docstringrunner(package)
File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/site-packages/docstring2argparse/init.py", line 340, in docstringrunner
run_args(pkgname,funcs,args)
File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/site-packages/docstring2argparse/init.py", line 330, in run_args
return func(*a,**ka)
File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/site-packages/dictys/network.py", line 790, in reconstruct
model=model('pyromodel',np.array(np.nonzero(mask)),mask.shape,npc,dc,floattype=dtype,device=device,scale_lyapunov=scale_lyapunov)
File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/site-packages/dictys/network.py", line 629, in init
super().init(name,npc,dcs,**ka)
File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/site-packages/dictys/network.py", line 499, in init
self.dcs=self.tensor(dcs)
File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/site-packages/dictys/network.py", line 211, in tensor
return torch.tensor(v,device=device,dtype=dtype,**ka)
File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/site-packages/torch/cuda/init.py", line 239, in _lazy_init
raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled
Traceback (most recent call last):
File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/site-packages/dictys/main.py", line 13, in
docstringrunner(package)
File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/site-packages/docstring2argparse/init.py", line 340, in docstringrunner
run_args(pkgname,funcs,args)
File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/site-packages/docstring2argparse/init.py", line 330, in run_args
return func(*a,**ka)
File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/site-packages/dictys/network.py", line 790, in reconstruct
model=model('pyromodel',np.array(np.nonzero(mask)),mask.shape,npc,dc,floattype=dtype,device=device,scale_lyapunov=scale_lyapunov)
File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/site-packages/dictys/network.py", line 629, in init
super().init(name,npc,dcs,**ka)
File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/site-packages/dictys/network.py", line 499, in init
self.dcs=self.tensor(dcs)
File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/site-packages/dictys/network.py", line 211, in tensor
return torch.tensor(v,device=device,dtype=dtype,**ka)
File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/site-packages/torch/cuda/init.py", line 239, in _lazy_init
raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled
make: *** [makefiles/common.mk:169: tmp_static/Subset9/net_weight.tsv.gz] Error 1
make: *** [makefiles/common.mk:169: tmp_static/Subset8/net_weight.tsv.gz] Error 1
make: Target 'gpu' not remade because of errors.

Please suggest!

Thanks
Suresh

Answer 11 · 2023-08-30T02:58:37.000Z

Hi Suresh,

This is the same issue you reported earlier. You should follow the FAQ at https://github.com/pinellolab/dictys#faq to address the issue.

Lingfei

Answer 12 · 2023-08-30T20:30:01.000Z

Hi Lingfei,
Because our GPU nodes on cluster don't have internet access. Therefore, when we install conda DICTYS environment on a CPU node with pytorch in a CPU version. Is DICTYS conda environment working for CPU node?

Thanks,
Suresh & Rong (IT Dept)

Answer 13 · 2023-08-30T21:04:54.000Z

Hi Suresh & Rong,

Yes. You can install and use the CPU version for pytorch. You need to set pytorch to CPU mode according to the tutorial. But it can be slow.

Alternatively, you can install the GPU version of pytorch on a CPU node and then use it on a GPU node. You need to complete the data downloading step on a CPU node in advance as well.

Dictys has a standard anaconda installation option. Most IT should support this.

Lingfei

Answer 14 · 2023-08-30T22:41:30.000Z

Hi Lingfei,

Thank you for your suggestion.
I am an HPC system administor, for a conda environment, if install pytorch on a cpu node, it is a cpu version of pytorch; if install pytorch on a gpu node, then it is a gpu version. On our cluster, pytorch for conda environment is cpu version.

Therefore, Dr. Satpati should be able to use cpu version of the software, as he uses command
make -f makefiles/static.mk -j 32 -k cpu || true
to do make file , the output is without gpu error as he reported earlier.

I have no problem to install dictys conda environment, However, not being a scientist, my running of the command in an active mode
$ make -f makefiles/static.mk -j 32 -k cpu || true
def is_extension_array_dtype(arr_or_dtype) -> bool:
KeyboardInterrupt
make: *** [makefiles/common.mk:124: tmp_static/Subset8/expression0.tsv.gz] Interrupt
make: *** [makefiles/common.mk:124: tmp_static/Subset2/expression0.tsv.gz] Interrupt
make: *** [makefiles/common.mk:124: tmp_static/Subset4/expression0.tsv.gz] Interrupt
make: *** [makefiles/common.mk:124: tmp_static/Subset10/expression0.tsv.gz] Interrupt
make: *** [makefiles/common.mk:124: tmp_static/Subset6/expression0.tsv.gz] Interrupt
make: *** [makefiles/common.mk:124: tmp_static/Subset3/expression0.tsv.gz] Interrupt
make: *** [makefiles/common.mk:124: tmp_static/Subset12/expression0.tsv.gz] Interrupt
make: *** [makefiles/common.mk:124: tmp_static/Subset11/expression0.tsv.gz] Interrupt
make: *** [makefiles/common.mk:124: tmp_static/Subset9/expression0.tsv.gz] Interrupt
make: *** [makefiles/common.mk:124: tmp_static/Subset5/expression0.tsv.gz] Interrupt
make: *** [makefiles/common.mk:124: tmp_static/Subset1/expression0.tsv.gz] Interrupt
make: *** [makefiles/common.mk:124: tmp_static/Subset7/expression0.tsv.gz] Error 1

It ends with the abovve message, I am not sure of the last sentense Error 1 does mean anything? However, I can see
the tmp_static directory filled with subdirectories Subset[1-12], each directories has two files names_rna.txt and names_atac0.txt. I wonder if this completes the command. I believe Scientifically, Suresh and you know much better than I do.

Besides that, I am curious the possibility you mentioned " install the GPU version of pytorch on a CPU node" for a conda environment, any reference for that?

Thank you,
Rong Yao

Answer 15 · 2023-08-31T05:44:04.000Z

Hi Rong,

The pytorch installation method worked for us. You should contact pytorch to install it properly on your cluster.

Your error message on KeyboardInterrupt appears a cluster issue too.

Lingfei

Answer 16 · 2023-09-01T19:52:06.000Z

Hi Lingfei,
I was able to successfully run GRN inference using CPU mode. But when I run aggregate command I see network files are missing in the Subset12 cluster.

Subset10/binding.tsv.gz Subset10/binlinking.tsv.gz Subset10/expression0.tsv.gz Subset10/expression.tsv.gz Subset10/footprints.bed Subset10/homer.tsv.gz Subset10/linking.tsv.gz Subset10/motifs.bed Subset10/names_atac0.txt Subset10/names_atac.txt Subset10/names_rna.txt Subset10/net_covfactor.tsv.gz Subset10/net_inweight.tsv.gz Subset10/net_iweight.tsv.gz Subset10/net_loss.tsv.gz Subset10/net_meanvar.tsv.gz Subset10/net_nweight.tsv.gz Subset10/net_stats.tsv.gz Subset10/net_weight.tsv.gz Subset10/peaks.bed Subset10/reads.bai Subset10/reads.bam Subset10/tssdist.tsv.gz Subset10/wellington.tsv.gz Subset11/binding.tsv.gz Subset11/binlinking.tsv.gz Subset11/expression0.tsv.gz Subset11/expression.tsv.gz Subset11/footprints.bed Subset11/homer.tsv.gz Subset11/linking.tsv.gz Subset11/motifs.bed Subset11/names_atac0.txt Subset11/names_atac.txt Subset11/names_rna.txt Subset11/net_covfactor.tsv.gz Subset11/net_inweight.tsv.gz Subset11/net_iweight.tsv.gz Subset11/net_loss.tsv.gz Subset11/net_meanvar.tsv.gz Subset2/expression.tsv.gz Subset3/net_weight.tsv.gz Subset5/names_atac.txt Subset7/binding.tsv.gz Subset8/net_meanvar.tsv.gz
Subset11/net_nweight.tsv.gz Subset2/footprints.bed Subset3/peaks.bed Subset5/names_rna.txt Subset7/binlinking.tsv.gz Subset8/net_nweight.tsv.gz
Subset11/net_stats.tsv.gz Subset2/homer.tsv.gz Subset3/reads.bai Subset5/net_covfactor.tsv.gz Subset7/expression0.tsv.gz Subset8/net_stats.tsv.gz
Subset11/net_weight.tsv.gz Subset2/linking.tsv.gz Subset3/reads.bam Subset5/net_inweight.tsv.gz Subset7/expression.tsv.gz Subset8/net_weight.tsv.gz
Subset11/peaks.bed Subset2/motifs.bed Subset3/tssdist.tsv.gz Subset5/net_iweight.tsv.gz Subset7/footprints.bed Subset8/peaks.bed
Subset11/reads.bai Subset2/names_atac0.txt Subset3/wellington.tsv.gz Subset5/net_loss.tsv.gz Subset7/homer.tsv.gz Subset8/reads.bai
Subset11/reads.bam Subset2/names_atac.txt Subset4/binding.tsv.gz Subset5/net_meanvar.tsv.gz Subset7/linking.tsv.gz Subset8/reads.bam
Subset11/tssdist.tsv.gz Subset2/names_rna.txt Subset4/binlinking.tsv.gz Subset5/net_nweight.tsv.gz Subset7/motifs.bed Subset8/tssdist.tsv.gz
Subset11/wellington.tsv.gz Subset2/net_covfactor.tsv.gz Subset4/expression0.tsv.gz Subset5/net_stats.tsv.gz Subset7/names_atac0.txt Subset8/wellington.tsv.gz
Subset12/expression0.tsv.gz Subset2/net_inweight.tsv.gz Subset4/expression.tsv.gz Subset5/net_weight.tsv.gz Subset7/names_atac.txt Subset9/binding.tsv.gz
Subset12/names_atac0.txt Subset2/net_iweight.tsv.gz Subset4/footprints.bed Subset5/peaks.bed Subset7/names_rna.txt Subset9/binlinking.tsv.gz
Subset12/names_rna.txt Subset2/net_loss.tsv.gz Subset4/homer.tsv.gz Subset5/reads.bai Subset7/net_covfactor.tsv.gz Subset9/expression0.tsv.gz
Subset1/binding.tsv.gz Subset2/net_meanvar.tsv.gz Subset4/linking.tsv.gz Subset5/reads.bam Subset7/net_inweight.tsv.gz Subset9/expression.tsv.gz
Subset1/binlinking.tsv.gz Subset2/net_nweight.tsv.gz Subset4/motifs.bed Subset5/tssdist.tsv.gz Subset7/net_iweight.tsv.gz Subset9/footprints.bed
Subset1/expression0.tsv.gz Subset2/net_stats.tsv.gz Subset4/names_atac0.txt Subset5/wellington.tsv.gz Subset7/net_loss.tsv.gz Subset9/homer.tsv.gz
Subset1/expression.tsv.gz Subset2/net_weight.tsv.gz Subset4/names_atac.txt Subset6/binding.tsv.gz Subset7/net_meanvar.tsv.gz Subset9/linking.tsv.gz
Subset1/footprints.bed Subset2/peaks.bed Subset4/names_rna.txt Subset6/binlinking.tsv.gz Subset7/net_nweight.tsv.gz Subset9/motifs.bed
Subset1/homer.tsv.gz Subset2/reads.bai Subset4/net_covfactor.tsv.gz Subset6/expression0.tsv.gz Subset7/net_stats.tsv.gz Subset9/names_atac0.txt
Subset1/linking.tsv.gz Subset2/reads.bam Subset4/net_inweight.tsv.gz Subset6/expression.tsv.gz Subset7/net_weight.tsv.gz Subset9/names_atac.txt
Subset1/motifs.bed Subset2/tssdist.tsv.gz Subset4/net_iweight.tsv.gz Subset6/footprints.bed Subset7/peaks.bed Subset9/names_rna.txt
Subset1/names_atac0.txt Subset2/wellington.tsv.gz Subset4/net_loss.tsv.gz Subset6/homer.tsv.gz Subset7/reads.bai Subset9/net_covfactor.tsv.gz
Subset1/names_atac.txt Subset3/binding.tsv.gz Subset4/net_meanvar.tsv.gz Subset6/linking.tsv.gz Subset7/reads.bam Subset9/net_inweight.tsv.gz
Subset1/names_rna.txt Subset3/binlinking.tsv.gz Subset4/net_nweight.tsv.gz Subset6/motifs.bed Subset7/tssdist.tsv.gz Subset9/net_iweight.tsv.gz
Subset1/net_covfactor.tsv.gz Subset3/expression0.tsv.gz Subset4/net_stats.tsv.gz Subset6/names_atac0.txt Subset7/wellington.tsv.gz Subset9/net_loss.tsv.gz
Subset1/net_inweight.tsv.gz Subset3/expression.tsv.gz Subset4/net_weight.tsv.gz Subset6/names_atac.txt Subset8/binding.tsv.gz Subset9/net_meanvar.tsv.gz
Subset1/net_iweight.tsv.gz Subset3/footprints.bed Subset4/peaks.bed Subset6/names_rna.txt Subset8/binlinking.tsv.gz Subset9/net_nweight.tsv.gz
Subset1/net_loss.tsv.gz Subset3/homer.tsv.gz Subset4/reads.bai Subset6/net_covfactor.tsv.gz Subset8/expression0.tsv.gz Subset9/net_stats.tsv.gz
Subset1/net_meanvar.tsv.gz Subset3/linking.tsv.gz Subset4/reads.bam Subset6/net_inweight.tsv.gz Subset8/expression.tsv.gz Subset9/net_weight.tsv.gz
Subset1/net_nweight.tsv.gz Subset3/motifs.bed Subset4/tssdist.tsv.gz Subset6/net_iweight.tsv.gz Subset8/footprints.bed Subset9/peaks.bed
Subset1/net_stats.tsv.gz Subset3/names_atac0.txt Subset4/wellington.tsv.gz Subset6/net_loss.tsv.gz Subset8/homer.tsv.gz Subset9/reads.bai
Subset1/net_weight.tsv.gz Subset3/names_atac.txt Subset5/binding.tsv.gz Subset6/net_meanvar.tsv.gz Subset8/linking.tsv.gz Subset9/reads.bam
Subset1/peaks.bed Subset3/names_rna.txt Subset5/binlinking.tsv.gz Subset6/net_nweight.tsv.gz Subset8/motifs.bed Subset9/tssdist.tsv.gz
Subset1/reads.bai Subset3/net_covfactor.tsv.gz Subset5/expression0.tsv.gz Subset6/net_stats.tsv.gz Subset8/names_atac0.txt Subset9/wellington.tsv.gz
Subset1/reads.bam Subset3/net_inweight.tsv.gz Subset5/expression.tsv.gz Subset6/net_weight.tsv.gz Subset8/names_atac.txt
Subset1/tssdist.tsv.gz Subset3/net_iweight.tsv.gz Subset5/footprints.bed Subset6/peaks.bed Subset8/names_rna.txt
Subset1/wellington.tsv.gz Subset3/net_loss.tsv.gz Subset5/homer.tsv.gz Subset6/reads.bai Subset8/net_covfactor.tsv.gz
Subset2/binding.tsv.gz Subset3/net_meanvar.tsv.gz Subset5/linking.tsv.gz Subset6/reads.bam Subset8/net_inweight.tsv.gz
Subset2/binlinking.tsv.gz Subset3/net_nweight.tsv.gz Subset5/motifs.bed Subset6/tssdist.tsv.gz Subset8/net_iweight.tsv.gz
Subset2/expression0.tsv.gz Subset3/net_stats.tsv.gz Subset5/names_atac0.txt Subset6/wellington.tsv.gz Subset8/net_loss.tsv.gz

What could be the reason and how to proceed further? Please suggest

I get the below error while running aggregate command:

WARNING:52151:2023-09-01 14:24:10,982:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset12 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:52151:2023-09-01 14:24:10,987:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset12 due to error: FileNotFoundError(2, 'No such file or directory')
Traceback (most recent call last):
File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/main.py", line 13, in
docstringrunner(package)
File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/docstring2argparse/init.py", line 340, in docstringrunner
run_args(pkgname,funcs,args)
File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/docstring2argparse/init.py", line 330, in run_args
return func(*a,**ka)
File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/network.py", line 1071, in tofile
n=network.from_folders(diri_data,diri_work,fi_subsets,dynamic=dynamic,nettype=nettype,optional=optional,fi_c=fi_c)
File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py", line 511, in from_folders
return cls(**params)
File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py", line 99, in init
self.check()
File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py", line 107, in check
raise ValueError('Cell/state/node names must be non-empty, unique, and matching their counts exactly.')
ValueError: Cell/state/node names must be non-empty, unique, and matching their counts exactly.
make: *** [makefiles/common.mk:175: output/static.h5] Error 1

Thanks,
Suresh

Answer 17 · 2023-09-05T16:07:44.000Z

Hi Suresh,

Great that the CPU version is working at least. Missing subset 12 is fine. Actually this is the error:

ValueError: Cell/state/node names must be non-empty, unique, and matching their counts exactly.

Could you update Dictys to the latest version for a more detailed error message? You can find the instructions at https://github.com/pinellolab/dictys#updating-dictys. You can also try a different tutorial and see if the error disappears there.

Lingfei

Answer 18 · 2023-09-07T21:14:52.000Z

Hi Lingfei,
Thanks for the information!

Actually I removed the subset 12 from the subset.txt file and it worked for remaining subset. I was able to generate the dotplot for this. But the issue I see that when I generate heatmap for the TFs I can only see the color key legend and not any image. I used the command shown in tutorial (fig,fig_colorbar,net=fig_heatmap_top(d0,tf,ntop=ntop,direction=direction,aspect=1,gann='all'). Please suggest!

Answer 19 · 2023-09-07T21:56:52.000Z

Hi Suresh,

Good to hear your feedbacks!

This seems like a jupyter-matplotlib compatibility issue. Can you try creating a notebook cell right after and run with the content:

fig

This should show the heatmap figure which is stored in variable fig. You should be able to use this trick to show any figure afterwards that is similarly hidden.

Lingfei

Answer 20 · 2023-09-11T16:02:43.000Z

Hi Lingfei,
Thanks for the response!
Actually, I am running the code in the HPC cluster and I don't think I can see the image right away. But for earlier images I don't see any compatibility issue, as all images generated properly. I don't have any idea on creating notebook. Can you help with the notebook generation.

Suresh

Answer 21 · 2023-09-11T16:38:44.000Z

Hi Suresh,

After GRN inference, you can download the h5 file to your local computer for visualization using this notebook.

Lingfei