Crash after completing "Working on chunk (xx / xx)"

[Thapeachydude]

Hi,

I'm getting the error message below when running it on a 10x snRNA-Seq dataset (5'HT). I've tried

cellbender remove-background --input raw_feature_bc_matrix.h5 --expected-cells 40000 --total-droplets-included 100000 --epochs 150 --output cellbender_feature_bc_matrix.h5 --cuda

Traceback (most recent call last):
  File "C:\Users\marce\anaconda3\envs\cellbender\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Users\marce\anaconda3\envs\cellbender\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\marce\anaconda3\envs\cellbender\Scripts\cellbender.exe\__main__.py", line 7, in <module>
  File "C:\Users\marce\anaconda3\envs\cellbender\lib\site-packages\cellbender\base_cli.py", line 118, in main
    cli_dict[args.tool].run(args)
  File "C:\Users\marce\anaconda3\envs\cellbender\lib\site-packages\cellbender\remove_background\cli.py", line 185, in run
    return main(args)
  File "C:\Users\marce\anaconda3\envs\cellbender\lib\site-packages\cellbender\remove_background\cli.py", line 230, in main
    posterior = run_remove_background(args)
  File "C:\Users\marce\anaconda3\envs\cellbender\lib\site-packages\cellbender\remove_background\run.py", line 107, in run_remove_background
    args=args,
  File "C:\Users\marce\anaconda3\envs\cellbender\lib\site-packages\cellbender\remove_background\posterior.py", line 125, in load_or_compute_posterior_and_save
    posterior.cell_noise_count_posterior_coo()
  File "C:\Users\marce\anaconda3\envs\cellbender\lib\site-packages\cellbender\remove_background\posterior.py", line 386, in cell_noise_count_posterior_coo
    self._get_cell_noise_count_posterior_coo(**kwargs)
  File "C:\Users\marce\anaconda3\envs\cellbender\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\marce\anaconda3\envs\cellbender\lib\site-packages\cellbender\remove_background\posterior.py", line 555, in _get_cell_noise_count_posterior_coo
    shape=[np.prod(self.count_matrix_shape), n_counts_max],
  File "C:\Users\marce\anaconda3\envs\cellbender\lib\site-packages\scipy\sparse\coo.py", line 196, in __init__
    self._check()
  File "C:\Users\marce\anaconda3\envs\cellbender\lib\site-packages\scipy\sparse\coo.py", line 283, in _check
    raise ValueError('row index exceeds matrix dimensions')

Also: Is there the option of running it without checkpoints? I'm setting it up to run on a cluster and the checkpoints saving frequently results in crashes.

Many thanks for any feedback,
M

[Bio-MingChen]

I encounter the same error,do you solve the problem?

[Thapeachydude]

I don't recall the specifics, but for me running it on a windows machine didn't work. I had to switch to a linux server. Regarding the check points there is an argument to set the amount of time between checkpoints. Just set that to longer than the run time.

[mdpatric]

Did anyone find a fix for this on windows? I'm on the sf_windows_checkpoint_file_rename_fix branch to fix the checkpoint save issue.

[antsmer]

Did anyone find a fix for this on windows? I'm on the sf_windows_checkpoint_file_rename_fix branch to fix the checkpoint save issue.

Did you or anyone else find a fix to this on windows? Thanks!