zinbsurf taking forever

[eltonjrv]

Dear Davide or anyone else from ZINBWaVE team,

I have this huge ~14k genes x ~270k cells SCE assay.
I've just tried to run zinbsurf for 48 hours on a high-memory node from my cluster without finishing:
The command that never ends is the following:
A1Bzinb_surf <- zinbsurf(A1Bmtx_sce, K=9, X="~batch", epsilon=1e3, BPPARAM=BiocParallel::DoparParam(), prop_fit = 0.25)

That's how I set my last SGE job script:
#$ -l h_rt=48:00:00,h_vmem=30G -pe smp 24
#$ -l node_type=24core-768G

It stopped due to reaching the time limit. Please see qacct below

qname 24core-768G.q
hostname db12mem3.arc3.leeds.ac.uk
group fbs
owner fbsev
project mem_FBS
department defaultdepartment
jobname tempZinbSurf.sh
jobnumber 4836924
taskid undefined
account sge
priority 0
qsub_time Fri Jan 27 17:21:48 2023
start_time Fri Jan 27 21:20:11 2023
end_time Sun Jan 29 21:21:07 2023
granted_pe smp
slots 24
failed 37 : qmaster enforced h_rt, h_cpu, or h_vmem limit
exit_status 137 (Killed)
ru_wallclock 172856s
ru_utime 0.045s
ru_stime 0.130s
ru_maxrss 4.039KB
ru_ixrss 0.000B
ru_ismrss 0.000B
ru_idrss 0.000B
ru_isrss 0.000B
ru_minflt 14894
ru_majflt 0
ru_nswap 0
ru_inblock 0
ru_oublock 16
ru_msgsnd 0
ru_msgrcv 0
ru_nsignals 0
ru_nvcsw 223
ru_nivcsw 25
cpu 172702.080s
mem 35325.167TBs
io 1.026GB
iow 0.000s
maxvmem 296.759GB
arid undefined
ar_sub_time undefined
category -U omics -l env=centos7,h_rt=172800,h_vmem=30G,node_type=24core-768G,project=arc -pe smp 24

It's reaching ~300Gb of memory, which is less than half of what I requested (720Gb), fine!
The problem seems to be on the CPUs, which zinbsurf appears to be using only one out of 24 cores from the node (divide cpu by ru_wallclock in bold above).

I'm using DoparParam from BiocParallel, as I've seen your recommendation for such in another issue herein.

I wonder if you have any tip to circumvent this.
A second question would be on, once I get zinbsurf completed, whether a zinb_surf object is suitable for DESeq downstream analysis, since surfing does not compute observational weights.

Many Thanks in advance,
Elton

[drisso]

Dear Elton,

we know that zinbwave is not suited for very large dataset. The zinbsurf function was one attempt to scale it up, but it has been superseded by the NewWave package that provides more efficient implementations based on mini-batch optimization. Please see here:
https://www.bioconductor.org/packages/release/bioc/html/NewWave.html