findEncodingDim always uses all avaialble cores
Opened this issue · 1 comments
Dear OUTRIDER authors,
thank you for this very useful package. These days i am doing some initial tests and unfortunately I am not able to limit the number of cores that are used by the findEncodingDim function: it ignores my BiocParallel setting, thus always using all avaialble cores. I'm having a hard time finding my mistake and unfortunately it is a very serious problem for me because I currently work on a shared server.
This is the simple code that I am running:
ncores <- 10
register(MulticoreParam(ncores))
ods <- findEncodingDim(ods, BPPARAM=bpparam())
I really hope you can help me. I also attach the session info, please let me know if you need additional information.
Thanks in advance!
Elisa
> sessionInfo()
R version 3.6.1 (2019-07-05)
Platform: x86_64-conda_cos6-linux-gnu (64-bit)
Running under: Ubuntu 18.04.4 LTS
Matrix products: default
BLAS/LAPACK: /scratch/Elisa/miniconda3/envs/rna-seq/lib/R/lib/libRblas.so
locale:
[1] LC_CTYPE=en_US.utf8 LC_NUMERIC=C
[3] LC_TIME=en_US.utf8 LC_COLLATE=en_US.utf8
[5] LC_MONETARY=en_US.utf8 LC_MESSAGES=en_US.utf8
[7] LC_PAPER=en_US.utf8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.utf8 LC_IDENTIFICATION=C
attached base packages:
[1] stats4 parallel stats graphics grDevices utils datasets
[8] methods base
other attached packages:
[1] tximport_1.14.2 DESeq2_1.26.0
[3] OUTRIDER_1.4.2 data.table_1.12.8
[5] SummarizedExperiment_1.16.1 DelayedArray_0.12.3
[7] matrixStats_0.56.0 GenomicFeatures_1.38.2
[9] AnnotationDbi_1.48.0 Biobase_2.46.0
[11] GenomicRanges_1.38.0 GenomeInfoDb_1.22.1
[13] IRanges_2.20.2 S4Vectors_0.24.4
[15] BiocGenerics_0.32.0 BiocParallel_1.20.1
loaded via a namespace (and not attached):
[1] colorspace_1.4-1 ellipsis_0.3.0 htmlTable_1.13.3
[4] XVector_0.26.0 base64enc_0.1-3 rstudioapi_0.11
[7] bit64_0.9-7 codetools_0.2-16 splines_3.6.1
[10] PRROC_1.3.1 geneplotter_1.64.0 knitr_1.28
[13] Formula_1.2-3 jsonlite_1.6.1 Rsamtools_2.2.3
[16] annotate_1.64.0 cluster_2.1.0 dbplyr_1.4.2
[19] png_0.1-7 pheatmap_1.0.12 compiler_3.6.1
[22] httr_1.4.1 backports_1.1.5 assertthat_0.2.1
[25] Matrix_1.2-17 lazyeval_0.2.2 cli_1.1.0
[28] acepack_1.4.1 htmltools_0.4.0 prettyunits_1.1.1
[31] tools_3.6.1 gtable_0.3.0 glue_1.3.1
[34] GenomeInfoDbData_1.2.2 reshape2_1.4.4 dplyr_0.8.5
[37] rappdirs_0.3.1 Rcpp_1.0.4 vctrs_0.3.1
[40] Biostrings_2.54.0 gdata_2.18.0 rtracklayer_1.46.0
[43] iterators_1.0.10 xfun_0.12 stringr_1.4.0
[46] lifecycle_0.2.0 gtools_3.8.1 XML_3.99-0.3
[49] dendextend_1.13.4 MASS_7.3-51.5 zlibbioc_1.32.0
[52] scales_1.1.0 TSP_1.1-10 pcaMethods_1.78.0
[55] hms_0.5.3 RColorBrewer_1.1-2 BBmisc_1.11
[58] curl_4.3 memoise_1.1.0 heatmaply_1.1.0
[61] gridExtra_2.3 ggplot2_3.3.0 biomaRt_2.42.1
[64] rpart_4.1-15 latticeExtra_0.6-29 stringi_1.4.3
[67] RSQLite_2.2.0 genefilter_1.68.0 gclus_1.3.2
[70] foreach_1.4.4 checkmate_2.0.0 seriation_1.2-8
[73] caTools_1.17.1.2 rlang_0.4.6 pkgconfig_2.0.3
[76] bitops_1.0-6 lattice_0.20-38 purrr_0.3.3
[79] GenomicAlignments_1.22.1 htmlwidgets_1.5.1 bit_1.1-15.2
[82] tidyselect_1.1.0 plyr_1.8.6 magrittr_1.5
[85] R6_2.4.0 gplots_3.0.1.1 Hmisc_4.4-0
[88] DBI_1.1.0 pillar_1.4.3 foreign_0.8-76
[91] survival_3.1-12 RCurl_1.98-1.1 nnet_7.3-13
[94] tibble_3.0.0 crayon_1.3.4 KernSmooth_2.23-15
[97] BiocFileCache_1.10.2 plotly_4.9.2.1 viridis_0.5.1
[100] jpeg_0.1-8.1 progress_1.2.2 locfit_1.5-9.4
[103] grid_3.6.1 blob_1.2.1 digest_0.6.18
[106] webshot_0.5.2 xtable_1.8-4 tidyr_1.1.0
[109] openssl_1.4.1 munsell_0.5.0 registry_0.5-1
[112] viridisLite_0.3.0 askpass_1.1
hey @emariella,
After looking into it, I'm not sure what is happening here. There are two parallelization mechanism that could trigger this.
- The
bplapplycall. But this is set by you and also appropriately passed on to internal functions. - The intrinsic parallelized computation of matrix multiplications with BLAS and openMP that already caused some headack (#18).
After reviewing the code again, the first option should not trigger it.
As we disabled in commit 1b9e5b5 openMP also this should not happen. To make sure, I explicitly set BLAS to use just one core in a new branch here f42b49c. As you use already 1.4.2 this should not happen, but maybe something went wrong with your Makevariables/CPPFlags while compiling the code.
Can you please try using the new branch version if this fixes the problem for you?
devtools::install_github("gagneurlab/OUTRIDER", ref="fix26")
# should give you the new line with `blas_set_num_threads(1L)`
OUTRIDER::findEncodingDim
# test it again
library(OUTRIDER)
ods <- makeExampleOutriderDataSet()
ncores <- 5
register(MulticoreParam(ncores, ncores*2, progressbar = TRUE))
ods <- findEncodingDim(ods, BPPARAM=bpparam(), params = seq(2,20,by=2))