BuildWijMatrix fails: invalid row or column index

Question

BuildWijMatrix fails: invalid row or column index

Closed this issue 7 years ago · 15 comments

R 3.2.5; latest version of largeVis; Ubuntu 12.04

Running largeVis by itself and its step-by-step components doesn't work. I narrowed it down to the BuildWijMatrix step failing, but don't know how to proceed.

sessionInfo()
R version 3.2.5 (2016-04-14)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu precise (12.04.5 LTS)

locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] largeVis_0.2 Matrix_1.2-8 Rcpp_0.12.10

loaded via a namespace (and not attached):
[1] colorspace_1.3-2 scales_0.4.1 assertthat_0.1 lazyeval_0.2.0
[5] plyr_1.8.4 tools_3.2.5 gtable_0.2.0 tibble_1.2
[9] ggplot2_2.2.1 grid_3.2.5 munsell_0.4.3 lattice_0.20-34

dim(as.matrix(seurat_sw480@data))
[1] 15843 1691
neighbors <- randomProjectionTreeSearch(as.matrix(seurat_sw480@data), n_trees = 5, max_iter = 1, verbose=T)
Searching for neighbors.
0% 10 20 30 40 50 60 70 80 90 100%
|----|----|----|----|----|----|----|----|----|----|
**************************************************|
edges <- buildEdgeMatrix(data = as.matrix(seurat_sw480@data), neighbors = neighbors)
gc()
used (Mb) gc trigger (Mb) max used (Mb)
Ncells 1124907 60.1 1770749 94.6 1770749 94.6
Vcells 118347552 903.0 196989268 1503.0 187146708 1427.9
rm(neighbors)
wij <- buildWijMatrix(edges)

error: SpMat::SpMat(): invalid row or column index
Error in referenceWij(is, x@i, x@x^2, as.integer(threads), perplexity) :
SpMat::SpMat(): invalid row or column index

Answer 1 · 2017-03-29T22:42:23.000Z

Well that definitely shouldn't be happening. Can you make that dataset available and I'll take a look?

…

On Mar 29, 2017, at 6:12 PM, billytcl ***@***.***> wrote: R 3.2.5; latest version of largeVis; Ubuntu 12.04 Running largeVis by itself and its step-by-step components doesn't work. I narrowed it down to the BuildWijMatrix step failing, but don't know how to proceed. sessionInfo() R version 3.2.5 (2016-04-14) Platform: x86_64-pc-linux-gnu (64-bit) Running under: Ubuntu precise (12.04.5 LTS) locale: [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 [7] LC_PAPER=en_US.UTF-8 LC_NAME=C [9] LC_ADDRESS=C LC_TELEPHONE=C [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] largeVis_0.2 Matrix_1.2-8 Rcpp_0.12.10 loaded via a namespace (and not attached): [1] colorspace_1.3-2 scales_0.4.1 assertthat_0.1 lazyeval_0.2.0 [5] plyr_1.8.4 tools_3.2.5 gtable_0.2.0 tibble_1.2 [9] ggplot2_2.2.1 grid_3.2.5 munsell_0.4.3 lattice_0.20-34 ***@***.***)) [1] 15843 1691 neighbors <- ***@***.***), n_trees = 5, max_iter = 1, verbose=T) Searching for neighbors. 0% 10 20 30 40 50 60 70 80 90 100% |----|----|----|----|----|----|----|----|----|----| **************************************************| edges <- buildEdgeMatrix(data = ***@***.***), neighbors = neighbors) gc() used (Mb) gc trigger (Mb) max used (Mb) Ncells 1124907 60.1 1770749 94.6 1770749 94.6 Vcells 118347552 903.0 196989268 1503.0 187146708 1427.9 rm(neighbors) wij <- buildWijMatrix(edges) error: SpMat::SpMat(): invalid row or column index Error in referenceWij(is, ***@***.***, ***@***.***^2, as.integer(threads), perplexity) : SpMat::SpMat(): invalid row or column index — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

Answer 2 · 2017-03-29T22:55:36.000Z

Here's a link to the as.matrix(seurat_sw480@data). The var name is now "mat".

https://www.dropbox.com/s/g8qgps5eatc9o27/seurat_sw480_matrix.RData?dl=0

Answer 3 · 2017-03-31T05:15:06.000Z

Hi there -- any updates?

Answer 4 · 2017-03-31T05:48:42.000Z

It'll be a few days before I have a chance to look at it. I'll let you know.

…

On Mar 31, 2017, at 1:15 AM, billytcl ***@***.***> wrote: Hi there -- any updates? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

Answer 5 · 2017-04-01T21:08:25.000Z

In your matrix, are the examples rows or columns?

largeVis requires that the examples be columns and the features, rows. I then don't reproduce the issue:

Answer 6 · 2017-04-01T21:25:32.000Z

The examples (cells) are columns and features (genes) are rows. I think there may be a memory issue because I got it to work when only using 500 columns instead of the full 1691. Is there any way to reduce usage more than what the tutorial recommends?

…

On Sat, Apr 1, 2017 at 5:08 PM elbamos ***@***.***> wrote: In your matrix, are the examples rows or columns? largeVis requires that the examples be columns and the features, rows. I then don't reproduce the issue: [image: screen shot 2017-04-01 at 5 02 31 pm] <https://cloud.githubusercontent.com/assets/10103420/24582408/59b92ff2-16fd-11e7-888c-f9a307767903.png> — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#43 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJ58T43EQst-I428rmD3Upw1SgZUUd96ks5rrrzJgaJpZM4Mtnzx> .

Answer 7 · 2017-04-01T21:28:33.000Z

Its not a memory issue.

So are there 15,000 examples and 1600 features, or 15,000 features and 1600 examples?

Answer 8 · 2017-04-01T23:40:36.000Z

Hi, there are 15,000 features and 1600 examples. Billy

…

On Sat, Apr 1, 2017 at 5:28 PM elbamos ***@***.***> wrote: Its not a memory issue. So are there 15,000 examples and 1600 features, or 15,000 features and 1600 examples? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#43 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJ58T3_85BN1sLhnfaJHyjLCsH-XdTE9ks5rrsGBgaJpZM4Mtnzx> .

Answer 9 · 2017-04-02T00:35:06.000Z

Ah. I think I may know what's going on. I'll check this weekend.

…

On Apr 1, 2017, at 7:40 PM, billytcl ***@***.***> wrote: Hi, there are 15,000 features and 1600 examples. Billy On Sat, Apr 1, 2017 at 5:28 PM elbamos ***@***.***> wrote: > Its not a memory issue. > > So are there 15,000 examples and 1600 features, or 15,000 features and > 1600 examples? > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#43 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AJ58T3_85BN1sLhnfaJHyjLCsH-XdTE9ks5rrsGBgaJpZM4Mtnzx> > . > — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

Answer 10 · 2017-04-03T07:34:10.000Z

@billytcl I think I've tracked this down. Thank you for posting it! It seems to be a subtle bug that creeped in late in the last update. It will take a few days to get a working fix. I will let you know.

Anyway, your data looks like this:

Answer 11 · 2017-04-04T05:10:00.000Z

@billytcl Try the version in branch hotfix/twobugs

Answer 12 · 2017-04-04T21:00:47.000Z

Thanks!! Will give it a shot.

Answer 13 · 2017-04-04T21:28:09.000Z

Looks like it's working! Thanks!!

Answer 14 · 2017-04-04T21:53:48.000Z

Cool! Btw, with datasets like the one you sent me, i think you'll be a lot happier if you convert the data to a sparse matrix. If you do that, then it'll scale to very, very, very large datasets before you hit a memory issue.

…

On Apr 4, 2017, at 5:28 PM, billytcl ***@***.***> wrote: Looks like it's working! Thanks!! — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

Answer 15 · 2017-05-12T16:27:40.000Z

I'm going to close this now. Please let me know if anything else comes up, and thank you for the report!