hdbscan-non-numeric argument to binary operator

Question

hdbscan-non-numeric argument to binary operator

Closed this issue 7 years ago · 24 comments

feng-1985 commented 7 years ago

library(largeVis)
set.seed(123)
ts_matrix_elec <- elect_data %>% scale() %>% t()
visObject <- largeVis(ts_matrix_elec, n_trees = 50,
K = 10)
plot(t(visObject$coords))

clusters <- hdbscan(visObject, verbose = FALSE) # failed
Error in stats::aggregate(probs, by = list(clusters), FUN = "max")$probs - :
non-numeric argument to binary operator

gplot(clusters, t(visObject$coords))

What happened? Is there any suggestion?

Answer 1 · 2017-04-01T20:47:57.000Z

Hey can you make elect_data available and I'll look? thanks.

Answer 2 · 2017-04-02T13:50:01.000Z

Thank you! and how to make the data avaliable for you?

Answer 3 · 2017-04-04T21:49:47.000Z

If you could Dropbox it to me today, I'm planning a bug fix update soon.

…

On Apr 2, 2017, at 9:50 AM, hu bifeng ***@***.***> wrote: Thank you! and how to make the data avaliable for you? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

Answer 4 · 2017-04-05T02:02:50.000Z

https://www.dropbox.com/s/0v41q45yvn9ahzh/test_gm_data.csv?dl=0
I upload the data to Dropbox, thank you!

It's time series from January 2012 to December 2015 monthly data, there are 11524 individuals, i want to cluster these individuals based on the time dimension.

Answer 5 · 2017-04-05T02:29:27.000Z

I can't reproduce it. Can you try the version currently in branch hotfix/twobugs and confirm if the issue is now resolved?

Loading required package: Matrix
> library(readr)
> elect_data <- read_csv("~/Downloads/test_gm_data.csv")
Parsed with column specification:
cols(
  .default = col_double()
)
See spec(...) for full column specifications.
> str(elect_data)
<snip>
> library(largeVis)
Loading required package: Rcpp
> library(magrittr)
> ts_matrix <- elect_data %>% scale() %>% t()
> visObj <- largeVis(ts_matrix, n_trees = 50, K = 10, verbose = TRUE)
Searching for neighbors.
0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
**************************************************|
Calculating edge weights...
0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
**************************************************|
Estimating embeddings.
0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
**************************************************|
Warning message:
In largeVis(ts_matrix, n_trees = 50, K = 10, verbose = TRUE) :
  The Distances between some neighbors are large enough to cause the calculation of p_{j|i} to overflow. Scaling the distance vector.
> plot(t(visObj$coords))
> clusters <- hdbscan(visObj, verbose = TRUE)
0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
**************************************************|
> gplot(clusters, t(visObj$coords))
Warning message:
Removed 1337 rows containing missing values (geom_segment). 
>

Answer 6 · 2017-04-05T04:06:02.000Z

how to use that version?

Answer 7 · 2017-04-05T04:15:27.000Z

devtools::install_github("elbamos/largeVis", ref = "hotfix/twobugs")

Answer 8 · 2017-04-05T06:21:04.000Z

too slow, is there any faster download method?

Answer 9 · 2017-04-05T06:21:59.000Z

No. It takes a fraction of a second from here. On April 5, 2017 at 2:21:12 AM, hu bifeng (notifications@github.com) wrote: too slow, is there any faster download method? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

Answer 10 · 2017-04-06T01:30:41.000Z

Yes, it works!

But why there is "NA" in the plot? I can't upload this image, did you see that in you plot?

Answer 11 · 2017-04-06T02:08:27.000Z

Yes. Points will have cluster NA if the algorithm does not put them in a cluster. You can review the documentation on the algorithm for detail if you'd like.

I'm going to close this now - feel free to reopen if anything comes up.

Answer 12 · 2017-04-06T03:10:32.000Z

Thank you!

Answer 13 · 2017-04-15T05:07:00.000Z

@bifeng There was a bug in the version of largeVis that you tested a week ago. The bug caused the hdbscan algorithm to fail to combine clusters that should be combined. If you try the version that I've just pushed, it should produce better results on your dataset.

Answer 14 · 2017-06-29T21:01:33.000Z

I am also encountering the same problem.

> load('C:/lab/normdata.Rdata')

> library(largeVis)

> library(ggplot2)

> norm <- scale(norm)

> l <- largeVis(norm,verbose=T)
Searching for neighbors.
0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
**************************************************|
Calculating edge weights...
0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
**************************************************|
Estimating embeddings.
0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
**************************************************|

> clusters <- largeVis::hdbscan(l,verbose=T)
0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
**************************************************|
********Error in stats::aggregate(probs, by = list(clusters), FUN = "max")$probs -  : 
  non-numeric argument to binary operator
In addition: Warning message:
In largeVis(norm, verbose = T) :
  The Distances between some neighbors are large enough to cause the calculation of p_{j|i} to overflow. Scaling the distance vector.

Answer 15 · 2017-06-29T21:05:34.000Z

Can you make your data available to me and I'll take a look tonight? I was sure I fixed this.

…

On Jun 29, 2017, at 5:01 PM, Christopher Lin ***@***.***> wrote: I am also encountering the same problem. > load('C:/lab/normdata.Rdata') > library(largeVis) > library(ggplot2) > norm <- scale(norm) > l <- largeVis(norm,verbose=T) Searching for neighbors. 0% 10 20 30 40 50 60 70 80 90 100% |----|----|----|----|----|----|----|----|----|----| **************************************************| Calculating edge weights... 0% 10 20 30 40 50 60 70 80 90 100% |----|----|----|----|----|----|----|----|----|----| **************************************************| Estimating embeddings. 0% 10 20 30 40 50 60 70 80 90 100% |----|----|----|----|----|----|----|----|----|----| **************************************************| > clusters <- largeVis::hdbscan(l,verbose=T) 0% 10 20 30 40 50 60 70 80 90 100% |----|----|----|----|----|----|----|----|----|----| **************************************************| ********Error in stats::aggregate(probs, by = list(clusters), FUN = "max")$probs - : non-numeric argument to binary operator In addition: Warning message: In largeVis(norm, verbose = T) : The Distances between some neighbors are large enough to cause the calculation of p_{j|i} to overflow. Scaling the distance vector. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or mute the thread.

Answer 16 · 2017-06-29T21:08:18.000Z

I'd rather not publicly post the data. Can I email it to you?

Answer 17 · 2017-06-29T21:54:02.000Z

Sure or email me a Dropbox link. My email is in the git

…

On Jun 29, 2017, at 5:08 PM, Christopher Lin ***@***.***> wrote: I'd rather not publicly post the data. Can I email it to you? — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or mute the thread.

Answer 18 · 2017-06-30T17:13:59.000Z

I couldn't reproduce it. Are you sure you're using a current version?

Answer 19 · 2017-06-30T18:05:38.000Z

I installed it with

devtools::install_github("elbamos/largeVis", ref = "hotfix/twobugs")

Is this correct?

Answer 20 · 2017-06-30T18:57:07.000Z

No - the hotfix was rolled in ages ago. Just install from master. Leave out the "ref" parameter.

…

On Jun 30, 2017, at 2:05 PM, Christopher Lin ***@***.***> wrote: I installed it with devtools::install_github("elbamos/largeVis", ref = "hotfix/twobugs") Is this correct? — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or mute the thread.

Answer 21 · 2017-07-05T20:14:38.000Z

I've reinstalled from master and it's still throwing the same error.

> h <- hdbscan(vis, verbose=T)
0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
**************************************************|
********Error in stats::aggregate(probs, by = list(clusters), FUN = "max")$probs -  : 
  non-numeric argument to binary operator
In addition: Warning message:
In largeVis(norm, verbose = T) :
  The Distances between some neighbors are large enough to cause the calculation of p_{j|i} to overflow. Scaling the distance vector.

Answer 22 · 2017-07-05T20:36:43.000Z

That's very odd. Can you send me the log or a screenshot of a complete session? Start from an empty environment, load largeVis, check the version, and try the commands in just the way I did them?

Answer 23 · 2017-07-07T18:50:04.000Z

> load('C:/lab/normdata.Rdata')

> library(largeVis)

> library(ggplot2)

> sessionInfo()
R version 3.4.0 (2017-04-21)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] ggplot2_2.2.1  largeVis_0.2.2 Matrix_1.2-9  

loaded via a namespace (and not attached):
 [1] colorspace_1.3-2 scales_0.4.1     compiler_3.4.0   lazyeval_0.2.0   plyr_1.8.4      
 [6] tools_3.4.0      gtable_0.2.0     tibble_1.3.3     Rcpp_0.12.11     grid_3.4.0      
[11] rlang_0.1.1      munsell_0.4.3    lattice_0.20-35 

> norm <- scale(norm)

> vis <- largeVis(norm,verbose=T)
Searching for neighbors.
0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
**************************************************|
Calculating edge weights...
0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
**************************************************|
Estimating embeddings.
0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
**************************************************|

> plot(t(vis$coords))

> h <- hdbscan(vis, verbose=T)
0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
**************************************************|
********Error in stats::aggregate(probs, by = list(clusters), FUN = "max")$probs -  : 
  non-numeric argument to binary operator
In addition: Warning message:
In largeVis(norm, verbose = T) :
  The Distances between some neighbors are large enough to cause the calculation of p_{j|i} to overflow. Scaling the distance vector.

Answer 24 · 2017-07-07T22:30:53.000Z

Thanks, I was able to reproduce this. The error is coming up because hdbscan can’t cluster this data. This is because it has a huge number of duplicate points in it. When the number of duplicates grows, then both largeVis and hdbscan become undefined since they depend on finding each point’s n-nearest neighbors. You can try to force a clustering by adjusting minPts and K (down). But really I think the question you want to ask is whether you want to de-dupe this data before you try to cluster and visualize it and, if you don’t want to de-dupe it, whether using a nearest-neighbor algorithm makes sense? I will include a check for this in the next largeVis version. Thanks again for reporting! On July 7, 2017 at 2:50:09 PM, Christopher Lin (notifications@github.com) wrote: h <- hdbscan(vis, verbose=T)