fdrtool error during diffnet
Closed this issue · 1 comments
Hello!
First of all, great package!
During analysis I have run into a problem with diffnet and specifically with fdrtool. Here is the code:
moss_16s_diffnet <- diffnet(moss_16s_comp, diffMethod = "permute", nPerm = 20, fileLoadAssoPerm = "moss_16s_assoPerm", storeCountsPerm = FALSE )
and the error:
Checking input arguments ... Done. Execute permutation tests ... |==========================================================================================| 100% Adjust for multiple testing using 'lfdr' ... Execute fdrtool() ... Error in if (max(x) > 1 | min(x) < 0) stop("input p-values must all be in the range 0 to 1!") : missing value where TRUE/FALSE needed
I am aware that my dataset is very suboptimal (2 groups, 3 repeats each) and also for permutation I had extremely suboptimal setting (I couldn't increase nPerm as the software was complaining):
moss_16s_groups <- netCompare(moss_16s_props, permTest = TRUE, nPerm = 20, storeAssoPerm = TRUE, fileStoreAssoPerm = "moss_16s_assoPerm", storeCountsPerm = FALSE, seed = 19860729 )
netCompare summary:
`Comparison of Network Properties
CALL:
netCompare(x = moss_16s_props, permTest = TRUE, nPerm = 20, seed = 19860729,
storeAssoPerm = TRUE, fileStoreAssoPerm = "moss_16s_assoPerm",
storeCountsPerm = FALSE)
Global network properties
Whole network:
M M+H abs.diff. p-value
Number of components 1.000 1.000 0.000 1.0
Clustering coefficient 0.934 0.862 0.073 0.5
Modularity 0.030 0.010 0.020 0.1 .
Positive edge percentage 61.448 52.603 8.844 0.1 .
Edge density 0.871 0.838 0.033 0.6
Natural connectivity 0.595 0.371 0.224 0.1 .
Vertex connectivity 53.000 100.000 47.000 0.4
Edge connectivity 58.000 101.000 43.000 0.5
Average dissimilarity* 0.454 0.597 0.143 0.1 .
Average path length** 0.498 0.620 0.122 0.1 .
-----
p-values: one-tailed test with null hypothesis diff=0
*: Dissimilarity = 1 - edge weight
**: Path length = Sum of dissimilarities along the path
______________________________
Jaccard index (similarity betw. sets of most central nodes)
Jacc P(<=Jacc) P(>=Jacc)
degree 0.000 0.000000 *** 1.000000
betweenness centr. 0.197 0.013914 * 0.993798
closeness centr. 0.049 0.000014 *** 0.999999
eigenvec. centr. 0.262 0.048043 * 0.968449
hub taxa 0.000 0.000059 *** 1.000000
Jaccard index in [0,1] (1 indicates perfect agreement)
Adjusted Rand index (similarity betw. clusterings)
wholeNet LCC
ARI 0.013 0.013
p-value 0.079 0.089
-----
ARI in [-1,1] with ARI=1: perfect agreement betw. clusterings
ARI=0: expected for two random clusterings
p-value: permutation test (n=1000) with null hypothesis ARI=0
______________________________
Graphlet Correlation Distance
`````````````````````````````
wholeNet LCC
GCD 2.714 2.714
p-value 0.700 0.700
-----
GCD >= 0 (GCD=0 indicates perfect agreement between GCMs)
p-value: permutation test with null hypothesis GCD=0
______________________________
Centrality measures
- In decreasing order
- Computed for the whole network
````````````````````````````````````
Degree (unnormalized):
M M+H abs.diff. adj.p-value
10056 58 149 91 0.350989
10132 59 149 90 0.350989
10109 59 148 89 0.350989
10080 58 146 88 0.350989
10071 61 147 86 0.616022
Betweenness centrality (unnormalized):
M M+H abs.diff. adj.p-value
10100 3081 2 3079 0.785714
10055 853 11 842 0.785714
10003 175 0 175 0.785714
10059 169 0 169 0.785714
10012 10 178 168 0.785714
Closeness centrality (unnormalized):
M M+H abs.diff. adj.p-value
10058 1.342184e+08 1.046614e+03 134217320 0.888462
10133 5.280217e+03 1.342219e+08 134216623 0.888462
10158 5.280217e+03 1.342219e+08 134216623 0.888462
10033 1.342184e+08 1.134760e+04 134207019 0.888462
10074 9.490665e+07 5.395190e+02 94906111 0.888462
_________________________________________________________
Significance codes: ***: 0.001, **: 0.01, *: 0.05, .: 0.1`
When I ran it on the filtrated dataset it went through without an error but there was no significant difference between groups, so I tried on full data set and error appeared.
I have tried to run it on the tutorial dataset, playing with different nPerm but even at nPerm = 3000 it did not result in any significant differences, so I still do not know if this is because my data or some issue with the diffnet?
Any help would be appreciated :)
Hi,
Unfortunately, a permutation test with 20 permutations does not make any sense. And with 6 samples in two groups, the possible number of unique permutations is 20. That's why the function is complaining when you try to increase the number of permutations ;)
The absolute minimum number of permutations to get an - at least to some extend - reliable p-value is 500.
Even with 1000 permutations it is difficult to see significant results after adjusting for multiple testing.
You should consider the following: With 20 permutations, the minimum possible p-value is 1/20, which will never be significant after adjusting for multiple testing.
So, I would recommend not doing permutation tests and reporting the results only with the group differences.
You should also consider that the correlations itself cannot be reliably estimated with such a low sample size.