Different results modularity_und in this module and in Matlab

[Pavel-IO]

Hi! I also thank you so much for implementing these tools in Python!

I'm trying to use the function modularity_und() compatible with the original Matlab version, but I'm getting different results in bctpy and BCT Matlab version.

Example of my input is here https://pastebin.com/KfzcqJmM.

Python version:

rho = numpy.loadtxt(my_file, float, delimiter=',')
_, q = bct.modularity_und(rho)
print(q)  # 0.1498

Matlab version:

rho = dlmread(myFile);
[~, q] = modularity_und(rho);
disp(q);  % 0.1626

Results:
In Python: 0.1498
In Matlab: 0.1626
Difference: 0.0128 is approximately 8 %.

This difference is quite essential in the subsequent group comparison (group t-test). The implementation in python gives me a significance level of p = 0.07, while the Matlab version returns p = 0.13.

Different is also the resulting array of community structure (ci). If I pass the community structure calculated in Matlab as an argument to kci, then the result already is same with Matlab's version.

rho = numpy.loadtxt(my_file, float, delimiter=',')
kci = [
    2, 2, 1, 2, 1, 1, 2, 2, 1, 1, 2, 2, 2, 2, 1, 1, 2, 2, 2, 2, 1, 1, 2, 2, 1, 1, 2, 1, 3, 3, 3, 3, 2, 2, 2, 2, 4, 1, 1,
    1, 1, 4, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 4, 4, 4, 4, 2, 2, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 3, 3, 4, 3, 3, 3,
    3, 3, 3, 2, 1, 1, 4, 1, 1, 1, 1, 1, 4, 4, 1, 1, 4, 4, 4, 4, 4, 4, 1, 1, 1, 1, 4, 1, 1, 1, 4, 4, 3, 4, 4, 4, 4, 4
]
_, q = bct.modularity_und(rho, kci=kci)
print(q)  # 0.1626

I noticed a comment in modularity_test.py: "matlab and bctpy appear to return different results due to the cross-package numerical instability of eigendecompositions", but is it possible that such a large difference is the result of numerical instability?

[aestrivex]

0.1498 and 0.1626 are pretty close. 8% of a small number is a small number in absolute terms. Both of these results suggest the module structure is fairly weak, it does not seem unexpected to have a difference of this magnitude. To the best of my knowledge, as the comment suggests in the code, the deterministic algorithm differs due to a difference in linear algebra libraries which calculate eigendecompositions. I remember preferring to have a precise copy of deterministic algorithms that produced the same results for testing, but that it was not feasible in this case without having a package that copies the linear algebra functions.

…

On Tue, Dec 29, 2020 at 3:47 PM Pavel-IO ***@***.***> wrote: Hi! I also thank you so much for implementing these tools in Python! I'm trying to use the function *modularity_und()* compatible with the original Matlab version, but I'm getting different results in bctpy and BCT Matlab version. Example of my input is here https://pastebin.com/KfzcqJmM. *Python version:* rho = numpy.loadtxt(my_file, float, delimiter=',') _, q = bct.modularity_und(rho) print(q) # 0.1498 *Matlab version:* rho = dlmread(myFile); [~, q] = modularity_und(rho); disp(q); % 0.1626 *Results:* In Python: *0.1498* In Matlab: *0.1626* Difference: *0.0128 is approximately 8 %*. This difference is quite essential in the subsequent group comparison (group t-test). The implementation in python gives me a significance level of p = 0.07, while the Matlab version returns p = 0.13. Different is also the resulting array of community structure (ci). If I pass the community structure calculated in Matlab as an argument to kci, then the result already is same with Matlab's version. rho = numpy.loadtxt(my_file, float, delimiter=',') kci = [ 2, 2, 1, 2, 1, 1, 2, 2, 1, 1, 2, 2, 2, 2, 1, 1, 2, 2, 2, 2, 1, 1, 2, 2, 1, 1, 2, 1, 3, 3, 3, 3, 2, 2, 2, 2, 4, 1, 1, 1, 1, 4, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 4, 4, 4, 4, 2, 2, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 3, 3, 4, 3, 3, 3, 3, 3, 3, 2, 1, 1, 4, 1, 1, 1, 1, 1, 4, 4, 1, 1, 4, 4, 4, 4, 4, 4, 1, 1, 1, 1, 4, 1, 1, 1, 4, 4, 3, 4, 4, 4, 4, 4 ] _, q = bct.modularity_und(rho, kci=kci) print(q) # 0.1626 I noticed a comment in modularity_test.py: *"matlab and bctpy appear to return different results due to the cross-package numerical instability of eigendecompositions"*, but is it possible that such a large difference is the result of numerical instability? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#93>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAET2HS3GOCB2CYVJPLCZ4DSXI55ZANCNFSM4VNTBCNQ> .

[Pavel-IO]

Hi aestrivex,
thank you for your help and confirmation. Mistakes happen sometimes, so I asked.

I agree that from an absolute point of view the difference is small, but to feel some certainty I wanted to get to the same numbers. Finally, I used modularity_finetune_und() and its matlab variant. Both implementations now give the same results.

Thanks again for porting BCT to Python. It's really useful.