qfo/benchmark-webservice

Data missing and provenance problems

Opened this issue · 0 comments

pmjklemm commented

Hey Adrian, I found some problems with the webservice and I have some suggestions:

  1. Some projects are missing in the public projects tab (https://orthology.benchmarkservice.org/proxy/projects/2020/)
    that are present in the public results tab (https://orthology.benchmarkservice.org/proxy/results/2020/)
    For example SonicParanoid2-sens, SonicParanoid2-... (only SonicParanoid is present)

  2. The names of the tools do not perfectly match (e.g. SonicParanoid_sensitive = sonicparanoid-sens) between the projects and results tab. Maybe crosslink them, so there is no confusion or use short IDs.

  3. Version numbers are missing for almost all projects (e.g. Domainoid+ has no tool version assigned).

  • idea: you could add this as a requirement field when submitting to the database
  1. Parameters are inconsistent, some projects are well described and others not.
  • idea: make the full command that generates the output a requirement to submit data, so that others can reproduce the results
  • I tried to reproduce the SonicParanoid-sens results with the description but got very different results, probably different versioning or some other post-processing is missing...
  • RBH/BBH: what is the program that generates these results? My RBH graph with 99% similarity and the specified e-value differs from this result in the benchmarks.

  1. It is clear to me that *-g represents a grouped upload, but maybe separate the results or make it clearer what is a paired upload and what is a grouped one.

  2. The 'EnsemblCompara-e56' has multiple entries and some of the data links go to your paper instead of the data uploaded for this result

Maybe this helps for the next qfo release!