AskNowQA/LC-QuAD

Enable automatic benchmarking

geraltofrivia opened this issue · 3 comments

geraltofrivia commented

Is your feature request related to a problem? Please describe.
Need a script to seamlessly use a QA solution, and benchmark its results over LC-QuAD

Describe the solution you'd like

  • Require them to open an API which returns answers or SPARQLS given a question.
  • Use over-arching platforms like Project Hobbit, but this requires dockerized systems and other added constraints.
  • What else?
geraltofrivia commented

Requesting suggestions from @mohnish-rygbee @debayan @saist1993

RicardoUsbeck commented

Hi, have you advanced in this point or do are you still searching for suggestions?

RicardoUsbeck commented

I will just leave a link here in case I forget this topic. https://github.com/dice-group/TextComparisonEvaluationTool/blob/43c263d905f8b05071c6034d484f9fe199ba2e8a/src/main/java/Engines/Run/Pipeline.java#L210 Source code from this link can be reused to call GERBIL/GERBILQA automatically from code. That is, all evaluation of a system (provided it has a webservice) is done via the GERBIL platform, the results are archived there but you can get all the results as JSON-LD as well as a citable URI.