frankmcsherry/dynamic-datalog

Add reference output

ryzhyk opened this issue · 3 comments

ryzhyk commented

Any chance we could ask benchmark developers to add reference output for each of the three benchmarks? If this is difficult, then at least knowing the size of output relations would help with debugging.

frankmcsherry commented

Can do. The doop and galen ones have clear output sizes. The crdt one is more awkward in that it only produces the "winning edit locations" not the characters that are there, but I'll put something together.

In the cases that Souffle runs (doop and galen) I confirmed that it produced the same numbers of output tuples.

frankmcsherry commented

I think this is at least partially addressed with 17a8b1f. Let me know what you think.

frankmcsherry commented

More detail:

  • CRDT: Martin, Michael, and Mistral went over the outputs with me in person in Cambridge, and we validated that it was producing the right results on smaller problems. I don't think we have a reference output for the provided problem. DD numbers reported, and independent reproduction (or not) would teach us something.

  • DOOP: Soufflé and DD produce the same cardinality outputs.

  • GALEN: Soufflé and DD produce the same cardinality outputs, and the same cardinality for p as a reference solution provided by the John Liagouris.