Question about the evaluation method

Question

Question about the evaluation method

Opened this issue 6 years ago · 5 comments

Hi Jianshu! Is the result (ExpRate, <=1, <=2, <=3) in the WAP paper are gotten from the translate.py file? I found that you used the official tool to evaluate the results at expresssion level in the WAP paper. But in the released code, you have also offered a script 'test.sh' to evaluate the WER and ExpRate. Will the two evaluate methods above generate the same result? Actually, I don't know how to use the official tool of CROHME, so i'm wondering whether your script can replace the tool? If not, can you please tell me how you used the official tool in your experiments? Thanks! By the way, your paper and code are excellent and help me a lot!

Answer 1 · 2019-01-22T11:14:50.000Z

Thank you for your appreciate, you need to use the CROHMELib tool (http://saskatoon.cs.rit.edu:10001/root/crohmelib) to get the <=1, <=2, <=3 and structure recognition rate, the ExpRate can be computed by using the 'compute-wer.py'. Note that, our evaluations did not count for the segmentation errors, if you want to evaluate segmentation errors, you can choose to use the LgEval tool (http://saskatoon.cs.rit.edu:10001/root/lgeval/tree/master), which is more recent official tool for handwritten mathematical expression recognition.

…

在 2019年1月22日，上午1:42，Jomensy ***@***.***> 写道： Hi Jianshu! Is the result (ExpRate, <=1, <=2, <=3) in the WAP paper are gotten from the translate.py file? I found that you used the official tool to evaluate the results at expresssion level in the WAP paper. But in the released code, you have also offered a script 'test.sh' to evaluate the WER and ExpRate. Will the two evaluate methods above generate the same result? Actually, I don't know how to use the official tool of CROHME, so i'm wondering whether your script can replace the tool? If not, can you please tell me how you used the official tool in your experiments? Thanks! By the way, your paper and code are excellent and help me a lot! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#18>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ASbU0UQaGGuEoTYLUUs7d4SDYwVkDmYxks5vFmx-gaJpZM4aLxdb>.

Answer 2 · 2019-01-23T09:07:42.000Z

Thanks for your quick answer! I have downloaded the tools. But I still don't know how to use those tools. Could you please tell me how to use these tools to translate the 'valid_decode_result.txt' or 'test_decode_result.txt' to the results in your experiments? Thanks for your generous help!

Answer 3 · 2019-01-23T11:52:20.000Z

As for CROHMELib, you should first use ‘LaTeXML’ tool to convert LaTeX strings into pmml representations, then use CROHMELib tool to evaluate the pmml results. There are detailed instructions on CROHMELib official websites, you can check and follow them. You may also need to use the inkml training files of CROHME competition, which you can find on CROHME2014 official website.

…

在 2019年1月23日，上午9:07，Jomensy ***@***.***> 写道： Thanks for your quick answer! I have downloaded the tools. But I still don't know how to use those tools. Could you please tell me how to use these tools to translate the 'valid_decode_result.txt' or 'test_decode_result.txt' to the results in your experiments? Thanks for your generous help! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#18 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ASbU0VTL-3bcD3Nd6_64tQv4AC-xL9TIks5vGCZegaJpZM4aLxdb>.

Answer 4 · 2019-01-23T13:00:49.000Z

Thanks again! But I'm so sorry to bother you again. I have installed the LaTeXML. i used it to translate the LaTeX strings to pmml representations by this command latexmlmath --pmml=- \\frac{b\\pm\\sqrt{b^2-4ac}}{2a} and i got the result below:
<?xml version="1.0" encoding="UTF-8"?> <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="\frac{b\pm\sqrt{b^{2}-4ac}}{2a}" display="block"> <mfrac> <mrow> <mi>b</mi> <mo>±</mo> <msqrt> <mrow> <msup> <mi>b</mi> <mn>2</mn> </msup> <mo>-</mo> <mrow> <mn>4</mn> <mo>⁢</mo> <mi>a</mi> <mo>⁢</mo> <mi>c</mi> </mrow> </mrow> </msqrt> </mrow> <mrow> <mn>2</mn> <mo>⁢</mo> <mi>a</mi> </mrow> </mfrac> </math>
I'm not sure whether my command is correct. Can you tell me?
It seems that the pmml representation is the mathML tree for the math expression. What still confusing me is that how to translate the pmml representation to .inkml file (which is used by the script 'evalInkml_v1.XX.pl' to evaluate the results). Can you tell me which exact tool in CROHMELib you have used to transform the pmml representation into inkml file? And which is the tool to evaluate the results? Is it evalInkml_v1.XX.pl? The description in CROHMELib does not seem clear enough for me. Thanks for your help!

Answer 5 · 2019-01-24T11:22:55.000Z

You should first convert LaTeX string into pmml representation, then replace the structure in inkml file with your converted pmml structure and remove the segmentation information in inkml file. Finally, you can use the evalInkml_v1.12.pl to evaluate your newly generated inkml files.

…

在 2019年1月23日，下午1:00，Jomensy ***@***.***> 写道： Thanks again! But I'm so sorry to bother you again. I have installed the LaTeXML. i used it to translate the LaTeX strings to pmml representations by this command latexmlmath --pmml=- \\frac{b\\pm\\sqrt{b^2-4ac}}{2a} and i got the result below: <?xml version="1.0" encoding="UTF-8"?> <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="\frac{b\pm\sqrt{b^{2}-4ac}}{2a}" display="block"> <mfrac> <mrow> <mi>b</mi> <mo>±</mo> <msqrt> <mrow> <msup> <mi>b</mi> <mn>2</mn> </msup> <mo>-</mo> <mrow> <mn>4</mn> <mo>⁢</mo> <mi>a</mi> <mo>⁢</mo> <mi>c</mi> </mrow> </mrow> </msqrt> </mrow> <mrow> <mn>2</mn> <mo>⁢</mo> <mi>a</mi> </mrow> </mfrac> </math> I'm not sure whether my command is correct. Can you tell me? It seems that the pmml representation is the mathML tree for the math expression. What still confusing me is that how to translate the pmml representation to .inkml file (which is used by the script 'evalInkml_v1.XX.pl' to evaluate the results). Can you tell me which exact tool in CROHMELib you have used to transform the pmml representation into inkml file? And which is the tool to evaluate the results? Is it evalInkml_v1.XX.pl? The description in CROHMELib does not seem clear enough for me. Thanks for your help! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#18 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ASbU0anY5a55ELI7_Knq_p8wU99CQt3qks5vGF0BgaJpZM4aLxdb>.