/StructTempRel-EMNLP17

Primary LanguageJava

Prerequisites

  • Have maven installed in your system
  • Run sh scripts/mvn_install.sh in your terminal to install the jar dependencies
  • Have Gurobi6.5.2 installed in your system and have env var GUROBI_HOME setup in your path. Please contact the author (qning2@illinois.edu) if you cannot find the official package for version 6.5.2. Note that if you only need the system output files, you can move forward without it.

Reproduce EMNLP Results

All the following cmd should be run from the root dir of the project, i.e., StructTempRel-EMNLP17/.

  • sh scripts/emnlp17_table3_line5.sh to generate the numbers on line 5 (SP+ILP), table 3.
    • The numbers on line 5 can be found at the end of logs/evaluation_process_20_0.01_0.0.txt (here it's slightly better than the reported values in the paper due to some minor changes). 
      === Temporal Awareness Score ===
Temporal Score  F1  P R
67.5557 69.7597 65.4867 
Overall Temporal Awareness Score (F1 score): 67.5557
    • The resulting .tml files can be found in output/pred_20_0.01_0.0/. The numbers above were generated by comparing these .tml files to the gold .tml files in data/TempEval3/Evaluation/te3-platinum/ using the TempEval-3 official evaluation script.
    • To get exactly the same results as in the paper, we have included the original system .tml files that generated those values, which you can find in output/pred_table3_SP+ILP. To evaluate it, run sh scripts/evaluate_test.sh table3_SP+ILP and go to the end of logs/evaluation_process_table3_SP+ILP, where you should see: 
      === Temporal Awareness Score ===
Temporal Score  F1  P R
67.2406 69.0909 65.4867 
Overall Temporal Awareness Score (F1 score): 67.240
  • sh scripts/emnlp17_table4_line6.sh to generate the numbers on line 6, table 4.
    • The numbers on line 6 can be found at the end of logs/evaluation_process_0.6_1.4_1.txt. 
      === Temporal Awareness Score ===
Temporal Score  F1  P R
40.2624 35.5236 46.4602 
Overall Temporal Awareness Score (F1 score): 40.2624
    • The system output .tml files can be found in output/pred_0.6_1.4_1/. The numbers above were generated by comparing these .tml files to the gold .tml files in data/TempEval3/Evaluation/te3-platinum/ using the TempEval-3 official evaluation script. If you simply want to evaluate it, run 
      sh scripts/evaluate_test.sh 0.6_1.4_1
    • Note: What emnlp17_table4_line6.sh does is to (1) (re)generate those .tml files in output/pred_0.6_1.4_1/ and (2) run evaluate_test.sh to evaluate it. Therefore, you can also skip the generation step and run sh scripts/evaluate_test.sh 0.6_1.4_1 directly. This situation is the same as "line 13" below.
  • sh scripts/emnlp17_table4_line13.sh to generate the numbers on line 13, table 4.
    • The numbers on line 13 can be found at the end of logs/evaluation_process_chambers_codl_tdtest_0.3_1.4_1.txt. 
      === Temporal Awareness Score ===
Temporal Score  F1  P R
48.5253 45.567  51.8944 
Overall Temporal Awareness Score (F1 score): 48.5253
    • The resulting .tml files can be found in output/Chambers/codl/TD-Test/0.3_1.4_1/. The numbers above were generated by comparing these .tml files to the gold .tml files in output/Chambers/gold/ using the TempEval-3 official evaluation script. If you simply want to evaluate it, run 
      sh scripts/evaluate_general.sh ./output/Chambers/gold ./output/Chambers/codl/TD-Test/0.3_1.4_1/ chambers_codl_tdtest_0.3_1.4_1

Citation

Qiang Ning, Zhili Feng, Dan Roth, "A structured learning approach to temporal relation extraction", EMNLP 2017 (paper, slides (pdf), slides (ppt))

@inproceedings{NingFeRo17,
    author = {Qiang Ning and Zhili Feng and Dan Roth},
    title = {A Structured Learning Approach to Temporal Relation Extraction},
    booktitle = {EMNLP},
    month = {9},
    year = {2017},
    address = {Copenhagen, Denmark},
    url = "http://cogcomp.org/papers/NingFeRo17.pdf",
    funding = {AI2, IBM-ILLINOIS C3SR, DARPA, ARL},
}