Human languages always involve the description of real-world events, and so do millions of videos on the Internet.
- ACE 2005: provides entity, value, time, relation, and event annotations for English, Chinese, and Arabic.
- REO: the Rich Event Ontology (REO) (Brown et al., 2017) unifies two existing knowledge resources (i.e., FrameNet (Fillmore et al., 2003) and VerbNet (Kipper et al., 2008)) and two event annotated datasets (i.e., ACE (Doddington et al., 2004) and ERE (Song et al., 2015)) to allow users to query multiple linguistic resources and combine event annotations.
- MAVEN: select 4,480 documents in total, covering 90 of the 95 major event topics defined in EventWiki.
- Lin et al., ACL'20: OneIE
- TimeML: a specification language for events and temporal expressions; EVENT, TIMEX3, SIGNAL, and LINK.
- 2012 i2b2 Challenge: The Sixth Informatics for Integrating Biology and the Bedside (i2b2) Natural Language Processing Challenge for Clinical Records focused on the temporal relations in clinical narratives.
- TB-Dense: 12,715 labeled relations over 36 TimeBank newspaper articles; TB-Dense has 1.1K verb events, between which 3.4K event-event (EE) relations are annotated (excerpt from MATRES paper); train / dev / test (doc: 22 / 5 / 9; pair: 4023 / 629 / 1427)
- RED: The Richer Event Description (RED) corpus presents 95 documents (totaling 54287 tokens) sampled both from news data and casual discussion forum interactions, which contain 8731 events, 1127 temporal expressions (TIMEX3s, section time, and document time labels), and 10320 entity markables. It contains 2390 identity chains, 1863 bridging relations, and 4969 event-event relations encompassing temporal, causal and subevent relations (as well as aspectual ALINK relations and reporting relations), as well as 8731 DOCTIMEREL temporal annotations linking these events to the document time.
- TCR: 25 newswire articles collected from CNN in 2010.
- MATRES: 275 documents; 72% of the events (0.8K) are anchored onto the main axis, resulting in 1.6K EE relations, and 16% (0.2K) are anchored onto orthogonal axes, resulting in 0.2K EE relations; train / dev / test (doc: 183 / 72 / 20; pair: 6332 / - / 827)
- McTaco: 13K tuples, in the form of (sentence, question, candidate answer)
- TORQUE: a new English reading comprehension benchmark built on 3.2k news snippets with 21k human-generated questions querying temporal relationships
- Pustejovsky et al., 2003; Chklovski and Pantel, 2004; Bethard, 2013; Llorens et al., 2010; D’Souza and Ng, 2013; Chambers et al., 2014
- Ning et al. (TCR), ACL'18: TCR (71.1 in F1)
- Ning et al. (MATRES), ACL'18: MATRES (69.0 in F1)
- Ning et al., EMNLP'19: MATRES (76.7 in F1), TCR (78.6 in F1)
- Han et al., EMNLP'19: MATRES (59.6 in F1, End-to-end), TB-Dense (49.4 in F1, End-to-end)
- Han et al., CoNLL'19: MATRES (81.7 in F1), TB-Dense (63.2 in F1), TCR (80.9 in F1)
- Han et al., EMNLP'20: TB-Dense (50.5 in F1, End-to-end), 2012 i2b2 Challenge (77.3 in F1, End-to-end)
- Yang et al., EMNLP'20 (Findings)
- Han et al., NAACL'21 Demo: TB-Dense (64.5 in F1), MATRES (75.5 in F1)
- Wen et al., NAACL'21 Demo: MATRES (78.8 in F1), MATRES-b (89.6 in Acc.)
- RED
- CaTeRs
- ESTER: five types of event semantic relations: CAUSAL, SUB-EVENT, CO-REFERENCE, CONDITIONAL and COUNTERFACTUAL
- Girju, 2003; Bethard and Martin, 2008; Riaz and Girju, 2010; Do et al., 2011; Riaz and Girju, 2013; Mirza and Tonelli, 2014, 2016
- Segers et al., LREC'18
- Ning et al., ACL'18: TCR (77.3 in F1)
- Gao et al., NAACL'19
- RED
- HiEve: randomly selected 100 documents from the GraphEve corpus, which contains gold-annotated event mentions; given a pair of event mentions, annotators were instructed to annotate one of the following relation types: SuperSub, SubSuper, Coref, NoRel
- IC: 100 texts in the IC domain; annotated only instances of full coreference, Subevent, and Member relations
- ESTER
- Araki et al., LREC'14: We used a corpus consisting of 65 newspaper articles in the IC domain.
- Badgett et al., EMNLP'16
- Yao et al., EMNLP'20
- Upadhyay et al., EVENTS WS'16
- Choubey et al., NAACL'18
- Jindal et al., COLING'20
- [Zhang et al., ongoing]
- Li et al., EMNLP'20: Event Graph Schema, where two event types are connected through multiple paths involving entities that fill important roles in a coherent story; introduce Path Language Model, an auto-regressive language model trained on event-event paths, and select salient and coherent paths to probabilistically construct these graph schemas.
- Zhang et al., EMNLP'20: leverages analogies among processes and conceptualization of sub-event instances to predict the whole sub-event sequence of previously unseen open-domain processes
- Dai et al., EMNLP'19: propose a regularization approach that tightly integratesthese constraints with contexts for deriving word representations
- Choubey et al., ACL'20
- Chen et al., AACL'20
- Chaturvedi et al., EMNLP'17: Event Prediction
- Peng et al., CoNLL'17:
- Peng et al., CoNLL'19: EventLM
- Zeng et al., TextGraphs'21: Event Network Embedding, aiming at representing events with low-dimensional and informative embeddings by incorporating neighboring events; Event Network constructed from one VOA news article, where events are connected through entities involved;
- Ning et al., NAACL'18
- Yao et al., ACL'18: explored narratology principles and built a weakly supervised approach that identifies 287k narrative paragraphs from three large text corpora; then extracted rich temporal event knowledge from these narrative paragraphs.
- Zhou et al., ACL'20
- Yao et al., EMNLP'20
- Choubey et al., EACL'21
- EventKG: A Multilingual Event-Centric Temporal Knowledge Graph
- ASER: Activities, States, Events, and their Relations
- EventWiki: EventWiki concentrate on major events, in which all entries in EventWiki are important events in mankind history.
- Li et al., ACL'20: Video M2E2
- Jin et al., IEEE Transactions on Multimedia'21
- Wen et al., NAACL'21 Demo: Video M2E2 (70.0 in Acc.)
- Finn et al., ICML'17
- Bansal et al., COLING'20
- Yin et al., arXiv preprint'20
- Deng et al., WSDM'20: propose a Dynamic-Memory-Based Prototypical Network (DMB-PN), which exploits Dynamic Memory Network (DMN) to not only learn better prototypes for event types, but also produce more robust sentence encodings for event mentions
- Han et al., EMNLP'20: propose a framework that enhances deep neural network with distributional constraints constructed by probabilistic domain knowledge. We solve the constrained inference problem via Lagrangian Relaxation and apply it to end-to-end event temporal relation extraction tasks
- Wang et al., EMNLP'20
- Chen et al., AACL'20: build graphs with candidate role filler extractions enriched by sentential embeddings as nodes, and use graph attention networks to identify event regions in a document and aggregate event information
- Wen et al., NAACL'21
- Zhou et al., ACL'20: proposes a novel sequence modeling approach that exploits explicit and implicit mentions of temporal common sense, extracted from a large corpus, to build TACOLM, a temporal common sense language model.
- He et al., ACL'20: QUASE learns representations from QA data, using BERT or other state-of-the-art contextual language models
- Dan Roth, Heng Ji, Ruihong Huang, Claire Cardie, Nanyun Peng, Chris Callison Burch, Martha Palmer, Eduard Hovy, Yadong Mu