Second Recognising Textual Entailment Challenge

New! TAC 2008 Recognizing Textual Entailment (RTE) Track

Third Recognising Textual Entailment Challenge (RTE-3) Website

Workshop links

Proceedings
Program & Slides

PASCAL Challenges Workshops

Video & Photos

 

new! preprocessed datasets are available again here.

 

Development set download problem notification: if you downloaded the RTE-2 development set between 26/3/2007 (after RTE-3 results submission deadline), and 27/4/2008, please download it again from the datasets section, as it was found that the download link there temporarily pointed to an incorrect version of the development set (there was no problem with the test set).

 

Citation

When referring to the RTE initiative and the first RTE challenge, please use the following citation:
Ido Dagan, Oren Glickman and Bernardo Magnini. The PASCAL Recognising Textual Entailment Challenge. In Quiñonero-Candela, J.; Dagan, I.; Magnini, B.; d'Alché-Buc, F. (Eds.) Machine Learning Challenges. Lecture Notes in Computer Science , Vol. 3944, pp. 177-190, Springer, 2006.

When referring to the second RTE challenge, please use the following citation:
Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini and Idan Szpektor. 2006. The Second PASCAL Recognising Textual Entailment Challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, Venice, Italy.         

Recognising Textual Entailment (RTE) Challenges

Textual Entailment Recognition has been proposed recently as a generic task that captures major semantic inference needs across many natural language processing applications, such as Question Answering (QA), Information Retrieval (IR), Information Extraction (IE), and (multi) document summarization. This task requires to recognise, given two text fragments, whether the meaning of one text is entailed (can be inferred) from the other text.

The first PASCAL Recognising Textual Entailment Challenge (15 June 2004 - 10 April 2005, see website and proceedings) provided the first benchmark for the entailment task. The challenge raised noticeable attention in the research community, attracting 17 submissions from research groups worldwide. The relatively low accuracy achieved by the participating systems suggests that the entailment task is indeed a challenging one, with a wide room for improvement.

The Second PASCAL Recognising Textual Entailment Challenge (RTE-2)

By introducing a second challenge we hope to keep the momentum going, and to further promote the formation of a research community around the applied entailment task. As in the previous challenge, the main task is judging whether a hypothesis (H) is entailed by a text (T). One of the main goals for the RTE-2 dataset is to provide more "realistic" text-hypothesis examples, based mostly on outputs of actual systems. We focus on the four application settings mentioned above: QA, IR, IE and multi-document summarization. Each portion of the dataset includes typical T-H examples that correspond to success and failure cases of such applications. The examples represent different levels of entailment reasoning, such as lexical, syntactic, morphological and logical.

RTE-2 was organized by Bar-Ilan University (Israel), CELCT (Trento, Italy), Microsoft Research (USA) and MITRE (USA). Data collection and annotation processes were improved this year, including cross-annotation of the examples across the organizing sites.

The RTE-2 workshop (April 10, 2006) was held in Venice, Italy, as part of the Second PASCAL Challenge Workshop.

Program committee:

Johan Bos (University of Rome "La Sapienza")
Ido Dagan (Bar Ilan University)
Robert Dale (Macquarie University)
Bill Dolan (Microsoft Research)
Lisa Ferro (MITRE)
Bernardo Magnini (ITC-irst)
Chris Manning (Stanford University)
Rada Mihalcea (University of North Texas)
Dan Moldovan (University of Texas at Dallas)
Maarten De Rijke (University of Amsterdam)
Dan Roth (University of Illinois at Urbana-Champaign)
Nicola Stokes (University of Melbourne)
Fabio Zanzotto (University of Milan)

Schedule

Note: the workshop is scheduled right after EACL 2006.

Contact: Roy Bar-Haim <barhair@cs.biu.ac.il>