trec_eval is the standard tool used by the TREC community for
evaluating an ad hoc retrieval run, given the results file and a
standard set of judged results.
------------------------------------------------------------------------------
Installation: Should be as easy as typing "make" in the source directory.
If you wish the trec_eval binary to be placed in a standard location, alter
the first line of Makefile appropriately.
------------------------------------------------------------------------------
Testing: sample input and output files are included in the directory test.
"make quicktest" will perform some sample simple evaluations and compare
the results.
------------------------------------------------------------------------------
Usage: Most options can be ignored. The only one most folks will need
is the "-q" flag, to indicate whether to output official results for individual
queries as well as the averages over all queries. Official TREC usage
might be something like
trec_eval -q -c -M1000 official_qrels submitted_results
to ensure correct evaluation if submitted_results doesn't have results
for all queries, or returns more than 1000 documents per query.
If you wish to output only one particular measure:
trec_eval -m measure[.params] rel_info_file results_file
------------------------------------------------------------------------------
Change Log (only recent)
------------------------------------------------------------------------------
12/31/08
comments and documentation of Zscore file format corrected.
trec_eval.c get_zscores.c
2/25/08 Version 9.0alpha.
Complete rewrite of entire trec_eval (needed for a long time!).
Complete separation of individual measure calculations -
computers are now fast enough so can afford recalculation of lots
of intermediate values.
Should be much easier to add measures to, and much easier to add
new input file formats with associated measures.
Parameters for measures (eg, cutoffs for P) can be specified on the
command line.
Choice of measures can be specifed on the command line.
An initial set of preference evaluation measures (with their own
input rel_info format) have been added.
Help now gives targeted measure and format descriptions. Try
trec_eval -h -m all_prefs -R prefs
to get info on preference measures and formats, for instance.
All internal calculations are in double rather than float. Yields
minor variations in output at rare times; mostly when going from a
double percentage to a corresponding doc cutoff (eg, in iprec_at_recall).
All globally known procedure names or variables now begin with 'te_'
to allow incorporation of procedures in other programs.
Measures added:
ndcg, ndcg_cut, set_F, success, map_avgjg, P_avgjg,
various preference evaluation measures.
Measures renamed:
Rprec-mult_* was *R-prec
set_P was exact_prec
set_recall was exact_recall
set_relative_P was exact_relative_prec
set_recall was exact_recall
set_map was exact_unranked_avg_prec
gm_map was gm_ap
11pt_avg was 11-pt_avg
P_* was P*
recall_* was recall*
relative_P_* was relative_P*
iprec_at_recall_* was ircl_prn.*
Measures dropped for now:
3-pt_avg
avg_doc_prec
avg_relative_prec
exact_relative_unranked_avg_prec
map_at_R
int_map
exact_int_R_rcl_prec
int_map_at_R
unranked_avg_prec*
relative_unranked_avg_prec*
rcl_at_142_nonrel
fallout_recall_*
int_*R-prec
micro_prec
micro_recall
micro_bpref
bpref variants
time base measures.
Input formats added:
prefs - allows expression of preferences
qrels_prefs - same as standard qrels, except treated as prefs
qrels_jg - same as standard qrels, except allows judgment sets
from multiple users (judgment groups).
Version 8.1, Added infAP, minor bug fixes
7/24/06 Improved infAP comments (implementation verified by Yilmaz).
trec_eval_help.c: allow longer measure explanations.
6/27/06 get_opt.c Fixed error message
6/22/06 Added measure infAP (Aslam et al) to allow judging only sample
of pools. -1 for rel in qrels file interpreted as pool doc not judged.
6/22/06 trvec_teval.c: fixed bugs in calculation of bpref if multiple
relevance levels were used and a non-default relevance level
was given. (Eg. A doc with rel level of 2 was counted as unjudged
rather than judged nonrel if a relevance level of 3 was needed
to consider relevant.)
4/5/06 Changed comments in README, trec_eval.c, trec_eval_help.c files
which incorrectly claimed queries with no relevant docs are
ignored (this was true with very old versions of trec_eval). Now
reads that queries with no relevance information are ignored.
Giorgio Di Nunzio and Nicola Ferro,
------------------------------------------------------------------------------
Version 8.0, full bpref bug fix, see file bpref_bug. I decided to up the
version number since bpref results are incompatible with previous
results (though the changes are small).
------------------------------------------------------------------------------
------------------------------------------------------------------------------
Files:
Main procedure:
trec_eval.c
---
Procedures to read input files of various formats:
formats.c Mapping names of input file formats to input procedures
get_qrels.c Read the standard judged documents (qrels format)
get_qrels_jg.c Read qrels format with multiple judgment groups per query
get_prefs.c Read preferences judgments instead of doc judgments - see
formats.c for full description.
get_qrels_prefs.c Read qrels_jg format file, interpret as prefs file.
get_trec_results.c Read the standard result file (trec_results format).
---
Procedures to merge rel_info and results from input form into form that measures
can easily use, if they wish:
form_res_rels.c 'qrels' and 'trec_results' into RES_RELS format.
form_res_rels_jg.c 'qrels_jg' and 'trec_results' into RES_RELS_JG format.
form_pref_counts.c ('prefs' or 'qrels_prefs') and 'trec_results' format
---
The actual measures:
measures.c Associates measure name with parameters and
init, calculation, accumulation, printing procedures
meas_*.c Common procedures used by many measures for init, acc, printing.
m_<measure_name>.c measure specific procedures
---
Miscellaneous:
Makefile Compile and test trec_eval
README This file
CHANGELOG Recent changes
test Directory of collection of sample input and output for trec_eval
trec_eval.h Basic evaluation structures.
functions.h Prototype decorations of measure procedures.
sysfunc.h
common.h
bpref_bug: Description of bug in bpref that existed in trec_eval versions 6
through 7.3.
------------------------------------------------------------------------------
------------------------------------------------------------------------------
Adding a new measure.
Assuming it uses standard input files:
1. In m_<new_measure>.c, write any needed measure specific procedures needed to
initialize measure
calculate measure
accumulate measure (adding one topic's value to summary value over topics)
calculate the ending average for a measure.
print single
print final query measure value (and cleanup if needed)
Most measures require only a new calculate measure procedure - the
other procedures are generic and already implemented depending on the type of
measure (has cutoffs and value for each cutoff, parameters, is a float, etc).
See functions.h to see fit for these generic procedures.
2. In same file, construct TREC_MEAS te_meas* entry pointing to above
procedures and any default cutoffs or parameters.
3. Add pointer to that TREC_MEAS entry in "measures.c"
4. Add measure source file to Makefile
------------------------------------------------------------------------------
Adding a new file format
1. Implement reading of new format in get_<new_format>.c, with returned top
level output of type ALL_REL_INFO or ALL_RESULTS. The individual topic
returned values will be in a format dependent form which will be passed
to the appropriate measures.
2. Add format to appropriate format list in formats.c
3. Add measures to take advantage of format (see above)
4. To use, invoke trec_eval with -R or -T values, and -m measures that
are appropriate.