Dev branch version-10.0-dev not giving output for measurements
Santosh-Gupta opened this issue · 2 comments
I am trying to use to dev branch to get some measurements (RBP), which are not available in the master branch.
The dev branch was used to judge trec-covid ( mentioned here https://groups.google.com/g/trec-covid/c/m0EQxNqSfd4 ).
The instructions were
At the time (not sure if it has been fixed yet), the file "m_num_nonjudged_ret.c" had to be removed from the target MEAS_SRCS before that version of trec_eval would make.
I'm guessing the version Ellen used is still the same, as Trec-covid was judged from Aug5th to Aug-29th, and the last update for the dev branch was 3 months ago.
I tried looking for "m_num_nonjudged_ret.c" in the MEAS_SRCS section of Makefile but it doesn't seem to be there https://github.com/usnistgov/trec_eval/blob/version-10.0-dev/Makefile#L30
I just tried installing without that instruction, and it seemed to install fine, however, make quicktest seemed to have failed
Makefile:60: recipe for target 'quicktest' failed
make: *** [quicktest] Error 1
Here is a colab notebook of my attempt at getting the dev branch to work
https://colab.research.google.com/drive/1GGTLNgxxat8g1jDTSRkuFXlVlGcKSEH6?usp=sharing
You can interact with the code directly by selecting 'save copy to drive' on the top right.
For comparison, here's a version for the main branch, which seems to work just fine, but does not have the RBP metric
https://colab.research.google.com/drive/1PUYNSAWqLDmUrjaziFYEMzXXwN8j1n1h?usp=sharing
Reproduced on Ubuntu:
soboroff@node99:~/baz$ git clone https://github.com/usnistgov/trec_eval
Cloning into 'trec_eval'...
remote: Enumerating objects: 576, done.
remote: Total 576 (delta 0), reused 0 (delta 0), pack-reused 576
Receiving objects: 100% (576/576), 456.48 KiB | 6.43 MiB/s, done.
Resolving deltas: 100% (400/400), done.
soboroff@node99:~/baz$ cd trec_eval/
soboroff@node99:~/baz/trec_eval$ git fetch --all --tags
Fetching origin
soboroff@node99:~/baz/trec_eval$ git checkout version-10.0-dev
Branch 'version-10.0-dev' set up to track remote branch 'version-10.0-dev' from 'origin'.
Switched to a new branch 'version-10.0-dev'
soboroff@node99:~/baz/trec_eval$ make clean
/bin/rm -f *.o *.BAK *~ trec_eval trec_eval.*.tar out.trec_eval Makefile.bak
soboroff@node99:~/baz/trec_eval$ make
gcc -g -I. -Wall -Wno-macro-redefined -DVERSIONID=\"10.0-dev\" -o trec_eval trec_eval.c formats.c meas_init.c meas_acc.c meas_avg.c meas_print_single.c meas_print_final.c gain_init.c get_qrels.c get_trec_results.c get_prefs.c get_qrels_prefs.c get_qrels_jg.c form_res_rels.c form_res_rels_jg.c form_prefs_counts.c utility_pool.c get_zscores.c convert_zscores.c measures.c m_map.c m_P.c m_num_q.c m_num_ret.c m_num_rel.c m_num_rel_ret.c m_gm_map.c m_Rprec.c m_recip_rank.c m_bpref.c m_iprec_at_recall.c m_recall.c m_Rprec_mult.c m_utility.c m_11pt_avg.c m_ndcg.c m_ndcg_cut.c m_Rndcg.c m_ndcg_rel.c m_binG.c m_G.c m_rel_P.c m_success.c m_infap.c m_map_cut.c m_gm_bpref.c m_runid.c m_relstring.c m_set_P.c m_set_recall.c m_set_rel_P.c m_set_map.c m_set_F.c m_num_nonrel_judged_ret.c m_prefs_num_prefs_poss.c m_prefs_num_prefs_ful.c m_prefs_num_prefs_ful_ret.c m_prefs_simp.c m_prefs_pair.c m_prefs_avgjg.c m_prefs_avgjg_Rnonrel.c m_prefs_simp_ret.c m_prefs_pair_ret.c m_prefs_avgjg_ret.c m_prefs_avgjg_Rnonrel_ret.c m_prefs_simp_imp.c m_prefs_pair_imp.c m_prefs_avgjg_imp.c m_map_avgjg.c m_Rprec_mult_avgjg.c m_P_avgjg.c m_yaap.c m_rbp.c m_rbp_resid.c -lm
soboroff@node99:~/baz/trec_eval$ make quicktest
./trec_eval test/qrels.test test/results.test | diff - test/out.test
0a1,30
> runid all STANDARD
> num_q all 3
> num_ret all 1500
> num_rel all 561
> num_rel_ret all 131
> map all 0.1785
> gm_map all 0.1051
> Rprec all 0.2174
> bpref all 0.1981
> recip_rank all 0.4064
> iprec_at_recall_0.00 all 0.4665
> iprec_at_recall_0.10 all 0.3884
> iprec_at_recall_0.20 all 0.3186
> iprec_at_recall_0.30 all 0.2852
> iprec_at_recall_0.40 all 0.2666
> iprec_at_recall_0.50 all 0.2184
> iprec_at_recall_0.60 all 0.0822
> iprec_at_recall_0.70 all 0.0348
> iprec_at_recall_0.80 all 0.0312
> iprec_at_recall_0.90 all 0.0312
> iprec_at_recall_1.00 all 0.0312
> P_5 all 0.2667
> P_10 all 0.3000
> P_15 all 0.3111
> P_20 all 0.3667
> P_30 all 0.3333
> P_100 all 0.2467
> P_200 all 0.1600
> P_500 all 0.0873
> P_1000 all 0.0437
Segmentation fault (core dumped)
Makefile:60: recipe for target 'quicktest' failed
make: *** [quicktest] Error 1