/Awesome-Embodied-Instruction-Following

A leaderboard for Embodied Instruction Following papers and BibTeX entries

Primary LanguageTeX

Awesome Embodied Instruction Following

This repository contains a leaderboard of the ALFRED benchmark and BibTeX entries (until Feb. 2024).

Low-level step-by-step instructions + High-level goal instructions

Method Test unseen Test seen
SR GC SR GC
Seq2Seq CVPR'20 0.39 (0.08) 7.03 (4.26) 3.98 (2.02) 9.42 (6.27)
MOCA ICCV'21 5.30 (2.72) 14.28 (9.99) 22.05 (15.10) 28.29 (22.05)
E.T. ICCV'21 8.57 (4.10) 18.56 (11.46) 38.42 (27.78) 45.44 (34.93)
LWIT IJCAI'21 8.37 (5.06) 19.13 (14.81) 29.16 (24.67) 38.82 (34.85)
LACMA EMNLP'23 9.2 20.1 32.4 40.5
HiTUT ACLF'21 13.87 (5.86) 20.31 (11.5) 21.27 (11.10) 29.97 (17.41)
ABP CVPRW'21 15.43 (1.08) 24.76 (2.22) 44.55 (3.88) 51.13 (4.92)
M-Track CVPR'22 16.29 (7.66) 22.60 (13.18) 24.79 (13.88) 33.35 (19.48)
LLM-Planner ICCV'23 16.42 23.37 18.20 26.77
MCR-Agent AAAI'24 17.04 (9.69) 30.13 (21.19)
MAT ICPR'22 21.84 (6.13) 32.41 (10.59) 33.01 (9.42) 43.65 (14.68)
AMSLAM IROS'22 23.48 (2.36) 34.64 (4.63) 29.48 (3.28) 40.88 (5.56)
Scene-LLM arXiv'24 25.15 33.75 26.52 37.09
FILM* ICLR'22 26.49 (10.55) 36.37 (14.30) 27.67 (11.23) 38.51 (15.06)
LGS-RPA RAL'22 35.41 (22.76) 45.24 (22.76) 40.05 (21.28) 48.66 (28.97)
Prompter* arXiv'22 45.32 (20.79) 56.57 (25.80) 51.17 (25.12) 60.22 (30.21)
CAPEAM ICCV'23 46.11 (19.45) 57.33 (24.06) 51.79 (21.60) 60.50 (25.88)
CAPEAM* ICCV'23 49.84 (22.61) 61.10 (27.00) 50.62 (22.61) 59.40 (27.49)
ThinkBot# arXiv'24 57.82 (26.93) 67.75 (30.73) 62.69 (32.02) 71.64 (37.01)
Human 91.00 (85.80) 94.50 (87.60)

High-level goal instructions only

Method Test unseen Test seen
SR GC SR GC
LAV arXiv'21 6.38 (3.12) 17.27 (10.47) 13.35 (6.31) 23.21 (13.18)
EmBERT arXiv'21 7.52 (3.58) 16.33 (10.42) 31.77 (23.41) 39.27 (31.32)
HiTUT ACLF'21 11.12 (4.50) 17.89 (9.77) 13.63 (5.57) 21.11 (11.00)
HLSM CoRL'22 20.27 (5.55) 30.31 (9.99) 29.94 (8.74) 41.21 (14.58)
FILM* ICLR'22 24.46 (9.67) 34.75 (13.13) 25.77 (10.39) 36.15 (14.17)
LGS-RPA RAL'22 27.80 (12.92) 38.55 (20.01) 33.01 (16.65) 41.71 (24.49)
EPA CVPRW'22 36.07 (2.92) 39.54 (3.91) 39.96 (2.56) 44.14 (3.47)
Prompter* arXiv'22 41.53 (18.84) 53.69 (24.20) 47.95 (23.29) 56.98 (28.42)
RoboGPT# arXiv'23 42.97 53.80 45.66 54.99
RoboGPT#* arXiv'23 44.57 55.01 49.45 58.39
CAPEAM ICCV'23 43.69 (17.64) 54.66 (22.76) 47.36 (19.03) 54.38 (23.78)
CAPEAM* ICCV'23 45.72 (20.15) 57.25 (24.73) 46.64 (20.81) 55.29 (25.47)
Human 91.00 (85.80) 94.50 (87.60)

The above methods are ranked by the SR on Test unseen

* for adopting templated action in FILM

# for very recent work