reproduce the results

[itsyoavshalev]

Hi, thanks a lot for sharing the code.

I tried reproducing the paper results with Recurrent-VLN-BERT and I got:
#####################################################
BEST RESULT TILL NOW: val_unseen | Iter 76000 iter 76000
val_seen | , nav_error: 3.3720, oracle_error: 2.2387, steps: 5.9481, lengths: 11.8580, success_rate: 0.7101, oracle_rate: 0.7640, spl: 0.6632
val_unseen | , nav_error: 3.6734, oracle_error: 2.3068, steps: 6.0681, lengths: 11.9416, success_rate: 0.6573, oracle_rate: 0.7395, spl: 0.6009
#####################################################

this is the script I used:

name=VLNBERT-train-GEO
flag="--vlnbert prevalent
--aug data/prevalent/prevalent_aug.json
--test_only 0
--train auglistener
--features clipresnet50_4
--maxAction 15
--batchSize 8
--feedback sample
--lr 1e-5
--iters 100000
--optim adamW
--mlWeight 0.20
--maxInput 80
--feature_size 640
--angleFeatSize 128
--featdropout 0.4
--dropout 0.5
--slot_attn
--slot_dropout 0.7
--slot_residual
--slot_local_mask
"
mkdir -p snap/$name
CUDA_VISIBLE_DEVICES=0 python r2r_src/train.py $flag --name $name

Can you please share the exact parameters for reproducing the paper results?

[jingyanghuo]

Hi, thank you for your interest in our work. Empirically, default settings like https://github.com/jingyanghuo/GeoVLN/blob/master/Recurrent-VLN-BERT/run/train_geo.bash give the best results more robustly.

Please try using the following parameter settings：

name=VLNBERT-train-GEO

flag="--vlnbert prevalent

  --aug data/prevalent/prevalent_aug.json
  --test_only 0
  
  --train auglistener

  --features clipresnet50_4
  --maxAction 15
  --batchSize 12
  --feedback sample
  --lr 1e-5
  --iters 300000
  --optim adamW

  --mlWeight 0.20
  --maxInput 80
  --feature_size 640
  --angleFeatSize 128
  --featdropout 0.4
  --dropout 0.5

  --slot_attn 
  --slot_dropout 0.7 
  --slot_residual 
  --slot_local_mask
  "

mkdir -p snap/$name
CUDA_VISIBLE_DEVICES=0 python r2r_src/train.py $flag --name $name

and the result with this setting is:
#####################################################
BEST RESULT TILL NOW: val_unseen | Iter 74000 iter 74000
val_seen | , nav_error: 3.1410, oracle_error: 2.2008, steps: 5.6141, lengths: 11.3989, success_rate: 0.7169, oracle_rate: 0.7581, spl: 0.6735
val_unseen | , nav_error: 3.5182, oracle_error: 2.3759, steps: 5.9523, lengths: 11.6563, success_rate: 0.6658, oracle_rate: 0.7280, spl: 0.6150
#####################################################