Please provide detailed parameters to reproduce the results in Table 2 and 3 in the paper

[hitercs]

Hi,

Thanks for your work.
I run your code with the default arguments by following your command in the readme.

Supervised Learning: python main.py --mode train --order offset --model_path model --method SL

Reinforcement Learning: python main.py --mode train --order offset --model_path model --method RL

After running for 500 epoches, I got results which is much lower than the paper's report results.
Specially, in SL setting:
best_aida_A_rlts [['aida-A', 0.9149358104581986], ['aida-B', 0.9279821627647714], ['msnbc', 0.9303749043611323], ['aquaint', 0.8433566433566434], ['ace2004', 0.8772635814889336], ['clueweb', 0.7220625224577794], ['wikipedia', 0.7024628355890836]]

In RL setting:
best_aida_A_rlts [['aida-A', 0.911178373864941], ['aida-B', 0.9208472686733556], ['msnbc', 0.9319051262433052], ['aquaint', 0.8657342657342657], ['ace2004', 0.8812877263581488], ['clueweb', 0.7107438016528925], ['wikipedia', 0.7360402337105243]]

The default order is offset, but in Fig.3 DCA-SL offset should be 94.35, while DCA-RL should be 93.70 on AIDA-B.

So could you please provide the full command including detailed arguments setting to reproduce the results?

Thanks a lot.
Looking forward to your reply.

[YoungXiyuan]

Thank you for your attention to our work. First, it is indeed our mistake to provide incomplete parameters, for that so many experiments in ablation study disturbs the original best parameter setting. Second, it seems that you just adopt our Berkeley-CNN baseline, because the provided results in E-mail are similar to our reported results of Berkeley-CNN + DCA-(SL/RL) in the paper. I would tell you how to switch to ETHZ-Attn baseline later. Best

…

------------------ Original ------------------ From: "Chen Shuang";<notifications@github.com>; Send time: Saturday, Oct 12, 2019 3:02 PM To: "YoungXiyuan/DCA"<DCA@noreply.github.com>; Cc: "Subscribed"<subscribed@noreply.github.com>; Subject: [YoungXiyuan/DCA] Please provide detailed parameters to reproduce theresults in Table 2 and 3 in the paper (#1) Hi, Thanks for your work. I run your code with the default arguments by following your command in the readme. Supervised Learning: python main.py --mode train --order offset --model_path model --method SL Reinforcement Learning: python main.py --mode train --order offset --model_path model --method RL After running for 500 epoches, I got results which is much lower than the paper's report results. Specially, in SL setting: best_aida_A_rlts [['aida-A', 0.9149358104581986], ['aida-B', 0.9279821627647714], ['msnbc', 0.9303749043611323], ['aquaint', 0.8433566433566434], ['ace2004', 0.8772635814889336], ['clueweb', 0.7220625224577794], ['wikipedia', 0.7024628355890836]] In RL setting: best_aida_A_rlts [['aida-A', 0.911178373864941], ['aida-B', 0.9208472686733556], ['msnbc', 0.9319051262433052], ['aquaint', 0.8657342657342657], ['ace2004', 0.8812877263581488], ['clueweb', 0.7107438016528925], ['wikipedia', 0.7360402337105243]] The default order is offset, but in Fig.3 DCA-SL offset should be 94.35, while DCA-RL should be 93.70 on AIDA-B. So could you please provide the full command including detailed arguments setting to reproduce the results? Thanks a lot. Looking forward to your reply. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[hitercs]

Thanks for your reply!
Waiting for your further update.
ps: please also add detailed parameters settings for your best results.

Thanks.

Best

[hitercs]

Hi,

Any updates on this Issue?

Thanks a lot.
Best

[StephanieWyt]

Hi,
I have the same problem with @hitercs about reproducing the reported best results.
Looking forward to your early reply.

Many thanks.
Best

[YoungXiyuan]

Hi, everyone. Sorry for later reply!

Now we are sure that your provided results is produced by our Berkeley-CNN baseline. (See Line # 125 and #295 in the mulrel_ranker.py)

We are trying to modify our original code, which would take one or two days.

Please excuse this clerical error.

[hitercs]

@YoungXiyuan
Thanks for your quick reply!
Got it. That's fine.

Best

[StephanieWyt]

@YoungXiyuan
Thanks for your quick response much appreciated.

Best

[YoungXiyuan]

Hi, everyone. We now update the "mulrel_ranker.py" file, and find that the default parameter setting in the "main.py" should be sufficient to reproduce results reported in our paper.

So please download the newly updated "mulrel_ranker.py" file and try again.

Best

[hitercs]

@YoungXiyuan , Great! Thanks.

Best

[hitercs]

I can reproduce the results now. Good work.
close this issue.