Dropout used in paper?
Closed this issue · 2 comments
JohnGiorgi commented
Hi,
I am just wondering if you used dropout in your paper? If so, would you mind telling me where the dropout layer was applied and what dropout ratio was used?
Thanks in advance!
yuzhimanhua commented
Hi,
Yes, we use dropout in our model, and its ratio is fixed to 0.5. It is applied in char_emb, char_lstm, word_emb, and word_lstm. You can also refer to our forward() function.
def forward(self, forw_sentence, forw_position, back_sentence, back_position, word_seq, file_no, hidden=None):
self.set_batch_seq_size(forw_position)
#embedding layer
forw_emb = self.char_embeds(forw_sentence)
back_emb = self.char_embeds(back_sentence)
#dropout
d_f_emb = self.dropout(forw_emb)
d_b_emb = self.dropout(back_emb)
#forward the whole sequence
forw_lstm_out, _ = self.forw_char_lstm(d_f_emb)#seq_len_char * batch * char_hidden_dim
back_lstm_out, _ = self.back_char_lstm(d_b_emb)#seq_len_char * batch * char_hidden_dim
#select predict point
forw_position = forw_position.unsqueeze(2).expand(self.word_seq_length, self.batch_size, self.char_hidden_dim)
select_forw_lstm_out = torch.gather(forw_lstm_out, 0, forw_position)
back_position = back_position.unsqueeze(2).expand(self.word_seq_length, self.batch_size, self.char_hidden_dim)
select_back_lstm_out = torch.gather(back_lstm_out, 0, back_position)
fb_lstm_out = self.dropout(torch.cat((select_forw_lstm_out, select_back_lstm_out), dim=2))
if self.if_highway:
char_out = self.fb2char(fb_lstm_out)
d_char_out = self.dropout(char_out)
else:
d_char_out = fb_lstm_out
#word
word_emb = self.word_embeds(word_seq)
d_word_emb = self.dropout(word_emb)
#combine
word_input = torch.cat((d_word_emb, d_char_out), dim = 2)
#word level lstm
lstm_out, _ = self.word_lstm(word_input)
d_lstm_out = self.dropout(lstm_out)
#convert to crf
crf_out = self.crflist[file_no](d_lstm_out)
crf_out = crf_out.view(self.word_seq_length, self.batch_size, self.tagset_size, self.tagset_size)
return crf_out
Thanks!
JohnGiorgi commented
Okay great, thanks for the quick response!