Test accuracy is very low
ChenMiya opened this issue ยท 21 comments
hello,I choose attention_pooling model when I run main.py. But I see a great difference from your results.And the train accuracy is so high, the test accuracy is so low.
My result in attention pooling model as follows:
Epoch: 1 Train: 28.94% Test: 42.63%
Epoch: 10 Train: 67.42% Test: 54.52%
Epoch: 50 Train: 92.29% Test: 55.44%
Epoch: 100 Train: 94.67% Test: 53.89%
Looking forward to your reply.
I can't reproduce the results of the paper, maybe the code is not right. I don't know what's wrong.
@FrankWork my version which has achieved acnn is using pytorch, some data handling code copy from yours. maybe you could see it
@FrankWork thanks for your recommendation
@lawlietAi What is your test accuracy?
that is the reason i didn't put it on README
@FrankWork i think the original paper didn't use test data but use the eval data which divided from train data
Is there anyone actually obtaining the results from the original paper? I couldn't reach the performance specified in the paper.
@lawlietAi I'm not familiar with pytorch while I'm implementing the paper using tensorflow. And I'm stuck on a specific problem. In original paper, the author views an entity as a word and calculate a similarity score between them. But in fact, there're entities composed of several words. How do you deal with that? how to calculate the score in such situation?
sorry, i could not understand certainly on โthemโ. ๐๐
@lawlietAi Sorry. In input attention layer, the author calculate a score f between word in entity and word in sentence based on inner product. The question is how to calculate the score if there're several words in an entity.
@GatsbyUSTC If an entity contains several words, we just use the last word to represent the entity. It is a common practice.
The question is how to calculate the score if there're several words in an entity.
@GatsbyUSTC In some papers, mean embeddings of those words are often used, too.
some reference
Recurrent Attention Network on Memory for Aspect Sentiment Analysis
Aspect Level Sentiment Classification with Deep Memory Network
Effective LSTMs for Target-Dependent Sentiment Classification
@wangtianyiftd Thanks very much. Could I ask you another question since you seem to read lots of papers about attention. The attention mechanism works well on translation model. A word can focus on several words from another language based on attention. But do you think the attention mechanism will work on relation classification problem? Inner product means euclidean distance between embeddings. when we do relation classification, I don't think more attention on words which are close to entities will help solving the relation classification. I didn't get a better result after adding attention mechanism on cnn model, and I don't know the problem comes from the attention or hyperparameter. It will be nice if you can answer this question.
@GatsbyUSTC Attention on words try to find the most relevant words related to the entity pair, of course it will help. Take the relation "President" as an example, the word 'president' itself contribute a lot to the classification results. In theory, attention mechanism will assign a large weight to the word 'president'.
Although we can't reproduce the results of this paper, we can't deny the entire attention mechanism. Maybe you can reproduce other attention based paper.
attention is not almighty. but it can let network get the keywords from articles. this paper used 2 attention layers to increase the depth of nn, but att reduced information a lot. maybe it is the reason of bad performance
in addition, maybe this paper gave us a way to process long articles. i would do more experiment further
@FrankWork I did not deny the entire attention mechanism. The question is how can the attention learn to find the most relevant words on relation classification task. The attention mechanism in input layer proposed by the author just compute a inner product between entity and words so that words that are similar to entity will be paid more attention. Does it really help relation classification? Take "The school master teaches the lesson with a stick" for example, keyword for relation classification should be "with" since the correct relation is Instrument-Agency. Word "with" is relevant to the relation classification but it doesn't seem to be similar with "master" or "stick" which means we can't get a high attention score based on inner product. I just want to know the function of the attention mechanism here. Of course I know in some applications the attention mechanism will help, but they also get a good reason.
@GatsbyUSTC The inner product does not guarantee that the two words must be similar semantically. It can also be interpreted as relevance since the embeddings are learned. "with + n." is common in English, why can't they have higher inner product?
@GatsbyUSTC I agree with you. If we are using pretrained embeddings and not tuning them during training, an attention at the input layer would be meaningless. However, if embeddings are randomly initialized and updated while training, the attention would be useful to choose words which are key indicators for a class. At the same time, this would require a considerably larger training set.
@ChenMiya ๆๆณ้ฎไธไธไฝ ็ฐๅจๆพๅฐไฝ ๆต่ฏ้็ๅ็กฎ็็นๅซไฝ็ๅๅ ไบๅ๏ผ
@FrankWork i think the original paper didn't use test data but use the eval data which divided from train data
You codes don't have the train or test data,So how to run your codes?