Do I just need to run the code by scripts？
Closed this issue · 8 comments
thanks you for your great works!
now I have a question that I run the scripts to get the results, but I got a great gap with the result in the paper . I think maybe something wrong in my progress, does it have the relationship with the environment of colab?
thanks
Thanks for your interest. One possible reason is the downstream task. e.g., classifying activity types or users. What's your command to run the classifier.py?
one is benchmark Deepsense on hhar.
the other one is classifier_bert on hhar.
all downstream are all har.
the result is about 74%.
tomorrow I will paste the details.
thanks for your quick replay！
python classifier_bert.py v1_v2 hhar 20_120 -f hhar -s limu_gru_v1 -l 0
Train Size: 43992, Vali Size: 5496, Test Size: 5508
Balance Label Size: 432, Unlabel Size: 43560; Real Label Rate: 0.010
cpu (0 GPUs)
Loading the model from saved/pretrain_base_hhar_20_120/hhar
The Total Epoch has been reached.
Best Accuracy: 0.995/0.769/0.765, F1: 0.995/0.763/0.758
Is there any error in the scripts？Thanks!
ps： Sorry， maybe I found an error that -l is not 0, in hhar it is 2?
And I finised the results is much better!
python classifier_bert.py v1_v2 hhar 20_120 -f hhar -s limu_gru_v1 -l 2
Train Size: 43975, Vali Size: 5495, Test Size: 5507
Balance Label Size: 438, Unlabel Size: 43537; Real Label Rate: 0.010
cuda:0 (1 GPUs)
The Total Epoch have been reached.
Best Accuracy: 0.995/0.918/0.919, F1: 0.995/0.906/0.908
I have another question when in my paper should I use the results in the article of LIMU-Bert or the results above?
They have an gap of about 5 points.
Thanks!
Have you tried the pretrained models provided in the saved folder? I just tested it and got the following results for hhar:
Best Accuracy: 0.998/0.958/0.955, F1: 0.998/0.953/0.953.
If you use the same setting as that in our paper, please give the above result or paper result. However, if you have other settings, e.g., different labeling rates, and methods of partition, feel free to report any actual result from our codes.
I just use the pre-trained models provided in the saved folder, in the result above it printed: Loading the model from saved/pretrain_base_hhar_20_120/hhar, there is a file called hhar.pt, is it right?
python classifier_bert.py v1_v2 hhar 20_120 -f hhar -s limu_gru_v1 -l 2, this command use this pre-trained model.
And if I want to use hhar dataset and the pre-trained model the command above have any error? I want have the same result as the paper of LIMU-Bert all the same.
Thanks!
It is not classifier_bert.py, please use classifier.py. The classifier_bert.py is to train the classifier by freezing the encoder. The command is the same.
python -u classifier.py v2 hhar 20_120 -f hhar -s limu_gru_v1 -l 2
Train Size: 43975, Vali Size: 5495, Test Size: 5507
Balance Label Size: 438, Unlabel Size: 43537; Real Label Rate: 0.010
cuda:0 (1 GPUs)
Best Accuracy: 0.995/0.962/0.967, F1: 0.995/0.958/0.965
Yeah, there's a great result!
But this result is generated by two steps： first, use LIMU-Bert to generate Embedding， second use the Embeddings to train the classifier， two parts can be seen as two individual models, when training the model only one part can update the parameters, and could not update params simultaneity.
Why not train the two parts together? Just because of the results? Or other reasons?
Yeah, you can train them together. The two steps are for improved efficiency. 1) To evaluate the performance of LIMU-BERT on different tasks, we are able to reuse the embeddings in the first step with such a design. 2) We do need to infer LIMU-BERT in the second step, which accelerates the classifier training process.