xiaojunxu/dnn-binary-code-similarity

training auc is equal validation auc?

Opened this issue · 0 comments

in line 140 of the train.py

        if (i % TEST_FREQ == 0):
            auc, fpr, tpr, thres = get_auc_epoch(gnn, Gs_train, classes_train,
                    BATCH_SIZE, load_data=valid_epoch)     #valid_epoch?
            gnn.say("Testing model: training auc = {0} @ {1}".format(
                auc, datetime.now()))
            auc, fpr, tpr, thres = get_auc_epoch(gnn, Gs_dev, classes_dev,
                    BATCH_SIZE, load_data=valid_epoch)
            gnn.say("Testing model: validation auc = {0} @ {1}".format(
                auc, datetime.now()))

partial results:

Initial training auc = 0.8385795480339078 @ 2021-01-14 10:49:18.571989
Initial validation auc = 0.8385795480339078 @ 2021-01-14 10:49:26.453895
EPOCH 1/100, loss = 0.6696747210700994 @ 2021-01-14 10:52:33.539342
Testing model: training auc = 0.9225966099742847 @ 2021-01-14 10:52:41.815191
Testing model: validation auc = 0.9225966099742847 @ 2021-01-14 10:52:49.352049
Model saved in ./saved_model/graphnn-model_best
EPOCH 2/100, loss = 0.6475605093730598 @ 2021-01-14 10:55:58.523499
Testing model: training auc = 0.9278334338892803 @ 2021-01-14 10:56:06.467269
Testing model: validation auc = 0.9278334338892803 @ 2021-01-14 10:56:14.051002

so should I change it as the following?

        if (i % TEST_FREQ == 0):
            auc, fpr, tpr, thres = get_auc_epoch(gnn, Gs_train, classes_train,
                    BATCH_SIZE, load_data=None) #change valid_epoch into None
            gnn.say("Testing model: training auc = {0} @ {1}".format(
                auc, datetime.now()))
            auc, fpr, tpr, thres = get_auc_epoch(gnn, Gs_dev, classes_dev,
                    BATCH_SIZE, load_data=valid_epoch)
            gnn.say("Testing model: validation auc = {0} @ {1}".format(
                auc, datetime.now()))