Generative Loss is negative
Opened this issue · 0 comments
wkgreat commented
I use t2vec to train ads-b data, the dataset is from opensky, download link is opensky
the date of dataset I used is from 2022-05-02 to 2022-06-27.
The hyper-parameters are:
{
"region": {
"cityname": "opensky",
"minlon": 73.49901302691623,
"minlat": 3.83788851813507,
"maxlon": 135.08737696307114,
"maxlat": 53.5616571938264,
"cellsize": 100.0,
"minfreq": 20
}
}args are:
Namespace(data='E:/codes/github/t2vec/data', checkpoint='E:/codes/github/t2vec/data/checkpoint.pt', prefix='opensky', pretrained_embedding=None, num_layers=3, bidirectional=True, hidden_size=256, embedding_size=256, dropout=0.2, max_grad_norm=5.0, learning_rate=0.001, batch=128, generator_batch=32, t2vec_batch=256, start_iteration=0, epochs=15, print_freq=50, save_freq=1000, cuda=True, use_discriminative=False, discriminative_w=0.1, criterion_name='KLDIV', knearestvocabs='data/opensky-vocab-dist-cell100.h5', dist_decay_speed=0.8, max_num_line=20000000, max_length=200, mode=0, vocab_size=20000, bucketsize=[(20, 30), (30, 30), (30, 50), (50, 50), (50, 70), (70, 70), (70, 100), (100, 100)])
training logs are:
Iteration: 0 Generative Loss: 2.175 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 50 Generative Loss: 1.196 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 100 Generative Loss: 1.089 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 150 Generative Loss: 0.731 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 200 Generative Loss: 0.528 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 250 Generative Loss: 0.376 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 300 Generative Loss: 0.278 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 350 Generative Loss: 0.226 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 400 Generative Loss: 0.195 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 450 Generative Loss: 0.092 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 500 Generative Loss: 0.077 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 550 Generative Loss: 0.077 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 600 Generative Loss: 0.076 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 650 Generative Loss: 0.047 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 700 Generative Loss: 0.052 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 750 Generative Loss: -0.023 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 800 Generative Loss: -0.016 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 850 Generative Loss: -0.010 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 900 Generative Loss: -0.031 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 950 Generative Loss: -0.023 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 1000 Generative Loss: -0.021 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Saving the model at iteration 1000 validation loss 76.77668204471983
Iteration: 1050 Generative Loss: -0.057 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 1100 Generative Loss: -0.052 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 1150 Generative Loss: -0.050 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 1200 Generative Loss: -0.056 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 1250 Generative Loss: -0.060 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 1300 Generative Loss: -0.081 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 1350 Generative Loss: -0.056 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 1400 Generative Loss: 0.070 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 1450 Generative Loss: -0.063 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 1500 Generative Loss: -0.068 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 1550 Generative Loss: -0.095 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 1600 Generative Loss: -0.076 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 1650 Generative Loss: -0.073 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 1700 Generative Loss: -0.002 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 1750 Generative Loss: -0.066 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 1800 Generative Loss: -0.069 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 1850 Generative Loss: -0.073 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 1900 Generative Loss: -0.069 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 1950 Generative Loss: -0.073 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 2000 Generative Loss: -0.100 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Saving the model at iteration 2000 validation loss 81.64155062971444
Iteration: 2050 Generative Loss: -0.077 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 2100 Generative Loss: -0.069 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 2150 Generative Loss: -0.072 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 2200 Generative Loss: -0.079 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 2250 Generative Loss: -0.074 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 2300 Generative Loss: -0.073 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 2350 Generative Loss: -0.076 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 2400 Generative Loss: -0.071 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 2450 Generative Loss: -0.070 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 2500 Generative Loss: -0.088 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 2550 Generative Loss: -0.104 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 2600 Generative Loss: -0.091 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 2650 Generative Loss: -0.075 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 2700 Generative Loss: -0.083 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 2750 Generative Loss: -0.063 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 2800 Generative Loss: -0.069 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 2850 Generative Loss: -0.071 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 2900 Generative Loss: -0.075 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 2950 Generative Loss: -0.072 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Iteration: 3000 Generative Loss: -0.100 Discriminative Cross Loss: 0.000 Discriminative Inner Loss: 0.000
Saving the model at iteration 3000 validation loss 85.59408001077587
As we can see, from iteration 750, the generative loss becomes negative, and validation loss ascendes.
How can explains this result?