Re-train PLAsTiCC model with 'weighted' Log-Loss metric and loss for optimisation
tallamjr opened this issue · 2 comments
Previous models have been optimised using the following line in astronet/t2/opt/hypertrain.py:118
# Evaluate the model accuracy on the validation set.
score = model.evaluate(X_val, y_val, verbose=0)
return score[1]
However, it would be desirable to optimise this using the metric defined by AI. Malz et. al of a weighted Log-Loss
An example implementation of this can be found in astrorapids
or perhaps with sklearn.metrics.log_loss
akin to:
def plasticc_log_loss(y_true, probs):
"""Implementation of weighted log loss used for the Kaggle challenge.
Parameters
----------
y_true: np.array of shape (# samples,)
Array of the true classes
probs : np.array of shape (# samples, # features)
Class probabilities for each sample. The order of the classes corresponds to
that in the attribute `classes_` of the classifier used.
Returns
-------
float
Weighted log loss used for the Kaggle challenge
"""
predictions = probs.copy()
labels = np.unique(y_true) # assumes the probabilities are also ordered in the same way
weights_dict = {6:1/18, 15:1/9, 16:1/18, 42:1/18, 52:1/18, 53:1/18, 62:1/18, 64:1/9,
65:1/18, 67:1/18, 88:1/18, 90:1/18, 92:1/18, 95:1/18, 99:1/19,
1:1/18, 2:1/18, 3:1/18}
# sanitize predictions
epsilon = sys.float_info.epsilon # this is machine dependent but essentially prevents log(0)
predictions = np.clip(predictions, epsilon, 1.0 - epsilon)
predictions = predictions / np.sum(predictions, axis=1)[:, np.newaxis]
predictions = np.log(predictions) # logarithm because we want a log loss
class_logloss, weights = [], [] # initialize the classes logloss and weights
for i in range(np.shape(predictions)[1]): # run for each class
current_label = labels[i]
result = np.average(predictions[y_true==current_label, i]) # only those events are from that class
class_logloss.append(result)
weights.append(weights_dict[current_label])
return -1 * np.average(class_logloss, weights=weights)
There, the above code in astronet/t2/opt/hypertrain.py:118
would be something like follows:
# Evaluate weighted Log Loss on the validation set.
probs = model.predict(X_val)
logloss = plasticc_log_loss(y_val, probs)
return -logloss # Minus since we want to maximise the objective
And then in astronet/t2/train.py
the model.compile( .. )
would need something like:
model.compile(
loss="categorical_crossentropy", optimizer=optimizers.Adam(lr=lr), metrics=[plasticc_log_loss]
)
And then in astronet/t2/train.py the model.compile( .. ) would need something like:
model.compile(
loss="categorical_crossentropy", optimizer=optimizers.Adam(lr=lr), metrics=[plasticc_log_loss]
)
Should really be:
loss = plasticc_log_loss
...
model.compile(
loss=loss, optimizer=optimizers.Adam(lr=lr), metrics=['auc']
)
Whilst it is now optimising using the loss function of "categorical_crossentropy"
with 3b7abb8, a "plasticc" implementation is not used just yet as it requires a tensorflow mathematical operation implementation as opposed to a numpy one due to this error:
Error:
NotImplementedError: Cannot convert a symbolic Tensor (IteratorGetNext:1) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported
See 7312f78.
Whilst this is possible with tf-nightly
, currently there is a bug that is preventing using it at the moment. Leaving this issue open until this is fixed and a full plasticc-tensorflow version of LogLoss implemented