Amp parameter issue

[lijinchang-shazhu]

When I run the training command to add the amp parameter for training, there is a NaN issue, and the specific error is as follows:

Warning: NaN values detected and removed from input data.
Traceback (most recent call last):
  File "/mnt/data0/LJC/Classification/MHIM-MIL/main.py", line 947, in <module>
    main(args=args)
  File "/mnt/data0/LJC/Classification/MHIM-MIL/main.py", line 87, in main
    ckc_metric = one_fold(args,k,ckc_metric,train_p, train_l, test_p, test_l,val_p,val_l)
  File "/mnt/data0/LJC/Classification/MHIM-MIL/main.py", line 388, in one_fold
    stop,accuracy, auc_value, precision, recall, fscore, test_loss, threshold_optimal = val_loop(args,model,val_loader,device,criteon,early_stopping,epoch,model_tea)
  File "/mnt/data0/LJC/Classification/MHIM-MIL/main.py", line 746, in val_loop
    accuracy, auc_value, precision, recall, fscore, threshold_optimal = five_scores(bag_labels, bag_logit)
  File "/mnt/data0/LJC/Classification/MHIM-MIL/utils.py", line 118, in five_scores
    fpr, tpr, threshold = roc_curve(bag_labels, bag_predictions, pos_label=1)
  File "/home/jw102/.conda/envs/mhim/lib/python3.9/site-packages/sklearn/utils/_param_validation.py", line 213, in wrapper
    return func(*args, **kwargs)
  File "/home/jw102/.conda/envs/mhim/lib/python3.9/site-packages/sklearn/metrics/_ranking.py", line 1108, in roc_curve
    fps, tps, thresholds = _binary_clf_curve(
  File "/home/jw102/.conda/envs/mhim/lib/python3.9/site-packages/sklearn/metrics/_ranking.py", line 855, in _binary_clf_curve
    tps = stable_cumsum(y_true * weight)[threshold_idxs]
  File "/home/jw102/.conda/envs/mhim/lib/python3.9/site-packages/sklearn/utils/extmath.py", line 1231, in stable_cumsum
    out.take(-1, axis=axis), expected, rtol=rtol, atol=atol, equal_nan=True
IndexError: cannot do a non-empty take from an empty axes.

[DearCaat]

In fact, the amp parameter of this script has not been tested, and none of the experiments applied this parameter. I may not be able to provide further assistance regarding amp-related issues. I apologize for any inconvenience. I have disabled the amp parameter option in the script.

[lijinchang-shazhu]

Okay, got it. Thank you for your work, as well as for your response and assistance. Thank you very much again.

[DearCaat]

Best wish for u!