broadinstitute/ml4h

issues running the `mnist_survival_analysis_demo.ipynb` notebook

amygdala opened this issue · 3 comments

I did a test run through:
https://github.com/broadinstitute/ml4h/blob/master/notebooks/mnist_survival_analysis_demo.ipynb
and hit a few glitches. (I was testing with TF 2.4.3 and TFP 0.12.2, but I don't think that's related to the errors I saw).

  • This line:
    # from ml4h.models import train_model_from_generators, make_multimodal_multitask_model, _inspect_model
    I believe should change (due to a refactoring) to:
from ml4h.models.train import train_model_from_generators
from ml4h.models.legacy_models import make_multimodal_multitask_model
from ml4h.models.inspect import _inspect_model
  • The training in the "Survival Analysis" section gives this error:
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-13-58156a64706b> in <module>
----> 1 train_multimodal_multitask(args)

/opt/conda/lib/python3.7/site-packages/ml4h/recipes.py in train_multimodal_multitask(args)
    153     return _predict_and_evaluate(
    154         model, test_data, test_labels, args.tensor_maps_in, args.tensor_maps_out, args.tensor_maps_protected,
--> 155         args.batch_size, args.hidden_layer, out_path, test_paths, args.embed_visualization, args.alpha,
    156     )
    157 

/opt/conda/lib/python3.7/site-packages/ml4h/recipes.py in _predict_and_evaluate(model, test_data, test_labels, tensor_maps_in, tensor_maps_out, tensor_maps_protected, batch_size, hidden_layer, plot_path, test_paths, embed_visualization, alpha)
    626             y = y_predictions
    627         y_truth = np.array(test_labels[tm.output_name()])
--> 628         performance_metrics.update(evaluate_predictions(tm, y, y_truth, protected_data, tm.name, plot_path, test_paths, rocs=rocs, scatters=scatters))
    629         if tm.is_language():
    630             sample_from_language_model(tensor_maps_in[0], tm, model, test_data, max_samples=16)

/opt/conda/lib/python3.7/site-packages/ml4h/plots.py in evaluate_predictions(tm, y_predictions, y_truth, protected, title, folder, test_paths, max_melt, rocs, scatters)
    319         ]
    320         logging.info(
--> 321             f"{[f'{label}: {value:.3f}' for label, value in zip(concordance_return_values, c_index)]}"
    322         )
    323         new_title = f"{title}_C_Index_{c_index[0]:0.3f}"

NameError: name 'c_index' is not defined
  • For the first training run in the "Command Line Friendly Jupyter Notebooks" section, it doesn't look like the plots are rendering successfully. They do render for the last section, but not this one. I just see the following, but no actual graphs:

image

In case it's helpful, I attached a zip of the html-ized notebook.
mnist_survival_analysis_demo.html.zip

/cc @deflaux

@amygdala @deflaux Just opened #458 which resolved these issues for me, let me know if it works for you. Thanks!

great, I'll give it a runthrough today.

Closed via #458