TypeError: can't convert CUDA tensor to numpy

[Hyperparticle]

When running with CUDA and generating tables, the labels are trying to be implicitly converted to numpy using scipy. I can see that there is already a .detach().cpu().numpy() in regimen.py, but for some reason it appears that the labels are being converted back to CUDA. Calling .cpu() before handing off the labels to scipy fixes it for me.

Full trace below.

Traceback (most recent call last):
  File "structural-probes/run_experiment.py", line 242, in <module>
    execute_experiment(yaml_args, train_probe=cli_args.train_probe, report_results=cli_args.report_results)
  File "structural-probes/run_experiment.py", line 182, in execute_experiment
    run_report_results(args, expt_probe, expt_dataset, expt_model, expt_loss, expt_reporter, expt_regimen)
  File "structural-probes/run_experiment.py", line 143, in run_report_results
    reporter(dev_predictions, dev_dataloader, 'dev')
  File "/home/hyper/Documents/repos/structural-probes/structural-probes/reporter.py", line 50, in __call__
    , dataloader, split_name)
  File "/home/hyper/Documents/repos/structural-probes/structural-probes/reporter.py", line 143, in report_image_examples
    ax = sns.heatmap(label)
  File "/home/hyper/Documents/anaconda3/envs/probe/lib/python3.7/site-packages/seaborn/matrix.py", line 517, in heatmap
    yticklabels, mask)
  File "/home/hyper/Documents/anaconda3/envs/probe/lib/python3.7/site-packages/seaborn/matrix.py", line 109, in __init__
    plot_data = np.asarray(data)
  File "/home/hyper/Documents/anaconda3/envs/probe/lib/python3.7/site-packages/numpy/core/numeric.py", line 538, in asarray
    return array(a, dtype, copy=False, order=order)
  File "/home/hyper/Documents/anaconda3/envs/probe/lib/python3.7/site-packages/torch/tensor.py", line 450, in __array__
    return self.numpy()
TypeError: can't convert CUDA tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.

[john-hewitt]

Thanks for the find! Could you note your pytorch, numpy, and scipy versions? I mostly tested this code on (regrettably) old installs of each of these, so this'll help me debug / make a fix that works for a lot of people's versions.

[Hyperparticle]

• numpy 1.16.2
• scipy 1.2.1
• torch 1.0.1

[howardhsu]

Got the same problem on PyTorch 1.0.1, change back to 0.4.1(0.4.1.post2) works.

[john-hewitt]

Should be fixed in PR #5 ; check it out if you'd like; I've tested it on old + new versions on parse-distance and parse-depth tasks. If all is well / no comments, I'll merge this weekend.