Weird memory leak behavior

[cyoon1729]

DQN-based algorithms have memory leak while training on Atari. The leak doesn't always happen (inconsistent), or happens suddenly in the middle of training.

[cyoon1729]

The function below (numpy to tensor conversion) likely causes the memory leak:

"""in rlcycle/common/utils/common_utils.py"""
def np2tensor(np_arr: np.ndarray, device: torch.device):
    """Convert numpy array to tensor"""
    tensor_output = torch.FloatTensor(np_arr).to(device)
    if device.type == "cuda":
        tensor_output.cuda(non_blocking=True)
    return tensor_output


def np2tensor2(np_arr: np.ndarray, device: torch.device):
    """Convert numpy array to tensor"""
    tensor_output = torch.from_numpy(np_arr).float().to(device)
    if device.type == "cuda":
        tensor_output.cuda(non_blocking=True)
    return tensor_output

[cyoon1729]

Fixed memory leak in PR #12
Here's what changed:

1. Numpy to tensor conversion:

def np2tensor(np_arr: np.ndarray, use_cuda: bool):
    """Convert numpy array to tensor"""
    if use_cuda:
        return torch.from_numpy(np_arr).cuda(non_blocking=True).float()
    return torch.from_numpy(np_arr).cpu().float()

This was recommended by Pytorch in this discussion thread.

2. Use cuda() instead `.to(device)' to cast cuda tensors. Recommended by Pytorch in this discussion thread.

3. Added a memory profiler for debugging.