CUDA MemoryError for loading adata variable names to cudf.Series method

[nullbyte91]

Hi Team,

I'm getting CUDA memory error when I call cudf.Series method from the hlca_lung_gpu_analysis-visualization notebook.

Error Log:

---------------------------------------------------------------------------
ArrowTypeError                            Traceback (most recent call last)
~/anaconda3/envs/rapidgenomics/lib/python3.7/site-packages/cudf/utils/utils.py in pyarrow_buffer_to_cudf_buffer(arrow_buf, mask_size)
    157     try:
--> 158         arrow_cuda_buf = arrowCudaBuffer.from_buffer(arrow_buf)
    159         buf = Buffer(

~/anaconda3/envs/rapidgenomics/lib/python3.7/site-packages/pyarrow/_cuda.pyx in pyarrow._cuda.CudaBuffer.from_buffer()

~/anaconda3/envs/rapidgenomics/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

~/anaconda3/envs/rapidgenomics/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowTypeError: buffer is not backed by a CudaBuffer

During handling of the above exception, another exception occurred:

MemoryError                               Traceback (most recent call last)
<timed exec> in <module>

~/anaconda3/envs/rapidgenomics/lib/python3.7/site-packages/cudf/core/series.py in __init__(self, data, index, dtype, name, nan_as_null)
    186 
    187         if not isinstance(data, column.ColumnBase):
--> 188             data = column.as_column(data, nan_as_null=nan_as_null, dtype=dtype)
    189 
    190         if index is not None and not isinstance(index, Index):

~/anaconda3/envs/rapidgenomics/lib/python3.7/site-packages/cudf/core/column/column.py in as_column(arbitrary, nan_as_null, dtype, length)
   1552         elif arb_dtype.kind in ("O", "U"):
   1553             data = as_column(
-> 1554                 pa.Array.from_pandas(arbitrary), dtype=arbitrary.dtype
   1555             )
   1556             # There is no cast operation available for pa.Array from int to

~/anaconda3/envs/rapidgenomics/lib/python3.7/site-packages/cudf/core/column/column.py in as_column(arbitrary, nan_as_null, dtype, length)
   1352     elif isinstance(arbitrary, pa.Array):
   1353         if isinstance(arbitrary, pa.StringArray):
-> 1354             data = cudf.core.column.StringColumn.from_arrow(arbitrary)
   1355         elif isinstance(arbitrary, pa.NullArray):
   1356             if type(dtype) == str and dtype == "empty":

~/anaconda3/envs/rapidgenomics/lib/python3.7/site-packages/cudf/core/column/string.py in from_arrow(cls, array)
   4515     @classmethod
   4516     def from_arrow(cls, array):
-> 4517         pa_size, pa_offset, nbuf, obuf, sbuf = buffers_from_pyarrow(array)
   4518         children = (
   4519             column.build_column(data=obuf, dtype="int32"),

~/anaconda3/envs/rapidgenomics/lib/python3.7/site-packages/cudf/utils/utils.py in buffers_from_pyarrow(pa_arr)
    129 
    130     if buffers[1]:
--> 131         padata = pyarrow_buffer_to_cudf_buffer(buffers[1])
    132     else:
    133         padata = Buffer.empty(0)

~/anaconda3/envs/rapidgenomics/lib/python3.7/site-packages/cudf/utils/utils.py in pyarrow_buffer_to_cudf_buffer(arrow_buf, mask_size)
    172             dbuf.copy_from_host(np.asarray(arrow_buf).view("u1"))
    173             return Buffer(dbuf)
--> 174         return Buffer(arrow_buf)
    175 
    176 

~/anaconda3/envs/rapidgenomics/lib/python3.7/site-packages/cudf/core/buffer.py in __init__(self, data, size, owner)
     55             except TypeError:
     56                 raise TypeError("data must be Buffer, array-like or integer")
---> 57             self._init_from_array_like(np.asarray(data), owner)
     58 
     59     def __len__(self):

~/anaconda3/envs/rapidgenomics/lib/python3.7/site-packages/cudf/core/buffer.py in _init_from_array_like(self, data, owner)
     95                 data.__array_interface__
     96             )
---> 97             dbuf = DeviceBuffer(ptr=ptr, size=size)
     98             self._init_from_array_like(dbuf, owner)
     99         else:

rmm/_lib/device_buffer.pyx in rmm._lib.device_buffer.DeviceBuffer.__cinit__()

MemoryError: std::bad_alloc: CUDA error at: ../include/rmm/mr/device/managed_memory_resource.hpp:72: cudaErrorIllegalAddress an illegal memory access was encountered

Please find the SW details,
Conda Version = conda 4.8.5
Yaml - rapidgenomics_cuda10.1.yml
CUDA on disk - cuda-10.1
Ubuntu - 18.04

Regards,
Jegathesan S

[rilango]

Hi Jegathesan,

Please point us to the cell in the notebook where this issue appears.

Regards,
Rajesh K Ilango

[nullbyte91]

@rilango I'm having an issue with Cell number 8 in this Notebook.
The code snippet is,

genes = cudf.Series(adata.var_names)
barcodes = cudf.Series(adata.obs_names)
sparse_gpu_array = cp.sparse.csr_matrix(adata.X)

[rilango]

@nullbyte91 I could not reproduce this issue. Can you please try using the container at https://hub.docker.com/r/claraparabricks/single-cell-examples_rapids_cuda10.2. Please let us know if it is still a problem.