NVIDIA/nccl-tests

Issues

• Test CUDA failure common.cu:941 'system not yet initialized'

  #264 opened 2 months ago by vijayaramaraju-kalidindi
  6

• nccl-tests result is only a half of the bandwidth on bond nics

  #265 opened a month ago by 913871734
  0

• Why is the NCCL test rate of my 4090x8 card so low?

  #269 opened a month ago by gogoman010310
  0

• Why is the NCCL test rate of my 4090x8 card so low?

  #270 opened a month ago by gogoman010310
  0

• question for NCCL write data size

  #266 opened a month ago by gabbychen
  4

• P2P performance with nccl-tests vs nvbandwidth

  #268 opened a month ago by goelayu
  0

• how overall throughout calculate about all2all

  #267 opened a month ago by ltm920716
  2

• Why the effective B/W for each NVlink is 20GB/s instead of 25GB/s

  #258 opened a month ago by gabbychen
  2

• How to get the latency and the package of NCCL

  #262 opened a month ago by gabbychen
  4

• Difference between in_place and out_of_place

  #261 opened 2 months ago by 17113325
  5

• Test CUDA failure common.cu:941 'invalid device ordinal' when test two nodes with nvhpc

  #263 opened 2 months ago by heya5
  3

• How to run nccl test in vm without nvswitch passthroughed?

  #260 opened 2 months ago by joydchh
  1

• nccl-tests did not perform as expected

  #257 opened 2 months ago by yalbaba
  3

• NCCL topology on the VM of H200

  #256 opened 3 months ago by wangjiafu0310
  4

• nccl-tests hangs when using HPCX

  #255 opened 3 months ago by ycm0k
  3

• Multiple MPI ranks using same GPU when conducting multi-node test

  #254 opened 3 months ago by ycm0k
  1

• Test NCCL failure common with network error.

  #252 opened 3 months ago by ismailguzel
  11

• question about pingpong example

  #253 opened 3 months ago by jinz2014
  5

• BW test on V100 4 GPUS is not matched with InfiniBand EDR (Connect-X4)

  #251 opened 3 months ago by javak87
  0

• Enable P2P on pcie in a nvlink machine

  #250 opened 4 months ago by cll24
  1

• Running in kubernetes pods Error

  #248 opened 4 months ago by drikster80
  2

• Getting Avg bus bandwidth = 0 when running all_reduce_perf in nccl-tests in my EC2 G5.8x large

  #249 opened 4 months ago by rajeshvenkata
  2

• all_reduce_perf core dumped on 4 L20

  #233 opened 4 months ago by songh11
  23

• NCCL all-reduce test failure due to TL_SHM ERROR, This case was happened on containers on same server.

  #247 opened 4 months ago by thsmfe001
  2

• 2 Node Nccl Test don’t work for A100

  #242 opened 4 months ago by jeffreyyjp
  4

• NCCL_Algo=Tree

  #246 opened 5 months ago by afattaholman
  1

• How do we comprehend the factor between algBw and busBw?

  #235 opened 5 months ago by lianghao208
  5

• What does dma_buf do when gpuDirectRdma is disabled ?

  #245 opened 5 months ago by Pavani-Panakanti
  1

• all_reduce_perf hangs; using a single GPU on a 4GPU machine

  #217 opened 7 months ago by isaacgerg
  21

• Test NCCL Hang

  #244 opened 5 months ago by sdonoso
  2

• AllReduce Bus Bandwidth decreases with larger network latency

  #241 opened 5 months ago by chenzhu99
  0

• 2 Node Nccl Test don’t work

  #236 opened 5 months ago by SdEnd
  7

• all_reduce algo factor for NVLink SHARP In network reductions

  #238 opened 5 months ago by OrenLeung
  0

• how to calculate the tree based allreduce ib bw?

  #237 opened 5 months ago by echobinarybytes
  0

• What's multi-allreduce ?

  #234 opened 5 months ago by ProHuper
  1

• NCCL Tree allreduce test cannot reach the theoretical bus bandwidth on 2 nodes with 4 nics

  #232 opened 5 months ago by ProHuper
  0

• Test NCCL failure common.cu:997 'internal error

  #231 opened 6 months ago by sdonoso
  9

• how to support One Device per Process?

  #221 opened 6 months ago by jiangxiaobin96
  4

• what is cu:990 error? how to solve this problem?

  #230 opened 6 months ago by MAKER-park
  5

• 2 Nodes nccl-test with mpi hangs

  #229 opened 6 months ago by sdonoso
  1

• has nvswitch, but uses 0 nvls channels

  #228 opened 6 months ago by MiyazonoKaori
  3

• Test fail caused by ibvwrap.c:160 NCCL WARN Call to ibv_modify_qp failed with error Connection timed out.

  #227 opened 6 months ago by thsmfe001
  2

• alltoall_perf: each rank is only sending to half of the other ranks

  #224 opened 6 months ago by russilwvong
  14

• NCCL WARN Cannot use cuda/gdr transports as part of specified UCX_TLS

  #222 opened 7 months ago by liuxingbo12138
  5

• stepbytes (increment size) argument does not support 1M notation

  #225 opened 6 months ago by StefanoSalsano
  1

• mpirun all_reduce_perf hang with multi-device test

  #223 opened 6 months ago by 913871734
  0

• 1 GiB headroom might be too small

  #220 opened 7 months ago by Namnamseo
  0

• Test NCCL failure common.cu:959 'internal error - please report this issue to the NCCL developers / '

  #219 opened 7 months ago by Assassin187
  9

• Rank Assignment Issue under four containers on two different servers.

  #218 opened 7 months ago by thsmfe001
  8

• NCCL initialization hangs with 4 GPUs, but works with 2 GPUs

  #216 opened 7 months ago by mickaelseznec
  4