aws/aws-ofi-nccl

Issues

• [Feature request] Topo file for g6e.48xlarge

  #689 opened a month ago by sean-smith
  2

• ncclInternalError: Internal check failed. | NET/OFI Unable to insert remote address into address vector for device 1

  #663 opened a month ago by emorikawa
  8

• "Request completed with error" log leads to p5e cluster collapse

  #611 opened a month ago by vmarkovtsev
  0

• register_mr_buffers:544 NCCL WARN NET/OFI Unable to register memory (type = 2) for device 0. RC: -22, Error: Invalid argument

  #584 opened 3 months ago by visatish
  9

• NCCL RDMA expects fi_cq_data_entry, but OPX provider fills CQ with fi_cq_tagged_entry

  #609 opened 2 months ago by lsavers
  2

• Initialization fails for OPX Libfabric Provider

  #606 opened 2 months ago by lsavers
  0

• Seg Fault during RDMA NCCL Connection with OPX Provider

  #599 opened 2 months ago by lsavers
  4

• Assistance to broader Tag releases

  #395 opened 7 months ago by caio-davi
  6

• Incorrect error message when setting configure flag: --enable-nvtx-trace-per-dev

  #477 opened 4 months ago by ryanhankins
  2

• RDMA protocol

  #364 opened 2 months ago by eliekozah
  9

• GPU direct

  #313 opened 2 months ago by tks2004
  1

• Topology Discovery Regression

  #298 opened 2 months ago by willgleich
  3

• Support Amazon Linux 2023 (AL2023)

  #282 opened a year ago by bryantbiggs
  4

• Version

  #391 opened 7 months ago by sean-smith
  3

• No include folder after installation

  #597 opened 2 months ago by YJHMITWEB
  5

• Support Red Hat Enterprise Linux 9+

  #290 opened 3 months ago by tmh97
  5

• Segfault after/during finalize with OpenMPI

  #322 opened 3 months ago by tmh97
  3

• RDMA support for g6e nodes

  #547 opened 3 months ago by Abhishek8394
  0

• rename library to match nccl docs

  #508 opened 4 months ago by aws-nslick
  0

• nccl tested failed on aws p5 nodes but works fine on p4 and p4de

  #476 opened 4 months ago by liangjuf
  1

• NCCL Cannot Find Tuner Symbols. Need to Export NCCL_TUNER_PLUGIN=/opt/aws-ofi-nccl/lib/libnccl-ofi-tuner.so

  #472 opened 4 months ago by zhanwenchen
  1

• NCCL topology file for g5.12xlarge

  #466 opened 5 months ago by abdulfatir
  2

• NCCL internal error after aws-ofi-nccl upgrade to version 1.7.4

  #316 opened 8 months ago by ps-stability
  6

• Building with release tarball throws `tuner_v1.h` not found error

  #382 opened 8 months ago by sirutBuasai
  1

• Unable to find libcudart.so (1.7.1)

  #244 opened 8 months ago by kwohlfahrt
  6

• Support Ubuntu 22.04

  #203 opened 8 months ago by tson1111
  4

• Propagate "Invalid address" to NCCL communicator

  #346 opened 9 months ago by vmarkovtsev
  0

• Add more examples with more recent cuda versions

  #296 opened a year ago by tchaton
  2

• Unable to force FI_HMEM to be used and FI_OPT_CUDA_API_PERMITTED is not respected by config scripts

  #278 opened a year ago by tmh97
  4

• Plugin fails if compiled against Libfabric 1.18 but run against Libfabric 1.17 or older.

  #241 opened a year ago by nvcastet
  11

• What are some AI/ML workloads users can utilize to test performance of the plugin?

  #269 opened a year ago by tmh97
  0

• Misleading comparison on unsigned integer

  #236 opened a year ago by rauteric
  0

• Running nccl-perf tests documentation is missing MPI instructions

  #255 opened a year ago by tmh97
  3

• Support FI_CONTEXT2

  #204 opened a year ago by tmh97
  2

• aws branch does not build on centos 7 with gcc 4.8.5

  #183 opened 2 years ago by wenduwan
  2

• potential reoccurrence of https://github.com/aws/aws-ofi-nccl/issues/69

  #179 opened 2 years ago by junpuf
  1

• NCCL WARN NET/OFI Only EFA provider is supported

  #174 opened 2 years ago by mkserge
  2

• PyTorch Distributed Training crashes with "Cannot allocate memory (-12)"

  #141 opened 2 years ago by airsplay
  14

• [Feature Request]Allow custom NCCL_TOPO_FILE location

  #150 opened 2 years ago by roywei
  2

• Mellanox and EFA in Docker Image

  #170 opened 2 years ago by mvpatel2000
  6

• aws-ofi-nccl makes unnecessary calls to ofi_iflush() when using the PSM3 transport.

  #151 opened 2 years ago by mwheinz
  2

• WARNING: unrecognized options: --with-nccl when attempting to install

  #167 opened 2 years ago by mvpatel2000
  10

• NCCL WARN NET/OFI Request completed with error. RC: 21. Error: unknown error

  #161 opened 2 years ago by Ridhamz-nd
  2

• Error running NCCL Tests via MPiJob: "prov_err Not a Directory"

  #156 opened 2 years ago by Csinclair0
  2

• How does ofi_iflush() work?

  #149 opened 2 years ago by mwheinz
  6

• Question - difference between main vs aws branches

  #130 opened 2 years ago by kiukchung
  1

• Training performance on p3dn.24xlarge on Amazon Linux 2 is worse than on Ubuntu 20.04 (with and without EFA)

  #106 opened 3 years ago by yukunlin
  2

• Error (and crash) when using EFA from docker running on ubuntu AMI

  #107 opened 3 years ago by yukunlin
  2

• Crash with multirail providers.

  #102 opened 3 years ago by dmaryin
  1

• [Question] Is RDMA available on p3dn instances?

  #104 opened 3 years ago by kiukchung
  7