dmonakhov/TentorRT-LLM-on-AWS

TensorRT-LLM with EFA libraries

TensorRT-LLM with EFA libraries

Container with TensorRT-LLM and EFA libraries.

FIXME: Code is not yet well tested

Build

make fetch
make build

# Build vanilla image similar to https://github.com/Azure/AI-benchmarking-guide/blob/main/Benchmarks/LLMBenchmark.py
make build-vanilla

Other documentation resources

• https://github.com/NVIDIA/TensorRT-LLM
• https://nvidia.github.io/TensorRT-LLM/quick-start-guide.html
• https://developer.nvidia.com/blog/turbocharging-meta-llama-3-performance-with-nvidia-tensorrt-llm-and-nvidia-triton-inference-server/
• https://github.com/Azure/AI-benchmarking-guide