How to use Tensorrt-LLM as backend

Question

How to use Tensorrt-LLM as backend

Worromots opened this issue 8 months ago · 4 comments

as describe in title

Answer 1 · 2024-05-20T09:25:20.000Z

You can set save_fp in llmc to True. Then you can use trt-llm ammo to convert a naive quant engine.

Answer 2 · 2024-05-22T12:30:59.000Z

THX for your reply. I have set save_fp in llmc to True, and these are files saved by llmc, how can I use trt-llm ammo to convert a naive quant engine.

Answer 3 · 2024-06-06T09:14:55.000Z

remark，I need your help

Answer 4 · 2024-06-07T21:27:33.000Z

The following process needs to modify some codes to change the default settings in TensorRT-LLM. To help users use our tool more conveniently, we are rushing an official doc page about the tool. Please wait for our news patient.