llama inference
Exploration of latency on various setups of inference with llama.
Caveats
- I didn't explore throughput. That is a deep rabbit hole - I was just exploring latency for a single request. You can tradeoff throughput and latency with various forms of batching requests.
- I tried my best to use tools based on the documentation provided.