Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding
Primary LanguageJavaScriptApache License 2.0Apache-2.0