AkeruAI/akeru

implement streaming in miner and validator

GuiBibeau opened this issue · 0 comments

Currentl calling the chat endpoint on validators results in a 5-10 second delay since the LLM workload is slow. We should implement streaming methods that are equivalent in both the miner and the validator to let them act closer to the open AI API