Raise thread cap for text gen.
Pyroserenus opened this issue · 1 comments
Pyroserenus commented
Certain configurations on Aphrodite-Engine and TGI will handle greater than 10 text threads on a single worker. My current record for optimal threads on a single worker is 25 but this could wind up greater as more testing is done on multi-GPU or if shorter context were deployed on high end hardware.
db0 commented
Done