perk11/large-model-proxy
Large Model Proxy is designed to make it easy to run multiple resource-heavy Large Models (LM) on the same machine with limited amount of VRAM/other resources. It listens on a dedicated port for each proxied LM, making them always available to the clients connecting to these ports.
GoGPL-2.0
Issues
- 3
Crash at main.canBeStopped({0xc00013a210?, 0x55ed17?}) /large-model-proxy/main.go:427 +0x5b
#7 opened by perk11 - 0
- 2
- 0
Implement priorities and scheduling to prevent single task from constantly hogging all the resources
#18 opened by perk11 - 0
- 0
Check config for duplicate names and ports
#16 opened by perk11 - 1
- 0
- 1
Custom healtcheck before connection
#8 opened by perk11 - 0
Monitor started process status
#13 opened by perk11 - 0
Support JSON5 format for config
#12 opened by perk11 - 0
After multiple parallel connections, a service might be never considered idle again
#11 opened by perk11 - 0
Test coverage for all the main functionality
#9 opened by perk11 - 1
- 3
Feature Request: Automatically kill the process after a configurable amount of time
#3 opened by jukofyork - 3
time.Time has no field or method Compar
#1 opened by fahdmirza