SluiceQoS-Aware Capacity Governance for Self-Hosted LLMs
Stops one noisy tenant from eating a paying customer's SLA.
Sluice is a FastAPI admission-control proxy for multi-tenant vLLM clusters. For every request it decides whether to pass, degrade, queue, fall back to a smaller model or reject. It combines GPU cache pressure, queue depth and per-tier SLA-violation rates to make that call. Redis coordinates fallback routing and cross-replica queueing, a PostgreSQL decision ledger records why each call was made, and Prometheus handles telemetry. It also has an offline replay tool that tests policy changes against recorded traffic.
Headline metric
Guaranteed-tier SLA violations cut from 30.0% to 2.0% under sustained overload, at ~0.006 ms median proxy overhead.
Synthetic benchmark against a mock backend.
- Python
- FastAPI
- vLLM
- Redis
- PostgreSQL
- Prometheus
- Docker
- Kubernetes