vllm

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

SGLang is a high-performance serving framework for large language models and multimodal models.

LMCache: Supercharge Your LLM with the Fastest KV Cache Layer