Mooncake

Community maintained hardware plugin for vLLM on Ascend

Production ready toolkit to run AI locally

Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM

A high-performance inference system for large language models, designed for production environments.