vllm-ascend

A high-throughput and memory-efficient inference and serving engine for LLMs

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs