kvcache-ai/ktransformers

framework for fast, resource-efficient LLM inference on local hardware

Open-source toolkit for building scalable LLM inference infrastructure
vllm-project/aibrix
Fast LLM inference and serving library
vllm-project/vllm
Minimal vLLM engine for fast offline LLM inference with a clean Python codebase
GeeeekExplorer/nano-vllm
run large language models on any device
mlc-ai/mlc-llm
Kubernetes-native framework for serving large language models at scale
llm-d/llm-d
OpenAI-compatible proxy that boosts LLM reasoning accuracy with 20+ optimization techniques, no retraining needed
codelion/optillm
LLM inference in C/C++ for running and serving models locally on various hardware
ggml-org/llama.cpp
lets you plug structured knowledge bases into LLMs for efficient, scalable augmentation
microsoft/KBLaM
fine-tune LLMs, generate synthetic data, collaborate on datasets
Kiln-AI/Kiln
1-bit LLM inference on CPUs and GPUs with BitNet
microsoft/BitNet