kvcache-ai/ktransformers

framework for fast, resource-efficient LLM inference on local hardware

View on index · View in 3D Map

Related repos

• Open-source toolkit for building scalable LLM inference infrastructure
• Fast LLM inference and serving library
• Minimal vLLM engine for fast offline LLM inference with a clean Python codebase
• run large language models on any device
• Kubernetes-native framework for serving large language models at scale
• OpenAI-compatible proxy that boosts LLM reasoning accuracy with 20+ optimization techniques, no retraining needed
• LLM inference in C/C++ for running and serving models locally on various hardware
• Resources for running LLMs on local hardware
• lets you plug structured knowledge bases into LLMs for efficient, scalable augmentation
• fine-tune LLMs, generate synthetic data, collaborate on datasets