vllm-project/speculators
Speeds up LLM inference with speculative decoding
View on index · View in 3D Map
Related repos
- Minimal vLLM engine for fast offline LLM inference with a clean Python codebase
- framework for fast, resource-efficient LLM inference on local hardware
- LLM inference engine with parallel speculative decoding
- Library for running and serving large language models
- LLM inference in C/C++ for running and serving models locally on various hardware
- AIBrix manages and scales large language model inference in the cloud for enterprise and developer workloads
- benchmark vLLM, SGLang, TensorRT-LLM on Modal
- Benchmark for evaluating LLMs on real-world GitHub issues
- run local LLMs fast on consumer GPUs with flexible quantization
- run large language models on any device
