turboderp-org/exllamav2

run local LLMs fast on consumer GPUs with flexible quantization

View on index · View in 3D Map

Related repos

• Run 70B LLMs on a 4GB GPU with layer-wise inference and memory optimization, quantization optional
• Run huge LLMs with 100k context on an 8GB GPU using SSD offload, no quantization
• Resources for running LLMs on local hardware
• Runs LLMs on AMD NPUs
• framework for fast, resource-efficient LLM inference on local hardware
• LLM inference in C/C++ for running and serving models locally on various hardware
• 1-bit LLM inference on CPUs and GPUs with BitNet
• LLMPEG: GPU-accelerated video transcoding tool
• Platform for automatic algorithm design using LLMs
• local LLM serving with GPU and NPU acceleration