intel/auto-round

Quantizes LLMs and VLMs to 2-4 bits using sign-gradient descent

View on index · View in 3D Map

Related repos

• framework for fast, resource-efficient LLM inference on local hardware
• Inference framework for 1-bit LLMs
• RAG and fine-tuning projects for LLMs
• Run 70B LLMs on a 4GB GPU with layer-wise inference and memory optimization, quantization optional
• Fine-tunes LLMs on Apple silicon
• OpenAI-compatible proxy that boosts LLM reasoning accuracy with 20+ optimization techniques, no retraining needed
• run local LLMs fast on consumer GPUs with flexible quantization
• Open-source toolkit for building scalable LLM inference infrastructure
• Speeds up LLM inference with speculative decoding
• Runs LLMs on Ryzen NPUs without GPUs