Mega4alik/ollm

Run huge LLMs with 100k context on an 8GB GPU using SSD offload, no quantization

View on index · View in 3D Map

Related repos

• Run 70B LLMs on a 4GB GPU with layer-wise inference and memory optimization, quantization optional
• run local LLMs fast on consumer GPUs with flexible quantization
• Resources for running LLMs on local hardware
• Runs LLMs on AMD NPUs
• framework for fast, resource-efficient LLM inference on local hardware
• efficient attention for long-context LLMs
• OpenAI-compatible proxy that boosts LLM reasoning accuracy with 20+ optimization techniques, no retraining needed
• minimal 4D-parallelism framework for distributed LLM training and experimentation
• Runs large language models on iOS and macOS
• run large language models on any device