leoheuler/flashtensors
Runs hundreds of large models on a single GPU by loading them from SSD to VRAM up to 10x faster than alternative loaders, achieving coldstarts under 2 seconds.
View on index · View in 3D Map
Related repos
- Runs 70B models on 4GB GPUs
- run local LLMs fast on consumer GPUs with flexible quantization
- High-performance OpenGL C++17 3D graphics engine
- GPU kernels written from scratch
- Realtime TTS without GPU
- Runs 405B Llama3.1 on 8GB VRAM
- Runs 397B model on laptop
- KV cache compression for LLM inference with 5.02x ratio
- Runs LLMs over a cluster of GPUs
- competes to train GPT-2 models super fast on 8xH100 GPUs using speedrun benchmarks
