tanishqkumar/ssd

LLM inference engine with parallel speculative decoding

View on index · View in 3D Map

Related repos

• multimodal LLM inference engine in Rust and Python
• Trains large models with distributed computing
• Static single-batch CUDA inference engine
• open-source memory engine for LLMs and AI agents
• Parallel disk analyzer that maxes out your SSD and lets you interactively delete files
• framework for fast, resource-efficient LLM inference on local hardware
• Run huge LLMs with 100k context on an 8GB GPU using SSD offload, no quantization
• LLM inference in C/C++ for running and serving models locally on various hardware
• minimal 4D-parallelism framework for distributed LLM training and experimentation
• Serve LLMs, speech, and multimodal models with one command