#quantization
Every summary, chronological. Filter by category, tag, or source from the rail.
Tag · #quantization
Compression at the Edge: Strategies for Efficient AI
Compression is not just about fitting models on consumer hardware; it is a strategic necessity for democratizing intelligence, increasing concurrency, and reducing operational costs by leveraging selective quantization and architecture-aware optimization.
AI EngineerLLM Inference: mmap Loading & Quantization Deep Dive
Efficient LLM inference hinges on mmap for lazy memory loading (e.g., <10s startup on llama.cpp) and quantization like GGUF K-Quants or AWQ/EXL2 to shrink 15GB models while preserving quality via salient weights and mixed precision.
Caleb Writes CodeShowing 4 of 4