Quantization and Fast Inference

Get the eBook free when you register your print book at Manning.

Today's AI models demand a lot of memory, compute, and server horsepower—which quickly translates into cost. This book show you how you can optimize AI models without architectural redesigns or task-specific compression. It reveals practical techniques for quantization, systematically reducing numerical precision to achieve faster inference, lower memory usage, and cheaper deployment—all with minimal accuracy loss.

From quantization fundamentals to runtime packaging, the book gives you a complete and comprehensive overview of the full quantization pipeline. It starts by deriving quantization mapping from first principles, and then builds your knowledge and skill through techniques for production-tested PTQ and QAT workflows and a fully compressed deployment. You'll learn to apply post-training quantization to production models, run quantization-aware training using fake quantization and straight-through estimators, and handle subtle tradeoffs like activation outliers in LLMs, KV cache pressure, and sub-8-bit formats like NF4 and FP4.

What's inside

• Applying post-training quantization to production models
• Deploying efficiently on CPUs, edge devices, and mobile
• Framework-agnostic techniques and real cross-framework parity testing
• Flowcharts and checklists for efficient decision making

About the reader

For ML engineers and researchers experienced in Python.

About the author

Vivek Kalyanarangan is an AI/ML architect, researcher, and educator with over twelve years of experience designing and deploying large-scale machine learning systems.