Browse the notebook

MLOps

Articles in this topic

Jan 29, 20269 min read

AI & ML

Quantizing Llama 3 8B to W8A8: A Complete Guide with LLMCompressor and vLLM

Step-by-step tutorial for quantizing Meta's Llama 3 8B Instruct model to INT8 weights and activations (W8A8) using LLMCompressor, then serving it with vLLM for production inference with KV cache and continuous batching.

Jul 06, 20258 min read

AI & ML

Building an End-to-End Chat Bot with ONNX Runtime and Rust

Step‑by‑step tutorial: export a HuggingFace Transformer model to ONNX, load it in Rust, and stream chat replies token‑by‑token.