NOTES / PART 3 · RUNNING IN PRODUCTION
Running models fast and affordably.
THIS SECTION COVERS
Latency and cost, KV cache, batching, quantisation, speculative decoding, mixture-of-experts, serving stacks, caching.
Related: LLM fundamentals · Fine-tuning and adaptation