From Dev.to
Decode Is 300x Memory-Bound. Batching Fixes That Until the Context Gets Long.
I spent last week arguing about KV cache sizes. The useful follow-up is simpler: multiply the cache by the context length and you get the number that actually sets your throughput. Decode does not run...