From Dev.to
What causes cold start latency in GPU inference and how do you avoid it?
Cold start latency in GPU inference is the delay a new replica adds before it can serve its first token: pulling the container image, loading model weights into GPU memory, and initializing CUDA (Comp...