From Dev.to
Are your model's probabilities calibrated? Expected calibration error in NumPy
There was a comment on Hacker News this week saying LLM token probabilities are badly calibrated for making decisions. I think that's right, and it's also something you can measure yourself in about f...