Calibration Check: When You Say 70%, Does It Happen 70% of the Time?

A reference curve that might help: the market's own calibration in the same range.
We log every match winner that the sharp market (Everygame, margin removed) prices at 65% or more, and score it against the 90-minute result. Since 18 October, 467 matches:
65-70%: shown 67.6%, happened 64.4% (160)
70-75%: shown 72.3%, happened 77.9% (104)
75-80%: shown 77.3%, happened 75.7% (74)
80%+: shown 85.8%, happened 84.0% (81)
All four bins sit inside their 95% intervals. With about 100 per bin, a 5-point miss is noise: at 70% and n=100 the interval is roughly ±9 points.
Two things follow for Klaus's 70-75 bin. First, the bets you take are by construction the ones where you disagree with the market, so a model that is well calibrated overall can still look overconfident on its bets (oddsfactory's point). If your 70s are spots where the market said 62, expect them to land nearer 62. Writing the market's devigged price next to your own number at bet time shows which of the two is off. Second, a hundred bets per bin is the minimum before reading anything into the curve.
Disclosure: I run TipsAudit, where this register is published.
 
Weather forecast for Cardiff says 30% chance of rain.
It is Cardiff.
Rain's not a percentage here butt it's a lifestyle.
 
The Brier score was devised for verifying weather forecasts.
Professional forecasters are among the best calibrated groups ever measured. When they say 30%, it rains on roughly 30% of those days.
 
Forecasters are scored every day. Bettors are scored on the result, never on the number.
 
That's the piece I keep coming back to. A forecaster says 30 and gets feedback every single day, so the number gets honest. We say 90 and get feedback on one game, and it either looked right or it didn't. The card exercise with my players works for the same reason. It makes the number the thing being graded, not the game.
 
Okay I'm gonna try tens. I looked at my last slip and I wanted to put 90 on it and I'm literally scared to see what I'd really write if I couldn't say lock lol 🏈
 
Princess, put 70 and see how it feels. (Nobody has ever regretted writing 70.) The 90 is where the lies live.
 
One useful distinction is between the probability implied by the odds and our own estimate. A price of -150 implies 60%, but that isn’t independent evidence that the outcome has a 60% chance.

Keeping those figures in separate columns could help reveal whether our estimates add information or simply follow the market. Do you also split your calibration results by sport or bet type?
 
Bronwyn doesn't use a spreadsheet.
Just says things.
Seventy percent you'll be home by ten. Eighty percent you'll say it was one more pint. Ninety percent you'll mention Wales before the starter.
She's right most weeks.
Years I've spent learning odds off websites mun.
Best forecast I've ever had is in my own kitchen.
 
emma never gave me a number... she just said sure you say that every time...
which i suppose is a probability... a pretty low one... and she wasnt wrong either...
 
Klaus, a 70-75% bin that's 5 points off while the middle bins hold is the shape I'd expect from a model built to disagree with the market: the biggest "edges" are partly model error. Two questions on your 18% shrink. Did you fit it on one season and test it on another, or on all 2,140 bets? And do you shrink toward the market price after removing the margin? (Disclosure: I run an automated football picks site, so this is exactly the problem I'm working on.)
Your first point is correct, and I would name the mechanism precisely: the selection itself manufactures the error. A model that bets only where it disagrees most with the market will, by construction, collect the cases where the model is most wrong. The 70 to 75% bin is not a random sample of the model’s opinions. It is the sample where it argued hardest with the price.
To your questions.
On the fit: all 2,140 bets, with no holdout. Ehrlich gesagt, that is a weakness. I introduced the 18% before I adopted a rule requiring an unseen holdout for every adjustment, so the 1.9 point gap I reported is an in-sample figure.
Since your question I have re-fitted on the first eleven seasons only, about 1,680 bets. The shrink comes out at 16%. Applied to the last three seasons, the pooled bins above 65% (118 bets) move from 4.6 points of overconfidence to 1.4. The standard error on that pooled hit rate is about 4.3 points, so I read this as consistent with the shrink and not as proof of it. The 70 to 75% bin alone holds roughly 40 bets in that window and cannot be tested.
On the target: yes, toward the market price after removing the margin, taken from the sharpest book at the time of the bet. I strip the margin proportionally. At prices between 1.33 and 1.43 the favourite-longshot bias is small, so the method matters less here than at the long end, but I have not tested the alternative.
A question back, since this is your problem as well: do you shrink per bin, or fit a single blend of model and market log-odds and let the data choose the weight? I suspect the second is more honest, and I have not yet done it.
 

Similar threads

Back
Top
GOALLLL!
Odds