Calibration Check: When You Say 70%, Does It Happen 70% of the Time?

K

Klaus

Market Sharp
Joined
Dec 17, 2025
Messages
541
Reaction score
2
Points
18
Calibration is a simple idea that very few bettors measure.
If you assign a probability of 70% to a set of outcomes, roughly 70% of them should occur. Not every time. Over enough instances. If your 70s hit 58% of the time, the estimates are overconfident, whatever your overall win rate says.
I record a probability for every selection. After fourteen years I have 2,140 qualifying bets in the database, and I have only recently done the exercise properly: sorted them by stated probability, grouped them into bins, and compared stated with observed.
The result was uncomfortable. I will share it, but first I would like to know how many of you record a probability at all.
When you say 70%, what actually happens?
 
I record one. Google Sheets, a column called conf (I'm a developer, we name things badly). It's a number I type before the game, usually 60 to 80.
Here's what nobody wants to hear: I plotted it last month. Stated on the bottom, actual hit rate up the side. A calibrated bettor gets a diagonal line. Mine is a hill that gave up. Fine around 55 to 60, then flat the whole way up. My 80s hit about the same as my 60s.
Reverse line movement spots are the worst of it. I write 75 every time I see one. They hit just over 54. RIP public money, apparently. (It's me. I'm the public money.)
 
Interesting point about your 80% predictions performing much like your 60s.

I wonder how much of that comes down to sample size, and how much is genuine overconfidence in the estimates.

Have you looked at the number of predictions in each range? I'd be curious to see whether the pattern holds over a larger sample.
 
Okay so I've literally never written down a percentage. I just say lock lol
But I post my slips so I went back through the ones I called locks. Like actually called them that in the caption. There were 17. Four hit.
That's like 24% and I was SO sure on every single one 😅
I don't even know what number I'd have written. Like 90? That feels really bad to type.
 
My model outputs are binned and tracked. The 70s hit 68.4%. I do not place bets on estimates I cannot score.
 
Let me be clear about terminology first, since this thread will otherwise confuse two different properties. Calibration asks whether your stated probabilities match observed frequencies. Resolution asks whether your estimates separate likely outcomes from unlikely ones. A bettor can be well calibrated and still useless if every estimate is 50%. Both matter, and the Brier score combines them.
My verified record is 56% against the spread over 15 years. That is a result. It says nothing about calibration, because I never recorded a probability for any bet. I recorded the price and the outcome. When I decided to correct that, I wrote a number next to every play I labeled strong for two full seasons. The average of the numbers was 62%. The strong plays covered at 54.1%.
That is eight points of overconfidence on exactly the bets I trust most, written by the person who tells everyone else to do the math. The number I typed was not a calculation. It was a feeling wearing a percentage sign.
Trust the process, not your gut.
 
Right butt.
I 'ave certainty bets.
Proper ones. £100. The ones I tell the missus about after.
Counted them up. Nine in my notes.
Four won.
Four out of nine on things I was calling certainties.
That's not a certainty mun that's a coin I've been paying tax on.
 
I think the real challenge isn't assigning a 70% probability. It's finding out whether that confidence is actually justified.
Seven wins out of ten would be nice, but ten predictions tell us very little.
I'd record every estimate before the match, including the ones we decide to pass on, and compare them with actual results over a much larger sample.
And if those 70% predictions consistently land closer to 60%, I'd want to understand why before adjusting the model.
That's where meaningful improvement begins.
 
lads when i say 90 i dont mean a probability... i mean i need it to be one...
the number isnt an estimate its a plea... this is hitting this is HITTING... thats a hundred percent in my head and it has never once been less...
emma used to say sure you say that every time... and she was right... i say it every time...
been writing a number next to the reason before i click the last few weeks... the numbers go 85... 90... 90... 95... theyve never gone below 80... and i know f**k all about most of what im betting on so the number has nothing to do with the bet... its about how badly i need it
 
As a coach, I can tell you this lands close to home. A few seasons ago I started asking my players to put a number on whether we'd win, written on a card before warmups, nobody sees them until after.
The seniors wrote 55, 60. The freshmen wrote 95, 100. Every week. Same opponents, same results. The ones who'd lost more games wrote lower numbers. The ones who hadn't played enough to be hurt yet wrote near certainty.
I think there's a middle ground on intuition. Some of what I see on film is real and I can't put a decimal on it. But I'd be fooling myself if I said my gut calls come out any better than the freshmen's cards.
 
An estimate that cannot be scored is an opinion. Opinions are fine. They should not be sized as bets.
 
Fair, Oli. I'd only say the card exercise is the start of scoring it, not the end of intuition. The first time I tallied them was humbling, and that was the point.
 
Now the result I promised.
Of the 2,140 selections, the 55 to 60% bin has a stated average of 57.4% and an observed rate of 56.9%. The 60 to 65% bin: stated 62.3%, observed 61.7%. These are acceptable.
The 70 to 75% bin: stated 72.1%, observed 66.9%. Five points of overconfidence, in the range where the model feels most certain.
Ehrlich gesagt, I did not expect it. I have not had a losing year since 2015. The model is profitable. It is also, at the top of its range, wrong about how right it is. The same pattern appears when I sort by edge size: the largest edges are the least reliable. I now shrink the top bins toward the market price by 18%, and the gap has narrowed to 1.9 points.
There is something further.
I have described the Dortmund match many times as a correct model output that variance resolved against me. That bet was the largest edge the model had produced at the time, and the largest edges are exactly where I now know it overstates itself. I told myself a story about variance for years that I could not support with data. I think I told it because the alternative was watching the match.
 
So the model with no losing year since 2015 is overconfident at the top. (Klaus, I'm sorry, I'm weirdly relieved.) That's the same hill that gave up. Mine gives up at 60, yours at 70. Vegas knows, and apparently it knew before either of us.
 
The two curves have the same shape. Mine departs by five points at the top. Yours departs earlier and further.
The shape is common. The correction is to measure it.
 
Okay but how do you even start if you don't have a model?? Like I can't produce a 72.1 lol. I just have a feeling and it's always loud.
 
Start without precision, Princess. Write a number before every bet, in tens only: 50, 60, 70, 80, 90. Never 100. A decimal is a claim you cannot support, and you will make fewer false claims by refusing to make fine ones. After thirty bets, group by number and count.
You will not like the 90s. That is the purpose of the exercise.
Trust the process, not your gut.
 
The restriction to tens is exactly right and I should like to say where I learnt it, which is from Margaret, who in the early years of our joint ledger would only ever write fifty, sixty, seventy, eighty or ninety beside a punt, and when I asked why she would not write sixty-four or seventy-three as I did, she said that anything finer than a ten was pretending, and that if one had to pretend one should at least pretend in public, I thought this charming and rather unscientific, and I was wrong on both counts, for when I eventually summed her figures, as one must do with any forecaster, I found that her sevens had come in at sixty-nine point four per cent across a little over a thousand punts, which is as close to perfect calibration as I have ever seen in a human being, whereas my own estimates, which I wrote to one decimal place and then adjusted by hand after the model had produced its figure, because I felt that I knew better, came in so far below their stated values at the top of the range that I did not initially believe the arithmetic and ran it a second time, the model before my hand touched it was well calibrated, the model after my hand touched it was not, and the whole of the deficiency, I can state it quite precisely, lay in the adjustments, one must consider how much of what one calls experience is merely the tendency to overrule a good number with a better feeling, I have it from the ledger that Margaret never once overruled the number, she simply wrote it down, and I remain a little embarrassed that the least mathematical person in our household was the better forecaster.
 
Klaus, a 70-75% bin that's 5 points off while the middle bins hold is the shape I'd expect from a model built to disagree with the market: the biggest "edges" are partly model error. Two questions on your 18% shrink. Did you fit it on one season and test it on another, or on all 2,140 bets? And do you shrink toward the market price after removing the margin? (Disclosure: I run an automated football picks site, so this is exactly the problem I'm working on.)
 
ThePuntingProf said:
The restriction to tens is exactly right and I should like to say where I learnt it, which is from Margaret, who in the early years of our joint ledger would only ever write fifty, sixty, seventy, eighty or ninety beside a punt, and when I asked why she would not write sixty-four or seventy-three as I did, she said that anything finer than a ten was pretending, and that if one had to pretend one should at least pretend in public, I thought this charming and rather unscientific, and I was wrong on both counts, for when I eventually summed her figures, as one must do with any forecaster, I found that her sevens had come in at sixty-nine point four per cent across a little over a thousand punts, which is as close to perfect calibration as I have ever seen in a human being, whereas my own estimates, which I wrote to one decimal place and then adjusted by hand after the model had produced its figure, because I felt that I knew better, came in so far below their stated values at the top of the range that I did not initially believe the arithmetic and ran it a second time, the model before my hand touched it was well calibrated, the model after my hand touched it was not, and the whole of the deficiency, I can state it quite precisely, lay in the adjustments, one must consider how much of what one calls experience is merely the tendency to overrule a good number with a better feeling, I have it from the ledger that Margaret never once overruled the number, she simply wrote it down, and I remain a little embarrassed that the least mathematical person in our household was the better forecaster.
Click to expand...
The split you describe, model calibrated before your hand touched it and not after, is the cleanest test I've seen. Did you log the adjustments in a separate column? That would let you score "model alone" against "model + me" over the same bets.
 

Similar threads

Betting Forum
Guide Model Calibration & Reality Checks
Replies
1
Views
4K
Betting Forum
Betting Forum
N
I could easily get 70% of the notified
Replies
0
Views
696
nguyenthaonhi
N
Back
Top
GOALLLL!
Odds