Which forecasts deserve your trust?
A public notebook for scoring forecasters by what actually happened, not by how confident they sounded.
Abstract
Agreement between forecasts is not evidence that they are right. We grade probability forecasts with proper scoring rules, draw their reliability on real outcomes, and weight a panel of forecasters by their earned record. Every number on this page is either computed in your browser from data you paste, or graded on live Hyperliquid prices; forecasts can be committed to Solana mainnet so the record cannot be rewritten later.
Keywords Brier score · calibration · reliability diagram · Hedge weighting · on-chain commitment
Introduction: confidence is cheap
A forecaster who says “90%” is making a promise about frequencies. Across all the times they say 90%, roughly nine in ten should come true. That promise can be checked. Most dashboards never check it: they show the forecast, not the record.
Calibro treats every forecast as a small bet against reality and keeps score with a proper scoring rule, one that a forecaster cannot game by shading their true belief. The simplest is the Brier score, the mean squared distance between the stated probability and what happened.
Here ft is the probability given before the event and ot is 1 if it happened and 0 if not. A perfect forecaster scores 0, a coin-flipper who always says 50% scores 0.25, and a confident forecaster who is often wrong can score far worse than the coin.
Equation 1, interactive: score a list of forecasts
Paste one forecast per line: a probability (0–1 or a percentage) and the outcome (1/0, yes/no, true/false). Everything runs in your browser; nothing is uploaded.
| Brier score | — | mean squared error, 0 best |
|---|---|---|
| Reliability | — | calibration error, 0 best |
| Resolution | — | ability to separate cases, higher better |
| Uncertainty | — | ō(1−ō), set by the events |
| Brier skill vs. base rate | — | 1 − BS / BSclim |
| Log score | — | mean −ln p(outcome), 0 best |
| Forecasts · hit rate | — | count and share that happened |
BS = REL − RES + UNC
4 The Murphy decomposition splits the score into three parts. Only reliability is the forecaster’s fault in the usual sense; uncertainty is the difficulty of the events themselves.
5 Log score punishes a confident miss without limit. A forecast of 0% on something that happens scores infinity; we clip at 0.1% to keep the table readable.
Reliability: does 70% mean 70%?
The same forecasts, binned by stated probability. Each point is a bin; its height is how often those events really happened. Bars show a 95% Wilson interval, so a small bin is honest about how little it proves.
| Stated | Happened | n | Gap |
|---|
Score some forecasts in §2 to draw this figure.
The live lab: five forecasters, graded on real prices
The question is fixed: will the price be higher at the end of the horizon than now? Five simple forecasters answer it every hour of the window using only data available at that moment. We then grade each answer against the real Hyperliquid close. The forecasters are deliberately simple models; the outcomes are real.
| Forecaster | Brier | Skill | Hit | Reliability | Now says |
|---|---|---|---|---|---|
- Coin flip
- Always 50%. The floor any forecaster must beat.
- Base rate
- Share of the last 72 hours that went up, smoothed.
- Momentum
- Recent 6-hour return in volatility units, through a gentle logistic.
- Reversion
- The mirror of momentum: bets that recent moves fade.
- Hot hand
- Momentum with five times the confidence. Built to show what overconfidence costs.
Anatomy of one graded forecast
- Made. A forecaster states 72% that SOL closes higher in an hour.
- Committed. Its SHA-256 fingerprint is written to Solana, so the 72% can never quietly become 80%.
- Resolved. The hour ends. The candle closes lower: the outcome is 0.
- Scored. The penalty is the square of the miss: (0.72 − 0)² = 0.518. It joins the running mean.
Scroll to step through.
A model of models: weight by earned record
Averaging every forecaster equally gives the overconfident one as much say as the careful one. The Hedge rule instead shrinks each forecaster’s weight by e−η·loss every time an outcome lands. Weights only use outcomes that had already resolved, so there is no peeking.
η = 0 is a plain average. Large η trusts the current leader almost completely.
| Equal weights | — |
|---|---|
| Hedge ensemble | — |
| Best single (hindsight) | — |
| Coin flip | 0.2500 |
The ledger: commit a forecast to Solana mainnet
A record is only worth something if it cannot be edited after the fact. Write your forecast, and Calibro fingerprints it with SHA-256 in your browser. You can then sign one SPL Memo transaction in your own wallet that stores only the fingerprint on mainnet, with the network’s timestamp. Later, anyone can check the text against the chain.
A · Write and fingerprint
—B · Verify any commitment
C · A forecaster’s on-chain ledger
Reads the address’s latest 100 transactions on mainnet and lists every Calibro commitment, with block time. Your local forecasts are matched by fingerprint and can be resolved here to build your own Brier score.
Methods: four clocks and three rules
A forecast’s value depends on when it was known. We record four times for every forecast and never fill a gap with a guess.
| # | Clock | Meaning | Source on this page |
|---|---|---|---|
| 01 | Made | When the forecaster formed the view | Your claim text |
| 02 | Issued | When it was written down | made field in the canonical text |
| 03 | Public | First proof anyone could see it | Solana block time of the memo |
| 04 | Graded | When the outcome was scored | Your browser, at resolution |
- Grade on the outcome source named in advance. If the claim says “Hyperliquid daily close”, no other price counts.
- Unknown stays unknown. A forecast without a public time is graded, but never counted as early.
- Small samples are shown as small. Every bin carries its interval; a 3-for-3 streak is not a record.
How the lab forecasters work
With k = 0.6 for momentum, −0.6 for reversion and 3 for the hot hand, and s1h the standard deviation of hourly log returns over the previous 72 hours. The base rate is (ups + 1) / (72 + 2). These are not trading signals. They are there to be graded.
How to use Calibro in practice
- i.
Collect
Export the probabilities a source gave you, with what happened. A newsletter, a model, a friend, yourself.
- ii.
Score
Paste them into §2. Read the reliability term first: it tells you whether their 80% means 80%.
- iii.
Weight
Give more say to sources with a better record, as §6 does, rather than to the loudest or the most recent.
- iv.
Commit
Write your own forecasts to the ledger before the event. After a season, your record is public and unarguable.
Data availability statement
| Element | Status | Source |
|---|---|---|
| Lab outcomes, Figure 1, Figure 3 | live | Hyperliquid candleSnapshot, mainnet prices |
| Lab forecasters | model | Equation 2, computed in your browser |
| Ensemble weights, Figure 4 | model | Hedge rule on the live outcomes |
| Scores in §2 and §3 | calculation | Your pasted data; example sets are generated |
| Commitments, ledger, slot | live | Solana mainnet-beta through this site’s read-only /api/rpc |
Calibro is an independent concept project. It never holds funds, never asks for a seed phrase, and the only transaction it can prepare is an SPL Memo that you review and sign in your own wallet; it moves no tokens. There is no audit, partnership or user count to report, so we report none.
Frequently asked questions
Is a lower Brier score always better?
For the same set of events, yes. Across different events, no: predicting tomorrow’s sunrise scores near 0 without any skill. That is why §2 also reports skill against the base rate.
Why not just count how often a forecaster was “right”?
Because it rewards bold rounding. Saying 51% and 99% count the same when they come true, and a forecaster learns to exaggerate. Proper scores pay for honesty.
Are the lab forecasters any good?
Mostly not, and that is the point. Short-horizon price direction is close to a coin flip. Watch the hot-hand forecaster lose to the coin: confidence without skill is costly.
What exactly goes on chain when I commit?
One SPL Memo instruction containing calibro:v1: followed by the 64-character SHA-256 of your canonical text. The text itself stays with you. You pay the normal network fee, shown before your wallet opens.
Can I delete a commitment?
No. Transactions on Solana are permanent. That is what makes the record trustworthy, so only commit what you are content to be judged on.
Does connecting a wallet give Calibro access to my funds?
No. Connecting shares your public address only. The site reads your ledger and balance from mainnet and can ask your wallet to sign a memo, which you can refuse.
Where do the prices come from?
Hyperliquid’s public candle endpoint, requested directly from your browser. Nothing is cached on a server of ours.
References
- Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1).
- Murphy, A. H. (1973). A new vector partition of the probability score. Journal of Applied Meteorology, 12(4).
- Gneiting, T. & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. JASA, 102(477).
- Freund, Y. & Schapire, R. E. (1997). A decision-theoretic generalization of on-line learning. JCSS, 55(1).
- Wilson, E. B. (1927). Probable inference, the law of succession, and statistical inference. JASA, 22(158).