Give a model a day of news it has never seen. Let it learn overnight. The next morning, ask it about that day with nothing added to the prompt. How much does it get right?
Each Monday we'll publish a fresh set of facts from that week's news. The questions get written on Thursday, after everyone's learnt weights are already on the Hub, so nobody can train on them, and none of it can be in any model's pretraining data. The first week will be announced here.
Archive
Twelve public news feeds, one day. Every setting was fixed before that day happened. The next morning a different model wrote one question per fact, in its own words, and each question went to the model exactly as written. Three judges, who couldn't see which system gave which answer, set aside 35 questions that gave their answer away or couldn't be answered fairly. That leaves 231.
| # | System | Starting model | All 266 | Fair 231 | Answers |
|---|---|---|---|---|---|
| 1 | LoRA on every layer, trained on practice questions from five other models, lr 2e-4 | Qwen2.5-7B-Instruct | 93.6% | reported | |
| All 266 of the day's facts pasted into the prompt | Qwen2.5-7B-Instruct | 91.7% | 93.9% | read | |
| 2 | Fintan | Gemma-2-9B-it | 91.0% | 91.8% | read |
| 2 | Fintan | Apertus-8B-Instruct | 91.0% | 90.9% | read |
| The right fact pasted into the prompt | Qwen2.5-7B-Instruct | 90.2% | 93.5% | read | |
| 4 | Fintan | Qwen2.5-14B-Instruct | 89.8% | 91.3% | read |
| 4 | Fintan | Llama-3.1-8B-Instruct | 89.8% | 90.0% | read |
| 6 | Fintan | OLMo-2-7B-Instruct | 89.5% | 90.5% | read |
| 7 | Fintan | Olmo-3-7B-Instruct | 88.7% | 88.7% | read |
| 8 | Fintan | Qwen2.5-7B-Instruct | 86.8% | 87.9% | read |
| 9 | Fintan | Olmo-3.1-32B-Instruct | 84.2% | 85.3% | read |
| Our notes lookup: facts picked by the subject the question names, put in the prompt | Qwen2.5-7B-Instruct | 76.7% | reported | ||
| 10 | Fintan | Mistral-7B-Instruct | 72.9% | 72.3% | read |
| 11 | LoRA on every layer, trained on the raw articles, lr 1e-3 | Qwen2.5-7B-Instruct | 16.2% | reported | |
| 12 | LoRA on every layer, trained on the raw articles, lr 2e-4 | Qwen2.5-7B-Instruct | 13.2% | reported | |
| As released | Olmo-3.1-32B-Instruct | 11.7% | 10.8% | read | |
| As released | Apertus-8B-Instruct | 11.7% | 10.8% | read | |
| As released | Gemma-2-9B-it | 11.3% | 10.8% | read | |
| As released | Mistral-7B-Instruct | 10.2% | 10.0% | read | |
| As released | Qwen2.5-7B-Instruct | 7.9% | 6.1% | read | |
| As released | Llama-3.1-8B-Instruct | 7.1% | 6.1% | read | |
| As released | Olmo-3-7B-Instruct | 6.8% | 5.2% | read | |
| As released | OLMo-2-7B-Instruct | 6.4% | 5.6% | read |
"Read" opens every answer that row gave. "Reported" means we measured it the same way but haven't published those answers yet. The Olmo 3 rows used OLMo-2's settings with nothing tuned for Olmo 3, and the Apertus row (Swiss AI's 8B, added 6 November) used the standard night with nothing tuned for Apertus. The LoRA rows are rank 16 on every linear layer; the top one trained on the same practice questions Fintan's night uses, which is why one day on its own isn't the hard part. The notes-lookup row is our own retrieval path with answers cut at 64 tokens. With the right fact pasted in, the model still misses about 1 in 10 under this strict scorer, mostly by rewording it ("$1.3 billion" for "$1.3bn"), so scores near 90% are close to what the scorer allows. With all 266 of the day's facts pasted in, it scores 91.7%, at a cost of about 7,900 extra prompt tokens on every question; the Fintan rows add none. These questions have been online since 15 October, so a new entry here isn't sealed; that's what the weekly test is for.
Sealed
One day isn't the hard part. Here Apertus-8B-Instruct learnt a new day of real news on each of four nights (2, 4, 5 and 6 November 2026), and after every night it was asked the questions about every day so far again, with nothing in the prompt. The comparison is a standard LoRA (rank 16 on every linear layer) trained night after night on the same practice questions. The pass bars were written down before any fact or question existed.
| After night | Day 1 (2 Nov) | Day 2 (4 Nov) | Day 3 (5 Nov) | Day 4 (6 Nov) |
|---|---|---|---|---|
| Fintan, night 1 | 90.7% | |||
| Fintan, night 2 | 90.7% | 84.7% | ||
| Fintan, night 3 | 90.7% | 79.2% | 90.8% | |
| Fintan, night 4 | 90.7% | 70.8% | 69.7% | 91.7% |
| LoRA, night 1 | 96.0% | |||
| LoRA, night 2 | 52.0% | 84.7% | ||
| LoRA, night 3 | 32.0% | 61.1% | 94.7% | |
| LoRA, night 4 | 26.7% | 45.8% | 51.3% | 93.1% |
| Apertus as released | 6.7% | 11.1% | 5.3% | 9.7% |
| After four nights | MMLU (570) | GSM8K (250) |
|---|---|---|
| Apertus as released | 57.4% | 69.6% |
| Fintan, as served | 58.4% | 69.2% |
| LoRA, after four nights | 52.8% | 4.4% |
72 to 76 questions a day, written the next morning by Qwen2.5-7B-Instruct, strict scorer. LoRA learns the newest day best but loses the earlier ones, and after four nights it stops working maths problems through, so GSM8K falls to 4.4%. Fintan keeps day 1 at 90.7%. Days 2 and 3 dipped to about 70% where the same stories came back in later days' news; that's what we're working on next. All three of the run's pre-registered claims passed: day 1 kept at 85% or more of night 1, day 1 at least 10 points above LoRA, and MMLU and GSM8K within 2 points of the model as released.
Check
Apertus-8B-Instruct after learning the 13 October day. MMLU: 570 questions, 10 from each of the 57 subjects, zero-shot. GSM8K: the first 250 test problems, step by step, scored on the final number. "As served" is how Fintan runs: what it learnt is used on questions about what it learnt, and every other question gets the model's own answer. "Always on" uses it on every question, which is the worst case.
| Benchmark | As released | Fintan, as served | Fintan, always on |
|---|---|---|---|
| MMLU (570) | 57.4% | 57.0% | 54.9% |
| GSM8K (250) | 69.6% | 69.6% | 34.8% |
As served, what it learnt came into play on 9 of the 570 MMLU questions and none of the GSM8K problems, which is where the 0.4-point MMLU change comes from. Always on, GSM8K halves: the model often stops its working after the first step, because a night of short news answers carries over into long reasoning. That's why Fintan doesn't run that way. The same check for the other models on this board is next.
Archive
This is the test SEAL, SLEEP and PaST report. The model learns one passage, the passage is taken away, and it answers that passage's questions from what it learnt.
| System | Judge | Right | Source |
|---|---|---|---|
| PaST (Peking University), two rounds | GPT-4.1 | 56.9% | paper |
| Fintan | Qwen2.5-14B-Instruct | 56.0% | every answer |
| SLEEP (Google Research) | GPT-4.1 | 48.9% | paper |
| SEAL (MIT) | GPT-4.1 | 47.0% | paper |
| Our notes: facts written from the passage, put in the prompt | Qwen2.5-14B-Instruct | 73.6% | every answer |
| Qwen2.5-7B-Instruct as released | Qwen2.5-14B-Instruct | 26.2% | every answer |
The paper numbers are as their authors report them. Ours used Qwen2.5-7B-Instruct, with 80 passages and 454 questions we'd never looked at and settings fixed beforehand (1 November 2026). Fintan's 95% interval is 50.8% to 61.0%. We used a different judge from the papers, so every one of our answers is public for anyone to re-grade.
| Each week (UTC) | What happens |
|---|---|
| Monday 00:00 | The week's facts and their sources go up. |
| Wednesday 23:59 | Push your learnt model to the Hub; the commit time proves when it was made. Closed systems post a sha256 of their weights instead. |
| Thursday 12:00 | The questions go up. |
| Friday 23:59 | Answers are due. |
| Saturday | Scores and every answer appear here. |
Pick two systems and see how each answered the same question from 13 October.
Learn from the 13 October facts, answer the questions in the same file, and score them here. It all runs in your browser; nothing gets uploaded. One line per question:
{"id": 0, "answer": "She is director of consumer finance at the FCA."}
The same rule in Python: scorer.py. It gives the same verdict as this page on all 5,054 published answers.
Open a discussion on this Space called "Submission: your method's name". Attach your answers, say which model you started from and what it saw while learning, and confirm nothing was added to the prompt. For the weekly test, add the Hub repo and the commit you pushed before the deadline, or a sha256 of your weights if they're closed.
We'll score it, check some answers by hand, and put it on the board with all its answers. Results from papers are welcome too, even if you're not one of the authors.