Overnight

Give a model a day of news it has never seen. Let it learn overnight. The next morning, ask it about that day with nothing added to the prompt. How much does it get right?

A new test every week

Each Monday we'll publish a fresh set of facts from that week's news. The questions get written on Thursday, after everyone's learnt weights are already on the Hub, so nobody can train on them, and none of it can be in any model's pretraining data. The first week will be announced here.

Archive

The news of 13 October 2026

Twelve public news feeds, one day. Every setting was fixed before that day happened. The next morning a different model wrote one question per fact, in its own words, and each question went to the model exactly as written. Three judges, who couldn't see which system gave which answer, set aside 35 questions that gave their answer away or couldn't be answered fairly. That leaves 231.

#SystemStarting modelAll 266Fair 231Answers
1LoRA on every layer, trained on practice questions from five other models, lr 2e-4Qwen2.5-7B-Instruct93.6% reported
All 266 of the day's facts pasted into the promptQwen2.5-7B-Instruct91.7%93.9%read
2FintanGemma-2-9B-it91.0%91.8%read
2FintanApertus-8B-Instruct91.0%90.9%read
The right fact pasted into the promptQwen2.5-7B-Instruct90.2%93.5%read
4FintanQwen2.5-14B-Instruct89.8%91.3%read
4FintanLlama-3.1-8B-Instruct89.8%90.0%read
6FintanOLMo-2-7B-Instruct89.5%90.5%read
7FintanOlmo-3-7B-Instruct88.7%88.7%read
8FintanQwen2.5-7B-Instruct86.8%87.9%read
9FintanOlmo-3.1-32B-Instruct84.2%85.3%read
Our notes lookup: facts picked by the subject the question names, put in the promptQwen2.5-7B-Instruct76.7% reported
10FintanMistral-7B-Instruct72.9%72.3%read
11LoRA on every layer, trained on the raw articles, lr 1e-3Qwen2.5-7B-Instruct16.2% reported
12LoRA on every layer, trained on the raw articles, lr 2e-4Qwen2.5-7B-Instruct13.2% reported
As releasedOlmo-3.1-32B-Instruct11.7%10.8%read
As releasedApertus-8B-Instruct11.7%10.8%read
As releasedGemma-2-9B-it11.3%10.8%read
As releasedMistral-7B-Instruct10.2%10.0%read
As releasedQwen2.5-7B-Instruct7.9%6.1%read
As releasedLlama-3.1-8B-Instruct7.1%6.1%read
As releasedOlmo-3-7B-Instruct6.8%5.2%read
As releasedOLMo-2-7B-Instruct6.4%5.6%read

"Read" opens every answer that row gave. "Reported" means we measured it the same way but haven't published those answers yet. The Olmo 3 rows used OLMo-2's settings with nothing tuned for Olmo 3, and the Apertus row (Swiss AI's 8B, added 6 November) used the standard night with nothing tuned for Apertus. The LoRA rows are rank 16 on every linear layer; the top one trained on the same practice questions Fintan's night uses, which is why one day on its own isn't the hard part. The notes-lookup row is our own retrieval path with answers cut at 64 tokens. With the right fact pasted in, the model still misses about 1 in 10 under this strict scorer, mostly by rewording it ("$1.3 billion" for "$1.3bn"), so scores near 90% are close to what the scorer allows. With all 266 of the day's facts pasted in, it scores 91.7%, at a cost of about 7,900 extra prompt tokens on every question; the Fintan rows add none. These questions have been online since 15 October, so a new entry here isn't sealed; that's what the weekly test is for.

Sealed

Four nights of real news

One day isn't the hard part. Here Apertus-8B-Instruct learnt a new day of real news on each of four nights (2, 4, 5 and 6 November 2026), and after every night it was asked the questions about every day so far again, with nothing in the prompt. The comparison is a standard LoRA (rank 16 on every linear layer) trained night after night on the same practice questions. The pass bars were written down before any fact or question existed.

After nightDay 1 (2 Nov)Day 2 (4 Nov)Day 3 (5 Nov)Day 4 (6 Nov)
Fintan, night 190.7%
Fintan, night 290.7%84.7%
Fintan, night 390.7%79.2%90.8%
Fintan, night 490.7%70.8%69.7%91.7%
LoRA, night 196.0%
LoRA, night 252.0%84.7%
LoRA, night 332.0%61.1%94.7%
LoRA, night 426.7%45.8%51.3%93.1%
Apertus as released6.7%11.1%5.3%9.7%
After four nightsMMLU (570)GSM8K (250)
Apertus as released57.4%69.6%
Fintan, as served58.4%69.2%
LoRA, after four nights52.8%4.4%

72 to 76 questions a day, written the next morning by Qwen2.5-7B-Instruct, strict scorer. LoRA learns the newest day best but loses the earlier ones, and after four nights it stops working maths problems through, so GSM8K falls to 4.4%. Fintan keeps day 1 at 90.7%. Days 2 and 3 dipped to about 70% where the same stories came back in later days' news; that's what we're working on next. All three of the run's pre-registered claims passed: day 1 kept at 85% or more of night 1, day 1 at least 10 points above LoRA, and MMLU and GSM8K within 2 points of the model as released.

Check

General ability, before and after

Apertus-8B-Instruct after learning the 13 October day. MMLU: 570 questions, 10 from each of the 57 subjects, zero-shot. GSM8K: the first 250 test problems, step by step, scored on the final number. "As served" is how Fintan runs: what it learnt is used on questions about what it learnt, and every other question gets the model's own answer. "Always on" uses it on every question, which is the worst case.

BenchmarkAs releasedFintan, as servedFintan, always on
MMLU (570)57.4%57.0%54.9%
GSM8K (250)69.6%69.6%34.8%

As served, what it learnt came into play on 9 of the 570 MMLU questions and none of the GSM8K problems, which is where the 0.4-point MMLU change comes from. Always on, GSM8K halves: the model often stops its working after the first step, because a night of short news answers carries over into long reasoning. That's why Fintan doesn't run that way. The same check for the other models on this board is next.

Archive

SQuAD, one passage at a time

This is the test SEAL, SLEEP and PaST report. The model learns one passage, the passage is taken away, and it answers that passage's questions from what it learnt.

SystemJudgeRightSource
PaST (Peking University), two roundsGPT-4.156.9%paper
FintanQwen2.5-14B-Instruct56.0%every answer
SLEEP (Google Research)GPT-4.148.9%paper
SEAL (MIT)GPT-4.147.0%paper
Our notes: facts written from the passage, put in the promptQwen2.5-14B-Instruct73.6%every answer
Qwen2.5-7B-Instruct as releasedQwen2.5-14B-Instruct26.2%every answer

The paper numbers are as their authors report them. Ours used Qwen2.5-7B-Instruct, with 80 passages and 454 questions we'd never looked at and settings fixed beforehand (1 November 2026). Fintan's 95% interval is 50.8% to 61.0%. We used a different judge from the papers, so every one of our answers is public for anyone to re-grade.

Rules

Each week (UTC)What happens
Monday 00:00The week's facts and their sources go up.
Wednesday 23:59Push your learnt model to the Hub; the commit time proves when it was made. Closed systems post a sha256 of their weights instead.
Thursday 12:00The questions go up.
Friday 23:59Answers are due.
SaturdayScores and every answer appear here.

Read the answers

Pick two systems and see how each answered the same question from 13 October.

Score your own

Learn from the 13 October facts, answer the questions in the same file, and score them here. It all runs in your browser; nothing gets uploaded. One line per question:

{"id": 0, "answer": "She is director of consumer finance at the FCA."}

The same rule in Python: scorer.py. It gives the same verdict as this page on all 5,054 published answers.

Submit

Open a discussion on this Space called "Submission: your method's name". Attach your answers, say which model you started from and what it saw while learning, and confirm nothing was added to the prompt. For the weekly test, add the Hub repo and the commit you pushed before the deadline, or a sha256 of your weights if they're closed.

We'll score it, check some answers by hand, and put it on the board with all its answers. Results from papers are welcome too, even if you're not one of the authors.