Two agents, and no way to tell if either one workedwhy I helped start a benchmark instead of building a third
LLMs hallucinate. I built two agents to keep made-up claims off the page, then wrote the checks that graded them. So I was grading myself. I helped start AI4S-Benchmark instead: scientists send in hard problems from their own field, and someone else holds the answer.
What a benchmark actually is
A benchmark is how you compare two systems that different people built. Neither builder marks the result.
For that to mean anything, five pieces have to exist.
- A task contract. What the agent is given, what it must hand back, and what it is not allowed to look at.
- A dataset. The material the agent can see, and the answer it cannot.
- A scorer. How an answer gets marked — by rule where possible, by a person where not.
- Leakage control. Some reason to believe the model did not simply memorise the answer before you asked.
- A baseline. At least one reference result that anybody can reproduce.
Miss any one of the five and what you have is a demonstration. Demonstrations are useful. They are not evidence that a thing works.
What went wrong the first time
The first agent reads public genetics databases and writes one page on whether a protein is worth making a drug against.
You cannot easily find the bugs by reading your own code. To improve it, I spent a summer having it attacked instead: 41 model agents, coming at it from four angles.
Each agent sees only its own angle, so a complaint from one of them may be biased. A complaint only became a finding once a second, separate model had checked it and agreed. Eighty-six problems were reported. Twenty-seven passed that check. Eighteen were real, and different from one another.
My own checks caught none of the eighteen.
Then a second number went wrong in the opposite direction. Ten cards out of ten passed validation, and I nearly wrote that down as progress — until I counted how much there had been to check. Across the whole batch of ten, three claims were checkable at all. The model had found the safe move: say almost nothing that can be tested. A pass rate with nothing behind it measures caution, not accuracy.
Both numbers were produced by me, about my own system, using checks I had written.
What went wrong the second time
The second agent asks the same kind of question starting from a molecule in blood rather than a protein. Different databases, a different way of naming things, an extra step to find the gene behind the molecule.
The machinery moved across without argument. The problems moved across with it. A request that could not be resolved. A source that failed and had to be reported as failed rather than as empty. A verdict that had to be computed rather than written, because the line most likely to be quoted back is the line a model should not be able to reach.
And once again, the only person deciding whether any of it worked was the person who built it.
What both had in common
I wrote the claim and I wrote the check. When the check passed, that told you the two agreed with each other. It did not tell you either one was right.
This is not a character flaw and it is not unusual. It is the normal condition of nearly every scientific agent being shown off right now: the team that built the system also decided what counted as success, which cases to show, and when to stop testing. Their results are not lies. Nobody outside can check them, which is a different kind of problem and in some ways a worse one.
The twenty target–disease pairs I eventually scored against real drug-development outcomes were labelled that way because I went and built the labels myself, from a published survey. That was the closest thing to an outside opinion available to me. It should not have been.
What is already measured, and what is not
There is no shortage of benchmarks in this area. Several good ones test whether a model can find a paper, answer a question about a database, or reason its way through an open scientific problem. Between them they cover literature retrieval and open-ended scientific question answering fairly well.
What none of them scores is the thing my two agents kept failing at.
They all ask whether it can answer. None asks whether the answer can be checked, step by step, or whether it will stop when the evidence runs out.
Those are different questions. An agent can produce a true sentence that nothing it retrieved supports, and every existing test will pass it, because the sentence is true. An agent can decline to commit and score badly, when declining was the correct behaviour. Until refusal counts for something and every step is checkable against a source, the leaderboard rewards fluency.
So we started one
Rather than build a third agent and grade it myself, I became one of the initiators of AI4S-Benchmark, an interdisciplinary benchmark for AI agents in science, and wrote the first three tasks in my own area.
A benchmark is usually a set of questions written by one team. Here, scientists contribute the problems themselves.
A task goes through five steps before it counts:
- Somebody proposes a problem out of their own work, through a form.
- Automated checks run over the proposal.
- People in that field review it, and they are asked two separate questions: is this scientifically worth anything, and can the answer actually be verified? A hard problem with no checkable answer does not get in.
- It becomes a structured pull request — the task, in a fixed shape, in public.
- Frontier agents are run against it, how far they get is recorded, and it enters a versioned release.
Step three is the one I care about, and it is why I am spending time on this rather than on a third agent. Somebody other than the author has to be willing to say that a task is unverifiable, or too easy, or measuring something other than what it claims.
Other tasks come from materials science, physics, chemistry, AI and applied mathematics. Every field has to be clear about what a checkable answer is.
The three tasks I wrote are in my own area:
- Protein evidence. Given a protein and a disease, rebuild the human-genetic causal evidence — the estimate, its direction, the checks that travel with it — and be scored on whether each piece traces to a source, not on whether the conclusion sounds right.
- Metabolite evidence. The same, starting from a molecule measured in blood, where identity is ambiguous and the gene behind the molecule has to be nominated before the question is even well posed.
- Reaction rules. Checking and repairing the gene-to-reaction rules inside a human metabolic model, where the answer is a formal expression that can be verified edge by edge and then run.
Reviewers are credited permanently for that work. Reviews are written on the website and synced to GitHub, so the argument about whether a task is sound can be read by anyone rather than settled in a room. No GitHub account needed. That is what keeps a benchmark from turning into one more system graded by the person who built it.
The honest state of it
My three tasks are specifications. For them there is no gold answer set, no hidden test set, and no score for any agent, including my own. Nothing on this page is a result.
The pipeline above is the design, and my tasks have not been through the far end of it. I would rather say that than describe a plan in the present tense.
Writing a task contract is the cheap part. The expensive parts are the ones still missing. Somebody has to produce answers that are right independently of any agent. Somebody has to hold those answers back. And somebody has to be willing to say the tasks are not hard enough, or not fair.
What it needs now
Three things, in order.
- Reviewers who will argue. People who can look at a task in genetics, metabolism or agent evaluation and say whether it is verifiable, whether it is genuinely difficult, and whether it is worth anyone’s time. Disagreement is the product here.
- Independent answers. Cases where the truth is known from somewhere other than the system being tested — a completed trial, a curated model, an expert who will commit to an answer in writing.
- Somebody else’s agent. A benchmark with one participant is still a self-assessment, just a more elaborate one.
If any of that is yours to give, write to me. I am most interested in hearing from people who think one of these tasks is wrong, or too easy, or measures something other than what it claims. That is more useful to me than agreement, and it is the entire reason for doing this in the open rather than in my own repository.
Proposing a task and signing up to review one both start at ai4sbench.org, and both are done on the site itself. Or write to me directly: shucheng.cao@mail.mcgill.ca.
This is the third of three posts. The first is about the failure that started all of this; the second is about what happened when the same design moved to a different kind of molecule.