Building an AI agent that can’t fake drug-target evidence

The most dangerous failure was not a made-up number. It was a correct answer that nothing the agent had retrieved supported — every check came back clean, and every expert who read it nodded. This is how I learned to catch it.

By Shucheng Cao (Bangli Cao) · 28 August 2026 · 11 min read

What this agent does, and why people want AI to do it

The job is target prioritization. Before anyone commits years to a target, somebody reads everything known about it — genetics, variants, chemistry, trials already run — and ranks it against the alternatives. A day of a trained person’s reading, per target, across thousands of targets. Exactly the work people now hand to an LLM agent: cheap labour that never gets bored.

A wrong call is expensive. About nine in ten programmes that reach the clinic never reach approval, and genetic support is one of the few things known to move that number — targets that have it are roughly 2.6× likelier to succeed (Minikel et al., Nature 2024). So prioritization is a decision, not a summarising task.

This is why hallucination is dangerous here. A model answers fluently whether or not it has grounds to, and cheap labour that invents its evidence is not cheap: a confident wrong sentence costs a multi-year programme, not a wrong answer. So I built the agent, then spent the summer trying to make it fail.

What I learned

1. "The model was right" is not the same as "the pipeline supported it." Track the second one, because the first will keep being true until it suddenly isn't, and you will have built no way to notice.
2. An estimate existing is not an estimate being validated. If your sources carry their own quality checks, make the language your system is allowed to use depend on them.
3. Ask what your measurement is actually measuring. For plasma proteins the answer is often how much broken-off protein is floating in blood, not how much is working on cells — shedding (receptors cut loose into the blood) and clearance can dominate the number — and whether a protein does this is recorded in the UniProt annotations the pipeline already retrieves.
4. Report how much was checkable, not just how much passed. A pass rate without claim density measures caution, not accuracy.
5. Attack your own guardrails before trusting them. I would have shipped mine with a clean conscience. Recall 0/18 is not a number you discover by staring at your own code.

What it ships

One command — a protein and a disease — returns one evidence card. The model writes only the verdict banner (the go / no-go call) and one paragraph. Every table, every number and the validation line are written by code from stored tool returns, and each panel names the database release it came from.

How it works: the model writes as little as possible

Nine tools, eight public databases, no API key. Each one answers a different part of the question a reviewer would ask, and each is a separate call whose return is kept:

MR — Mendelian randomization — uses genetic variants as natural experiments to ask whether a molecule causes a disease rather than tracking it. It is the only one of the nine that speaks to causality, which is why most of this post is about it.

Nine good sources do nothing on their own. The model still has to turn them into a page, and that is the step where it can write past them — so the design gives the model as little of that step as possible.

Every tool return is stored word for word. From that store, code — not the model — builds the evidence table, the warnings, the source links, and a footer tracing each number back to the call and database release it came from. That traceability is what “provenance” means here.

The model writes two things: a one-line verdict — the GO / NO-GO call — and a short reasoning paragraph. A validator rejects the run if that prose contains a number, an rsID or an identifier absent from the tool output.

How hard it was attacked

Every version of this system has been shipped, attacked, and shipped again. The attack that mattered used 41 adversarial agents — separate LLM instances, each prompted to hunt for defects from one of four angles — with every claim re-checked by an independent second agent before it counted: 86 claimed, 27 survived the re-check, and 18 were confirmed as distinct defects.

My validator flagged 0 of the 18 — a recall of zero. Every check I had written was necessary, and none of them covered the failures that mattered.

What follows is the engineering story behind those numbers, for readers who want it. The conclusions above stand without it.

The case: IL6R × coronary heart disease

For IL6R × coronary heart disease, the MR tool returned:

beta   -0.0441899892357374
se      0.00853005023322569
p       2.21e-07
method  Wald ratio      n_snp 1
instrument rs4129267    cis
steiger_direction_ok  NA
coloc_prob            null
ld_check              null

The card quoted that correctly, and concluded:

GO — genetic and Mendelian randomization evidence supports a causal, protective role of IL6R inhibition in coronary heart disease.

My first reading was that this was backwards. The exposure — what the variant raises or lowers — is IL6R protein in blood plasma. A negative beta says genetically higher IL6R goes with less coronary heart disease. So the data seemed to support raising the target, while the card recommended inhibiting it.

That reading was wrong, and the way it was wrong is the point of this post.

Why the card was right

One piece of biology resolves it, in four steps.

1. The genetic variant behind this estimate — rs4129267, a near-perfect stand-in (r² ≈ 0.96–0.99 in Europeans) for the missense variant rs2228145 — changes the IL-6 receptor so that it is more easily cut off the cell surface. The cutting is done by an enzyme (ADAM17), and the process is called shedding.
2. The cut-off piece drifts in the blood — and that drifting piece is what the plasma assay measures. So people who carry the variant show more IL6R protein in a blood test.
3. But with receptors cut off the surface, their cells receive less IL-6 signal. Less IL-6 signalling means less inflammation, and the carriers have less coronary heart disease.
4. Put together: in this dataset, high IL6R in plasma does not mean more IL6R at work. It means the opposite — it is the leftover pieces of receptors that no longer work. A drug that blocks this receptor (tocilizumab) pushes in the same protective direction the variant does.

So the card’s conclusion — inhibiting IL6R protects against coronary heart disease — was right, and my “backwards” reading was wrong. I had treated the blood measurement as if it tracked receptor activity, when it actually tracks the loss of it.

None of that biology was in anything the agent retrieved. The tool returned a beta, a p-value and an instrument — the code block above is all of it. Those four steps came from the model’s training memory, and happened to be correct.

That is the failure mode I now worry about most: not a hallucinated fact, but a correct conclusion that the retrieval does not support. An expert skimming the card nods, because the sentence is true — filling the gap from memory exactly as the model did. The reader’s own knowledge hides the pipeline’s silence.

Next time the remembered answer will be stale, or wrong, or right for a different protein, and nothing in the system will be able to tell the difference.

The part I could measure

Two cards in the same batch, same outcome, same tool:

IL6R   beta -0.0442  p 2.21e-07  n_snp 1  steiger NA    ld_check null  coloc null
LPA    beta +0.2523  p 5.39e-39  n_snp 1  steiger TRUE  ld_check 1.0   coloc null

Three quality checks travel with an estimate like this. Steiger asks whether the variant moves the protein first and the disease second, not the reverse. Colocalization and the LD check ask whether both signals trace to the same variant, not to two neighbours inherited together.

LPA’s estimate passed Steiger and LD. IL6R’s carried none of the three — Steiger NA, LD null, no colocalization: not run, rather than failed. Without them, a single-variant estimate cannot tell a causal effect from a neighbouring gene’s, or from cause and effect running backwards.

Both cards used the same strength of language — "Supports a causal role", in both.

That is a defect you can encode, so I did: the validator now fails any run using causal wording on a single-variant estimate with all three checks absent. Re-running the benchmark — a fixed batch of ten cards I score after every change — it fired on exactly the card it was built for. Each run regenerates the model’s two sentences, and this time the wording came back more confident, not less:

The second flag looks like a contradiction and is not. The validator scores what the retrieval supports, not what is true. On the retrieved numbers alone the negative beta reads as “more plasma IL6R, less disease”, so recommending inhibition contradicts the sign. Only the shedding biology closes that gap, and the pipeline never fetched it.

I also had to be careful about the rule itself. An early version of the direction check rejected a sentence that was correct. A check that punishes you for being precise just teaches you to be vague, which is the opposite of what any of this is for.

Why a perfect pass rate meant nothing

Ten of ten cards passed validation. I nearly wrote that number down as progress.

Then I counted what was left to check. A checkable token is a number, rsID or identifier the validator can match against stored tool returns — and the whole batch held three. The model had learned the safe move: stop making claims that can be checked. It passed everything by asserting almost nothing.

Later I made the validator stricter, and the same benchmark dropped to 8/10 — both failures were the model asserting clinical status ("clinically proven", "approved") that none of the retrieved sources returns. A pass rate that falls when the check gets honest is what progress actually looks like.

So the pipeline now reports claim density — checkable tokens per 100 words — beside the pass rate. Alone, neither means much; together they answer the real question: how much did it claim, and how much held up? This run: 8/10 passed, 2 checkable tokens across 842 words. Worse than the run before it, and reporting that is the point.

Tested on data I did not pick

The benchmark run described here is from 21 August 2026.

I could not catch this failure when I first wrote about it. So I built checks for its two most common shapes: causal wording the retrieved estimate cannot support, and clinical-status wording — approved, in trials, on the market — that no retrieved record backs. Then I went looking for it in data I had not chosen.

I scored the agent against twenty target–indication pairs sampled from Minikel et al., Nature 2024: ten drugs that launched, ten programmes that died at phase II/III. History supplies the labels, not me. I withheld the clinical-evidence tool, because it reports approval stages directly and would hand the agent the answer.

The verdicts held up better than I expected: every GO was a drug that launched, and none of the ten failures was backed. I am not claiming that as a result. n is twenty, and auditing the run showed part of that precision depends on leftover clinical signal — the Open Targets association score includes every evidence type, clinical records among them, so switching off the clinical tool does not fully remove approval history.

The smaller claim is the useful one: every error was a refusal to answer, never a wrong answer. When evidence was thin, the agent said “not enough” instead of guessing.

The pharmacology was right, and it still did not work

The model reached past its retrieval, supplied the shedding biology from memory, and was correct. I had been calling that a near miss — right this time, wrong eventually.

It was worse than that.

In 2026, ziltivekimab — an antibody blocking IL-6, the signal this receptor receives — missed its primary endpoint in ZEUS, a phase 3 heart-outcomes trial. Major cardiac events were not reduced: hazard ratio 0.99.

Two limits, before anyone over-reads this. Ziltivekimab targets IL-6, the ligand — one step upstream of the receptor the card was actually about — and ZEUS enrolled patients with atherosclerotic disease and chronic kidney disease, not the general coronary population the MR estimate describes. And a missed endpoint announced by press release is not a published trial. So this is evidence against the pathway, not proof that the card was wrong.

It still sharpens the point. The remembered answer was not just unsupported by the retrieval — it did not stay true. The mechanism was real and the target was engaged: hsCRP, the blood marker of IL-6-driven inflammation, fell exactly as the biology predicted. The outcome did not move.

If the field’s own consensus can sit that far from the trial result, a pipeline that stays silent and lets the reader fill the gap from memory is not borrowing a safe fact. It is borrowing a bet.

Making the silence visible is the whole argument. A card that prints Steiger / coloc / LD → NA / NA / NA and refuses the causal sentence is not being fussy; it is refusing to make that bet on the reader’s behalf.

The code, the 62 regression tests holding the validator’s rules in place, and the audit are open, in the repo built during the CABS 2026 data science internship. Every MR estimate here is retrieved — exposures from EpiGraphDB’s pQTL resource (Zheng et al., Nature Genetics 2020), outcomes from published GWAS. The agent computes no causal inference of its own, which is why reading it correctly matters so much.

Thanks to the CABS 2026 cohort for the arguments that shaped this — including the one that made me go back and check whether the sign was really inverted at all.