Building an AI agent that can’t fake drug-target evidencefrom a protein to whether it is worth a drug

LLMs hallucinate. Researchers in academia and industry still search by hand for the mechanism behind their results, and it takes a great deal of human labour. So I built an agent for biomedicine that only prints what its sources returned.

By Shucheng Cao (Bangli Cao) · 28 August 2026 · 11 min read

Background

To make a new drug, you first decide which protein in the body it should act on. That protein is called the target. Pick the wrong one and everything after it is wasted: designing a molecule that acts on that protein, testing it in animals, then testing it in people (a clinical trial). You usually find that out years later.

For most of the last century that choice came out of the lab. You put a chemical on cells, or give it to mice, and see if anything improves. That shows the chemical does something. It does not show that the protein is why people get the disease.

This goes wrong often. About nine in ten drugs that reach trials in people are never approved. Drugs aimed at a protein that human genetics also points to do better — about 2.6 times more likely to be approved (Minikel et al., Nature 2024).

To get that kind of evidence, scientists use a method for telling cause from coincidence in genetic data (Mendelian randomization). It uses the gene versions a person is born with to test whether a protein actually causes a disease, instead of just appearing alongside it.

That evidence is already public and free. Collecting it is the problem. For one protein and one disease you open seven or eight websites, each calling the same gene something different, and piece it together by hand. That is about a day’s work, and there are thousands of proteins.

This post is about the agent I built to do that in minutes, and what it took to make its answers checkable.

What the agent does

It is for people who want to know whether a protein has anything to do with a disease: a researcher looking for the mechanism behind their own results, someone in pharma R&D, or anyone who is curious about that link and wants a clear answer.

You give it two things: a protein and a disease. It gives back one page — an evidence card — showing what is known about that pair, where every fact came from, and whether the protein looks worth going after.

To build that page it asks nine questions across eight free public databases: EpiGraphDB, Open Targets, ChEMBL, gnomAD, ClinVar, the GWAS Catalog, UniProt and ClinPGx. Between them they cover whether anyone has shown the protein causes the disease, how strongly the two are linked, whether a drug against it has already been tried and what happened, whether any chemical is known to hit it, and whether switching it off is likely to be safe. None of them needs an account or a paid key.

Giving this job to a language model looks like the easy answer. It never gets bored, it costs almost nothing, and there are thousands of proteins to get through.

The problem is that a model writes a fluent answer whether or not it has anything to back it up. Here a confident wrong sentence can point years of drug development at the wrong protein. So the model is only cheap if it does not invent its evidence.

Whether a sentence on the card is true and whether the retrieved evidence supports it are two different questions. I built the whole design around the second one. A sentence can be true with nothing behind it in what the agent fetched, and nothing on the card would show that.

That is why I spent the summer attacking my own agent rather than adding to it. The rest of this post is what the attacks found.

What it ships

Open the app and there is one screen: a box for the protein, a box for the disease, and a button. About twenty seconds later the card is on the page — a verdict at the top, then one panel per database.

Down the side is the list of databases, each with a tick. Not everyone wants all of them. Un-tick the ones you do not need and the run is faster; where they would have been, the card says tool not called in this run.

What it helps with, and what it does not. It retrieves published MR estimates and computes none, so it does not stand in for an analysis. It answers the question that comes before one: does an estimate already exist for this protein and this outcome, did that estimate carry its direction and shared-variant checks, and if nothing comes back, is that because no instrument exists or because nobody has run it. The same sweep covers 991 proteins; restricted to each protein’s fifteen strongest disease associations, 7,524 protein–disease pairs carry a genetic association with no MR estimate in the resource behind this tool. That set is a list of analyses nobody has run, not a list of results.

What it asks each database

Nine tools, eight public databases, no API key. Each one answers a different part of the question a reviewer would ask, and each is a separate call whose return is kept:

MR — Mendelian randomization — uses genetic variants as natural experiments to ask whether a molecule causes a disease rather than tracking it. It is the only one of the nine that speaks to causality, which is why most of this post is about it.

How I keep it from making things up (reducing hallucination)

The card used to be written by the model. Now it is built by code from what the tools returned, and the model writes two sentences that a checker has to approve.

Having nine good sources is not enough by itself. Someone still has to turn what they return into a page. If that someone is the model, this is the step where it can add things the sources never said. So the design leaves the model as little of that step as possible.

Every tool return is stored word for word. From that store, code — not the model — builds the evidence table, the warnings, the source links, and a footer tracing each number back to the call and database release it came from. That traceability is what “provenance” means here.

The model writes two things: a one-line verdict — the GO / NO-GO call — and a short reasoning paragraph. A validator rejects the run if that prose contains a number, an rsID or an identifier absent from the tool output.

How hard it was attacked

An audit of my own agent found 18 real defects. The checks I had already written caught none of them.

Every version of this system has been shipped, attacked, and shipped again. The attack that mattered used 41 adversarial agents — separate LLM instances, each prompted to hunt for defects from one of four angles — with every claim re-checked by an independent second agent before it counted: 86 claimed, 27 survived the re-check, and 18 were confirmed as distinct defects.

My validator flagged 0 of the 18. Every check I had written was necessary, and none of them covered the failures that mattered.

What follows is the engineering story behind that number, for readers who want it.

The case: IL6R × coronary heart disease

For IL6R × coronary heart disease, the MR tool returned:

beta   -0.0441899892357374
se      0.00853005023322569
p       2.21e-07
method  Wald ratio      n_snp 1
instrument rs4129267    cis
steiger_direction_ok  NA
coloc_prob            null
ld_check              null

The card quoted that correctly, and concluded:

GO — genetic and Mendelian randomization evidence supports a causal, protective role of IL6R inhibition in coronary heart disease.

My first reading was that this was backwards. The exposure — what the variant raises or lowers — is IL6R protein in blood plasma. A negative beta says genetically higher IL6R goes with less coronary heart disease. So the data seemed to support raising the target, while the card recommended inhibiting it.

My reading was wrong. Working out why is what the rest of this post is about.

Why the card was right

One piece of biology resolves it, in four steps.

1. The genetic variant behind this estimate — rs4129267, which is inherited together with rs2228145 almost every time, so one stands in for the other (r² ≈ 0.96–0.99 in Europeans) — changes the IL-6 receptor so that it is more easily cut off the cell surface. The cutting is done by an enzyme (ADAM17), and the process is called shedding.
2. The cut-off piece drifts in the blood — and that drifting piece is what the plasma assay measures. So people who carry the variant show more IL6R protein in a blood test.
3. But with receptors cut off the surface, their cells receive less IL-6 signal. Less IL-6 signalling means less inflammation, and the carriers have less coronary heart disease.
4. Put together: in this dataset, high IL6R in plasma does not mean more IL6R at work. It means the opposite — it is the leftover pieces of receptors that no longer work. A drug that blocks this receptor (tocilizumab) pushes in the same protective direction the variant does.

So the card’s conclusion — inhibiting IL6R protects against coronary heart disease — was right, and my “backwards” reading was wrong. I had treated the blood measurement as if it tracked receptor activity, when it actually tracks the loss of it.

None of that biology was in anything the agent retrieved. The tool returned a beta, a p-value and an instrument — the code block above is all of it. Those four steps came from the model’s training memory, and happened to be correct.

That is the failure mode I now worry about most: not a hallucinated fact, but a correct conclusion that the retrieval does not support. An invented fact at least looks wrong to an expert. This does not. The expert reads the card, nods because the sentence is true, and fills the gap from their own memory exactly as the model did. So nobody notices that the pipeline fetched nothing to back it.

Next time the remembered answer will be stale, or wrong, or right for a different protein, and nothing in the system will be able to tell the difference.

The part I could measure

Two cards in the same batch, same outcome, same tool:

IL6R   beta -0.0442  p 2.21e-07  n_snp 1  steiger NA    ld_check null  coloc null
LPA    beta +0.2523  p 5.39e-39  n_snp 1  steiger TRUE  ld_check 1.0   coloc null

Two questions travel with an estimate like this, under three names. The first: does the variant move the protein first and the disease second, rather than the other way round? That check is called Steiger. The second: do both signals trace to the same variant, rather than to two neighbours inherited together? That one is answered either by colocalization or, where the data colocalization needs is missing, by an LD check. The two are alternatives and never appear together — across all 101,543 estimates in this resource, not one carries both.

LPA’s estimate answered both questions: Steiger, and the LD check. IL6R’s answered neither — Steiger NA, LD null, no colocalization: not run, rather than failed. Without them, a single-variant estimate cannot tell a causal effect from a neighbouring gene’s, or from cause and effect running backwards.

Both cards used the same strength of language — "Supports a causal role", in both.

That is a defect you can encode, so I did: the validator now fails any run using causal wording on a single-variant estimate with neither question answered. Re-running the benchmark — a fixed batch of ten cards I score after every change — it fired on exactly the card it was built for. Each run regenerates the model’s two sentences, and this time the wording came back more confident, not less:

The second flag looks like a contradiction and is not. The validator scores what the retrieval supports, not what is true. On the retrieved numbers alone the negative beta reads as “more plasma IL6R, less disease”, so recommending inhibition contradicts the sign. Only the shedding biology closes that gap, and the pipeline never fetched it.

I also had to be careful with the rule itself. An early version of the direction check rejected a sentence that was correct. If a check rejects correct, precise sentences, the model learns to write vague ones instead, and vague sentences are exactly what I am trying to get rid of.

Why a perfect pass rate meant nothing

Ten of ten cards passed validation. I nearly wrote that number down as progress.

Then I counted what was left to check. A checkable token is a number, rsID or identifier the validator can match against stored tool returns — and the whole batch held three. The model had learned the safe move: stop making claims that can be checked. It passed everything by asserting almost nothing.

Later I made the validator stricter, and the same benchmark dropped to 8/10 — both failures were the model asserting clinical status ("clinically proven", "approved") that none of the retrieved sources returns. The score dropped because the check got stricter, and a lower score from a stricter check is progress.

So the pipeline now reports claim density — checkable tokens per 100 words — beside the pass rate. Alone, neither means much; together they answer the real question: how much did it claim, and how much held up? This run: 8 of 10 passed, with 2 checkable tokens across 842 words. That is worse than the run before it, and I report it as it is.

Tested on data I did not pick

The benchmark run described here is from 21 August 2026.

I could not catch this failure when I first wrote about it. So I built checks for its two most common shapes: causal wording the retrieved estimate cannot support, and clinical-status wording — approved, in trials, on the market — that no retrieved record backs. Then I went looking for it in data I had not chosen.

I scored the agent against twenty target–indication pairs sampled from Minikel et al., Nature 2024: ten drugs that launched, ten programmes that died at phase II/III. Whether each pair counts as a success or a failure was decided by what happened in the clinic, not by me. I withheld the clinical-evidence tool, because it reports approval stages directly and would hand the agent the answer.

The verdicts held up better than I expected: every GO was a drug that launched, and none of the ten failures was backed. I am not claiming that as a result. n is twenty, and auditing the run showed part of that precision depends on leftover clinical signal — the Open Targets association score includes every evidence type, clinical records among them, so switching off the clinical tool does not fully remove approval history.

The smaller claim is the useful one: every error was a refusal to answer, never a wrong answer. When evidence was thin, the agent said “not enough” instead of guessing.

The pharmacology was right, and it still did not work

In 2012 the genetics said this pathway lowers heart disease. In 2026 the phase 3 trial missed its endpoint.

The model reached past its retrieval, supplied the shedding biology from memory, and was correct. I had been calling that a near miss: right this time, and likely wrong some other time. Then the answer itself stopped being right.

In 2026, ziltivekimab — an antibody blocking IL-6, the signal this receptor receives — missed its primary endpoint in ZEUS, a phase 3 heart-outcomes trial. Major cardiac events were not reduced: hazard ratio 0.99.

Two limits, before anyone over-reads this. Ziltivekimab targets IL-6, the ligand — one step upstream of the receptor the card was actually about — and ZEUS enrolled patients with atherosclerotic disease and chronic kidney disease, not the general coronary population the MR estimate describes. And a missed endpoint announced by press release is not a published trial. So this is evidence against the pathway, not proof that the card was wrong.

It still sharpens the point. The remembered answer was not just unsupported by the retrieval — it did not stay true. The mechanism was real and the target was engaged: hsCRP, the blood marker of IL-6-driven inflammation, fell exactly as the biology predicted. The outcome did not move.

If the whole field’s best guess could miss the trial result by this much, then a card that stays silent and lets the reader fill the gap from memory is passing on a guess, not a fact.

That is why the card has to show where it has nothing. When it prints Steiger / coloc / LD → NA / NA / NA and will not use the word causal, it is leaving that guess to the reader instead of making it for them.

What I learned

1. "The model was right" is not the same as "the pipeline supported it." Track the second one, because the first will keep being true until it suddenly isn't, and you will have built no way to notice.
2. An estimate existing is not an estimate being validated. If your sources carry their own quality checks, make the language your system is allowed to use depend on them.
3. Ask what your measurement is actually measuring. For plasma proteins the answer is often how much broken-off protein is floating in blood, not how much is working on cells — shedding (receptors cut loose into the blood) and clearance can dominate the number — and whether a protein does this is recorded in the UniProt annotations the pipeline already retrieves.
4. Report how much was checkable, not just how much passed. A pass rate without claim density measures caution, not accuracy.
5. Attack your own guardrails before trusting them. I would have shipped mine believing they worked. I only found out they had caught none of the 18 because other agents went looking for the defects I had not thought of.

The code, the 62 regression tests holding the validator’s rules in place, and the audit are open, in the repo built during the CABS 2026 data science internship. Every MR estimate here is retrieved — exposures from EpiGraphDB’s pQTL resource (Zheng et al., Nature Genetics 2020), outcomes from published GWAS. The agent computes no causal inference of its own, which is why reading it correctly matters so much.

Questions and corrections are both welcome. If you run it on a protein you care about and the card gets something wrong, that is the most useful message I can get. If you work on the same problem — keeping an agent’s claims tied to its sources, or the genetics itself — I am glad to talk. Write to shucheng.cao@mail.mcgill.ca.

Thanks to the CABS 2026 cohort for the arguments that shaped this — including the one that made me go back and check whether the sign was really inverted at all.