Shucheng Cao

Ideas / failures / validation

Blog

Longer accounts of how a result failed, what the failure exposed, and which checks were worth keeping.

  1. Share card: my own checks caught none of the eighteen defects an audit found

    · Engineering

    Two agents, and no way to tell if either one workedwhy I helped start a benchmark instead of building a third

    Part 3. I wrote the agents and I wrote the checks that graded them — which is a self-assessment, not a test. Why that led to helping start a benchmark, and the honest state of it.

    benchmarks evaluation LLM agents
    Read the post
  2. Share card: it refuses the question it cannot read — a garbled request answered with, I could not read a pair out of that

    · Engineering

    Building an AI agent that can’t fake drug-target evidencefrom a metabolite in blood, through the gene behind it, to whether it is worth a drug

    Part 2. LLMs hallucinate. The same agent rebuilt to start from a molecule in blood: it names the gene behind the molecule, refuses a question it cannot read, and computes its verdict instead of letting a model write one.

    LLM agents evidence provenance human genetics
    Read the post
  3. A research timeline connecting an IL6R genetic prediction to a phase 3 clinical trial

    · Engineering

    Building an AI agent that can’t fake drug-target evidencefrom a protein to whether it is worth a drug

    Part 1. LLMs hallucinate, and gathering the evidence behind a result is still done by hand. An agent for biomedicine that only prints what its sources returned.

    LLM agents evaluation causal inference
    Read the post