They asked for AI.
What they needed first
was a protocol a machine
could read.

Both problems shared a cause

The company came with two asks and assumed they needed two answers. They shared a root.

A third problem, which nobody had written down

About three weeks in, a pattern showed up that neither ask covered. Both problems start upstream, when the trial gets designed, in a conversation between the company and its research partner with almost nothing behind it. We added it as a third track. It carries the biggest prize and it appears early here, at section 08, because everything after it reads differently once you've seen it.

Three questions we had to answer

We followed one rule through four sites in eight weeks

We sat with coordinators during screening visits in three countries, walked the systems they use, read the support desk export, and followed a single eligibility rule through every document that repeats it. That last exercise took an afternoon and changed the shape of the whole engagement.

Eight working sessions with the company, its research partner and two site teams. Each one ended in a decision.

How the diagnosis changed as we went

Three things we saw differently by the end:

• Week two. We arrived expecting a search-quality problem and spent the first fortnight on model improvements. The rule trace in section 03 pointed the work somewhere else entirely.
• Week four. We had France down as a staffing problem, because that's how the country team described it. Comparing it with Portugal showed four possible causes tangled together, and we went back to separate them.
• Week six. Track C wasn't in the original scope. It came from noticing the same rules kept turning up in both problem sets, and it's now the part of the work we'd lead with.

How confident we are, line by line

Four classes, marked at the point of the claim.

Running a trial on site is a cross-checking job

We expected people short of tools. We found people surrounded by tools and short of one fact: which version of the rules is in force at this site, today.

Modelled Figures show the shape of what we observed.

Following one question end to end

An ordinary moment. A coordinator, mid-visit, isn't sure whether a candidate meets an eligibility rule. Nothing here is smoothed. The phone call is drawn because the phone call is where the answer comes from, and no system in the company can see it.

Why the version question has no answer

We took one eligibility rule and followed it into every document that repeats it.

The data underneath, scored on whether it can do its job

A field can be completely filled in and still worthless. The clearest example here looks healthy in every report the company produces.

Which decisions carry the weight, and what they were missing

We anchored the design on decisions, not processes, because a decision is the only unit that tells you what evidence it needs. Three questions per decision: what would a competent person need to know, does it exist anywhere, and can they reach it in time. The third is where nearly everything fails, and the usual data assessments don't ask it.

What this does to the person holding it

We ran the coordinator's role through a work-system analysis, looking at the person, the tasks, the tools, the environment and the organisation together instead of blaming any one of them. The finding is specific and uncomfortable.

So where can AI help on Track A? Q1

In three places, and only after one non-AI thing is fixed.

Two of the three screening errors are invisible

You can't reduce an error rate nobody measures. Before designing anything, we had to work out what kinds of error screening produces and which of them the company can currently see.

Where screening loses people

This constraint shapes everything here

The company has no lawful route to identifiable records for anyone screened who never joins the trial. That's the shape of the problem, and the design starts from it.

You can't build screening intelligence by moving patient data to the company, because the patients who matter most (the ones turned away) are the ones whose data can never move. So the computation goes to the hospital, and only rule-level outcomes come back. Section 15 is what that looks like.

What the company believed, and what held up

The brief arrived with confident positions. We treated each as something to test against what the evidence could support. Three held, three needed reframing, two turned out the other way. The reframings mattered most, because they redirected effort that was about to be spent in the wrong place.

Arguing for it in a room that had reasons to say no

Recommending a search assistant into a regulated clinical environment is a governance argument. We knew which objections would land before we walked in, because they're the same six every time, and we built the case around answering them.

Below is what a company's quality, medical, data and privacy functions each need satisfied before anything like this gets approved. Working through them is what turned a technology recommendation into a service design, and each one produced a design constraint we wouldn't otherwise have written.

Every objection came back to the same thing: the system had no way to know what was true, when, for whom, and no way to say so when it didn't.

Every objection became a requirement

We stopped defending the recommendation and rewrote it against what they'd said. This is the pivot of the engagement.

We assessed six kinds of solution

Six categories, assessed as at June 2024. Named examples illustrate each category. They aren't endorsements or a competitive assessment.

Designing the trial together

Neither ask covered this. But both problems are made upstream, when the trial is designed, in a conversation that has almost nothing behind it. This is the section with the biggest prize and the least prior art.

How they communicate today

The company writes the protocol. The partner and the hospitals run it. Between those two facts sits a set of channels that are all either slow, thin, or leave no record, and usually all three.

What we built instead: run the protocol before you sign it off

Two things we'd already built made this possible without either party seeing the other's data. Rules are now structured records (14), and there's a local engine at each hospital that can check a rule against that hospital's own patients and return only counts above a minimum group size (15). Put those together and a draft protocol becomes a query you can run.

The second shared object: what a change costs

The other conversation that goes badly is amendments. The company proposes a change, the partner estimates effort, and neither has a shared picture of what it touches, so the negotiation is about the number instead of the change. Once the protocol is a graph, what it touches is computable and both parties look at the same figure.

The operating model that goes with it

Software on its own doesn't change how two organisations talk. What changes the conversation is having a shared thing to point at, and a defined moment to point at it.

The AI service blueprint

Two tracks drawn separately. They're different services with different users, data and legal exposure. What they share is the layer underneath. Track C gets its own section, because it changes who's in the room.

Track A · running the trial on site

The same moment as Fig 01, redesigned. We asked what the coordinator stops doing.

Track B · patient screening

A different service. The computation runs at the hospital because the data can't leave it, and the output is evidence rule by rule instead of a yes or no.

We assessed fourteen techniques and built five

Before designing anything we assessed the available AI techniques against this company. Two questions per technique: could it do the job, and could this company run it. The second decided more than the first.

Where each one sits

Could this company run it?

Capability assessments usually stop at the technique. The more useful question is about the organisation: five dimensions, scored one to five, per capability. A capability with strong data and weak governance fails just as completely as one with no data.

We split every task four ways before automating it

You can't automate a task. You automate a stage inside it. Every significant activity got split into four stages and allocated separately at each one. This is the piece of work that stops “AI-assisted” from being a phrase that means nothing.

Exactly what the AI does, and what it doesn't

Ten capabilities. Five use a language model. Five are ordinary software that gets called AI far too often. Naming which is which is the difference between a system you can validate and one you can only hope about.

The rules between three sources of truth

Most descriptions of this kind of system assume one pile of documents. There are three here, they aren't comparable, and the interesting engineering is in how a question gets routed between them and what happens when they disagree.

Routing happens before scoring

The classifier's job isn't to pick an answer. It decides which sources can legitimately answer the question, and a large share of questions never reach the search at all, because the structured store holds an exact value.

None of the three precedence rules is a weight

Inside route B: the version gate runs first

The scoring, and why it isn't off the shelf

Relevance alone ranks a training slide above a protocol section, because the slide is written in the same plain language as the question. Five domain features fix that.

Two numbers decide whether it answers at all

How this scales to forty-one studies without moving any patient data

The distributed architecture is the backbone of both Track B and Track C, so it belongs in the open. What follows is the ladder, what blocks each rung, and where we stopped.

The topology

The five levels

What blocks a shared model here

Not computing power, and not model architecture. Six things, in the order they bite.

The economics of stopping at level 3

Level 4 becomes worth it in one specific circumstance: when you need a prediction that a count can't give, and no single hospital has enough examples to build it alone. The realistic candidate here is deciding which chart to review first. That's a genuine distributed-learning problem, and it earns a second look once levels 1 to 3 have run long enough to produce the labels.

Running it on Claude

Claude isn't the search engine and isn't the system of record. It does five bounded jobs at three cost tiers, wrapped in plain code that decides what it's allowed to see.

Getting staff and patients onto the system

Access control in clinical research usually gets rebuilt from scratch for every tool, badly. There's a record that already exists, is already kept current, and is already inspected, and almost nobody wires systems to it.

Staff: the delegation log is the access layer

Trial regulations already require a log naming who may perform which trial duties and from when. It's signed by the lead doctor, kept current, and examined at inspection. It's also, in most organisations, a spreadsheet that no system reads.

Patients: mostly they never touch it, and that's deliberate

Everything in Track B runs behind the hospital's own systems and is used by the coordinator and the doctor. The patient experiences it as a shorter wait and fewer repeat visits. That's the intended relationship, and there's a reason to be careful about changing it.

Running the trial on site

Four changes. Only one of them is a model. Together they close the cross-checking loop that was eating the coordinator's day.

Reducing the screening error rate

Five mechanisms, in the order they contribute. Only the third involves a model reading a patient note, and it's the narrowest possible use of one.

Mechanism 5 reduces nothing. It's what turns an invisible error into a measurable one.

The evidence this approach works

Not speculative. In June 2024, the same month as this engagement, Mass General Brigham published RECTIFIER in NEJM AI: a system answering thirteen eligibility rules as separate questions against clinical notes, for a heart-failure trial. It's the closest published analogue to this architecture, and the result is instructive in a specific way.

Which lever moves the number

Why the human line sits where it does

Every checkpoint is justified by something specific: an action you can't undo, one kind of error costing far more than the other, or accountability a regulator requires. “Healthcare is sensitive” wasn't accepted as a reason for any of them.

The four states the interface can be in

No percentage confidence score reaches the user. A named state tells someone what kind of checking to do. A percentage invites them to invent a private threshold. The escalation route is visible in all four states, including the confident one, so escalating never reads as the tool having failed.

Spotting errors and trouble on site

Once questions, answers and decisions exist as records, eight things become visible that previously weren't. None of them need a model. All of them need the event record from section 17.

What the country manager sees on a Monday

What this is worth, and when it lands

Five layers of value, each traced from the problem through the mechanism to something you could measure. Each row names the number that would prove it and the baseline that has to exist first.

When each layer lands

The layers don't arrive together, and presenting them as though they do is how these programmes lose credibility in year two.

From recorded events to decisions someone can act on

Once questions flow through one place, the company starts producing operational data it's never had. Here's how that becomes something a country manager can act on, traced from one real question all the way up.

What we measure

The earlier pilot reported one number, 64.9% adoption, meaning a single login. Two levels down, use that had changed how work was done stood at 4.1%. Nothing measured whether the answers were right.

Adoption, redefined so it can't flatter anyone

Rungs 5 and 6 didn't exist in the pilot. They're the only two that can support a decision to scale, and both depend on the re-review baseline being established first.

Ordered by dependency, not by quarter

No dates. Each step is gated by something the previous step has to produce. A phased plan survives none of those.

What people see

Three screens, placed last. The interface is the smallest part of this work and the part that would have been designed first if we'd got the sequence wrong.