They asked for AI.
What they needed first
was a protocol a machine
could read.

Two problems, one shared cause

The sponsor came with two asks and assumed they needed two answers. They shared a root, which is the only reason the work was affordable.

A third problem, which nobody had written down

Roughly three weeks into the diagnosis a pattern emerged that neither ask covered. The site problem and the screening problem were both being created upstream, at protocol design, in a conversation between the sponsor and the CRO with almost no mechanism behind it. We added it as a third track. It carries the largest prize in the engagement and it appears early in this piece, at section 08, because everything after it reads differently once you have seen it.

Three questions we had to answer

Eight weeks, four sites, one criterion followed all the way

We sat with coordinators during screening visits in three countries, walked the systems they use, read the support desk export, and traced a single inclusion criterion through every document that restates it. That last exercise took an afternoon and changed the shape of the engagement.

Eight working sessions with the sponsor, the CRO and two site teams, each one ending in a decision rather than a summary.

Where our own view changed

Three things we got wrong early, and when we noticed:

• Week two. We arrived expecting a retrieval-quality problem and spent the first fortnight scoping model improvements. The criterion trace in section 03 ended that line of work.
• Week four. We had France down as a capacity problem, because that is how the country team described it. Comparing it with Portugal made clear that four candidate causes were unseparated, and we withdrew the recommendation we had drafted.
• Week six. Track C did not exist in the original scope. It emerged from noticing that the same criteria kept appearing in both problem sets, and it is now the part of the work we would lead with.

What we could stand behind, and what we couldn't

Four classes, marked at the point of the claim rather than buried in a footnote.

On-site trial management is a reconciliation job

We expected people short of tools. We found people surrounded by tools and short of one fact: which version of the protocol is in force at this site, today.

Modelled Figures show the shape of what we observed, not audited client data.

One question, followed all the way

An ordinary event: a coordinator, mid-visit, isn't sure whether a candidate meets an inclusion criterion. Nothing here is smoothed. The phone call is drawn because the phone call is where the answer actually comes from, and no system in the organisation can see it.

Why the version question has no answer

We took one inclusion criterion and followed it into every artefact that restates it.

The data underneath, scored on fitness rather than completeness

A field can be fully populated and worthless. The clearest example in this case looks healthy in every report the sponsor produces.

Which decisions actually matter, and what they were missing

We anchored the whole design on decisions rather than processes, because a decision is the only unit that tells you what evidence a decision requires. Three questions per decision: what evidence would a competent person need, does it exist anywhere, and can they reach it in time. The third is where nearly everything fails, and conventional data-maturity assessments don't ask it.

What this does to the person holding it

We ran the coordinator's role through a work-system analysis — person, tasks, tools, environment and organisation examined together rather than blaming any one of them. The finding is specific and uncomfortable.

So where can AI help on Track A? Q1

In three places, and only after one non-AI thing is fixed.

Two of the three screening errors are invisible

You cannot reduce an error rate nobody measures. Before designing anything, we had to establish what kinds of error screening produces and which of them the organisation can currently see.

Where screening loses people

The constraint that shapes everything here

The sponsor has no lawful route to identifiable records for anyone screened and never enrolled. That is not a gap to close. It is the shape of the problem.

Screening intelligence cannot be built by moving patient data to the sponsor, because the patients who matter most — the ones turned away — are the ones whose data can never move. So the computation goes to the site and only criterion-level outcomes come back. Section 15 is what that looks like.

What the organisation believed, and what survived

The brief arrived with confident positions. We treated each as a hypothesis and tested it against what the material could actually support. Three survived, three were reframed, two were contradicted. The reframings mattered most, because they redirected effort that was about to be spent in the wrong place.

Arguing for retrieval in a room that had reasons to say no

Recommending a retrieval assistant into a GxP environment is not a technical argument, it is a governance one. We knew which objections would land before we walked in, because they are the same six every time, and we built the case around answering them rather than around the technology.

The list below is what a sponsor's quality, medical, data and privacy functions each need satisfied before anything like this gets approved. Working through them is what turned a technology recommendation into a service design, and each one produced a design constraint we would not otherwise have written.

Every objection came back to the same thing: the system had no way to know what was true, when, for whom, and no way to say so when it didn't.

Six objections, six requirements

We stopped defending the recommendation and rewrote it against what they'd said. This is the pivot of the engagement.

Six solution archetypes, assessed

Six archetypes, assessed as at June 2024. Named examples illustrate each category — they are not endorsements, evaluations or a competitive assessment.

Sponsor and CRO, designing the trial together

Neither ask covered this. But both problems are manufactured upstream, at protocol design, in a conversation that has almost no mechanism behind it. This is the section with the largest prize and the least prior art.

How they actually communicate

The sponsor writes the protocol. The CRO and the sites execute it. Between those two facts sits a set of channels that are all either slow, low-bandwidth, or leave no record — and usually all three.

What we built instead: run the protocol before you lock it

Two things we'd already built made this possible without either party seeing the other's data. Criteria are now structured objects (14), and there is a local engine at each site that can evaluate a criterion against that site's own population and return only counts above a cohort threshold (15). Put those together and a draft protocol becomes a query you can run.

The second shared object: what a change actually costs

The other conversation that goes badly is amendments. The sponsor proposes a change; the CRO estimates effort; neither has a shared picture of the blast radius, so the negotiation is about the number rather than the change. Once the protocol is a graph, the blast radius is computable and both parties look at the same figure.

The operating model that goes with it

Software on its own does not change how two organisations talk. What changes the conversation is having a shared object to point at, and a defined moment to point at it.

The AI service blueprint

Two tracks drawn separately. They are different services with different users, data and legal exposure. What they share is the layer underneath. Track C gets its own section, because it changes who is in the room rather than what happens in it.

Track A · on-site trial management

The same event as Fig 01, redesigned. The test we applied wasn't “what does the product add?” but “what does the coordinator stop doing?”

Track B · patient screening

A different service. Computation runs at the site because the data can't leave it, and the output is criterion-level evidence rather than a verdict.

Fourteen techniques considered, five built

Before designing anything we assessed the available AI techniques against this organisation rather than against the field. Two questions per technique: could it do the job, and could this company run it. The second killed more candidates than the first.

Where each one sits

Could this organisation actually run it?

Capability assessments usually stop at the technique. The more useful question is organisational: five dimensions, scored one to five, per capability area. A capability with strong data and weak governance fails just as completely as one with no data.

Where the machine was allowed to touch the work

Automation is not a property of a task. It's a property of a stage within a task. Every significant activity was split into four stages and allocated separately at each one. This is the piece of work that stops “AI-assisted” from being a phrase that means nothing.

Exactly what the AI does — and what it doesn't

Ten capabilities. Five use a language model. Five are ordinary deterministic software that gets called AI far too often. Naming which is which is the difference between a system you can validate and one you can only hope about.

Three sources of truth, and the rules between them

Most descriptions of RAG assume one corpus. There are three here, they are not comparable, and the interesting engineering is in how a question gets routed between them and what happens when they disagree.

Routing happens before scoring

The classifier's job is not to pick an answer. It decides which substrates can legitimately answer the question, and a large share of questions never reach retrieval at all because the structured store holds an exact value.

Three precedence rules, none of them weights

Inside route B: the version gate runs first

The scoring function, and why it isn't off the shelf

Relevance alone ranks a training slide above a protocol section, because the slide is written in the same plain language as the question. Five domain features fix that.

Two numbers decide whether it answers at all

How this scales to forty-one studies without moving any patient data

Federated architecture is the backbone of both Track B and Track C, so it belongs in the open rather than in an appendix. What follows is the ladder, what actually blocks each rung, and where we stopped.

The topology

The five levels

What actually blocks federated learning here

Not compute, and not model architecture. Six things, in the order they bite.

The economics of stopping at level 3

Level 4 becomes justifiable in one specific circumstance: when you need a prediction that a count cannot give, and no single site has enough positive examples to build it alone. The realistic candidate here is pre-screening prioritisation — ranking which chart to review first. That is a genuine federated-learning problem, and it is worth revisiting once levels 1 to 3 have run long enough to produce labels. It was not worth starting with.

Running it on Claude

Claude is not the retriever and not the system of record. It does five bounded jobs at three cost tiers, wrapped in deterministic code that decides what it's allowed to see.

Getting staff and patients onto the system

Access control in clinical research is usually rebuilt from scratch for every tool, badly. There is an artefact that already exists, is already maintained, and is already inspected, and almost nobody wires systems to it.

Staff: the delegation log is the access layer

Good clinical practice already requires a delegation of authority log for every study, naming who may perform which trial-related duties and from when. It is signed by the investigator, kept current, and examined at inspection. It is also, in most organisations, a spreadsheet that no system reads.

Patients: mostly they never touch it, and that is a design decision

Everything in Track B runs behind the site's own systems and is used by the coordinator and investigator. The patient experiences it as a shorter wait and fewer repeat visits. That is the intended relationship, and there is a reason to be careful about changing it.

On-site trial management

Four changes. Only one of them is a model. Together they close the reconciliation loop that was consuming the coordinator's day.

Reducing the screening error rate

Five mechanisms, in the order they contribute. Only the third involves a model reading a patient note, and it's the narrowest possible use of one.

Mechanism 5 reduces nothing. It's what turns an invisible error into a measurable one, which is why it was built first.

The evidence this approach works

Not speculative. In June 2024 — the same month as this engagement — Mass General Brigham published RECTIFIER in NEJM AI: a retrieval-augmented GPT-4 system answering thirteen eligibility criteria as separate questions against clinical notes, for a heart-failure trial. It's the closest published analogue to this architecture, and the result is instructive in a specific way.

Which lever actually moves the number

Where the human line sits, and why there

Every checkpoint is justified by something specific — an irreversible action, an asymmetric error cost, or an accountability a regulator requires. “Healthcare is sensitive” wasn't accepted as a reason for any of them.

The four states the interface can be in

No numeric confidence score reaches the user. A named state tells someone what kind of checking to do; a percentage invites them to invent a private threshold. The escalation route is visible in all four states, including the confident one, so escalating never reads as the tool having failed.

Spotting errors and trouble on site

Once questions, answers and decisions exist as events, eight things become detectable that previously were not. None of them need a model. All of them need the event record from section 17.

What the country manager sees on a Monday

What this is worth, and when it lands

Five layers of value, each traced from the problem through the mechanism to a measurement that would confirm it. Nothing here is a claimed saving. Each row names the number that would prove it and the baseline that has to exist first.

When each layer lands

The layers do not arrive together, and presenting them as though they do is how these programmes lose credibility in year two.

From recorded events to decisions someone can act on

Once questions flow through one surface, the organisation starts producing operational data it has never had. Here's how that becomes something a country manager can act on — one real question, traced all the way up.

What we measure, and what would make us stop

The previous pilot reported one number — 64.9% adoption — meaning a single authenticated login. Two levels down, use that had actually changed how work was done stood at 4.1%. Nothing measured whether the answers were right.

Adoption, redefined so it can't flatter anyone

Rungs 5 and 6 didn't exist in the pilot. They're the only two that can support a decision to scale, and both depend on the re-adjudication baseline being established first.

Assumptions we tested before building

Section 04 tested the organisation's assumptions about the problem. This tests ours about the solution. Each row is something we believed at the point of designing, how we checked it, and what we did when it did not hold.

Still open

Six things this work did not settle. They are listed because a portfolio piece that resolves everything is describing something that did not happen.

Ordered by dependency, not by quarter

No dates. Each step is gated by an artefact the previous step has to produce, which is what lets the sequence survive a slipped quarter, a reorganisation or a change of priority — none of which a phased plan survives.

What people actually see

Three screens, placed last. The interface is the smallest part of this work and the part that would have been designed first if we'd got the sequence wrong.