Willis eCOA

I started with immersion: regulatory training, the persona research I inherited, and recorded interviews with sponsors and clinicians to understand each role more deeply.

The research revolved around 7 roles, each sitting somewhere between the participant and the data. Participants answered their own questionnaires. Raters focused on participants and their own training, and coordinators on enrollment and scheduling. PIs² covered participants and data, plus how their raters performed. Sponsors and CROs³ focused on data across sites, and our ops team on setting up studies and keeping them running.

From what we had in writing and recordings, I mapped the product plan, core workflows and permission matrices to see where more research was needed. We hadn’t hired a front-end engineer yet, so I set up a starter UI kit, built on Untitled UI, for when one joined.

The persona research gave a general profile of sponsors, PIs, coordinators and clinicians, and the recordings were of the founder’s own interviews. Our own sessions in December filled in more. I also inherited detailed product requirements. The initial team was small: a PM, a few scientists and a CTO with a couple of back-end engineers.

7 roles meant 7 versions of every flow, more than our timeline allowed. So we designed most flows twice: once for sponsors, CROs and our ops team watching many sites, and once for site staff running visits.

Walking both groups through mid-fidelity screens of every core flow showed how much still lived on paper: PHI⁴ in a physical book, notes on loose sheets, certifications and past visits in binders. Even clinicians with permission were uneasy seeing PHI on screen.

We had planned full participant management in Willis: names, dates of birth, contact details and, for participants who needed one, a care partner. In testing, a few sites were keen to use it. Most wanted control over what they entered, so PHI became configurable per study, with a participant ID as the only requirement. Notes moved into each assessment, and sites were relieved that having certifications and past visits in the app meant less time tracking down binders and files.

On a participant’s visit page, each assessment is a card showing its status, rater and score. Our early cards also showed where the score fell on the scale’s range, with its severity label (e.g., MADRS⁵ 19 = mild depression). Raters read a visit as a whole and already knew what the numbers meant, so the marker and label were noise that could mislead. We kept the score alone.

To test the response to an AI score placed beside the raters’, we started with a slider to show the comparison between the two scores: WillisAI’s range as a band, with the rater’s score marked against it.

The same two numbers read differently depending on who was reviewing. Sponsors saw another layer of evidence behind every score in their study. Raters and PIs saw their judgment graded against a single recording of a participant they had seen visit after visit. That made the item-by-item breakdown we’d already planned more important: WillisAI had to show the reasoning behind its score, so raters could treat it as a second opinion to check against their own.

Initial requirements described a far larger platform than we could build by May, so we kept only what a visit needed to run from site to WillisAI to sponsor.

For sites, that meant profiles, visit timelines and an assessment template most ClinRO and PRO scales could be built into. Raters would see their certification status on each required scale and train for the rest. Sponsors would follow the study through an overview, flags and analytics, each score beside WillisAI’s. Queries and data change requests, which question or correct data after a visit, were tabled for later releases.

The launch was timed to the ASCP⁶ conference in late May. Real visits came first, since WillisAI needed recordings to evaluate. Only qualified clinicians could run them, so staff management put certified raters on each study, and rater profiles showed sponsors who was ready to rate.

Development and staging ran on mock data, analytics included, so we could build and demo before a single real visit, with no participant data outside production.

Onboarding turned out more complex than anticipated, largely because of regulation. Under 21 CFR Part 11⁷, each electronic signature in a trial must belong to one verified person, so every account needed a confirmed identity before anyone could sign anything.

That made a forwarded invitation a real risk: whoever held it could set a password first, lock out the intended user and strand the account. So an emailed code came before any credential, and no one counted as logged in until they’d verified.

Brooklyn Health set up each organization (a sponsor, CRO or site) and named its admin, who was invited first, finished the organization’s details, then invited their own staff. Each account had one email but could belong to several organizations, so an admin of more than one had to complete each organization’s details.

Password creation moved from the first screen into personal details, closing the window where an unverified account already had a credential. Anyone who left after verifying lost what they’d entered, so invitations stayed valid for 72 hours, and users could request a new one or their admin could resend it. SSO users authenticated through Google or Microsoft, so only password users required MFA.

We also planned a PIN so raters sharing a visit could pass one tablet without a full logout. We planned to provide sites with tablets once the iOS app launched, so like most iOS flows, the PIN was tabled until after the MVP.

With launch two months out, we took on the raters’ core task: administering an assessment during visits. We optimized it for tablets, the device typically used during visits, and designed it in segments engineering could assemble into a template for any scale: clinician-rated scales with required recording, item navigation and notes, and patient-reported questionnaires simplified for the participant.

Every step also had to stand up to an audit. Visits and assessments record who started and finished them and when, and each Part 11 signature joins a history built for the data change requests to come.

Recording was required for clinician-rated scales, since WillisAI needs every interview. Raters had to be able to skip items, take notes on each, and pause if needed, but could not cancel. A cancel option was dropped after sites said it could count as malpractice.

With rater PINs deferred, raters signed off on assessments with their username and password, and a history panel recorded the date, time, name and score at each signing.

Patient-reported screens used larger text, recorded nothing and locked answers once submitted. Handoff screens, with short animated loops by illustrator Doug Fuchs, marked the tablet passing to the participant and back. Taking it back required the rater’s signature.

Willis would run in trials across countries, so the platform had to be ready for more than one language. At sites that ran visits in several languages, each participant’s language was recorded when they were added, so the rater knew which language the visit would be in.

The visit header tracked each visit from pending to in progress to completed, and starting before the window required a reason.

After launch, Willis grew release by release: from one product to two (Willis eCOA and Willis Academy), from web to web and native iOS, and from 7 roles to 9, with permission layers each study could configure.