An AI note is a draft clinical record produced by software. Reviewing it is a different task from writing it.
Consider a synthetic visit in which a patient mentions a symptom they had last year and says it has not returned. The draft moves that symptom into the current history. The sentence is clear. The detail came from the conversation. The timing is wrong.
A spelling check will not catch that. Neither will counting how many sections the note contains.
We think this is where an AI documentation evaluation should begin: can the person reviewing the draft find the difference between what was said and what the record now claims?
In brief
Treat review as work with its own skills and measurements. Check facts, timing and clinical meaning. Measure what the reviewer catches and what remains after correction, alongside the time needed to finish. A readable draft is not yet a finished record.
A new framework asks what reviewers actually know
A commentary published October 12 in npj Digital Medicine proposes a framework for teaching trainees to review ambient AI documentation. Ambient AI tools capture a clinical conversation and generate a draft note.
The authors name four skills: detecting errors and omissions, reconciling the note with the reasoning of the encounter, writing independently when AI is unavailable, and judging when to rely on the draft or question it.
Their proposal is preliminary. It has not established that this training prevents errors or improves patient outcomes. It is also written for medical education, not as a validated checklist for every independent practice.
Still, it asks a useful question for anyone buying documentation software. If the safety plan says a doctor reviews the note, what does that review involve, and how do we know it happened well?
More complete and more factual are different judgments
An October 9 Scientific Reports paper gives a reason to keep those questions separate. Its prospective evaluation of SMART ASSISTANT, a research system using abdominal pain as the example, rated AI records higher for integrity but lower for factuality than physician records.
The prospective rating scores were 3.50 versus 3.21 for integrity, and 3.34 versus 4.01 for factuality. These are rating scores, not error percentages. The paper also reports other evaluation stages. Those should not be mixed together as if they were one result.
This does not describe every AI scribe. It does not evaluate Thyra. What it does show is why a fuller record and a more accurate record should not be treated as the same outcome.
A practice can value both. It should measure both.
Try the review task before measuring the shortcut
For a first evaluation, use a synthetic encounter and a reference record whose intended facts and reasoning are known. Do not put patient information into an unapproved tool to run the test.
Create a draft with a few deliberate differences. One could move a historical symptom into the present. Another could omit a relevant negative finding. A third could turn a plan discussed as an option into a plan that appears agreed.
These examples test the review process. They are not claims about errors observed in Thyra or a particular vendor.
Ask a reviewer to correct the draft. Then compare the corrected version with the reference. Did they notice each difference? Did a correction introduce another problem? Did the note preserve the clinician's reasoning, rather than merely include the right words?
The training commentary proposes simulations with planted errors. Our suggestion applies that idea to a small practice evaluation. It is a proposed exercise, not a validated safety test or a substitute for the practice's clinical and privacy controls.
Four questions worth keeping beside the draft
Does each statement match its source? Check who said it, what was said and whether uncertainty was preserved.
Is the timing right? A previous treatment, an old symptom and today's plan should not become interchangeable.
Does the note tell the same clinical story? Correct facts can still be given the wrong emphasis or imply a decision the clinician did not make.
Can the reviewer explain the correction? A changed sentence should reflect the encounter, not simply sound better.
These questions are deliberately concrete. They help separate editing for style from checking the medical record.
Count the work after generation
An evaluation that measures only the time to produce a draft stops too early.
Record review time, correction time and the types of changes needed. With synthetic test cases, also record deliberately planted differences that remain after review. Keep those findings separate from normal clinical production data and from patient outcome claims.
A faster review is useful only in context. It may reflect a better draft, a simpler visit or a reviewer who missed something. A longer review may reflect careful work on a difficult encounter. Time alone cannot tell those explanations apart.
At Thyra, we think the meaningful endpoint is a record the clinician can stand behind. The evaluation should follow the work to that point, rather than call a generated paragraph the result.
Questions for a documentation evaluation
What does a planted error exercise tell a practice?
It shows how reviewers handle known differences under controlled conditions. It does not establish the tool's real clinical error rate or prove patient safety.
Should a practice compare note completeness with factual accuracy?
Yes. A draft can cover more information while getting part of it wrong. Keep the judgments separate and define how each is assessed.
What belongs in the timing measure after an AI draft appears?
The work needed to review, correct and finish the note. Report that alongside the quality assessment, not in place of it.
The review is part of the job. It deserves more than a checkbox.
Sources
Review competency framework, npj Digital Medicine, October 12, 2026
SMART ASSISTANT evaluation, Scientific Reports, October 9, 2026