The AI Scribe, the False Mushroom, and the Rural Clinic

Audio article

Uses the speech voice supplied by your browser or device.

Acronyms and terms used in this post:

• AI: Artificial Intelligence, software that generates, classifies, predicts, summarises, or acts on patterns in data.
• ASR: Automatic Speech Recognition, the layer that converts recorded speech into text.
• EHR: Electronic Health Record, the clinical system in which patient care is documented and managed.
• IT: Information Technology, the people, systems, and operating practices that keep digital healthcare working.
• LLM: Large Language Model, a model that generates or transforms text from learned statistical patterns.
• PHC: Primary Health Centre, a frontline public healthcare facility in India.
• RLHF: Reinforcement Learning from Human Feedback, one method used to tune some AI models towards preferred responses; it is not used in the same way by every model, and it is not necessarily the source of a scribe error.

---

A woman went to a urologist and, some weeks later, discovered another woman living inside her medical record.

This second woman microdosed psychedelic mushrooms.

The real woman, Rebecca Green, said she had never taken them. She said mushrooms had not entered the consultation at all. Yet a specialist letter sent to her general practitioner stated that she used them. According to the ABC News report below, an AI scribe had been used during the appointment, the doctor had not caught the false statement before it travelled, and the doctor later apologised.

The clip cannot tell us exactly where the machinery failed. It does not identify the product or provide the audio, intermediate transcript, prompt, generated draft, edit history, or system logs. The mistake attributed to the scribe may have begun in speech recognition, summarisation, speaker attribution, a template, integration, or some unrecorded interaction between machine and human. That uncertainty matters because patient safety is not improved by confidently blaming the wrong layer.

The false statement matters too.

This is not proof that every AI scribe is dangerous or every doctor is careless.

It is a reported example of something narrower and more useful: a plausible, clinically consequential sentence appeared without being true, and an ordinary clinical workflow allowed it to leave the room.

That is enough to deserve our attention.

We are in the useful and dangerous middle. These tools are good enough to spread, imperfectly characterised enough to require close supervision, and commercially urgent enough that organisations are tempted to collect the benefit before paying for the controls.

This is not an argument against the scribe

Doctors did not begin using ambient scribes because they had developed a sudden aesthetic affection for microphones. Clinical documentation is exhausting. A consultation can become a three-way custody dispute between the patient, the keyboard, and the clock. The patient speaks. The doctor types. Eye contact becomes an intermittent service.

If a scribe lets a clinician look at the patient instead of pecking at an EHR template, that is a real benefit. If it reduces the work waiting after clinic, that is a real benefit. If it helps a tired doctor preserve more detail, that is a real benefit.

The evidence is promising without being magical. A JAMA Network Open quality-improvement study associated ambient-scribe use with lower cognitive burden and a modest reduction in note time, although it did not find a statistically significant reduction in burnout or lower off-hours EHR work. Other studies have reported stronger improvements in perceived burden. The sensible conclusion is neither miracle nor fraud. These tools help some clinicians in some workflows, and the size of the benefit remains dependent on the product, setting, and way it is used.

Human documentation is not pure either. Doctors omit things. Human scribes misunderstand things. Notes are copied forward. Laterality gets reversed. Negations disappear. Medication histories breed like damp fungus in old problem lists. A small Australian simulation comparing four commercial AI scribes with notes written by two GPs across four consultations found no statistically significant difference in overall scores across the five documentation sources. Some products performed better in particular domains, but the sample was tiny and the authors asked for diverse real-world studies. It is a useful antidote to the fantasy that the human baseline is a marble statue of perfection.

The question is therefore not whether machines make mistakes while humans float above the ward on wings.

The question is what happens to a mistake after software makes it easy to generate, easy to approve, easy to distribute, and easy to repeat.

A medical record is a rumour with institutional privileges

A false sentence in an ordinary conversation may die in the room. A false sentence in a medical record acquires a badge.

It can enter a referral letter. It can be copied into another note. It can alter how a later clinician interprets pain, confusion, anxiety, adherence, or requests for medication. It can affect what questions are asked and which answers are believed. Depending on the health system and jurisdiction, inaccurate information may also reach insurers, auditors, care coordinators, or other organisations that were never present when the original error occurred.

The patient then faces an ugly asymmetry. The computer already looks official. She must prove that the official-looking thing invented her.

This is why editing the first record is necessary but may not be sufficient. Australia’s privacy regulator explains that a person can ask a healthcare provider to correct health information and notify other parties that received the inaccurate version. In operational language, correction needs a blast radius. Which letter was sent? Which chart absorbed it? Which decision used it? Who received the old version? Can the organisation prove that the repair travelled as far as the mistake?

Healthcare IT people understand interface queues, document repositories, audit trails, downstream feeds, and copy-forward behaviour. We must now understand correction as a distributed systems problem.

The false fact is not repaired merely because one screen looks cleaner.

The error that scales is a different animal

One poorly checked note can hurt one patient. A shared product defect, model change, prompt error, template problem, or systematic weakness with a population can touch many patients before anybody recognises the shape.

Software multiplies productivity.

It also multiplies failure modes.

This does not make software uniquely evil. A bad institutional policy, defective device, contaminated supply, or foolish documentation template can also create many victims. Healthcare already knows systemic harm. AI simply gives it a new delivery vehicle: fast, plausible, centrally updated, and capable of changing behind a familiar product name.

A 2025 study of seven commercial clinical AI scribes found that none produced error-free summaries across eight standardised consultations. Most counted errors were omissions. Hallucinations and factual inaccuracies were less common, but proportionally more likely to be clinically serious. The authors also described overall summarisation accuracy as high and found large differences among products and scenarios.

That combination is exactly why this transition is difficult.

The systems can be good enough to become routine and imperfect enough to require vigilance.

A useless tool is easy to reject. A useful tool with a quiet tail of serious errors is an operations problem. It saves enough time that people want it, sounds polished enough that people trust it, and fails rarely enough that checking every line begins to feel ceremonial. Then one day a mushroom grows in the chart.

The danger is not only the individual mistake. It is the moment the organisation corrects Rebecca’s record, closes the ticket, and fails to ask whether the same mechanism has written a different fiction under twenty other names.

Every model arrives with invisible luggage

An AI scribe is usually not one intelligence sitting politely in the consultation. One common architecture is a chain.

The microphone captures sound. An ASR model estimates what was said. A language model or related summarisation system decides what belongs in the note and how to organise it. Vendor prompts and templates shape the result. Integration software places the draft into a clinical workflow. A clinician reviews, edits, or approves it. Each layer can preserve an error, correct it, or manufacture a new one.

Each layer also carries assumptions.

It assumes something about language, accent, speaking speed, room noise, microphone quality, interruptions, medical vocabulary, turn-taking, and who is speaking. The summariser assumes something about what counts as clinically relevant. The template assumes something about how a consultation is structured. The workflow assumes that the clinician has enough time and attention to inspect the draft. The consent process assumes the patient can refuse without quietly losing access or goodwill.

Then there is the data.

We often do not know whether the speech model heard enough Bengali-accented English, Bangla, or code-switched clinical conversation. We may not know whether the summariser was tested on Indian medicine brands, local abbreviations, public-sector workflows, or descriptions of illness that do not resemble a textbook sentence. We may not know which foundation model sits underneath the product at all. Digital Rights Watch reported that none of the Australian-based vendors it reviewed publicly disclosed the underlying model.

It is reasonable to worry that some training corpora, preference data, and evaluation sets overrepresent well-recorded populations in affluent countries. But this incident cannot tell us that RLHF caused the error, and speech-recognition systems and language models are trained and tuned through different mixtures of data and methods. The sharper problem is undisclosed provenance: a clinic may be unable to learn whether its languages, accents, patients, and workflows were adequately represented at all.

We do not know enough, yet deployment can proceed as if the missing knowledge does not matter.

The World Health Organization’s guidance on AI for health warns that training datasets often underrepresent women, ethnic minorities, older people, rural communities, and disadvantaged groups. The Australian Commission on Safety and Quality in Health Care similarly tells clinicians to confirm that an ambient scribe was trained and tested on data consistent with the population in which it will be used.

“Works” is an incomplete sentence.

Works for whom, speaking which language, into which microphone, in what kind of room, under whose workflow, with what opportunity for correction?

The rural West Bengal clinic is not an edge case

Imagine the product travelling from a polished demonstration into a PHC in rural West Bengal.

The consultation may contain Bangla, Bengali-accented English, Hindi, code-switching, locally pronounced medicine brands, a relative answering from the side, background noise, and a clinician moving through a queue that has already eaten the afternoon. The microphone may be a modest phone. The patient may describe a symptom through metaphor, family history, embarrassment, or a word absent from the vendor’s test set.

None of this makes the clinic backward.

It makes the clinic real.

It may also be exactly where a good scribe could help most. A clinician carrying an impossible documentation burden does not become less tired because the clinic is under-resourced. A tool that preserves a clearer note or gives attention back to the patient could be valuable in ways a comfortable conference room has not imagined.

The access benefit and the validation gap can arrive in the same box.

This is the point affluent technology debates regularly miss. Rural and low-resource settings are presented either as grateful recipients of innovation or as picturesque victims of it. They are neither. They are deployment settings with their own languages, disease patterns, infrastructure, workflows, expectations, and expertise. Local clinicians and patients should be co-design partners with authority to reject or stop the system, not names at the bottom of a slide titled “future expansion”.

India already has relevant national guidance. The Indian Council of Medical Research’s 2023 ethical guidelines address development, deployment, adoption, governance, and informed consent. The 2026 Strategy for Artificial Intelligence in Healthcare for India [SAHI, a national guidance framework rather than binding regulation] places safety, equity, public trust, data quality, workforce readiness, and continuous evaluation near the centre of the approach.

That is encouraging.

A framework, however, does not listen to a single Bengali consultation. A policy cannot discover that a particular drug name becomes another drug when spoken under a fan. Strategy becomes safety only when somebody tests the actual system in the actual clinic, measures the actual errors, and retains the authority to stop deployment when the results are poor.

The rural clinic is not where validation ends.

It is where imported assumptions meet a patient.

Drift enters without ringing the bell

Nothing in the Rebecca Green report proves model drift. It demonstrates that a false statement escaped into a clinical document. Drift is the next question, not the explanation we should paste over missing evidence.

In practice, several different phenomena are crowded beneath the word.

A mismatch present on day one—different people, languages, accents, devices, or environments—is deployment or distribution mismatch. Data drift is a change in operational inputs over time. Concept drift is a change in the relationship a system is expected to capture. A vendor swapping a speech model, foundation model, prompt, template, or feature is a controlled product change, not merely weather that happened to the algorithm. Any of these can alter real-world performance, but naming them properly helps us investigate the right cause.

Then there is automation bias and review decay, which are human and organisational changes rather than model drift. On Monday, the clinician checks every sentence. After a month of mostly good notes, the review becomes a quick scroll. After three months, the signature is muscle memory. The human remains “in the loop” in the architectural diagram while their attention has quietly left the building.

The Australian safety guidance notes that evolving underlying models can produce different summaries from the same input and recommends regular review when accuracy or quality deteriorates. The TGA likewise tells clinicians to consider whether software updates have changed a product’s purpose or performance. These are modest sentences with large operational consequences.

The version validated in March is not automatically the version running in August.

Healthcare IT must therefore keep a local, clinically designed regression set: consented or synthetic consultations covering accents, code-switching, interpreters, noisy rooms, speech impairments, multi-speaker visits, ambiguous histories, and the dangerous little words that machines and tired humans both lose. No. Not. Never. Left. Right. Before. After. Milligram. Microgram. Allergy. Family history. Patient denies.

Test after meaningful changes. Monitor by relevant subgroup, not only across a flattering average. Track clinically weighted errors: negation, laterality, medicine and dose, allergies, diagnosis, substance use, examination findings, temporal context, and speaker attribution. Record the product, speech model, language model, prompt or template version where available. Require vendor change notices. Use shadow or canary deployment. Keep a rollback path.

If a product cannot provide meaningful version and change information, the hospital cannot safely assume that the system it validated is still the system in use.

It has a subscription with bedside consequences.

“Human in the loop” is not a magic spell

The clinician must review the note. That responsibility is real. The Australian clinical safety scenario says AI-generated summaries should be checked for accuracy, missing information, over-summarisation, bias, and hallucinations before the medical record is completed.

But placing the entire safety burden on a final human check can become responsibility-shifting dressed as oversight.

If the business case assumes the scribe saves time, while the safety case assumes a clinician compares every important statement against memory or audio, the organisation must budget the same minutes honestly. If the interface presents a polished note without showing uncertainty, transcript differences, or high-risk fields, the clinician is being asked to find a needle after the software has upholstered it. If seventy patients are waiting, “the doctor should check” may be legally tidy and operationally unserious.

A real human control needs time, interface support, training, and escalation. High-risk statements should be easy to inspect. Nothing should enter a final note, referral, letter, order, diagnosis code, allergy list, or problem list without clinician authorisation appropriate to its consequence. AI involvement should be traceable. Near misses should be reportable without requiring a small personal act of martyrdom. Repeated errors should reach the vendor, governance team, and regulator where applicable.

Patients also need a place in the loop that is not merely decorative.

Consent should be specific, understandable, and voluntary. Refusing an AI scribe should not become a secret fee paid in poorer access or colder care. Patients should know that the consultation is being recorded or processed, what happens to the audio, which parties receive the data, whether it is retained or reused, and how to request an ordinary consultation instead. They should be able to see and challenge the resulting record.

And when the system writes a fiction, the patient should not be appointed unpaid chief investigator of the hospital’s data lineage.

The regulatory gap is real, but it has edges

The ABC clip says AI medical scribes are not currently regulated. The more precise Australian position is narrower.

Under the TGA’s current digital-scribe guidance, software intended only to transcribe or translate a consultation without analysis or interpretation is not treated as a medical device. If the intended functionality analyses or interprets—for example, by generating a diagnosis, differential diagnosis, or treatment recommendation not stated by the clinician—the product is considered a medical device and must meet the corresponding requirements. Privacy law, professional obligations, consumer law, consent duties, and responsibility for record accuracy do not disappear merely because a product sits outside one regulatory category.

This boundary is awkward because modern scribes do not merely type. Summarisation selects, compresses, reorganises, and omits. A product may be marketed as administrative plumbing while performing small acts of interpretation. Australia’s TGA is reviewing products’ functionality, and Digital Rights Watch has argued that adoption has outrun meaningful oversight.

Regulation matters. So does not waiting for regulation to perform basic engineering.

A hospital does not need parliamentary permission to test Bengali speech. A vendor does not need a new statute to disclose material model changes. A clinic does not need a commission of inquiry to give a patient a genuine choice. An EHR team does not need a national AI act to build a correction-and-recall procedure.

Law sets a floor.

Patients are injured in the space between the floor and the workflow.

What responsive healthcare IT looks like now

The transition period does not need one grand global rule before anybody can behave responsibly. It needs organisations willing to perform dull work before a patient performs the suffering.

Before deployment, define intended and prohibited uses, identify the population and setting, validate locally, include groups likely to disappear inside an average, and run the tool in shadow mode before its notes can travel. Examine errors for clinical severity, not merely word accuracy.

During use, obtain consent, preserve an equivalent non-AI route, require timely clinician authorisation, label AI involvement, record versions, audit corrections, watch subgroup performance, and give staff a visible stop mechanism. After an incident, correct the record, trace downstream copies, preserve evidence, identify the failure layer, test whether the pattern affects other patients, and share the lesson far enough that another clinic does not have to rediscover it through another person’s pain.

This is where industry self-interest and patient interest become the same thing. Defensive responses can injure people, erode trust, harden procurement, invite blunter regulation, and discourage clinicians from using tools that may otherwise improve care.

Cavalier adoption can hurt the patient first and the industry immediately after.

The record must remain answerable

The worst response to this story would be panic. The second worst would be a shrug.

AI scribes may become ordinary clinical infrastructure. They may reduce clerical labour, return attention to patients, and preserve better documentation than some exhausted humans can produce alone. Rural and understaffed systems may have more to gain from them, not less.

That promise raises the standard of care around deployment. It does not lower it.

Healthcare IT cannot answer a reported falsehood only with “the doctor should have checked” and return to the sales dashboard. It cannot answer population mismatch with one global accuracy number, product change with a familiar logo, consent with a laminated notice, or a distributed error by quietly deleting one line.

The purpose is not to keep AI out of the clinic.

The purpose is to make it answerable to the clinic it enters.

Somewhere in rural West Bengal, a doctor may soon press a button, turn away from the keyboard, and finally look directly at the patient. That could be a small act of progress. But while several languages, medicine names, and human uncertainty enter the microphone together, somebody must remain responsible for the distance between what was said and what the record claims.

The machine may draft the memory.

It must not be allowed to invent the patient.

Related Posts

• The Linear Algebra Blind Spot in Healthcare AI
• The Shadow Architecture Behind Clinical AI
• AI in Healthcare: Beware
• Why India’s EHR, HIE, and AI Future Will Be Constrained by Reality