Poisson Distribution: Counting Rare Trouble Before It Becomes a Queue, a Claim, or a Crisis

Acronyms used in this post:

EHR: Electronic Health Record, the clinical system where patient care is documented, ordered, reviewed, coded, and often argued with.

ED: Emergency Department, the part of a hospital where unscheduled acute care arrives with a cough, a fracture, a fever, chest pain, confusion, paperwork, and usually relatives.

ADT: Admission, Discharge, Transfer, the event stream that describes a patient’s movement through care settings.

HL7 v2: Health Level Seven version 2, the old but stubbornly alive healthcare messaging standard that still carries much of the world’s hospital data.

FHIR: Fast Healthcare Interoperability Resources, a modern healthcare data exchange standard that represents clinical information as modular resources.

CDA: Clinical Document Architecture, a document-based healthcare standard that mixes human-readable narrative with structured data.

HIE: Health Information Exchange, the sharing of health information across organizations, systems, regions, or networks.

ETL: Extract, Transform, Load, the process of pulling data from source systems, reshaping it, and loading it into a target system.

ELT: Extract, Load, Transform, a related approach where raw data is loaded first and transformed later inside the target platform.

SQL: Structured Query Language, the common language used to query relational databases.

SPC: Statistical Process Control, a method for monitoring whether variation in a process looks routine or unusual.

AI: Artificial Intelligence, software that performs tasks involving prediction, classification, generation, reasoning, or pattern recognition.

CTMS: Clinical Trial Management System, software used to manage the operational side of clinical trials.

CDMS: Clinical Data Management System, software used to collect, clean, validate, and manage clinical trial data.

CDISC: Clinical Data Interchange Standards Consortium, the organization that defines important clinical research data standards.

SDTM: Study Data Tabulation Model, a CDISC model for organizing clinical trial data for regulatory submission and analysis.

---

The Poisson distribution is the mathematics of rare trouble arriving in countable packets.

One failed interface message. One medication error. One patient fall. One infection. One ED arrival at 2:17 in the morning. One duplicate patient record born because two systems disagreed about a middle initial and behaved, as systems often do, like two goats on a narrow bridge.

At first glance this sounds too small for a grand mathematical story. We are not measuring planets. We are not estimating the age of the universe. We are counting little nuisances. But this is the catch: in healthcare, little nuisances are often the advance scouts of disaster.

A hospital does not collapse all at once. It frays.

A queue lengthens. A lab feed slows. A bed request waits. A patient identity merge gets delayed. A sepsis alert fires too often, then gets ignored, then fires once when it should not be ignored, and now everybody is suddenly very interested in governance.

The Poisson distribution lives in that awkward middle zone between “nothing happened” and “why is the director calling?”

Its basic question is plain enough for a tea stall and deep enough for a doctoral seminar: how many times should we expect a rare event to happen in a fixed window?

Not whether it can happen.

How many times.

That tiny shift is the whole game.

The binomial distribution asks a more buttoned-up question. Suppose there are $n$ trials. Each trial has probability $p$ of success. How many successes occur? This is the world of exams, coin tosses, drug response, claim denial, appointment attendance, and other situations where you can count the number of opportunities one by one.

The binomial formula is:

$P(X=k)=\binom{n}{k}p^k(1-p)^{n-k}$

It is a respectable formula. It has combed hair.

Now imagine $n$ becomes very large and $p$ becomes very small. Many chances. Tiny probability each time. But the expected number stays steady. That expected number is called $\lambda$.

Then the binomial begins to change its clothes in a dark room.

Out comes the Poisson distribution:

$P(X=k)=\frac{e^{-\lambda}\lambda^k}{k!}$

Here $X$ is the number of events. The value $k$ is the count we are asking about: zero events, one event, two events, fifteen events, whatever the window permits. The parameter $\lambda$ is the expected number of events in that window.

If a hospital interface usually has 3 failed messages per hour, $\lambda=3$ for one hour.

If a unit usually has 2 falls per 1,000 patient-days, $\lambda=2$ for that exposure.

If an ED usually sees 14 arrivals between midnight and 1 a.m., $\lambda=14$ for that hour.

This is where the distribution becomes useful and dangerous. Useful because it gives us a clean expectation. Dangerous because healthcare will cheerfully ruin clean expectations before lunch.

The history is worth a small detour.

Probability did not begin as a hospital quality dashboard. It began with gamblers, astronomers, insurers, demographers, philosophers, and the sort of men who looked at death records and thought, “There must be a pattern here,” which is either scientific curiosity or an alarming way to spend an afternoon.

Pascal and Fermat worried about games of chance in the seventeenth century. Bernoulli gave us the law of large numbers. De Moivre worked on approximations. Laplace made probability feel as if it had swallowed the sky. Then, in 1837, Siméon Denis Poisson, a French mathematician with a name now permanently attached to rare events, studied probability in the context of legal judgments.

Legal judgments, not lab interfaces.

History is like that. You open a cupboard looking for healthcare analytics and find a French court wearing powdered seriousness.

Poisson’s work showed how a rare-event distribution could emerge from the binomial when there are many opportunities and each has a tiny chance. Later, Ladislaus von Bortkiewicz popularized the “law of small numbers,” including the famous example of Prussian soldiers killed by horse kicks. This example has now been dragged through so many statistics textbooks that one feels faintly sorry for the horses, who never asked to become adjunct faculty.

But the example worked because it had the shape Poisson liked: rare events, many opportunities, fixed exposure, countable outcomes.

Healthcare is full of that shape.

A large hospital gives thousands of medications. Only a few administrations cause serious harm.

A laboratory sends millions of results. Only a few fail routing.

A clinical rule evaluates thousands of encounters. Only some trigger alerts.

A registry reviews large populations. Only a few cases meet strict criteria.

An interface engine processes rivers of HL7 v2 messages. Only some drown on the rocks.

The ordinary person may ask, quite reasonably, why we need a special distribution for this. Why not just take the average and keep moving? After all, life is short, electricity bills are long, and in Calcutta the ceiling fan already makes enough philosophical noise without adding $\lambda$.

Because an average alone does not tell us what amount of variation is normal.

Suppose a system usually gets 4 failed messages per hour. One hour it gets 5. Fine. Another hour it gets 7. Maybe fine. Then it gets 18.

Is that bad luck, or has something broken?

The Poisson distribution helps answer that. It gives us a way to say, “If the usual process were still operating, how surprising would this count be?”

That sentence should be printed and pasted above many dashboards.

Not “is the number large?”

Not “does the chart look scary?”

Not “who can we blame before the meeting?”

But: if the process has not changed, how surprising is this count?

That is a civilized question. Rare in committee rooms.

The Poisson distribution has one peculiar property that looks harmless until it bites. Its mean and variance are both $\lambda$.

If the expected count is 5, the variance is also expected to be 5.

In actual healthcare data, the variance often arrives inflated, sweating, and carrying three extra bags. Counts cluster. Mondays differ from Sundays. Winter differs from summer. A new documentation template changes recording behavior. A payer rule changes coding behavior. A software patch changes message behavior. A resident discovers a shortcut. A senior nurse enforces a workflow. A public holiday bends the week into a strange shape.

Then the observed variance is much larger than the mean.

This is called overdispersion.

Overdispersion is not a statistical nuisance. It is the data whispering, “Your simple story is missing something.”

Too often the organization replies, “Data quality issue,” and everybody nods, because “data quality issue” is the healthcare IT version of blaming the weather.

But many so-called data quality problems are not data quality problems.

They are representation problems.

A data quality problem means the data is wrong relative to what it claims to represent. A representation problem means the data is faithfully recording the wrong abstraction.

That difference is not academic. It is the difference between a dirty window and a map of the wrong city.

If a medication was given at 8:05 but documented at 10:40, the timestamp may be technically valid. It records documentation time, not clinical administration time. The database is not necessarily lying. It is telling the truth in the language of workflow.

If an ADT message says a patient moved units at 14:32, does that mean the patient physically moved, the bed board changed, the order was entered, the registration status updated, or the downstream system received the message? Each answer can be true in its own little kingdom. Only one may be useful for your model.

This is why Poisson thinking is so valuable in healthcare IT. It forces the question nobody wants to ask: what exactly are we counting?

A count is not a pebble. You cannot simply pick it up and say, “Here is one.”

A count is a decision.

One ED arrival. But arrival according to whom? Front desk registration? Door time? Triage time? First provider contact? ADT event? Billing encounter? Ambulance handoff? Each one can generate a different count by hour.

One infection. But detected how? Lab culture? Surveillance definition? Diagnosis code? Manual review? Public health report? Registry abstraction?

One alert. But first firing only, or every retrigger? Suppressed alerts included? Test patients excluded? Overrides counted? Silent alerts counted? What about alerts that failed to render because a dependency was down?

You think you are counting events.

Actually, you are counting institutional decisions about what becomes visible.

That is the little trapdoor in the floor.

In healthcare, the event and the record of the event are not the same animal. The event happens in the ward, clinic, ambulance, lab, pharmacy, operating room, or patient’s body. The record happens inside software, under rules, through screens, messages, workflows, codes, queues, approvals, and human exhaustion.

Between the event and the count stands a whole bazaar.

This is where standards enter, carrying clipboards.

HL7 v2 can transport a lab result. FHIR can expose an Observation. CDA can package a discharge summary. HIE can move information across organizations. ETL can reshape source data into a warehouse. SQL can count the rows.

None of this guarantees meaning.

Data transport is not semantic meaning.

A parcel can arrive at the right house and still contain the wrong shoes.

This distinction is so important that it should be taught before anyone is allowed near a dashboard. Transport asks: did the data move? Semantics asks: what does the data mean, in this workflow, at this time, for this purpose?

An HL7 v2 result message may be syntactically correct. It may pass validation. It may reach the receiver. Lovely. But is it final, preliminary, corrected, canceled, duplicated, reflex-generated, panel-level, component-level, suppressed, or amended? The wire does not answer all that by itself.

FHIR improves the package. It does not abolish ambiguity.

A FHIR Observation still needs context. A FHIR Encounter still depends on local encounter logic. A FHIR Condition may represent billing, clinical concern, problem-list memory, historical residue, or a diagnosis that nobody has had the courage to retire because some old report in the basement may start coughing.

Healthcare data is not raw reality. It is reality after it has passed through clerks, clinicians, interfaces, forms, policies, incentives, and software vendors with release notes that read like fog trying to be useful.

Now return to Poisson.

The formula is small:

$P(X=k)=\frac{e^{-\lambda}\lambda^k}{k!}$

But the real work is not in the formula. The real work is in choosing $\lambda$ honestly.

Usually $\lambda$ is rate multiplied by exposure:

$\lambda=r\times t$

The rate is $r$. The exposure is $t$.

In ordinary life, if one mosquito bites you every 10 minutes, and you sit outside for an hour, you expect about 6 bites. That is the kind of denominator even a sleepy man in a vest on a humid Kolkata evening understands at once.

Healthcare has fancier mosquitoes.

Infections per 1,000 device-days.

Falls per 1,000 patient-days.

No-shows per 100 scheduled visits.

Message failures per 100,000 messages.

Denials per 1,000 claims.

ED arrivals per hour.

Sepsis alerts per eligible encounter.

The denominator is not decoration. It is the floor.

Ten infections mean very different things in 500 device-days and 50,000 device-days. Fifty interface failures mean very different things if the engine processed 2,000 messages or 2 million. A dashboard that shows only counts without exposure is not analytics. It is gossip with colors.

This is a practical architecture lesson: denominators must be modeled, stored, governed, and versioned like serious data objects.

A denominator can break.

A census feed can change. A device-day calculation can shift. A message volume counter can exclude retries after a patch. A no-show denominator can change when telehealth visits are added. A claim population can shift when a payer contract changes.

When the denominator changes, the rate changes even if the underlying world does not.

Then people run around looking for a clinical cause, and the real culprit sits quietly in an ETL job, sipping tea.

The Poisson distribution is especially useful in patient safety.

Patient harm events are often rare but consequential. A unit may have zero falls for several weeks, then three in two days. Is that a signal? A cluster? Bad luck? A staffing issue? A documentation catch-up? A change in fall definition? A new nurse actually reporting events others used to ignore?

This last possibility matters. Higher counts do not always mean worse care. Sometimes they mean better detection.

A hospital that reports more safety events may be more dangerous. Or more honest. Or more surveilled. Or more burdened. Or simply better at making invisible things visible.

The count does not bring its own moral certificate.

This is one of the non-obvious lessons of healthcare analytics: a rising event count can indicate worsening reality, improving measurement, changing workflow, changing incentives, or changing exposure. The chart cannot tell you which one without architecture.

In infection surveillance, Poisson models and their cousins help compare observed counts with expected counts. But infection counts are not just biology. They depend on culturing practice, lab sensitivity, clinical suspicion, device utilization, patient mix, surveillance definitions, documentation behavior, and reporting systems.

A unit that cultures aggressively may look worse than a unit that prefers ignorance wrapped in calm Excel files.

In ED operations, Poisson processes appear in arrival modeling and queueing theory. Many basic queue models assume arrivals are Poisson. This can be useful. It can also be comically incomplete.

People do not arrive at an ED like raindrops obeying a university textbook. They arrive after office hours, after road accidents, during heat waves, during respiratory surges, after festivals, after panic, after neglect, after Google has terrified them at 1 a.m., and after relatives have debated for four hours whether the symptom is serious.

Still, at the right scale, Poisson arrival models can help. They can estimate staffing pressure, triage load, imaging demand, lab volume, and bed request patterns. The model is not reality. It is a torch. It helps if you point it at the right part of the room.

In interface monitoring, the Poisson idea is almost embarrassingly useful.

Suppose an interface normally fails 2 messages per hour. Suddenly it fails 20. A Poisson-based threshold can flag the spike. But the alert is only the beginning. The architecture must support drill-down by source, target, message type, facility, order code, acknowledgment status, transformation step, terminology mapping, and deployment timestamp.

Otherwise the dashboard screams and nobody knows where the smoke is coming from.

This is the old tragedy of IT operations: excellent alarm, poor map.

A proper monitoring system should not merely say, “Failures are high.” It should say, “Failures are high for outbound microbiology results from facility A to system B after the 09:15 terminology update, concentrated in messages using this local code.”

That is not a statistic anymore. That is a flashlight with batteries.

Poisson regression extends the same idea.

It models expected counts using predictors:

$\log(\lambda_i)=\beta_0+\beta_1x_{1i}+\beta_2x_{2i}+\log(t_i)$

The $\log(t_i)$ part is the offset. It adjusts for exposure. Without it, the model may confuse “more events because more opportunity” with “more events because higher risk.”

This is not a small mistake. It is the kind of mistake that can make a large hospital look unsafe merely because it is large, or a busy clinic look inefficient because it actually sees patients instead of arranging chairs for accreditation visits.

Poisson regression can model readmissions, falls, ED arrivals, imaging demand, claim denials, missed appointments, adverse events, query counts in clinical trials, and alert burden.

But when the variance exceeds the mean, plain Poisson regression becomes too confident. It starts speaking in crisp statistical sentences that reality has not authorized. Then negative binomial models may be better. If there are many zeros, zero-inflated models may help. If the rate changes over time, a time-varying model may be needed. If hospitals, units, clinicians, or patient groups differ deeply, hierarchical models may be necessary.

There is no shame in starting with Poisson.

There is shame in refusing to notice when it is wrong.

In clinical research, the same logic appears in adverse event counts, protocol deviations, site queries, recruitment rates, and monitoring findings. CTMS and CDMS platforms can count everything with an enthusiasm that sometimes exceeds understanding. CDISC and SDTM can standardize structure, but structure alone does not settle meaning.

A site with fewer adverse events may be safer.

Or underreporting.

Or less sick.

Or poorly monitored.

Or staffed by people who interpret the protocol differently.

Or operating in a context where reporting creates burden without visible benefit, which is a polite way of saying that humans do not feed machines unless the machines feed something back.

This is why governance must sit close to measurement.

Not as a bureaucratic umbrella. More like a bicycle brake. You do not admire it every day, but you miss it sharply when going downhill.

For healthcare AI, rare-event modeling becomes even more treacherous.

AI systems love data the way goats love laundry. They will chew what they are given. If the rare event is poorly defined, underreported, workflow-biased, or denominator-confused, the model will learn the distortion with great confidence.

If one hospital documents complications aggressively and another does not, an AI model may learn documentation culture instead of clinical risk.

If voluntary safety reports depend on staff morale, the model may learn morale.

If lab alerts depend on order sets, the model may learn order sets.

If sepsis labels depend on billing, the model may learn reimbursement.

Then everyone says the AI is biased, which may be true, but incomplete. The deeper issue is that the target variable was not reality. It was reality after passing through a paperwork machine wearing clinical clothing.

Poisson thinking does not fix AI.

It makes AI harder to fool.

It asks: what is the event, what is the opportunity, what is the rate, what is the time window, what is the exposure, what is the source, what changed, and what does zero mean?

Zero is especially slippery.

Zero infections may mean excellent care. Or no devices. Or no tests. Or no reporting. Or a broken feed. Or a denominator mismatch. Or an extraction job that failed silently and now lies on the floor like a gecko pretending to be dead.

In healthcare data, zero is not always nothing.

Sometimes zero is a confession waiting for cross-examination.

This is where the history loops back. Poisson emerged from the attempt to understand small numbers in large fields of opportunity. That is still the problem. Only now the field is not French legal statistics or Prussian cavalry. It is hospitals, data warehouses, public health systems, claims pipelines, registries, HIE networks, interface engines, and AI models being asked to make sense of clinical life from digital traces.

The old question remains sharp.

How many rare events should we expect?

But in healthcare IT, we must add several more.

Expected by whom?

Counted from which system?

Generated by which workflow?

Divided by which denominator?

Changed by which policy?

Transported through which standard?

Interpreted under which definition?

And when the number jumps, what exactly jumped: reality, recording, exposure, or meaning?

That is why the Poisson distribution deserves more respect than it gets. It is not merely a formula students memorize before an exam and forget with relief. It is a small door into a large room. Behind that door are rate, exposure, time, workflow, semantics, governance, surveillance, operations, and the uncomfortable fact that healthcare systems often count shadows and then argue about the furniture.

A clean solution is not available, because healthcare does not have one clean truth. Clinical care, billing, research, operations, public health, compliance, and quality reporting all cut the world differently. The EHR wants a care record. The claims system wants payment logic. The registry wants classified cases. The researcher wants a cohort. The interface engine wants valid messages. The regulator wants standardized reporting. The patient wants to get better and go home.

All are real.

None is complete.

So the practical answer is not purity. It is disciplined imperfection.

Define events brutally. Preserve denominators. Separate event time from documentation time. Store provenance. Monitor semantic change. Expect overdispersion. Treat representation failures as architecture problems, not as moral defects in data. Build dashboards that allow drill-down before blame. Let Poisson be the first question, not the final verdict.

At my age, sitting on the edge of Calcutta where the city thins into broken pavements, tea stalls, half-built ambition, and the eternal cough of traffic, I find something oddly comforting in this distribution. It does not promise certainty. It does not say the world is tidy. It says only this: if rare things happen at a certain rate, here is what ordinary randomness looks like.

That is a modest gift.

And in healthcare IT, modest gifts matter. Because before we predict the future, automate medicine, or install another shiny AI lantern over the same old swamp, we should be able to count one rare event, over one honest denominator, in one clear window, without forgetting what the count actually means.

The Poisson distribution counts the drops.

Healthcare IT must explain the rain.