A one-size-fits-all AI review can miss meaningful privacy risk while slowing narrow models designed for specific tasks. Here are five questions hospital AI committees can use to evaluate the system that actually exists.

I recently explored a growing problem in hospital AI governance in a Fierce Healthcare article I coauthored with David Knobel: committees are increasingly asking whether protected health information (PHI) was used to train an AI model, but too often treating the answer as a proxy for the model’s actual privacy risk.

It is an important question. It just isn’t enough.

Two models can both be developed using PHI and present dramatically different privacy risks once deployed. One may accept open-ended prompts, retrieve information from clinical records, retain prompts and outputs, and generate unrestricted text. Another may perform one fixed task and return only a label, timestamp, coordinate, or redaction mask.

The presence of PHI during development tells us something about how data must be protected. It does not, by itself, tell us what information the deployed system can expose.

The better approach is to govern the model that actually exists: its architecture, inputs and outputs, user interface, accessible components, data flow, retention, and realistic pathways to sensitive information. At Aimbient, privacy and security shape the task definition, product interface, access model, validation, and deployment architecture from the beginning.

Here are five questions I believe hospital AI committees should be asking.

1. What can the model accept and produce?

Start with the actual interface.

Can the system accept arbitrary clinical text, images, audio, or open-ended prompts? Does it retrieve information from other hospital systems? Can it generate free-form responses, or does it return a constrained output such as a label, timestamp, bounding box, or mask?

Inputs and outputs define a major part of an AI system’s exposure surface.

A generative model that can accept broad prompts and produce unrestricted responses creates a different set of privacy considerations from a model designed to perform one narrow technical task. A system that returns a binary classification, for example, does not provide the same opportunity for information disclosure as one capable of reproducing arbitrary text.

That distinction informs how we build Aimbient products. Models used to identify room states, detect procedural events, generate timestamps, or create de-identification masks are designed around a defined workflow and a constrained output. We expose the result needed for that workflow—not a general-purpose channel into the model, its internal representations, or its training data.

The practical question is straightforward: What information can enter this system, and what information can come back out?

2. Can a user probe the model?

Privacy risk also depends on how interactively a system can be queried.

A user who can submit repeated, adaptive prompts has considerably more opportunity to explore a model’s behavior than someone interacting with a fixed-function system with no exploratory query interface.

Aimbient deliberately does not expose an open interface through which hospital users can query production models unrestrictedly. Users interact with permissioned product workflows and defined features. They cannot craft arbitrary prompts to probe for training examples, inspect embeddings, retrieve model weights, or traverse internal development systems.

That is not merely a user-experience decision. It reduces practical paths for model extraction, membership inference, prompt-based disclosure, and other exploratory abuse.

Where generative functionality is appropriate, it should be reviewed as a distinct product surface. Prompts, retrieval sources, context windows, conversation history, retention, and generated outputs all require their own controls.

Hospital reviewers should understand whether users can repeatedly probe the system, what information it can retrieve, whether responses change based on supplied context, and what limits that interaction. A theoretical ability for information to exist somewhere inside an AI system is different from a practical ability for a user to elicit it.

3. What is actually accessible?

Not every component of an AI system is exposed to the same people.

There is a meaningful difference between access to a model’s intended output and access to its confidence scores, embeddings, logs, stored prompts, model weights, training infrastructure, APIs, or administrative tooling.

At Aimbient, hospital users are not given access to model weights, training datasets, internal embeddings, development environments, or unrestricted inference tooling. Product access is separated from the systems used to develop, validate, deploy, and monitor models. Access to PHI and sensitive technical assets is limited to defined purposes and controlled roles rather than being broadly available because someone uses the product.

A constrained user interface is not enough if logs, prompts, retrieved records, or administrative systems are broadly accessible behind the scenes. Strong healthcare AI security must address the entire data path.

The relevant question is not simply what exists within the technical stack. It is: What can the anticipated user, administrator, vendor employee, or attacker actually reach?

4. What is the credible path from the deployed system to an identifiable patient?

A serious privacy assessment should be able to describe the exposure pathway, not merely identify a theoretical possibility.

What access would someone need? What sequence of actions would be required? What additional information would they need? How repeatable is the pathway? What safeguards interrupt it? And how likely is it to succeed in the actual deployment environment?

This is why our development reviews at Aimbient examine the complete product path, not just the model file: how data is captured, transferred, processed, stored, de-identified, displayed, exported, retained, and deleted.

That forces concrete design questions. Are we collecting more data than the workflow requires? Can a derived result replace exposure to raw data? Can identifiable information be removed earlier? Which users need which outputs? What must be logged for security without unnecessarily replicating PHI? What retention is actually required?

HHS guidance on HIPAA de-identification recognizes that identifiability is contextual. Under the Expert Determination method, the relevant question is whether the risk of identification is very small given the information itself and other reasonably available information. That is not a universal test for AI models, but the principle is useful: privacy risk should be evaluated in context and based on realistic pathways.

A committee should push beyond “Could this model theoretically reveal something?” and ask, “How, specifically, would that happen in the system we are being asked to approve—and which controls block that path?”

5. What risk is created by saying no?

This may be the most overlooked question.

Good governance should evaluate both the risk of adopting a technology and the risk of preserving the status quo.

Consider a narrow AI model designed specifically to identify or remove sensitive information. Preventing it from learning from representative clinical data could make it less effective at the very task intended to reduce privacy exposure. Rejecting the technology might preserve a workflow in which humans continue to manually view and handle identifiable information.

This matters directly in procedural care. Automated de-identification, event detection, structured timestamps, and other constrained outputs can reduce the number of people who need to inspect raw, identifiable data. A review focused only on whether PHI supported development can miss the PHI exposure the finished system is designed to eliminate.

That does not mean the correct decision is automatically to approve the model. It means inaction should not be treated as risk-free.

The committee should compare the alternatives explicitly: What exposure exists if the system is approved? What remains if it is rejected? Can additional controls reduce one or both? Does the technology eliminate an existing source of human access, manual handling, or unnecessary retention?

Governance is stronger when it compares the credible risks of each path rather than evaluating only the risks introduced by change.

Govern the model—and the product—that actually exists

Differentiated governance is not weaker governance. It is more precise governance.

A narrow model used in a critical clinical workflow may present limited privacy exposure and still deserve intense scrutiny for accuracy, robustness, bias, cybersecurity, and patient safety. A generative model operating on de-identified information may create fewer PHI concerns while still requiring serious review for hallucination, inappropriate reliance, security vulnerabilities, or operational risk.

NIST’s AI Risk Management Framework reflects this broader reality by treating trustworthy AI as multidimensional, encompassing validity, safety, security, privacy, transparency, explainability, accountability, and fairness.

Secure PHI handling is likewise a system property built from data minimization, constrained interfaces, least-privilege access, de-identification, retention controls, secure infrastructure, logging, validation, and monitoring. No answer about whether PHI was used for training can substitute for that architecture.

At Aimbient, this posture is deliberate. We build for defined procedural workflows, restrict how production models can be queried, separate user-facing functionality from model internals, and evaluate credible paths by which sensitive information could be exposed. The goal is not to claim that risk disappears. It is to make risk concrete, constrained, testable, and governable.

The objective is to identify which dimensions of risk matter for the specific system, understand how it operates in practice, and apply scrutiny and controls accordingly.

Asking whether PHI was used to train a model is reasonable. Treating the answer as a substitute for understanding the model, the product, and the development process is not.

The strongest hospital AI governance asks the harder question: What can this system actually expose, through the interfaces and under the conditions in which it will actually be used?

For a deeper discussion of this argument, read my recent Fierce Healthcare Industry Voices article with David Knobel, “Hospital AI committees are asking the wrong privacy question.”

This article is for general informational purposes and does not constitute legal advice.