meilynx
← All posts

What a HIPAA Examiner Asks After PHI Goes Into an AI Chatbot

PHI sent to an unapproved AI tool is presumed a breach until a four-factor risk assessment shows otherwise. What an examiner asks for, and what to have ready.

Julia MeloJulia MeloCo-Founder10 min readCompliance

A nurse pastes a discharge summary into a public AI chatbot to tidy the wording. A prior-authorization assistant sends a member's whole chart to a model provider for a request that needed three fields. Privacy and compliance teams we talk to describe versions of both. Once one of them reaches the privacy office, the questions that follow come mostly from HIPAA's breach notification rule, and it helps to know them before they arrive.

01 · The presumption

An impermissible disclosure to an AI service is presumed a breach

HIPAA has no AI provision. An assistant that reads a chart or drafts a patient message handles protected health information. The Privacy, Security and Breach Notification Rules apply to it as they apply to any other system. Our reference page on the Security Rule and AI workloads covers those rules in full.

Start with the definition of breach in 45 CFR 164.402. A use or disclosure of PHI that the Privacy Rule does not permit is presumed to be a breach. The covered entity or business associate can rebut that by showing a low probability that the PHI was compromised, based on a risk assessment of at least four factors, and under 45 CFR 164.414(b) the burden of showing it sits with the organization.

The clock runs while you assess. Individuals are notified of a breach of unsecured PHI without unreasonable delay and no later than 60 calendar days after discovery (164.404), and a business associate notifies the covered entity on the same terms (164.410). Discovery is the first day the breach is known, or would have been known with reasonable diligence. On day one we tell teams to write down the date and how they found out, because every other date counts from it.

60

calendar days from discovery, at most, to notify individuals

4

factors, at a minimum, in the risk assessment

The clock runs while the assessment is under way

What the assessment decides

Starting point

A use or disclosure the Privacy Rule does not permit is presumed a breach

A business associate notifies the covered entity on the same 60-day terms.

Low probability of compromise shown

The presumption is rebutted. The documented assessment is the evidence.

Not shown, and the PHI was unsecured

The presumption stands and individuals are notified.

How the presumption and the notification clock work. Breach definition at 45 CFR 164.402, individual notice at 164.404, business associate notice at 164.410, burden of proof at 164.414(b).

02 · The recipient

The first facts are the vendor and the agreement

Take the nurse. The privacy officer's first calls have little to do with what the summary said. Which chatbot was it, and who runs it? Which model sits behind it? Does the clinic have an agreement with that company, and does that company have one with the model provider?

If she used a free consumer account under its standard terms, there is no business associate agreement behind it. The disclosure was impermissible and the four-factor assessment starts. Now take the clinic's approved scribe, where the vendor signed a business associate agreement, the vendor has a subcontractor agreement with its model provider, and the use is one the agreements cover. That may involve no impermissible disclosure at all, and the file stays short.

The rules behind this sit in HHS's business associate guidance, which lists a patient-portal AI chatbot among its examples of a business associate. A covered entity may disclose PHI to a business associate once an agreement gives satisfactory assurance that it will be safeguarded, and a business associate needs the same kind of agreement with any subcontractor. A model provider that receives prompts from an AI vendor is that subcontractor. HHS limits the conduit exception to services that only transmit data, so it does not reach a vendor that works with the PHI to perform a service.

In our experience the second link is what slows teams down. They can usually name the tools they have approved. The model provider behind each one, and the agreement that covers it, takes longer to find. A one-page list of approved tools, each with its vendor, model provider and agreement, turns that first call into a lookup.

Before the four factors: was the disclosure permitted?

1

Does the AI vendor sit under a business associate agreement?

HHS names a third-party AI chatbot on a patient portal as a business associate. The conduit exception covers only entities that provide transmission services.

NoImpermissible disclosure. The four-factor assessment begins.
YesNext question
2

Does the model provider have a subcontractor agreement with that vendor?

A business associate needs one before passing PHI on. The model provider stands in that position.

NoImpermissible disclosure. The four-factor assessment begins.
YesNext question
3

Is the use one those agreements permit?

An agreement covers the purposes it names, and the minimum necessary standard still applies.

NoImpermissible disclosure. The four-factor assessment begins.
YesNo impermissible disclosure

Path ends here

Clinical copilot under contract

Vendor agreement, subcontractor agreement and a permitted purpose all in place.

May involve no impermissible disclosure.

Exits at gate 1

Consumer chatbot, standard terms

No agreement behind the disclosure.

The four-factor assessment begins.

Three questions decide whether the disclosure was permitted. Business associate and conduit guidance from HHS; subcontractor and minimum necessary standards at 45 CFR 164.502. In both cases the examiner needs the tool, the vendor, the model and the agreement by name.

03 · Four factors

The risk assessment weighs at least four factors

The factors are listed in 45 CFR 164.402 and explained in the preamble to the 2013 Omnibus Rule (78 FR 5566). We read each one as a question an organization should be able to answer about any AI request on the day it is made.

Nature and extent of the PHI. Which identifiers, and how much clinical detail? HHS asks entities to weigh treatment plans, diagnoses, medications, medical history and test results, and a prompt built from a chart can carry all of them in one request. Pull the categories of PHI the request carried and which of them were removed before it left.

The unauthorized person. Who received it. HHS notes that the probability of compromise may be lower when the recipient is itself bound by HIPAA. Pull the provider, the model, and the terms that govern what the provider can keep and use.

Whether the PHI was actually acquired or viewed. HHS's example is a stolen laptop that was later recovered, where forensic analysis showed the data on it was never accessed. A request a model answered did reach the provider's systems. A request that was blocked, or redacted before it left, is a different case, as long as a record shows the decision.

The extent of mitigation. HHS points to the recipient's satisfactory assurances that the information will not be further used or disclosed, or will be destroyed, and cautions that such assurances from some third parties may not be enough. With an AI vendor we would ask for written confirmation of deletion in the first week, change the workflow so it cannot recur, and keep evidence that the change held.

Presumed a breach unless shown otherwise

To rebut the presumption, the covered entity or business associate shows a low probability that the PHI was compromised, based on a risk assessment of at least these four factors.

1

Nature and extent of the PHI

The types of identifiers and the likelihood of re-identification.

Asked of an AI request

Which identifiers, and how much clinical detail, did the request carry?

The record that answers it

The categories of PHI detected in the request, and which of them were redacted before it left.

2

The unauthorized person

Who used the PHI, or to whom it was disclosed.

Asked of an AI request

Which tool, vendor and model received it, and under which agreement?

The record that answers it

The model and provider for each request, matched to the signed business associate agreements.

3

Actually acquired or viewed

Or whether only the opportunity existed.

Asked of an AI request

Did the PHI reach the model, or was the request blocked or redacted first?

The record that answers it

The decision on the request (delivered, redacted or blocked) and the time it was made.

4

Extent of mitigation

Assurances of no further use or disclosure, or of destruction.

Asked of an AI request

What changed once the disclosure was found, and did the change hold?

The record that answers it

The vendor's written assurance, the date the workflow changed, and the requests recorded after it.

Each factor maps to a record. Factors from paragraph (2) of the breach definition at 45 CFR 164.402. The burden of proof sits with the covered entity or business associate under 45 CFR 164.414(b).

Three of the four factors turn on facts about the request: what it carried, where it went and whether it got there. An organization that cannot establish those facts has little to put in the assessment, and the presumption stands.

04 · Audit controls

The answers come from audit controls

The Security Rule's audit controls standard, 45 CFR 164.312(b), requires mechanisms that record and examine activity in systems that contain or use ePHI. An AI workflow that reads a chart is one of those systems. HHS's January 2025 proposal to update the rule goes further for AI: its preamble says a risk analysis must consider the type and amount of ePHI the AI tool accesses, to whom the data is disclosed, and to whom the output goes. The proposal is not final and the current rule stands. The first two items on that list are the first two breach factors, asked before an incident.

Minimum necessary (45 CFR 164.502) applies to what goes into the tool. A disclosure to a business associate is limited, with reasonable efforts, to the minimum necessary for its purpose, and the treatment exception for disclosures to health care providers does not cover an AI vendor. Sending the whole chart for a task that needed three fields raises a minimum-necessary question before any breach question arises.

Evidence the organization does not hold is weak evidence. A vendor's retention and access terms govern its logs, and screenshots gathered after the fact read as reconstruction. When we ask teams how they would answer these questions today, a vendor's logs and screenshots are usually the first things they mention. The record that works is written when the request is made, kept by the organization, and shows whether anyone has changed it since.

Inside the organization’s environment

1

Clinician or patient

PHI enters

Asks a question, pastes a note, or speaks during a visit.

2

AI tool

PHI in the prompt

A clinical copilot, ambient scribe, message drafter or prior-authorization assistant builds the prompt, often from the chart.

3

Boundary check

Last point of control

PHI is detected and redacted in flight, before the request leaves. The decision is written to a tamper-evident record.

What the record holds

  • The workflow that sent the request, and the model and provider that received it
  • The categories of PHI detected, and what was redacted
  • Whether the request was delivered, redacted or blocked
  • Sealed in a tamper-evident chain in a write-once store

The raw prompt stays inside the organization’s environment.

The request leaves the organization

4

Model provider

Detected PHI redacted

Receives the request as it crossed the boundary.

One AI request from start to finish. The boundary is the last point the organization controls before PHI reaches a model provider, and a record written there shows which PHI left and who received it.

05 · Beyond HIPAA

Section 1557 and state laws ask their own questions

HHS's Office for Civil Rights enforces both HIPAA and Section 1557 of the Affordable Care Act. Under 45 CFR 92.210, a covered entity may not discriminate on the basis of race, color, national origin, sex, age or disability through any automated or non-automated tool it uses to support clinical decision-making, and a copilot that suggests a diagnosis is one. Since 1 May 2025, covered entities have also had to make reasonable efforts to identify the tools whose inputs measure those characteristics, and to mitigate the risk of discrimination from each. The court orders against the 2024 rule do not reach section 92.210.

Our reference pages on Section 1557 and on ONC HTI-1 cover the detail. HTI-1 is the EHR certification rule for decision support tools, and ONC finalized HTI-5 in October 2026, which narrows it.

Three states have added duties that reach the same workflows. The timeline sets them beside the federal Section 1557 duty in date order (details and penalties).

Health AI duties, in order of effect

Not yet in force

1 Jan2025

CA

AB 3030

Binds: Provider

Generative AI patient messages about clinical information carry a disclaimer and a way to reach a human, unless a licensed or certified provider read and reviewed them.

1 Jan2025

CA

SB 1120

Binds: Plan

AI may not deny, delay or modify care on medical necessity. A licensed physician or other qualified professional decides.

1 May2025

US

Section 1557, 45 CFR 92.210

Binds: Covered entity

Make reasonable efforts to identify the decision support tools whose inputs measure protected characteristics, and mitigate the risk of discrimination from each.

1 Sep2025

TX

SB 1188

Binds: Provider

Review all records created with diagnostic AI and tell patients about the use. EHRs stay physically in the United States, including at cloud vendors.

1 Jan2026

TX

HB 149

Binds: Provider

Disclose AI use in a health care service or treatment to the patient by the date it is first provided.

1 Jan2027

CO

HB 26-1139

Binds: Plan

AI in utilization review produces and keeps documentation, audit logs and model-governance records that show compliance.

Six duties with five effective dates. Summaries of the enrolled texts and the Section 1557 final rule; see the source notes for each citation.

Texas SB 1188 adds a location rule, so where an AI service stores what it receives belongs in the same vendor review.

06 · Meilynx

Where the record comes from

Before the next incident

If we had to pick where a privacy office should start, it would be two things: the list of approved AI tools with the vendor, model provider and agreement for each, and a record of each request that the organization keeps itself. The breach rule gives at most 60 days from discovery to notify individuals and puts the burden of proof on the organization. A team that has both starts the assessment with the answers in hand.

This post summarizes 45 CFR Parts 92 and 164 and the state laws named above. It is not legal advice; verify against the primary sources and consult counsel.

Frequently asked

Is PHI pasted into a public AI chatbot automatically a reportable breach?

It is presumed to be a breach. Under 45 CFR 164.402, the covered entity or business associate can rebut the presumption by showing a low probability that the PHI was compromised, based on a documented assessment of at least four factors. If the presumption stands and the PHI was unsecured, individuals are notified without unreasonable delay and no later than 60 calendar days after discovery.

Does an AI vendor that handles PHI need a business associate agreement?

HHS lists a third-party AI chatbot on a provider's patient portal that handles patients' PHI as an example of a business associate. A business associate in turn needs an agreement with any subcontractor that receives the PHI. A model provider that receives prompts from an AI vendor stands in that subcontractor position.

Does Section 1557 apply to generative AI in clinical care?

Section 92.210 covers any automated or non-automated tool a covered entity uses to support clinical decision-making. This includes a generative AI tool that suggests a diagnosis or a course of treatment. Since 1 May 2025, covered entities have had to make reasonable efforts to identify such tools whose inputs measure protected characteristics, and to mitigate the risk of discrimination from each.

More from the blog

Regulatory updates

When a regulator changes what an AI examination asks for, hear about it first.

Short notes on SR 26-2, NYDFS 500, FINRA, the NAIC bulletin, the EU AI Act, and the employment-AI statutes, plus what we ship. A few emails a month.