Healthcare Voice AI: Answer These Questions Before You Ask for a Demo

A good voice AI demo can be dangerous.

Not because the technology is automatically unsafe. The problem is that a polished conversation can make a healthcare team feel farther along than it is. The voice sounds natural. The appointment gets booked. The transfer works. Everyone leaves impressed, but nobody has agreed on what the system may say, what data it may touch, or when a person must take over.

The team starts comparing vendors before it has defined the job.

Pause the shortlist. Before you ask for a demo, pick one patient-access workflow and write down its boundaries. The first buying decision is whether that workflow is safe, useful, measurable, and realistic enough to test.

This is different from a general AI receptionist evaluation. Healthcare adds clinical sensitivity, protected health information, patient expectations, and more consequential failure modes. A demo should test your written requirements, not create them for you.

Name one workflow

“Improve patient access” is not a use case. It is a broad goal that can include scheduling, reminders, referrals, billing questions, prescription requests, triage, prior authorization, and dozens of other call types.

Choose one narrow job. For example:

Answer approved questions about location, hours, and parking
Route calls to the right department or location
Confirm an existing appointment without changing clinical information
Collect a callback request for a defined team
Cover one after-hours queue with a limited menu of actions

Then document the starting event and the finished outcome. If the workflow is appointment confirmation, does success mean the patient verbally confirms, the scheduling system is updated, and the action appears in an audit log? If a system update fails, is the call still counted as complete?

Do not let a vendor define success as “the AI handled the call.” That phrase can hide a bad transfer, a duplicate appointment, an incomplete message, or a patient who hangs up and calls again.

Ask these questions first:

Which exact call type are we considering?
What may the voice agent do during that call?
What must it never do?
Which team owns the workflow today?
What record proves the call ended correctly?

If the answers are vague, you are not ready for a useful demo.

Draw the data boundary before discussing HIPAA

A vendor saying “HIPAA ready” does not tell you whether your proposed workflow is acceptable. The answer depends on the service, contract, configuration, data flow, and how your organization uses it.

Map what the system may hear, create, retrieve, store, and send. Include caller phone numbers, recordings, transcripts, patient names, dates of birth, appointment details, messages, authentication answers, call summaries, and metadata. Then map every destination: the voice platform, model provider, telephony carrier, EHR, scheduling system, contact center, analytics tool, logging platform, and vendor support environment.

Under the HIPAA Rules, a covered entity can use a business associate to handle protected health information only under the applicable requirements, including satisfactory assurances that the information will be safeguarded. The written arrangement must establish permitted uses and disclosures and address safeguards, incident reporting, subcontractors, and what happens to the information when the relationship ends. Review the current requirements in 45 CFR 164.502 and 45 CFR 164.504 with your privacy and legal teams.

That means the BAA question needs more precision:

Will the vendor sign a BAA for this specific voice AI service?
Which affiliates and subprocessors handle the data?
Are model providers and telephony services inside the documented boundary?
Which data uses are permitted under the agreement?
Is customer data used to train or improve shared models?
How long are audio, transcripts, logs, and backups retained?
Can retention be shortened, and can deletion be verified?
What data can vendor support personnel access?
What is returned or destroyed at termination?

Do not accept the compliance page for the vendor’s main platform as proof that every new AI feature, connector, and model is covered.

The HIPAA minimum necessary standard also matters for many uses and disclosures of PHI, subject to its exceptions. A scheduling workflow should not receive broad record access because another future use case might need it. Start with the least data the approved job requires, then have the right internal reviewers confirm the boundary.

For a broader contract review, use these AI vendor data retention questions before employees or callers supply sensitive information.

Mark the human-only boundary

Healthcare voice automation needs an obvious exit.

Write down the topics and conditions that require a person. This may include reported urgent symptoms, clinical questions, medication requests, complaints, distressed callers, identity uncertainty, repeated misunderstanding, language or accessibility needs the system cannot support, and any request outside the approved workflow.

These are evaluation hypotheses, not universal clinical rules. Clinical, privacy, legal, operations, and patient-access leaders need to approve the final boundaries for their setting.

The handoff design should answer more than “Can it transfer a call?”

Ask:

What words, intents, or failure conditions trigger transfer?
Can the caller ask for a person at any time?
What happens when the destination queue is closed or overloaded?
Does the employee receive the reason for the call and approved context?
Does the patient have to repeat sensitive information?
Who owns a message when a live transfer fails?
How is an urgent exception routed after hours?
Can staff interrupt or take control of a live conversation?

Run those scenarios in the demo. A perfect happy-path booking tells you very little about how the product behaves when the call matters most.

Define every system action

Voice AI becomes more useful when it can read schedules, create appointments, send messages, open cases, and update patient records. It also becomes harder to control.

For each proposed integration, document the objects the system can read and the fields it can write. Identify the service account, permission scope, authentication method, timeout behavior, retry logic, duplicate prevention, and audit record. Decide what happens when the EHR or scheduling system is unavailable.

The HIPAA Security Rule requires covered entities and business associates to conduct risk analysis and apply administrative and technical safeguards. Current regulatory text addresses security management processes in 45 CFR 164.308 and controls such as access control, audit controls, authentication, and transmission security in 45 CFR 164.312.

Translate that into demo evidence. Ask the vendor to show how an administrator limits access, how a failed write appears, how an action connects to a caller and session, and how the team can reconstruct what happened after a complaint or incident.

A slide about security is not the same as seeing the control work.

Decide who is buying and who can stop the pilot

Voice AI can touch IT, security, privacy, clinical operations, patient access, legal, procurement, and finance. If all of them are “stakeholders” but nobody owns the outcome, the project will drift.

Name five roles before vendor selection:

A business owner accountable for the patient-access result
A technical owner for telephony, integrations, and support
Privacy and security reviewers for the data and control boundary
An operational owner for scripts, exceptions, and human staffing
An executive sponsor who can approve expansion or stop the work

One person may cover more than one role in a smaller organization. The accountability still needs to be explicit.

NIST’s Generative AI Profile recommends documenting AI systems, human oversight roles, and pre-deployment testing. Use that as an operating discipline. Put the voice agent in the AI inventory before it reaches production, not after an incident forces someone to find it.

Build the scorecard before the vendor controls the room

A demo should test whether the product can support your workflow. Give each vendor the same scenarios, integrations, constraints, and questions.

Score the evidence across four areas:

Workflow completion: Did the approved outcome happen correctly?
Safety and handoff: Did the agent stop and transfer when required?
Data and controls: Could the team verify access, retention, actions, and logs?
Operating fit: Can staff maintain the knowledge, review failures, and support the workflow?

Add cost only after the scope is clear. Model implementation, integration, telephony, usage, support, monitoring, tuning, and internal review time. A cheap per-minute rate can still produce an expensive workflow if employees spend their day repairing bad appointments or chasing dropped messages.

Set pilot entry criteria and stop conditions in writing. Stop if the system exposes information to the wrong person, performs an unauthorized action, repeatedly mishandles a high-risk call, blocks human escalation, or fails without creating a recoverable work item.

The pilot should end with one decision: expand, revise, or stop. “Keep experimenting” is not a decision.

Your pre-demo decision packet

Before scheduling vendor demos, require a short packet containing:

One workflow and named owner
Approved actions and prohibited actions
Data-flow map and PHI boundary
Required systems and integration permissions
Human-only topics and transfer rules
Security, privacy, and contract questions
Test scenarios and expected outcomes
Pilot measures, stop conditions, and decision owner
Expected volume and full cost model
Open questions that remain open

That last item matters. Unknowns should stay unknown until evidence closes them. Do not let a confident demo fill gaps with assumptions.

Healthcare voice AI may be a good fit for a narrow patient-access workflow. It may also be the wrong tool for the call type, data boundary, staffing model, or integration environment. You need to know that before a long contract turns curiosity into an operating problem.

Pause the vendor shortlist until one workflow has a named owner and written boundaries. Catch Advisors helps healthcare IT leaders define requirements, compare voice AI and contact center options, and run a vendor-neutral evaluation. If your team is considering patient-access automation, book a voice AI discovery session before you ask vendors to show you what they want to sell.