A clinician does not care which model you used. They care whether the note is right, whether their patient's data stayed where it was supposed to, and whether anyone can explain a decision later if someone asks. That is the bar an AI feature has to clear in healthcare, and it is higher than most demos are built for.
I build in this space, so I have watched promising features stall at exactly this point. The model works in the prototype. Then someone asks where the data goes, who signs off on the output, and what happens when the service is down during a clinic's busiest hour. That is where the real product work starts.
Decide what the model is allowed to see
Start with the data, because that is where the real risk lives. Before any prompt leaves your system, ask what the model needs to do the job and send only that. A summarization feature rarely needs a full chart. It needs the few fields the summary is about.
If you use a hosted model provider, you need a signed business associate agreement and a written guarantee that your data will not be used to train their models. No BAA means protected health information should not reach that endpoint, full stop. Where you can de-identify before the call and still get a useful result, do it. The cheapest way to handle sensitive data is to not send it.
Let the model draft, keep the human deciding
The features that hold up in clinical settings tend to share one trait. The AI proposes and a person disposes. A model can draft a visit summary, suggest a code, or flag a possible interaction. A licensed human still reviews and signs. That line keeps a confident wrong answer from turning into a clinical action.
This also changes how you design the screen. The reviewer needs to see what the model suggested, why, and what it was working from, with an edit always one click away. If accepting the AI output is easier than checking it, you have built a rubber stamp, and eventually that bites someone.
Keep a trail you would be willing to show
Log enough that you can reconstruct any single decision months later. For us that means the input the model saw, the model and version, the raw output, who reviewed it, and what they changed before it became part of the record. The point is the day a patient, an auditor, or your own engineer asks what happened on a specific encounter.
Good logging has a quieter benefit. When a feature starts drifting, the trail is also your evaluation set. You can see where humans keep overriding the model and fix the feature instead of guessing.
Assume the model will be slow, wrong, or down
Clinical work does not pause because an API is having a bad day. Every AI feature needs a defined behavior for when the model is unavailable, too slow, or clearly off. Usually that means the workflow falls back to the manual path the team already knows, without trapping anyone. Until that path exists, the feature is still a prototype with a nicer uptime chart.
Test the fallback before launch
For example, simulate a timeout while a clinician is drafting a visit note. The unsaved text should remain available, manual editing should still work, and no partial AI draft should enter the signed record. When the service recovers, a late response must not overwrite the clinician's edits. Run that scenario before enabling the feature for a clinic.
If your team is working through this, I am always glad to compare notes.