AWS’ head of health AI on Amazon’s new clinical documentation play

Amazon Web Services generated significant buzz earlier this week when it announced HealthScribe, its own clinical documentation tool that uses generative artificial intelligence to summarize doctor-patient visits.

With the launch, Amazon joined a number of other tech companies including Microsoft in the space, jockeying for a share of the market as providers look for ways to cut documentation burden on physicians.

In an interview on the sidelines of the AWS Summit on Wednesday, AWS Head of Health AI Tehsin Syed told Healthcare Dive what differentiates HealthScribe from other medical notetakers, how the technology is expected to drive client demand and how Amazon is approaching pernicious accuracy, ethics and liability issues around generative AI in healthcare.

Editor’s note: This interview has been edited for clarity and brevity.

TEHSIN SYED: HealthScribe transcribes the conversation between a doctor and a patient. It identifies the speakers and segments the transcript. If you start with some chitchat, it’ll recognize, okay, this is not relevant to the clinical note. It extracts specific medical information — things like medications, tests and references to anatomy — to take those as structured elements. This is where the generative AI capability comes in. That information, and the large language model we’ve trained, is used to generate summary documentation, in the shape of chief complaint, history of present illness, test and plan.

So they take a Titan model — that’s an Amazon first-party model — and they do what’s called grounding. They adapt the model to this specific purpose of learning how to generate this kind of documentation.

Finally, they show the end user the transcript and the summary, and cite conclusions in the summary back to the dialogue. So it’s a really natural and easy review process for clinicians.

SYED: HealthScribe is an API that Amazon is selling to its partners for them to provide an end user experience. That’s a very different business model. We’re saying, you don’t have to have all the machine learning expertise and all of the infrastructure that’s needed to manage a large language model.

AWS head of health AI Tehsin Syed

Permission granted by AWS

AWS is going to do that part of the work, and maintain the security and privacy of the data. They provide this capability to anyone who wants to use it, and they have public pricing based on usage, so it’s clear what’s going on.

What we’re trying to do is say, whoever you are, whatever software builder you are, you get to market much faster with this capability. You can concentrate on the end user experience.

There’s also innovation in the technology itself. Some of the grounding techniques are patent-pending. Underlying HealthScribe itself — at a high level everyone’s using an LLM and whatever — but part of this is the expertise that they bring.

SYED: There’s a number of others who are interested. I absolutely anticipate new customers coming to the table who previously didn’t have the ability to do this [work] themselves. We’ve seen a lot of interest from software vendors, because we’re providing an API and you do have to have some expertise to build. When they talk to provider systems, they’re interested, and they usually ask who’s a good partner, and how do they work with them to incorporate this.

SYED: The best way to show accuracy and build trust in the system is the method of showing end users what in the summary came from the dialogue. I don’t think showing them an accuracy stat is going to help as much.

Internally, obviously, they use a number of approaches. Some are about factual completeness — is what was in the dialogue reflected in the summary? Did the model come up with something that wasn’t in the dialogue? There’s a couple other metrics they track internally but it’s not shared externally.

SYED: I think we’re directly addressing the concern about quality by showing the user every time why a decision was made.

SYED: This is part of why we’re announcing in preview, and we’re working with their partners to understand what’s necessary to do here. I don’t have a definitive answer for you. There’s a lot of complexity to it, right? You have an ambient listening device, but maybe it’s too far away, or the air conditioner’s too loud. It’s not as simple as HealthScribe sharing accuracy numbers, because there could be an issue with the recorded conversation that we’re getting.

I agree that there’s a lot of think about in doing this responsibly.

What we’re saying is here’s what they can do, and as HealthScribe is deployed in the workflow there are other things that their customers have to do to ensure it’s the right level of accuracy. They are very clear that it’s generating the summary but it is not automating the documentation. The idea is, this is a tool, and then the tool is applied in a workflow. And the workflow has a lot of those checks and balances in it.

SYED: The way they look at that is, we’re providing a capability that has to be incorporated in an end-to-end workflow, and that workflow has to deal with those aspects you’re talking about. It’s the vendor who’s incorporating the capability that has to make sure that workflow is appropriate, and the clinician also has to make sure.

I worked at Cerner a long time before this. The dictation thing has been around for a very long time. It’s the same thing, right? Is it accurate or not? So folks have mechanisms in place to handle the workflow around this. But across the board with AI, there’s lots of new issues.

SYED: I don’t have an opinion on moving too quickly or not quickly enough. They work with their customers on very specific use cases. With HealthScribe, we’re iterating during this preview period to make sure it is actually fit for its purpose, and it meets criteria that would result in real use in a responsible way.

AWS is ready to power AI agents that can handle busywork instead of just chatting

By Emilia David, a reporter who covers AI. Prior to joining The Verge, she covered the intersection between technology, finance, and the economy.

During the AWS Summit in New York on Wednesday, Amazon launched Agents for Bedrock, which will let companies build AI apps that can automatically do tasks on their own, like booking a flight for users instead of just telling them about it. AI agents are the assistant that actually gets a restaurant reservation instead of just giving suggestions on where to eat.

“I believe this will supercharge developers who wanted an easier way to build agents and at the same time customize the data the models read,” Swami Sivasubramanian, vice president of data and machine learning at AWS, tells The Verge. “Building agents took so much time even with how advanced generative AI is now, but we’re making it so developers can access exactly the models they need.”

Generative AI models like GPT-4 or Llama 2 are powerful but do not actually automate some tasks for users without extra help, like plug-ins. With agents, a travel company could use generative AI to suggest trips, then build one agent that will take in a user’s travel history and interests, another agent that finds flight schedules, and finally, an agent that books the chosen flight.

Amazon, along with other AI companies like Meta, Google, and OpenAI, signed an agreement with the White House last week pledging to develop responsible AI. 

Amazon itself hasn’t been a big player in the generative AI arms race in the way Meta, Google, OpenAI, and Microsoft have been. Still, it has positioned itself, via AWS, as a key infrastructure provider to the space. StabilityAI and Anthropic released new versions of their large language models, Stable Diffusion XL 1.0 and the chatbot Claude 2, that are now available on Bedrock. 

Agents for Bedrock is aimed at startups and enterprises, letting companies use their own data to teach foundational models, like image-to-text models or large language models, and then build additional apps to finish tasks. A developer can choose which foundational model to use, provide some instructions, and choose which data for the model to read.

Other tech companies are also working on agents; Meta CEO Mark Zuckerberg told investors there’s an opportunity to bring AI agents “to billions of people in ways that will be useful and meaningful.” OpenAI CEO Sam Altman, in a wide-ranging interview with The Atlantic, ruminated on AI agents and how to approach their creation best. 

Bedrock, a platform where organizations can access foundational generative AI models like Stable Diffusion, Claude, Jurassic, and Amazon’s own Titan language, was launched in April this year. 

Other AWS announcements included bringing generative AI to healthcare through a new service called AWS HealthScribe and a partnership with Nvidia allowing AWS to handle larger amounts of memory and data by using Nvidia H100 Tensor Core GPUs.

AWS HealthScribe automates clinical documentation, basically notes summarizing a patient’s complaint, and makes medical notes for diagnosis. AWS partnered with 3M Health Information System to deploy the tool. Sivasubramanian said using generative AI to create clinical documentation saves time for healthcare professionals and lets them interact with more patients. 

Bringing AI to healthcare, though, will be fraught. Healthcare is one of the most regulated spaces in the US. AI companies face questions about privacy protection and security. Data privacy is one of the main focuses of the Federal Trade Commission’s investigation into OpenAI. AWS reiterated that HealthScribe is HIPAA-eligible and will not use results from HealthScribe to train the tool. 

“Healthcare is one of the spaces where AI can dramatically reduce the burden on people,” Sivasubramanian said.

AWS made its machine learning-powered analytics tool, Entity Resolution, that helps people analyze and match documents across applications generally available. AWS also made its code assistant CodeWhisperer available in Glue Studio — a platform that simplifies coding for data analysts — to allow access for data engineers and analysts. It gives code suggestions in real time. 

AWS Introduces Amazon Route 53 Resolver on AWS Outposts Rack

AWS recently announced that Amazon Route 53 Resolver is now available on AWS Outposts rack providing on-premises services and applications with local Domain Name Service (DNS) resolution directly from Outposts. In addition, local Route 53 Resolver endpoints also enable DNS resolution between Outposts and on-premises DNS servers.

AWS Outposts Rack is a fully managed service that extends AWS infrastructure, services, APIs, and tools on-premises. While Amazon Route 53 is a service for managing DNS names and answering DNS queries. The Amazon Route 53 Resolver (earlier called Amazon DNS server) responds recursively to DNS queries from AWS resources for public records, Amazon VPC-specific DNS names, and Amazon Route 53 private hosted zones and is available by default in all Virtual Private Clouds (VPC).

With the availability of Amazon Route 53 Resolver on AWS Outposts rack, the company is providing a solution that can Excellerate on-premises applications availability and performance since earlier, applications and services running on an Outpost forwarded their DNS queries to the parent AWS Region the Outpost is connected to. This could lead to temporary site disconnections due to weather events or fiber cuts, which subsequently leads to local DNS resolution fails.

When opting for local Route 53 Resolvers, the company states, "applications and services will continue to benefit from local DNS resolution to discover other services—even in a parent AWS Region connectivity loss event."

Furthermore, Sébastien Stormacq, a principal developer advocate at AWS, explains in an AWS news blog post:

Local Resolvers also help to reduce latency for DNS resolutions as query results are cached and served locally from the Outposts, eliminating unnecessary round-trips to the parent AWS Region. All the DNS resolutions for applications in Outposts VPCs using private DNS are served locally.

The local Route 53 Resolver endpoints are part of the launch of the availability of Amazon Route 53 Resolver on the AWS Outposts rack, providing cloud network administrators with the capability to establish endpoints within the VPC on Outposts.

With Route 53 Resolver outbound endpoints, cloud network administrators can direct DNS queries from Route 53 Resolvers to DNS resolvers under their management, such as those on their on-premises network. On the other hand, Route 53 Resolver inbound endpoints forward DNS queries received from outside the VPC to the Resolver running on Outposts. This feature facilitates sending DNS queries for services deployed on a private Outposts VPC from external sources beyond that VPC.

Screenshot of the Create Route 53 resolver in the AWS Management Console (Source: AWS News blog post)

Another public cloud provider Microsoft offers similar services to AWS Outpost Rack. For instance, the Azure Stack family offers Azure Stack Hub, which provides a subset of Azure services that can run locally, allowing organizations to build and deploy applications like they would on the public Azure cloud. It includes an integration with DNS.

To start with Amazon Route 53 Resolver on AWS Outposts, the company recommends that customers enable Route 53 Resolver when ordering a new Outpost or work with them to add Route 53 Resolver to an existing Outpost. Furthermore, they can create and manage the Route 53 Resolver on Outposts using Route 53 APIs, the AWS Management Console, the AWS CLI, or the AWS SDKs.

Lastly, users can configure local Resolvers and endpoints in all AWS Regions where Outposts racks are available, except in AWS GovCloud (US) Regions. Furthermore, the local Resolvers are provided without additional cost, and the endpoints are charged per elastic network interface (ENI) per hour. And finally, the pricing details of the Outposts rack are available on the pricing page.