For many, frustration with the American healthcare system begins with the very first step: finding doctors. Although most health plans offer online directories to search for in-network providers, these tools often fall short in connecting patients with comprehensive care options.

Now, Kaiser Permanente, a health consortium with thousands of doctors in its network, has teamed up with researchers at USC Viterbi’s Information Sciences Institute (ISI) to improve the quality of its search application using AI. The team engineered a new version of Kaiser’s “Find Doctors and Locations” tool, one that uses a semantic search engine to deliver higher-quality results.

“We present a novel approach to an under-studied, but high-impact, search problem in the healthcare domain called Finding Doctors and Locations,” Kaiser researchers wrote in the paper. “We implemented and deployed an advanced knowledge graph-based pipeline drawing from best-of-breed AI and data analytics research.”

Compared to the previous system, the novel tool improved patients’ chances of finding a doctor in their region by 20%. The researchers will present their findings at the 47th International ACM SIGIR Conference in July 2024.

Previously, Kaiser’s Find Doctors and Locations tool utilized a string-based search engine, which worked by matching the exact keywords entered by the user to the information available from the provider. 

But string-based search proved inadequate for the complex world of medicine jargon. If a patient misspelled a specialty, searched via their symptoms (“chest pain” versus “cardiologist”), or resorted to colloquial terminology (“kidney doctor” versus “nephrologist”), the system returned incomplete or irrelevant results. 

Building a healthcare knowledge graph

Enter semantic search, a type of AI that can understand the context and relationship between words in a query. At the core of this system is what is known as a knowledge graph, a network that represents real-world entities — such as doctors, locations, specialties and symptoms — as well as the relationship between them. 

In a user query, the system then matches the keywords (e.g. “kidney”) to the relevant categories within the healthcare knowledge graph, providing users with better results than directly comparing text.

“We like to say that a knowledge graph represents things not strings,” Mayank Kejriwal, an ISI Research Lead who worked on the study, said. 

Researchers built the healthcare knowledge graph using data from various sources, including the Kaiser Permanente Find Doctors & Locations API, electronic health records and user search logs. The team also employed OpenAI’s ChatGPT to map body organ terms to relevant medical specialties, further enriching the knowledge graph’s semantic network.

To evaluate the effectiveness of their search engine, the researchers conducted an experiment using real query logs from Kaiser Permanente’s Find Doctors & Locations system. 

The results were promising. When combined with the current Find Doctors & Location system, the semantic-based search engine provided relevant results for more than half of the queries, demonstrating a significant 20% advantage over the baseline system. 

Results in milliseconds

Semantic search also helped the researchers achieve real-world deployment. “A very challenging aspect of the system is that it has to work in real time,” Kejriwal explained. “There are a lot of great algorithms out there, but it’s very difficult to implement them on a real website where you get results quickly.” 

If the researchers had built their tool using more complex deep neural networks or large language models, it may not have provided the fast response times that real users expect from a search engine, possibly discouraging adoption of the new tool. 

But Kaiser’s new tool works quickly, capable of returning results in milliseconds. The health network publicly deployed the new version of the Find Doctors & Location application on their website at the end of 2023, and it has been utilized by over 12 million users. 

“We hope that our implementation can serve as a successful case study in applying AI research that is traditionally only applied in domains such as e-commerce and generic web search, with the goal of serving the needs of patients,” they wrote.

The researchers believe that the improved search quality could have a long-term impact on patient behavior. Previously, discouraged by poor-quality results, patients might have abandoned searches for what they truly needed. Now, the system encourages patients to be more specific and complex with their queries, such as searching for a practice that offers multiple specialities in one location — a scenario that would be very unlikely to find via an exact text match. 

“In some ways, users get conditioned by the system,” Kejriwal said. “Now that it’s up, we are hoping that users will try more complicated queries, and that system will be able to answer them.” 

Published on June 11th, 2024

Last updated on June 11th, 2024