Google has unveiled a significant upgrade to its AI-powered clinical search-and-answer tool, Vertex AI Search for healthcare, enabling it to extract and interpret data from medical images such as charts, tables, and diagrams. This new capability, part of a broader push into multimodal AI, aims to provide clinicians with a more comprehensive view of patient health by integrating insights from both text-based clinical notes and visual data. The tool is now also powered by Google’s latest AI model, Gemini 2.0, which the company claims is faster and more accurate than previous versions. The announcement comes as tech giants and developers showcase their latest innovations at the HIMSS health IT conference in Las Vegas this week.
Since its launch in March 2023, Vertex AI Search for healthcare has been designed to help clinicians quickly search through patient notes and clinical data using generative AI. The tool can answer questions about a patient’s medical history, streamlining workflows and improving decision-making. The latest update introduces Visual Q&A, a feature that allows the tool to process images directly without converting them into text first. This is particularly significant in healthcare, where nearly 90% of data is in image form, such as X-rays, scans, and diagnostic charts. By incorporating visual data, Vertex AI Search can deliver more relevant and actionable insights to healthcare providers.
The integration of multimodal AI into Vertex AI Search marks a major step forward in healthcare technology. Traditionally, AI tools have relied heavily on text-based inputs, but the ability to analyze images opens up new possibilities for diagnosis and treatment planning. For example, a clinician could upload a medical diagram or chart, and the tool would extract relevant information to include in its search results. This functionality is expected to enhance the accuracy and depth of clinical insights, ultimately improving patient care.
Google’s Vertex AI Search for healthcare is already integrated with two of the company’s large language models, Gemini 1.5 Flash and MedLM, and now benefits from the enhanced capabilities of Gemini 2.0. While Google has not disclosed the number of healthcare organizations currently using the tool, it has seen adoption by major health systems such as Community Health Systems and Highmark Health. Additionally, electronic health record (EHR) vendor Meditech has incorporated Google’s AI into its hospital software, and AI documentation company Suki has integrated Google’s patient summarization and Q&A technology into its clinical assistant tool.
In another development, Google announced a partnership with Counterpart Health, a subsidiary of health insurer Clover, to integrate Vertex AI Search into Counterpart’s clinical decision support platform. Clover, which operates Medicare Advantage plans in five states, launched Counterpart Assistant last spring to offer its technology to third-party clients. This collaboration underscores the growing interest in AI-driven tools among healthcare organizations, as providers seek to leverage advanced technologies to improve efficiency and patient outcomes.
The race to develop and deploy AI solutions in healthcare is intensifying, with tech giants like Google, Microsoft, and Amazon competing to establish themselves as leaders in the space. Google, in particular, has been focusing on tailoring its large language models to healthcare data and recently introduced a platform to help organizations create semi-autonomous AI agents. However, the rapid adoption of AI in healthcare has raised concerns about privacy, security, and the potential for errors. Unlike other industries, healthcare faces unique regulatory and ethical challenges, and the lack of federal oversight has left some experts wary of moving too quickly.
The introduction of multimodal AI, while promising, also introduces new complexities. For instance, a study by the National Institutes of Health last July found that while AI models performed well on text-based medical quizzes, they struggled with image-based questions, often making errors when interpreting visual data. This highlights the need for rigorous testing and validation as healthcare organizations increasingly rely on AI tools for critical decision-making.
Despite these challenges, the potential benefits of AI in healthcare are undeniable. By enabling clinicians to access and analyze a wider range of data, tools like Vertex AI Search for healthcare could revolutionize how medical professionals diagnose and treat patients. As the industry continues to embrace AI, the focus will be on balancing innovation with safety, ensuring that these technologies deliver on their promise without compromising patient care.
