AI doctor seems like a concept from science fiction, but it may be closer to the reality than we think.

Recently, researchers from Google and DeepMind published a study in Nature, showcasing the amazing capabilities of their developed model, Med-Pathways Language Model (hereafter referred to as Med-PaLM) which can answer medical questions on a par with physicians.

This seems to imply that AI doctors are not far from replacing physicians.

01 Comparable to physicians

In this paper, the research team introduced a whole new MultiMedQA benchmark to evaluate Google’s PaLM large language model and its variant Flan-PaLM. This benchmark combines six existing medical question-answering datasets (MedQA, MedMCQA, PubMedQA, LiveQA, MedicationQA and MMLU), covering professional medicine, medical research and consumer medical question. Flan-PaLM reached an extremely high accuracy in evaluations of different datasets, with an accuracy of 67.6% on MedQA (United States Medical Licensing Examination), which was over 17% higher than previous level.

In addition, this team invited a group of physicians and outside experts to rate and assess the answer of Med-PaLM on multiple dimensions, including factuality, comprehension, reasoning, safety, equity, and bias.

02 Can AI doctors replace physicians?

This research by Google and DeepMind is definitely a breakthrough, displaying the tremendous potential of AI doctors in the medical field. However, does this mean that AI doctors can replace physicians?

The answer is not necessarily.

We should know that Med-PaLM is only a large language model but not a real physician. It can only learn from textual data and generate answers based on the given questions and prompts, but cannot understand the emotion, need and preference of patients, as well as complex and ambiguous questions. It is unable to diagnose and treat patients according to the actual condition, and even fails to adapt to different contexts and scenarios. Sometimes bias and misleading may occur. .

After all, this is a preliminary research achievement, and many limitations and challenges require to be overcome. For instance, the model can only answer questions raised in English, as well as multiple-choice or simple questions, but there are many complex and open questions worldwide; it is currently only capable of learning from existing dataset, but new findings and changes always emerge globally; and its performance still needs human evaluation, while there are many objective and standard evaluation methods.

Therefore, more testing, validations and improvements are necessary before clinical application of Med-PaLM and other similar large language model. Meanwhile, more comprehensive regulatory, ethical, and legal frameworks should be established to guarantee the quality, responsibility and transparency of AI doctors. More importantly, extensive dialogue and cooperation should be carried out with stakeholders, including patients, consumers, AI researchers, physicians, social scientists, ethicists, and policy makers, so as to responsibly transform these preliminary research achievements into real application that benefits human health and well-being.

Collectively, AI doctors cannot replace physicians, but can serve as an auxiliary tool to rapidly provide accurate information and suggestions, and help physicians save time and elevate efficiency.

What do you think of AI doctors?