With seed funds from the Data Science Institute, Kaveri Thakoor is helping doctors and AI train each other, and the early results are promising.
Although the president of the American Medical Association recently warned physicians that those who don’t use AI will be left behind many doctors remain skeptical of artificial intelligence. According to a recent survey, while two-thirds of doctors are interested in using AI, only one-third have taken the plunge. Holding the others back are a range of concerns about risks, including potential for bias, errors, and harm to the doctor-patient relationship.
But what if AI could function more like a medical intern, learning from and collaborating with senior physicians, then building on their expertise? Imagine AI tools that not only analyze data and suggest diagnoses but also provide transparent reasoning, akin to a human clinician. This is the vision driving Kaveri Thakoor, Assistant Professor of Ophthalmic Sciences at the Columbia University Vagelos College of Physicians and Surgeons and a member of the Data Science Institute. Thakoor has created an AI-powered decision support tool to assist in diagnosing eye disease.
By recording eye movements when expert ophthalmologists scan reports, and feeding that data into an AI model, Thakoor has built a tool that improves upon the accuracy of AI algorithms that did not use eye-tracking data.
What’s more, this tool is designed to act as a partner—trained by physicians for physicians—and is capable of explaining its diagnostic choices, akin to a human clinician.
“Human clinicians and AI bring distinct strengths to the table,” said Thakoor, who received her PhD in Biomedical Engineering at Columbia in 2022. “I wanted to create a tool that is a true collaborator, one that learns from the best doctors and, in turn, elevates the doctor’s ability to understand and treat diseases.”
Unlocking Expert Insight Through Eye-Tracking
Thakoor, who describes herself as an “engineer amongst ophthalmologists,” has been interested in eye-tracking since she was a graduate student and learned about the research of Harold Kundel and Calvin Nodine, pioneering researchers in the field of medical image perception. By studying the eye movements of radiologists as they scanned lung X-rays for cancer, they helped reveal the cognitive processes and visual strategies that radiologists use when scanning for early signs of cancer.
Experts cannot always explain their decisions objectively, said Thakoor, but studying their gaze is one way to get at the “otherwise mysterious” process.
Thakoor was drawn to the technique to create a diagnostic tool for early detection of glaucoma, one of the leading causes of irreversible blindness globally.
Building the Medical Expert-AI Team
Thakoor envisioned a two-pronged approach using AI and expert eye-tracking data. First, she wanted to create a tool to train novice doctors using data from senior physicians. And second, she proposed merging doctor eye movement data with existing AI diagnostic algorithms. This would create a “medical expert-AI team” that was more accurate than either one alone.
The project was a collaboration with Steven Feiner, Professor of Computer Science, and funded by Data Science Institute Seed Funds which, since 2018, have supported 38 promising new research collaborations between domain experts and data scientists across Columbia University. These grants have propelled advances in a range of areas, from understanding hiring inequities to supporting the transition to renewable power.
To train the model, Thakoor recruited 13 ophthalmology residents, fellows, and specialist physicians and told them to analyze a series of ophthalmology reports that contain images of different layers of the retina, which doctors use to diagnose glaucoma.
Each physician participant wore an eye-tracking headset as they reviewed and scored 20 patient reports. The tracker was equipped with one camera that pointed outwards towards the report and another pointed inwards to track the participant’s pupils.
It recorded pauses in gaze, called “fixations,” the areas of the report that physicians fixated on the most in their eye movements across the report. The patient reports were segmented into a grid and assigned corresponding numbers. This allowed the system to pinpoint which specific report areas, or “patches,” attracted the doctors’ gaze most consistently.
Surprising Findings and AI Training Insights
The research, published in 2023 in Frontiers in Medicine and in the Conference of the IEEE Engineering in Medicine and BIology Society, found something that surprised Thakoor—expert ophthalmologists only fixated on 10 percent of the “patches” in the report, whereas the novices tended to examine the entire report in detail. In addition, experts did not scan the page in the typical top-to-bottom, left-to-right direction, but they used a unique sequence of eye fixations.
To merge the expert physician data with existing AI, Thakoor’s team fed this data into an existing AI diagnostic system, telling it what patches it should focus on and in which order. She makes the analogy to ChatGPT’s ability to learn how to generate text. In language models, AI predicts which words belong together and what order they should appear in the text. “It is almost like we are learning the grammar of the clinicians’ gaze trajectory across the image in the same way ChatGPT learns the grammar of a language,” she said.
While the algorithm’s development was crucial, making the data usable for doctors was key. Steven Feiner’s expertise in graphical interfaces and augmented reality (AR) helped turn eye-tracking data into effective educational visual aids. Working together with Thakoor, their team refined the design to an intuitive orange highlighting system and built it on an interactive platform, making the tool easier to use and allowing for impact measurement.
“Our goal was to ensure that the technology not only functioned well but was also intuitive for doctors to use in real-world settings,” said Dr. Feiner. “By refining the user interface and visualization, we aimed to make complex data easy to interpret, empowering clinicians to make faster, more informed decisions.”
A Breakthrough in Diagnostic Accuracy
The results were promising: the AI diagnostic tool improved after it was fed the physician gaze data.
The AI model that incorporated gaze data diagnosed glaucoma more accurately than the AI trained without gaze data. Because the AI was trained on the data provided by experienced clinicians, it is able to explain which features on the reports it used to make a diagnosis, building trust among its human counterparts.
“Dr. Thakoor’s research demonstrates the promise of AI in healthcare, which is to support and augment clinicians’ abilities,” said Noémie Elhadad, Chair and Associate Professor in of the Department of Biomedical Informatics. “This helps physicians deliver better care while maintaining autonomy and trust. Work like this could have broad applications across medicine.”
Broader Implications and Future Directions
Next, Thakoor hopes it will revolutionize training. Using the data collected on gaze fixations and sequences, and working with the support of Data Science Institute students, she has developed a tool that allows trainees to practice diagnosing glaucoma by examining simulated clinical reports.
The tool gently nudges them to focus on specific patches of the report. It shows them where to focus attention and guides them through the correct sequence. The tool, which she is currently fine tuning and testing, will eventually be able to explain its rationale, and trainees will be able to ask for more information as needed.
“It’s very exciting,” said Thakoor. “We are creating something new, a medical expert-AI team, which is hopefully better than the sum of its parts.”