A new artificial intelligence model can predict a cancer patient’s risk of dying from a single photograph of their face, according to new research. While not perfectly accurate, the model found that people with higher risk-of-death scores, in general, had a higher risk of dying.

The AI builds on how human doctors evaluate their patients’ health, says Ray Mak, a radiation oncologist at Mass General Brigham Cancer Institute, who conducted the research with his colleagues.

“We’re trained in medical school to walk into a room, look at the patient and make an assessment—an impression of how healthy, frail or sick they are,” Mak says. That decision can help determine what treatment a patient receives—surgery, chemotherapy, radiation, and more. The problem, Mak says, is “that’s really a subjective assessment.”

On supporting science journalism

If you’re enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.

The AI, Mak and his team hope, could help doctors and patients make more informed, objective decisions about cancer treatment and end-of-life care.

Mak and his colleagues trained an algorithm to analyze and “learn” 40 million images of faces without giving it a specific task. They then tweaked the model to analyze new photographs in two distinct ways: first, to estimate a person’s biological age and, second, to predict cancer patients’ odds of survival.

The outcome was surprisingly accurate. In a test group of 3,820 cancer patients, more than half of those with the highest risk score—64 out of 122—died within just 90 days of when their photograph was taken. In a separate test group that included more than 38,000 cancer patients, those that looked older than their chronological age also had a higher risk of dying.

“To our knowledge, this is the first report of an AI algorithm that can directly predict cancer patient survival from a simple face photograph,” the researchers write in the study, which was published on Monday in the journal Lancet Digital Health.

The question is: How does the AI do this? For both age and death risk predictions, Mak and his team tracked where the algorithm was “looking” in its analysis, a technique they call “attention mapping.” For age prediction, they found that the AI tended to base its predictions on facial areas such as the nasolabial folds—the crease between your nose and mouth—and parts of the forehead. For risk of death, the AI tended to focus on the bridge of the nose and areas below the eyes. It’s unclear why those areas specifically correlated with a person’s risk of death.

The two analyses work better together. “When we put both the face age and face survival scores into a clinical prognostic model, we do better at survival prediction than either one alone,” Mak says.

In some cases, the AI may have been picking up on known health risks, such as smoking and alcohol use. When it came to the age estimate, smokers, for instance, were more likely to look older than their age, Mak says. The same was true of heavy alcohol drinkers.

It also didn’t matter if a photograph showed a person frowning or smiling, with or without makeup or in various lighting. For age prediction, the AI was able to account for common forms of plastic surgery, Mak says. A major exception was rhinoplasty—the AI tended to rate people who’d had a nose job as slightly older in images taken after the surgery compared with images taken before it.

Mak’s hope is that, eventually, an AI model like this could be deployed in hospitals and serve as an extra data point in doctors’ decision-making. It could also help serve as an early-warning system of disease if a patient appears to be aging faster than expected. Mass General Brigham has a patent pending for face health algorithms.

Right now the AI is not ready for the clinic. For one, the cancer survival model was trained on a Boston-area cancer patient population that was majority white. Mak and his colleagues are working to expand their training sets to include a wider diversity of faces. (To that end, if you feel comfortable doing so, you can help train the algorithm and receive a “face age” estimate by sharing your own photographs here.)

But the possibility of AI having any role in determining a patient’s treatment plan raises clear ethical questions.

Matthew Liao, a professor and director of the Center for Bioethics at New York University, who was not involved in the new research, says that the paper was “interesting” and that this kind of tool could be useful for identifying people who might be sick and in need of extra care. But he notes it’s possible that an AI algorithm could slip up—and misidentify patients as being healthy or not. “If it gets it wrong, we might be sort of not providing care to someone who actually needs it,” he says.

Liao also raised the question of insurance companies turning to this sort of technology to grant or deny coverage or employers using it to hire a given job candidate deemed to be “healthier” than another. “We really need to think through the societal implications of this thing as we try as we develop it further,” he says.

Mak, for his part, is optimistic. “There’s still a lot of development ahead and clinical testing,” he says. “But we’re really excited that that we’re launching a completely new field in medicine.”