AI-based clinical predication models cannot be generalised
The results of new study involving the University of Augsburg led by Yale University show that algorithms fail to adapt to new data sets.
Scientific models generated using algorithms based on large data sets cannot be generalised and transferred to other patient data. This is the finding of a new international study involving Prof. Dr Alkomiet Hasan, Chair of Psychiatry and Psychotherapy, at the University of Augsburg. The study was led by Prof. Dr Adam Chekroud from Yale University. Precision medicine using modern algorithms and machine learning is currently one of the most important fields in medicine. It aims to offer individualised treatment adapted to each patient. Large amounts of data and data points are therefore processed upon which individual therapy response predictions or predictions about the development of a disease are then developed. This facilitates physicians in selecting the most promising medications for successfully treating individual patients.
Email:
alkomiet.hasan@bkh-augsburgbkh-augsburg.de ()
Email:
corina.haerning@presse.uni-augsburgpresse.uni-augsburg.de ()
“However, there is still too little research and discussion about the challenges involved in this approach,” says Prof. Dr Alkomiet Hasan, who holds the Chair of Psychiatry and Psychotherapy at the Faculty of Medicine. Hasan took part in an international study led by Prof. Dr Adam Chekroud (Yale University and founder of the unicorn startup Spring Health) that showed that clinical prediction models for mental illness still have a long and rocky road ahead of them. The sobering results of a study recently published in the renowned journal Science and cited in the equally prestigious journal Nature make it clear that models for predicting treatment response in schizophrenia, one of the most serious mental illnesses, can be developed very well using various methods. However, when these algorithms were applied to data from patients who were not part of the study, they failed, meaning that predictions cannot be made.
The results of the study show that algorithms developed on the basis of one data set cannot be generalised and transferred to other groups of patients, which means they cannot be used in practice. Chekroud emphasises in a commentary on the study that the development of algorithms for predicting therapy response must be developed in the same way as new medications. This means that not every success in a study should be understood as evidence of its applicability to practice. “The results of this study make clear that not only in psychiatry and psychotherapy but also in other areas of medicine there is still a long way to go to individualised therapy,” says Hasan, adding “This project, which began in 2016, shows how important it is to collaborate internationally across disciplinary boundaries and work with large, publicly accessible data to overcome hurdles.” “I am proud that Augsburg was part of this international study and did not shy away from tackling this important topic. I am sure that the results will attract attention among the scientific community,” adds Prof. Dr Martina Kadmon, dean of the Faculty of Medicine at the University of Augsburg.
Publication in Science:
https://www.science.org/doi/10.1126/science.adg8538
Commentary on the study in Science:
https://www.science.org/doi/10.1126/science.adm9218
Commentary on the study in Nature:
https://www.nature.com/articles/d41586-024-00094-9
Scientific Contact
Media Contact