Researchers Analyze Accuracy of Machine Learning Model Relating to Medical Treatments for Schizophrenia

Tulsa, Okla. – As it is widely hoped that statistical models can improve decision-making related to medical treatments, and because the cost and scarcity of medical outcomes data can make testing data from patients in a different context prohibitive, researchers including Dr. Martin Paulus, Scientific Director and President at Laureate Institute for Brain Research (LIBR) in Tulsa, Okla., examined how well a machine learning model performed across several independent clinical trials of antipsychotic medicine for schizophrenia.
 
Researchers found models predicted patient outcomes with high accuracy within the trial in which the model was developed, but performed no better than chance when applied out-of-sample. Pooling data across trials to predict outcomes in the trial left out did not improve predictions. These results suggest that models predicting treatment outcomes in schizophrenia are highly context-dependent and may have limited generalizability.
 
The research findings, “Illusory Generalizability of Clinical Prediction Models,” were published in the January 11, 2024 edition of Science, the weekly journal of the American Association for the Advancement of Science (AAAS).
 
Researchers used treatment data from five international, multisite randomized controlled trials (RCTs) obtained through the YODA Project (https://yoda.yale.edu/), selected because of their comparability and consistency. All patients had a current DSM-IV diagnosis of schizophrenia at the start of the trial, randomized patients to an antipsychotic medication or placebo, and used the same scale to measure treatment outcomes (the Positive and Negative Syndrome Scale, PANSS). All trials also included a four-week timepoint to measure outcomes and collected similar data about the patients at baseline. The team applied machine learning methods using baseline data to predict whether a patient would achieve clinically significant improvement in symptoms over four weeks of anti-psychotic treatment. Further, the team evaluated the applicability of machine learning models across four distinct scenarios to gain insights into their generalizability: assessing within-trial performance without any external validation; employing within-trial cross-validation; assessing out-of-sample performance in a paired-trial validation; and assessing performance through an extension of the paired-trial validation with a leave-one-trial-out approach.
 
Results suggest that predictive models are fragile and that excellent performance in one clinical context is not a strong indicator of performance on future patients. Researchers from the team say this is “highly concerning” as most predictive studies today rely on internal samples for testing and validation. Further, researchers say the present study offers an “underwhelming but realistic picture of our current ability to develop truly useful predictive models” for schizophrenia treatment outcomes, and that “we should a priori remain skeptical” of any predictive model findings that lack an independent sample for validation.
 
The research team included first author Dr. Adam Chekroud, [email protected], Spring Health, Co-Founder and President, Yale University School of Medicine; Dr. Martin Paulus, [email protected], Laureate Institute for Brain Research; and collaborators from University of Cologne, Max-Planck-Gesellschaft, United Partners Consulting, LLC, Yale University School of Medicine, University Augsburg, and Yale University.
 
No funding source had any role in the study design, data collection, data analysis, data interpretation, writing, or submission of the report. All trials were originally funded by Janssen Research and Development.
 
# # #
 
CONTACT: For more information about the project, contact Martin Paulus, M.D., at Laureate Institute for Brain Research at [email protected]. For press inquiries, contact: Aimee Tonquest Mehl, Kingmaker Public Relations at [email protected].
 
ABOUT LAUREATE INSTITUTE FOR BRAIN RESEARCH (LIBR)
Launched in 2009, the Laureate Institute for Brain Research (laureateinstitute.org) is home to a multidisciplinary team of scientists and clinical research staff who apply neuroimaging, generic, pharmacological and neuropsychological tools to investigate the biology of neuropsychiatric disorders. LIBR’s creation was supported by The William K. Warren Foundation for the purpose of conducting studies aimed at developing more effective treatments and/or prevention strategies for these disorders. The studies are led by scientists from diverse backgrounds, including physics, cognitive neuroscience, psychology, psychiatry, neurology, developmental neuroscience, computer science, and genetics.