Predicting Synergistic Cancer Drug Combinations Using a Disease-Agnostic Machine Learning Approach

Overview

Cancer combination therapy may enhance therapeutic efficacy by concurrently targeting multiple survival pathways; however, experimentally evaluating millions of potential drug pairs is neither biologically practical nor financially sustainable. This study utilized a cross-dataset machine learning framework to predict synergistic cancer drug combinations while contextualizing the biological mechanisms underlying those predictions. DrugComb, Genomics of Drug Sensitivity in Cancer (GDSC), Library of Integrated Network-Based Cellular Signatures (LINCS L1000), ChEMBL, and Molecular Signatures Database (MSigDB) were integrated into a unified Drug A × Drug B × Cell Line feature matrix containing chemical descriptors, drug-response features, transcriptomic signatures, and pathway-level information across 19,086 drug combinations and 48 cancer cell lines. Five models, Random Forest, XGBoost, LightGBM, Elastic Net, and a deep neural network, were trained and evaluated using stratified cross-validation and repeated validation with cell-line-aware partitioning to reduce information leakage. XGBoost achieved the strongest overall performance, with an accuracy of 0.999 and F1-score of 0.998. To interpret XGBoost’s drug combination predictions, SHapley Additive exPlanations (SHAP) was used to quantify each gene-associated input’s contribution to the model output and rank the genes by their influence on predicted synergy, after over-representation analysis (ORA) was applied to the top-ranked genes to test whether they were significantly enriched in known biological pathways. The framework was further examined through ovarian (OVCAR-8) and prostate cancer (PC-3) case studies, selected to represent female- and male-associated malignancies, respectively, and to evaluate whether the model’s biologically contextualized predictions remained informative across distinct sex-linked disease settings. In ovarian cancer, docetaxel + nilotinib achieved a predicted synergy probability of 0.996 in OVCAR-8 cells, with SHAP highlighting HtrA Serine Peptidase 1 (HTRA1), a gene associated with replication stress and checkpoint vulnerability, and ORA identifying enrichment of E2F, G2/M checkpoint, cell-cycle, and signaling pathways, consistent with vulnerability in proliferative and checkpoint control. In prostate cancer, docetaxel + vismodegib achieved a predicted synergy probability of 0.977 in PC-3 cells, with SHAP highlighting FAT Atypical Cadherin 1 (FAT1) and RNA-binding protein with serine-rich domain 1 (RNPS1), genes linked to survival signaling and stress adaptation, and ORA identifying enrichment of cancer, cell-cycle, apoptosis, focal adhesion, HIF-1, and related signaling pathways, consistent with dependence on adaptive survival programs. By linking model predictions to gene-target, gene-level, and pathway-level biology across integrated public datasets, this framework supports drug synergy prioritization with biological context. The prioritized combinations also showed high predicted binding affinity (<-9 kJ/mol) to their intended targets. Future wet-lab testing in OVCAR-8 and PC-3 could determine whether these combinations are more efficacious than current treatment approaches and whether targeting multiple survival pathways may help reduce relapse by limiting treatment resistance.