Skip to main content

Conference Program

Subpage Hero

Loading

Testing concordance of pathogenicity predictions from orthogonal machine learning algorithms and signal to noise analysis

Clinical Genetics and Therapeutics
  • Primary Categories:
    • Basic Research
  • Secondary Categories:
    • Basic Research
Introduction:
Since the adoption of the ACMG/AMP guidelines (Richards et al., 2015), many methods and algorithms, using diverse and distinct types of data, have been developed to predict the pathogenicity and assess the functional effects of variants. While concordance between different models has traditionally been desired to build confidence in predictions, the lack of agreement between individually validated models that rely on different types of data can tell us important information about specific genes and genetic variants. Here, we compare the predictions of several machine learning models developed by a clinical laboratory to a recently created computational tool (DiscoVari) that is based on signal to noise (S:N) analysis in nine cardiomyopathy- and channelopathy-related genes. 

Methods:
DiscoVari predicts disease-associated genetic hotspots by comparing the ratio of the frequency of disease-causing variants (the signal, S) for a given amino acid position to the frequency of rare non-disease causing variants (the noise, N) from gnomAD at the same position. We tabulated the DiscoVari S:N score for all amino acid positions in one arrhythmogenic cardiomyopathy (ACM) related gene (DSC2), one catecholaminergic polymorphic ventricular tachycardia (CPVT) related gene (RYR2), three long-QT syndrome (LQTS) related genes (KCNQ1, KCNH2, SCN5A), and four hypertrophic cardiomyopathy (HCM) related genes (ACTC1, MYBPC3, MYH7, TNNT2) and determined whether the score exceeded the gene-specific threshold for pathogenicity. We compared these predictions to the predictions made by five different clinically-validated machine learning (ML) models developed within our clinical genetic testing laboratory: one based on population frequency related data (PFM), one based on protein sequence and structure (GSE), one based on evolutionary conservation (gwEVE), one based on evolutionary conservation and molecular stability (FIM), and one based on phenotypic data and penetrance (CVM). We assessed the agreement between methods by calculating Cohen’s Kappa, percent concordant predictions, percent positive agreement (PPA), and percent negative agreement (PNA). Cohen’s Kappa values of 0.2, 0.4, 0.6, and 0.8 correspond to fair, moderate, substantial, and almost perfect agreement. Statistical significance for Cohen’s Kappa was assessed using Z-tests with a Bonferroni-corrected alpha value of 0.05.

Results:
Cohen’s Kappa ranged from -0.046 to 0.748 (mean = 0.159, sd = 0.216), showing relatively low levels of agreement overall, however, the level of agreement varied significantly between genes and models. The highest level of agreement was seen for KCNQ1-FIM and the lowest level of agreement seen for ACTC1-gwEVE. Cohen’s Kappa was significantly higher than 0 for KCNH2-CVM, KCNQ1-CVM, KCNQ1-PFM, MYBPC-CVM, MYH7-GSE, RYR2-GSE. PPA varied greatly between models, ranging between 0.127 to 0.980 (mean = 0.554, sd = 0.297).  DiscoVari showed moderate-to-high PPA when compared to CVM and FIM, but low-to-moderate PPA when compared to GSE and gwEVE. Alternatively, DiscoVari showed moderate-to-high levels of PNA (range = 0.509-0.920, mean = 0.710, sd = 0.126), and percent concordance (range = 0.519-0.873, mean = 0.646, sd = 0.093) for all models. 

Conclusion:
Agreement between DiscoVari and other ML-based models was highly contingent on the gene and model, with low-to-moderate levels of agreement overall. As these models utilize different underlying data and methods, it is not surprising that agreement was relatively low. Importantly , as they are all independently validated, they provide unique information for each variant and gene, which can lead to better variant classifications and inform gene-disease properties. Comparing the pathogenicity predictions of these orthogonal methods highlights the utility of using multiple, independently validated computational methods for variant classification. Future work using these tools should focus on gaining further insight into the dynamics of predicting variant pathogenicity vs. penetrance. 

Agenda

Sponsors