A Replicable Protocol for Benchmarking Wastewater-Based Genomic Surveillance Methods

Overview

Wastewater-Based Genomic Surveillance (WBGS) is an emerging technology that has demonstrated tremendous potential in combating pandemics by enabling rapid detection of infections through testing for pathogens in the sewage system. Low overhead costs make it an attractive alternative to individual testing. However, the lack of a standardized approach to the benchmarking of computational WBGS technologies prevents a scalable program. This paper presents a set of potential benchmarking norms and strategies to simulate real-world conditions most accurately during in silico analysis. The paper focuses on CliqueSNV, a haplotype reconstruction tool, as a case study. Different alignment methods do not impact CliqueSNV’s performance; with each alignment method, CliqueSNV found a similar number of haplotypes with similar characteristics. However, the different alignment tools took vastly different times to run; the Burrows-Wheeler Aligner (BWA) was by far the fastest. Additionally, the position of short reads from similar variants greatly impacted results. When the reads generated from the SARS-CoV-2 Alpha sequence were grouped together at the beginning of the simulated metagenome, CliqueSNV would only be able to reconstruct a haplotype equivalent to Alpha. However, when the ordering was random, as would be encountered in real-world sequencing, CliqueSNV had difficulty finding any variants. The reproducible protocol presented in this paper and the opportunities to eliminate order bias in WBGS technologies can be used to both benchmark and improve the accuracy of other wastewater-based genomic surveillance technologies; if other technologies could overcome the order bias of CliqueSNV, they could be more suited for large-scale deployment.