Nucleic Acids Research · 2025
Pipeline Olympics
Continuable benchmarking of computational workflows for DNA methylation sequencing data against an experimental gold standard.
We compared ten complete workflows that turn bisulfite and enzymatic methylation sequencing reads into DNA methylation calls. We ran them on data from five library protocols and scored them against highly accurate locus-specific measurements. The benchmark stays open: the results can be explored and re-ranked interactively, and new workflows can be run on the same data.
Lin Y-Y, Breuer K, Weichenhan D, et al. Nucleic Acids Res. 2025;53(19):gkaf970. doi:10.1093/nar/gkaf970 · open access (CC BY)
The study
DNA methylation data are usually processed in four steps: read trimming, conversion-aware alignment, post-alignment filtering (such as PCR duplicate removal) and methylation calling. Many tools exist for each step, but complete workflows had never been compared on real data with a known answer. Pipeline Olympics fills that gap.
Samples with a known answer
Two pairs of colon cancer and adjacent normal tissue (patients 5 and 6) from the BLUEPRINT benchmark of targeted methylation assays. In that study, 16 assays in multiple labs established consensus corridors for the true methylation levels at selected loci. We use these as the gold standard.
Five library protocols
Each sample was sequenced with standard WGBS, three low-input protocols (Swift Accel-NGS, tagmentation-based T-WGBS and PBAT) and the bisulfite-free enzymatic EM-seq. That gave 300 to 986 million read pairs per sample.
Reproducible execution
All workflows were implemented in the Common Workflow Language with containers, and run in the same controlled environment. Methylation calls were aggregated with methrix.
| Metric | What it measures |
|---|---|
| Coverage | Area under the curve of CpGs retained versus read-depth threshold |
| Genome-wide deviation | Distance from a data-driven consensus corridor across all CpGs (WGBS, Swift, EM-seq) |
| Gold-standard deviation | Mean absolute deviation from the experimental consensus at the 46 BLUEPRINT loci |
| Differential methylation | Recovery of tumor–normal differences found with Infinium 450k arrays (AUC and correlation) |
| Run time | Wall-clock time with all resources of a compute node |
| Peak memory | Maximum memory usage |
Key findings
- BSBolt, methylCtools and bwa-meth perform best. BSBolt ranked first, and methylCtools placed second in three protocols. All three top workflows use three-letter alignment with BWA as the engine.
- The workflow matters most for low-input data. On deep WGBS, Swift and EM-seq data, most workflows agree closely. On PBAT and T-WGBS, low coverage magnifies the differences, and the workflow had a significant effect on accuracy at the gold-standard loci.
- PBAT is the hardest case. Workflows lost 20–77% of PBAT reads during alignment, partly because of chimeric reads. Biscuit, BSBolt and methylCtools handled it best. Biscuit varied the most across protocols overall, but did very well on PBAT, the protocol closest to single-cell methods.
- Fast workflows are not the most accurate. Run time ranged from about 4 hours (gemBS) to 14 days (GSNAP), and peak memory from 16 GB (Bismark) to 319 GB (GSNAP). gemBS, FAME, Biscuit and Bismark were the most efficient, but tended to score lower on accuracy.
- A benchmark can find bugs. Inspecting outliers at gold-standard loci showed that methylpy fails to restore some in-silico converted bases before calling, so it underestimates methylation.
- Good documentation does not predict performance. Biscuit, methylpy and Bismark scored highest for installation and documentation quality. This score was unrelated to the benchmark results.
The overall ranking weights all metrics equally. Your priorities may differ, so the results portal lets you change the weights and re-rank the workflows for your protocol.
Benchmarked workflows
The CWL implementation of each workflow is in CWL/workflows on GitHub.
Resources
Results portal
An interactive Shiny app with coverage and methylation statistics, the 46 gold-standard loci, and a ranking table you can customize.
Open the results portal →Benchmarking service
A public workflUX server on the de.NBI cloud. It runs the benchmarked workflows on downscaled versions of the study data, with ready-made parameter sheets.
How to use it →Code and workflows
CWL workflows with container details, and the R analysis scripts. Archived on Zenodo.
GitHub · ZenodoRun your own instance
Setup bundles install the benchmarking service or the results portal on your own Ubuntu 24.04 server. One script per service, with HTTPS and hardening included.
Download from GitHub releases · detailsSequencing data
Raw reads for all samples and protocols, under controlled access.
EGA: EGAS50000000541How to cite
Lin Y-Y, Breuer K, Weichenhan D, et al. Pipeline Olympics: continuable benchmarking of computational workflows for DNA methylation sequencing data against an experimental gold standard. Nucleic Acids Research 2025;53(19):gkaf970. https://doi.org/10.1093/nar/gkaf970
All authors
BibTeX
@article{lin2025pipelineolympics,
author = {Lin, Yu-Yu and Breuer, Kersten and Weichenhan, Dieter and others},
title = {Pipeline {Olympics}: continuable benchmarking of computational
workflows for {DNA} methylation sequencing data against an
experimental gold standard},
journal = {Nucleic Acids Research},
year = {2025},
volume = {53},
number = {19},
pages = {gkaf970},
doi = {10.1093/nar/gkaf970}
}