Benchmarking service
Run the workflows on the study data
The Pipeline Olympics workflUX server runs the benchmarked workflows on downscaled versions of the study data: four samples, five protocols. Everything is set up in advance. Pick a parameter sheet, upload it and start the runs. No account is needed.
How to run a workflow
Each parameter sheet describes one complete job: one workflow and one protocol, with four runs, one per test sample. 5N and 6N are the adjacent normal samples of patients 5 and 6, and 5T and 6T are the tumors. The settings are the same as in the study.
-
Download a sheet
Choose the workflow and protocol in the table below and download the
.xlsxfile. -
Create the job
On the workflUX server, open Create Job and select the workflow named in the sheet's file name, which is the part before the protocol. For example,
bsbolt_singlelib_EMSEQ.xlsxbelongs tobsbolt_singlelib. -
Import the sheet
Enter a job name with letters, digits,
.,_or-only, and no spaces. Under Provide parameter using, choose Spreadsheet. Skip the export step: under 3. Import and create job, select the downloaded sheet and import it. -
Start the runs
Open Job Execution & Results, select your job and start its four runs. You can follow their progress there, stop them, and view and download the output files once they finish.
Please note: this is a shared public demo. Visitors don't log in, so the jobs you create are visible to all other visitors. Runs are queued, with at most two running at once, and stopped after 48 hours. Jobs are deleted after 14 days. Only the test data on the server can be used as input, and files cannot be uploaded.
Parameter sheets
One sheet per workflow and protocol. A dash means the study had no settings for that combination. For example, BAT does not support the non-directional PBAT protocol. All sheets are also in docs/workflux/sheets on GitHub.
| Workflow on the server | EM-seq | PBAT | Swift | WGBS | T-WGBS |
|---|---|---|---|---|---|
BAT_workflow | xlsx | – | xlsx | xlsx | xlsx |
BAT_workflow_start_with_trimmed | xlsx | – | xlsx | xlsx | xlsx |
biscuit_singlelib | xlsx | xlsx | xlsx | xlsx | – |
biscuit_singlelib_start_with_trimmed | xlsx | xlsx | xlsx | xlsx | – |
bsbolt_singlelib | xlsx | xlsx | xlsx | xlsx | – |
bsbolt_singlelib_start_with_trimmed | xlsx | xlsx | xlsx | xlsx | – |
bwameth_singlelib | xlsx | xlsx | xlsx | xlsx | – |
fame_multiple | xlsx | xlsx | xlsx | xlsx | xlsx |
gembs | xlsx | – | xlsx | xlsx | xlsx |
gsnap_singlelib | xlsx | xlsx | xlsx | xlsx | – |
methylCtools_singlelib | xlsx | xlsx | xlsx | xlsx | – |
methylCtools_singlelib_start_with_trimmed | xlsx | xlsx | xlsx | xlsx | – |
methylpy_singlelib_start_with_trimmed | xlsx | xlsx | xlsx | xlsx | – |
The *_start_with_trimmed workflows skip adapter trimming and expect trimmed reads. The study has no separate settings for them, so their sheets use the settings of the matching full workflow and the untrimmed test reads. For meaningful results, replace the reads with trimmed FASTQ files.
Changing the settings in a sheet
Open the sheet in Excel or LibreOffice, edit it, save it as .xlsx and import it as above.
| Tab | Contents |
|---|---|
global single values | Settings shared by all four runs (null = not set) |
run-specific single values | One row per run, for example the output name |
read1, read2 | The FASTQ files of each run, one row per run |
config, metadata | Used by workflUX, leave unchanged |
Input files must be on the server under /data/data. To use your own settings without a sheet, choose HTML form instead of Spreadsheet in step 3.
Benchmark your own workflow
Pipeline Olympics is meant to keep growing. Workflows are added to the server by its maintainers, so visitors cannot import workflows themselves. To add yours:
-
Describe it in CWL
Implement the workflow in the Common Workflow Language, with Docker containers for its tools. A simple single-command wrapper is enough. The workflows in
CWL/workflowsshow the expected inputs and outputs. -
Submit it
Open a pull request or an issue on GitHub with the CWL files and a job file for each protocol. We will add it to the server.
-
Compare the results
Run it on the test data like the other workflows, and compare its methylation calls with theirs. The analysis scripts compute the metrics used in the study.
Run the services on your own server
Both Pipeline Olympics services can be installed on your own premises, for example to benchmark workflows on private infrastructure or with more compute. Each setup bundle contains one script that installs and configures everything on a fresh Ubuntu 24.04 server, including HTTPS (Let's Encrypt, self-signed or your own certificate) and the hardening of the public instances.
Benchmarking service
workflUX with all importable workflows, their container images and the parameter sheets on this page. Needs about 1 TB of disk for the test data and results.
The test data are derived from controlled-access patient data, so they are not included. The bundle's README describes the required files and how to prepare them, starting from the raw reads in EGA study EGAS50000000541.
Results portal
The Shiny app with R, Bioconductor and Shiny Server. A small server is enough, and no data have to be prepared.