Skip to content

feat: implement somatic SNV/indel benchmarking pipeline - #1

Open
apsteinberg wants to merge 13 commits into
devfrom
benchmark
Open

feat: implement somatic SNV/indel benchmarking pipeline#1
apsteinberg wants to merge 13 commits into
devfrom
benchmark

Conversation

@apsteinberg

@apsteinberg apsteinberg commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

Implements the core benchmarking workflow for evaluating somatic SNV/indel callers against constructed truth sets.

Pipeline steps

  1. Preprocess — bcftools view to filter PASS variants and restrict to specified contigs
  2. Normalize — bcftools norm to split multiallelics and left-align against reference
  3. Build truth — Union (concat + dedup + sort), intersect, or single-caller strategies
  4. Benchmark — bcftools isec to classify variants as TP/FP/FN
  5. Stratify — bedtools intersect against user-provided BED regions
  6. Aggregate — Per-variant-type summary tables with precision/recall/F1

Key additions

  • stratify_variants — BED stratification of TP/FP/FN VCFs
  • aggregate_results — Cross-sample summary statistics (pandas)
  • stratify_variants.py + aggregate_results.py — Pipeline scripts
  • Installed nf-core modules: bcftools/{view,norm,concat,sort,isec}
  • Input schema: sample, variant (snv/indel), caller, category (query/truth), vcf
  • Params: --stratification, --contigs, --snv_truth, --indel_truth, --fasta
  • Test profile with sarscov2 data from nf-core/test-datasets
  • Containers built with wave, hosted on quay.io/shahlab_singularity

PR checklist

  • This comment contains a description of changes (with reason).
  • If you've fixed a bug or added code that should be tested, add tests!
  • If you've added a new tool - have you followed the pipeline conventions in the contribution docs
  • Make sure your code lints (nf-core pipelines lint).
  • Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
  • Check for unexpected warnings in debug mode (nextflow run . -profile debug,test,docker --outdir <OUTDIR>).
  • Usage Documentation in docs/usage.md is updated.
  • Output Documentation in docs/output.md is updated.
  • CHANGELOG.md is updated.
  • README.md is updated (including new tool citations and authors/contributors).

@github-actions

github-actions Bot commented Jun 9, 2026 •
edited
Loading

Copy link
Copy Markdown

nf-core pipelines lint overall result: Passed ✅ ⚠️

Posted for pipeline commit 5841232

+| ✅ 186 tests passed       |+
#| ❔  19 tests were ignored |#
!| ❗  23 tests had warnings |!
Details

❗ Test warnings:

  • readme - README contains the placeholder zenodo.XXXXXXX. This should be replaced with the zenodo doi (after the first release).
  • pipeline_todos - TODO string in README.md: TODO nf-core:
  • pipeline_todos - TODO string in README.md: Include a figure that guides the user through the major workflow steps. Many nf-core
  • pipeline_todos - TODO string in README.md: Fill in short bullet-pointed list of the default steps in the pipeline 2. Present QC for raw reads (MultiQC)
  • pipeline_todos - TODO string in README.md: Describe the minimum required steps to execute the pipeline, e.g. how to prepare samplesheets.
  • pipeline_todos - TODO string in README.md: update the following command to include all required parameters for a minimal example
  • pipeline_todos - TODO string in README.md: If applicable, make list of people who have also contributed
  • pipeline_todos - TODO string in README.md: Add citation for pipeline after first release. Uncomment lines below and update Zenodo doi and badge at the top of this file.
  • pipeline_todos - TODO string in README.md: Add bibliography of tools and data used in your pipeline
  • pipeline_todos - TODO string in nextflow.config: Optionally, you can add a pipeline-specific nf-core config at https://github.com/nf-core/configs
  • pipeline_todos - TODO string in nextflow.config: Update the field with the details of the contributors to your pipeline. New with Nextflow version 24.10.0
  • pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
  • pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
  • pipeline_todos - TODO string in main.nf: Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!
  • pipeline_todos - TODO string in CONTRIBUTING.md: Add any pipeline specific contribution guidelines here, such as coding styles, procedures, checklists etc.
  • pipeline_todos - TODO string in usage.md: Add documentation about anything specific to running your pipeline. For general topics, please point to (and add to) the main nf-core website.
  • pipeline_todos - TODO string in output.md: Write this documentation describing your workflow's output
  • pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline
  • pipeline_todos - TODO string in nextflow.config: Specify any additional parameters here
  • pipeline_todos - TODO string in test_full.config: Specify the paths to your full test data ( on nf-core/test-datasets or directly in repositories, e.g. SRA)
  • pipeline_todos - TODO string in test_full.config: Give any required params for the test so that command line flags are not needed
  • pipeline_todos - TODO string in base.config: Check the defaults for all processes
  • pipeline_todos - TODO string in base.config: Customise requirements for specific processes.

❔ Tests ignored:

  • files_exist - File is ignored: CODE_OF_CONDUCT.md
  • files_exist - File is ignored: assets/nf-core-onemorevariant_logo_light.png
  • files_exist - File is ignored: docs/images/nf-core-onemorevariant_logo_light.png
  • files_exist - File is ignored: docs/images/nf-core-onemorevariant_logo_dark.png
  • files_exist - File is ignored: .github/ISSUE_TEMPLATE/config.yml
  • files_exist - File is ignored: .github/workflows/awstest.yml
  • files_exist - File is ignored: .github/workflows/awsfulltest.yml
  • nextflow_config - Config variable ignored: manifest.name
  • nextflow_config - Config variable ignored: manifest.homePage
  • nf_test_content - nf_test_content
  • files_unchanged - File ignored due to lint config: CODE_OF_CONDUCT.md
  • files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
  • files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
  • files_unchanged - File ignored due to lint config: assets/email_template.txt
  • files_unchanged - File ignored due to lint config: assets/nf-core-onemorevariant_logo_light.png
  • files_unchanged - File ignored due to lint config: docs/images/nf-core-onemorevariant_logo_light.png
  • files_unchanged - File ignored due to lint config: docs/images/nf-core-onemorevariant_logo_dark.png
  • files_unchanged - File ignored due to lint config: docs/README.md
  • actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/onemorevariant/onemorevariant/.github/workflows/awstest.yml

✅ Tests passed:

Run details

  • nf-core/tools version 4.0.2
  • Run at 2026-06-11 19:57:21

apsteinberg and others added 10 commits June 9, 2026 15:57
@apsteinberg
precommit linting
4049e2b
@apsteinberg
ignore nf-test content in linting
8f98cb3
@apsteinberg
allow for different sample names when concatenating truth vcfs
2127cd9
@apsteinberg
reheader vcfs from the same sample to fix downstream bcftools concat …
df5a1cc 
…issue
@apsteinberg
carry through tabix indices
c34376e
@apsteinberg
fix: BCFTOOLS_VIEW tuple arity for Nextflow 26 compatibility
f7a9aea
@apsteinberg
add additional stats to stratification output
517f552
@apsteinberg
remove truth_qual column (Illumina somatic VCFs have no QUAL)
0873ee5
@apsteinberg
feat: add consensus truth strategy (>=2/N agreement via bcftools isec)
d3056b0 
Adds a 'consensus' option for --snv_truth and --indel_truth that builds
the truth set from variants called by at least 2 of N input callers.
Uses bcftools isec -n+2 → concat → sort (all nf-core modules, no custom
scripts). For N=2, this degenerates to intersection.
@apsteinberg
linting
5841232
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@apsteinberg