Computational Reproducibility Audit of 501 Cochrane Meta-Analyses

Can Cochrane meta-analyses be computationally reproduced when an automated pipeline re-extracts effect sizes from source trial publications? We audited 501 Cochrane systematic reviews encompassing 14,340 individual studies using MetaReproducer, a deterministic pipeline that parses RevMan data files, retrieves open-access PDFs, extracts effects via RCT Extractor v10.3, and re-pools results using inverse-variance random-effects models. The pipeline infers effect type by back-computing candidate log-odds and log-risk ratios from two-by-two tables, matching against Cochrane reference values within a natural-scale relative tolerance of 0.001. Only 1,688 of 14,340 studies had accessible PDFs, yielding an open-access prevalence of 11.8 percent (95% CI 11.3-12.3), leaving most evidence computationally unverifiable. Among six reviews with sufficient coverage for classification, two showed major discrepancies including one complete direction change. The primary barrier to reproducibility is infrastructural access rather than methodology, suggesting that mandating structured data deposition could transform verification. This fundamental limitation of open-access coverage constrains any automated reproducibility audit of the published evidence base.

Live dashboard: https://mahmood726-cyber.github.io/metareproducer/

Run

Open index.html (landing page) or dashboard/index.html (the interactive dashboard) in any modern browser. No build step.

Pipeline scripts auto-discover external inputs from common workspace locations. Use environment overrides when needed:

• METAREPRODUCER_RDA_DIR
• METAREPRODUCER_RCT_EXTRACTOR_DIR
• METAREPRODUCER_MEGA_DIR
• METAREPRODUCER_CTGOV_ENV

For local development:

python -m http.server 8000
# then open http://localhost:8000/

Test

python -m pytest -q

The suite under tests/ covers the pipeline, dashboard, and portability contracts.

Repo layout

Path	Purpose
dashboard/index.html	the interactive dashboard (main artifact)
index.html	landing page
tests/	pytest tests
e156-submission/	E156 micro-paper bundle
E156-PROTOCOL.md	project metadata (E156 entry #106)

License

See LICENSE (MIT).