This repository attempts to assemble and categorize a body of emerging literature that leverages computational tools, big data sources, and machine learning to evaluate the replicability and reproducibility of research results across sciences. The focus is on novel works that develop scalable, rapid, and updating approaches to evaluation of research results. Although an evaluation system that is simultaneosuly scalable, rapid, and constantly updating is not (yet) in place, each literature source adds a piece from the puzzle.
Scalable: can simultaneously evaluate numerous published studies, hypotheses, results, and claims against purpose-build (e.g., prediction markets) or repurposed (e.g., high-throughput experiments) verification data.
Rapid: can efficiently screen research publications to promptly uncover false positive results, possibly before they propagate in the literature.
Updating: can incorporate next results in a continuously updating manner.
The distinction between readymade and custommade data Salganik made in Bit By Bit: Social Research in the Digital Age is useful here. We extend Salganik's distinction to tentatively categorize automated approaches to research replicability and reproducibility into four classes:
- Reuse (e.g., aggregation of published results aka meta-analysis or scalable computational reproducibility)
- Repurpose (e.g., use of Genome-wide association studies and high-throughout experiments to evaluate published results)
- Crowdsource (e.g., use of prediction markets to aggregate individual beliefs)
- Simulate (e.g., computer simulations)
- Reproducibility in Cancer Biology: Making sense of replications
- What does research reproducibility mean?
- DARPA Wants to Build a BS Detector for Science
- “PICkLE” – Path to Iterative Confidence Level
- Reproducibility of computational workflows is automated using continuous analysis
- An empirical analysis of journal policy effectiveness for computational reproducibility
- Centralized “big science” communities more likely generate non-replicable results
- Large-scale Efforts to Massively Replicate Reported Candidate-gene Associations (Table 1)
- Repeatability of published microarray gene expression analyses
- Using prediction markets to estimate the reproducibility of scientific research
- Evaluating replicability of laboratory experiments in economics
- Reproducibility of preclinical animal research improves with heterogeneity of study samples
- Randomly auditing research labs could be an affordable way to improve research quality: A simulation study
- The natural selection of bad science
The idea of assembling literature sources on an emerging topic was inspired by pimentel, greenelab, and the Social Media Collective.
This project is licensed under the GNU General Public License v3.0—see the LICENSE.md file for details