What Is Reproducible Research?

Learn what makes research computationally reproducible and how reproducibility, replicability and robustness differ in one worked example.

LanguageEnglish

Start with one result

Computational reproducibility means recovering a reported result using the same data and analysis steps. Start with one result. A colleague reports that sessions in a quiet room averaged 25 minutes. If you can find the two quiet-room records and repeat the calculation, you can check that result.

The quiet-room records are 20 and 30 minutes, which average 25 minutes. The background-noise records are 15 and 25 minutes, which average 20 minutes.

The records here are fictional. The five-minute difference illustrates the calculation. It provides no basis for a conclusion about the effect of noise on studying.

Reproducibility vs. replicability

Fields use these terms differently. Here is the convention used in this series.

QuestionWhat to doStudy-session example
ReproducibilityUse the same data and analysis.Recalculate the two means and check whether you get 25 and 20.
ReplicabilityCollect new data under a comparable design.Run a new study and check whether it supports the original finding.
RobustnessTry another reasonable analysis on the same data.Check whether a documented exclusion rule or different summary changes the interpretation.

Make a result easy to trace

Put yourself in your colleague’s shoes. They need the input file and an explanation of its columns, the exact commands or steps you ran, the relevant software versions, and an expected result to compare against. If a result depends on a random draw, record the seed and how it was used.

For restricted data, document who can request access and which steps can be tried with synthetic data. Keep sensitive records out of public repositories.

Try it on one result

Pick a table or figure from your work. Write down the input file, the analysis step, and the expected output in one sentence each. Ask a colleague to point to the exact input they would open.

See a worked answer

Input: data/sessions.csv. Step: group rows by condition and take the mean of minutes. Expected output: background-noise,20.0 and quiet,25.0 in the result table.

References

Adapted from a chapter by The Turing Way Community (CC BY 4.0). Miss Formula reorganized and expanded the explanations and wrote the worked example and exercises.