Replication seems a sensible way to assess whether a scientific result is right. The intuition is clear: if a result is right, you should get a similar answer when repeating the work; if it it’s wrong, your answer should be quite different. Statisticians have devised numerous statistical tests for deciding whether a replication passes or fails thus validating of refuting the original result. I simulate many of these tests across a range of conditions. For exact replications, simple significance testing works fine as a validation test, but when replicas diverge from the original studies no test works well. Much of the replication literature focuses, perhaps unwittingly, on methods for testing whether the studies are similar; these tests work quite poorly under all conditions analyzed here. Many caveats apply, but if correct, my results bode ill for large, systematic replication efforts, which typically prioritize uniformity over fidelity to run lots of studies at reasonable cost.

## Introduction

The basic replication rationale goes something like this: (1) many published papers are wrong; (2) this is a serious problem the community must fix; and (3) systematic replication is an effective solution. (In recent months, I’ve seen an uptick in pre-registration as another solution. That’s a topic for another day.) In this post, I focus on the third point and ask: viewed as a statistical test, how well does systematic replication work; how well does it tell the difference between valid and invalid results?

By “systematic replication” I mean projects like Many Lab, Reproducibility Project: Psychology (RPP), Experimental Economics Replication Project (EERP), and Social Sciences Replication Project (SSRP) that systematically select studies in a particular field and repeat them in a uniform fashion. The main publications for these projects are Many Lab, RPP, EERP, SSRP.

I consider a basic replication scheme in which each original study is repeated once. This is like RPP and EERP, but unlike Many Lab as published which repeated each study 36 times and SSRP which used a two-stage replication strategy. I imagine that the replicators are trying to closely match the original study (direct replication) while doing the replications in a uniform fashion for cost and logistical reasons.

Various authors have proposed tests for deciding whether a replication succeeds or fails. These include significance testing of the replica study, seeing whether the observed effect size of one study falls in the confidence or prediction interval of the other, checking whether the confidence or prediction intervals of the two studies overlap, significance testing of the studies combined via meta-analysis, and comparing the observed effect size of the replica study with a “small telescope” threshold derived from the smallest true effect size the original study could have plausibly detected. These tests answer different questions, but each is used by one or more authors to accept or reject replications.

I simulate these rules (and more) across many replication conditions and compute false positive and false negative rates across a range of conditions. The simulations vary the sample size from 10 to 1000 and true effect size from 0 to 1; in total, I simulate 14,641 conditions each with $$10^4$$ simulated replications, for a grand total of almost 150 million simulated instances.

A replication is exact if the two studies are sampling the same population and near-exact if the populations differ slightly. This is an obvious replication scenario. You have a study you think may be a false positive; to check it out, you repeat the study, typically with a larger sample size, taking care to ensure that the replica closely matches the original.

A replication is inexact if the two studies are sampling very different populations. This scenario seems disconnected from the goal of validating an existing study. Since the populations are different, there’s no reason to expect the studies to get similar answers and little basis for declaring a study invalid if the replication fails. What motivates this scenario, I think, is generalizability not validity. You have a study that demonstrates an interesting effect in a limited setting and want to know whether it generalizes to other settings. Some authors call this conceptual replication; in meta-analysis, people use the term study heterogeneity for situations in which the study populations are very different.

Significance testing of the replica works fine as a validation test for exact and near-exact replications, but error rates increase rapidly as the populations diverge. All other tests have excessive error rates under all conditions I analyzed. All tests have unacceptable error rates when used to check whether the original and replica studies are similar.

Many caveats apply. Perhaps my correctness criteria are wrong or error thresholds too low. Maybe I’m testing the methods on unrealistic conditions. I assume throughout that all population effect sizes are equally likely within the range studied; in particular, I don’t consider publication bias or the possibility that many results are expected to be false a priori. The ever-present danger of software bugs also lurks.

If correct, my results suggest that systematic replication projects should change goals or methods. Replication only works as a validation test when replicas closely match the original studies. This is bad news for systematic replication efforts, which typically prioritize uniformity over fidelity to run lots of studies at reasonable cost. An alternative is to switch gears, give up on validation, and focus on generalizability.

The software supporting this post is open source in my repwr GitHub repository.

## Literature

Here are the main papers and blog posts that led to this work.

David Colquhoun reminded me that it’s important to take into account the prior probability that studies are true, a topic I explored in a previous post. Relevant papers by Colquhoun include his recent proposal to supplement p-values with false positive risk and 2014 and 2017 papers discussing why p-values don’t tell you the probability that a result is a false positive.

Please see my incipient website for related blog posts and working papers including a detailed description of the replication tests.

## The Simulation

The software first simulates studies across a range of conditions, then combines pairs of studies into pairwise replications, applies rules (called measures) for deciding which pairwise replications pass, summarizes the results as counts and pass rates, and finally computes true and false positive and negative rates for measures and conditions of interest.

The studies are simple two group comparisons parameterized by sample size $$n$$ and population effect size $$d_{pop}$$ ($$d_{pop}\ge0$$). For each study, I generate two groups of random numbers, each of size $$n$$. One group, group0, comes from a standard normal distribution with $$mean=0$$; the other, group1, is standard normal with $$mean=d_{pop}$$. I then calculate basic statistics of interest, most notably the standardized observed effect size, $$d_{sdz}$$, aka Cohen’s d, as the mean of group1 minus the mean of group0 divided by the pooled standard deviation of the two groups. The software simulates many studies (default $$10^4$$) for each combination of $$n$$ and $$d_{pop}$$.

When I need to be pedantic, I use the terms study instance for each individual study and study set for the ensemble of study instances for a given combination of $$n$$ and $$d_{pop}$$.

The program varies $$n$$ from 10 to 1000 and $$d_{pop}$$ from 0 to 1. When analyzing results, I interpolate to get values that weren’t simulated directly.

To generate pairwise replications, I consider all (ordered) pairs of study sets. For each pair, the software permutes the instances of each study, then combines the instances row-by-row to get pairwise replication instances. It’s often convenient to think of the first study of the pair as the original and the second as the replica.

A pairwise replication set is the ensemble of pairwise replication instances for a given pair of study sets. Four variables parameterize each pairwise replication set: $$n1$$, $$n2$$, $$d1_{pop}$$, $$d2_{pop}$$. These are, naturally enough, the sample and population effect sizes for the two study sets. In meta-analysis, the situation where $$d1_{pop} \ne d2_{pop}$$ is called study heterogeneity.

After forming the pairwise replications, I apply the measures, i.e., rules for deciding which replications pass. Each measure takes a pairwise replication set as input and returns a vector of boolean values telling which instances pass or fail. The result is a boolean matrix whose rows represent instances and columns represent measures.

This post focuses on eight measures that seem most important. These measures answer different questions but all appear in papers or posts as tests to accept or reject replications.

• sig2: the second study of the pair has a significant p-value
• sigm: the fixed effect meta-analysis of the studies has a significant p-value
• d1.c2 (resp. d2.c1): $$d1_{sdz}$$ (resp. $$d2_{sdz}$$), the standardized observed effect size (aka Cohen’s d) of the first (resp. second) study is in the confidence interval of the second (resp. first) study; my implementation of confidence intervals is based on Uri Simonsohn’s code supporting this post
• d1.p2 (resp. d2.p1): $$d1_{sdz}$$ (resp. $$d2_{sdz}$$) is in the prediction interval of the second (resp. first) study; my implementation of prediction intervals adapts code from David Stanley’s predictionInterval package and Spence and Stanley’s paper
• c1.c2 (resp. p1.p2): the confidence (resp. prediction) intervals of the two studies overlap

All measures assume that the first study is significant (sig1 in my notation) and the observed effect sizes of the two studies have the same sign (both positive or both negative).

I briefly examine Uri Simonsohn’s small telescopes method (d2.scp1 in my notation) from his paper and post. My implementation is based on Uri’s code supporting that post. Small telescopes, unlike the others, assumes that sig2 holds. For this reason, it needs a separate analysis.

The software summarizes the results by counting the number of positive results for each measure, taking into account the assumptions in the preceding paragraphs, and then converts the counts into pass rates. The final step is to convert pass rates into true positive, false positive, true negative, and false negative rates. This requires a definition of true and false instances, which in turn requires an explicit statement as to which replication instances represent replications that should succeed vs. ones that should fail.

## Correctness Criteria

I found no concise, rigorous definition of replication correctness anywhere. Many authors rely on some variant of “A replication should succeed if my method says so”. It’s impossible to define error rates with such circular definitions.

Replication researchers study two aspects of correctness.

1. The most basic concern is that the original study is a false positive.
2. The other concern is that the observed effect sizes of the two studies are inconsistent with each other. This might mean that the population effect sizes are different or that one result is an outlier.

The first concern seems sensible (at least, as sensible as any other use of the null hypothesis testing framework). The second concern baffles me: if the two studies are sampling different populations or one is an outlier, why does this invalidate the first study?

I’m tempted to ignore the second concern, but most proposed replication methods address it. Sadly, I’m stuck with it for purposes of this post.

Here are the precise technical definitions.

1. non-zero: a replication instance is true if $$d1_{pop}\ne0$$, i.e., the population effect size of the first study is non-zero
2. same-effect: a replication instance is true if $$d1_{pop}=d2_{pop}$$, i.e., both studies have the same population effect size; with tolerance $$\epsilon$$, a replication instance is true if $$abs(d1_{pop}-d2_{pop})\le\epsilon$$, i.e., the two population effect sizes differ by at most $$\epsilon$$

Note that these criteria depend only on the population effect sizes.

## Nomenclature

I’ve already introduced most of my nomenclature. Here is a concise reprise.

Studies

• study instance is a single simulated study
• $$n$$ is the sample size
• $$d_{pop}$$ is the population effect size
• $$d_{sdz}$$ is the standardized observed effect size (aka Cohen’s d)
• study set is the set of study instances for a given $$n$$ and $$d_{pop}$$

Pairwise replications

• pairwise replication instance (or simply replication instance) is an ordered pair of study instances
• s1 and s2 refer to the first and second study of the pair
• $$n1$$, $$d1_{pop}$$, $$d1_{sdz}$$ are the sample size, population effect size, and Cohen’s d of the first study
• $$n2$$, $$d2_{pop}$$, $$d2_{sdz}$$ are the sample size, population effect size, and Cohen’s d of the second study
• pairwise replication set (or simply replication set) is the set of replication instances for a given $$n1$$, $$n2$$, $$d1_{pop}$$, and $$d2_{pop}$$
• a replication instance or set is exact if $$d1_{pop}=d2_{pop}$$
• true and false refer to the answers given by correctness criteria when applied to replication instances
• positive and negative refer to the results of applying measures to replication instances
• true positive, false positive, true negative, and false negative combine the notions of true vs. false and positive vs. negative instance. For a given correctness criterion and measure
• true positive instance is a replication instance for which the correctness criterion and measure both return true
• false positive instance is an instance for which the correctness criterion is false, but the measure returns true
• true negative instance is an instance for which the correctness criterion and measure both return false
• false negative instance is an instance for which the correctness criterion is true, but the measure returns false

Correctness criteria

• non-zero is true if $$d1_{pop}=0$$
• same-effect is true if $$d1_{pop}=d2_{pop}$$; with tolerance $$\epsilon$$, same-effect is true if $$abs(d1_{pop}-d2_{pop})\le\epsilon$$
• for non-zero
• false positive occurs when $$d1_{pop}=0$$, but the measure returns true
• false negative occurs when $$d1_{pop}\neq0$$, but the measure returns false
• for same-effect with tolerance $$\epsilon$$
• false positive occurs when $$abs(d1_{pop}-d2_{pop})>\epsilon$$, but the measure returns true
• false negative occurs when $$abs(d1_{pop}-d2_{pop})\le\epsilon$$, but the measure returns false

To simplify the presentation

• s1 (the first study of the replication instance) represents the original study and s2 (the second study) represents the replica
• error rates are relative to sig1; e.g., an error rate of 5% means 5% of the replications satisfying sig1 exhibit the error

## Graph Types

The parameter space is vast, because it’s possible to vary each of the four parameters ($$n1$$, $$n2$$, $$d1_{pop}$$, $$d2_{pop}$$) across a considerable range. I use several kinds of graphs to present the data in a comprehensive, yet concise, manner.

1. line graphs: simple and intuitive, I think, but not good at showing data that varies across many parameters
2. heatmaps: still reasonably intuitive and somewhat better at depicting multiple parameters
3. rate-vs-rate scatter plots: able to display error rates across large swaths of parameter space but with less parameter resolution and perhaps less intuitive clarity; inspired by receiver operating characteristic (ROC) curves
4. aggregate line graphs: same data as rate-vs-rate scatter plots but for fewer measures and with better parameter resolution

## Results

### Replication used for validation

When replication is used to validate a study, non-zero is the correctness criterion that matters. I first present results for exact replications, then inexact, and finally near-exact.

#### Exact replications

Figures 1-4 show false positive rates for two values of $$n1$$ (20 and 40) and a range of values for $$n2$$. The smallest $$n2$$ in each case satisfies the recommendation in Uri Simonsohn’s post on the 90x75x50 heuristic that $$n2=2.5 \times n1$$ is big enough. The x-axis shows all four parameters using $$d1$$, $$d2$$ as shorthand for $$d1_{pop}$$, $$d2_{pop}$$ to conserve space. $$d1_{pop}=d2_{pop}=0$$ throughout because this is the only way to get false positives for non-zero with exact replications.

The first two figures are line graphs with false positive rate along the y-axis; these graphs use color and line width to distinguish the measures. The latter two are heatmaps with measures along the y-axis; these plots use color (shades of red and blue) to depict false positive rate; the switch from red to blue is set at the conventionally accepted threshold of 0.05 for false positives.