Using Permutation Inference to Strengthen Regression Discontinuity Analysis

Introduction

Regression Discontinuity (RD) designs are a powerful tool for causal inference, but traditional parametric approaches can be sensitive to model specification. Permutation inference offers a robust alternative that doesn’t rely on distributional assumptions and can provide more credible statistical inference.

What is Permutation Inference?

Permutation inference tests the null hypothesis by randomly reassigning treatment status and comparing the observed effect to the distribution of effects under random assignment. In RD contexts, this typically involves:

  1. Randomly reassigning the treatment cutoff
  2. Calculating the treatment effect at each permuted cutoff
  3. Comparing the true effect to this null distribution

A Simple Example

Let’s consider a scholarship program where students with test scores above 75 receive financial aid.

Sample Data

Here’s what our data might look like:

Student Test Score Received Aid Outcome
1 73.2 No 2.8
2 76.1 Yes 3.4
3 74.8 No 2.9
4 77.3 Yes 3.6
5 72.5 No 2.7

Note: This is simulated data for illustration purposes.

Implementation Steps

Step 1: Calculate the True Effect

Observed effect = Mean(Outcome | Score ≥ 75) - Mean(Outcome | Score < 75)
                = 3.5 - 2.8 = 0.7

Step 2: Generate Permutation Distribution

  1. Randomly select 1000 alternative cutoff points
  2. For each cutoff, calculate the treatment effect
  3. Build a null distribution of these effects

Step 3: Calculate P-value

p-value = Proportion of permuted effects ≥ observed effect

Advantages of This Approach

  • Assumption-free: No need to specify functional form
  • Robust: Less sensitive to outliers and model misspecification
  • Intuitive: Clear interpretation of statistical significance
  • Flexible: Works with any test statistic

Key Considerations

Window Selection: Choose permutation range carefully - too narrow misses the point, too wide includes irrelevant comparisons.

The permutation window should typically:

  • Exclude the true cutoff region
  • Cover a reasonable range of the running variable
  • Maintain sufficient sample size for each permutation

Conclusion

Permutation inference provides a valuable complement to traditional RD analysis. While it requires more computational effort, the robust statistical properties and assumption-free framework make it particularly valuable when:

  • Sample sizes are modest
  • Functional form is uncertain
  • Traditional methods show sensitivity to specification

By combining traditional parametric estimates with permutation-based inference, researchers can build more convincing causal arguments.

For more details on implementation, see Cattaneo, Idrobo, and Titiunik (2020) or explore the rdrobust package in R.