I’ve seen a lot of experimental economics papers as a coeditor of the Journal of Public Economics and a frequent reviewer for many journals. There are some recurring design and analysis decisions that lead authors astray. I’ll discuss a series of them. The first is Not Randomizing Treatment. It’s more common than you might think!

Not randomizing treatment. Randomly assigning participants to treatment is one of the key benefits of lab-based economics experiments. When we want to test the effect of a treatment, we want treatment to be orthogonal to everything else. It’s pretty clear how to do this with participant-level randomization—a random number generator assigns each participant to a treatment.

Things often go awry when we move to group-level randomization. For instance, you want to run a public goods game under one set of rules (“Basic Rules”), and then see how contributions change when it is run under another set of rules (“Enhanced Rules”).

Ideally, for each session, you would randomly assign half of the participants who show up to play Basic Rules (and interact with other participants in the Basic Rules condition), and half to play Enhanced Rules (and interact with other participants in the Enhanced Rules condition). This is great, you’ve actually achieved participant-level randomization.

But it’s logistically complicated to have different rules going on simultaneously (and perhaps the lab cannot handle enough people). So instead, you do session-level randomization. You run a “large-enough” number of sessions. You create a random order of sessions, so you might run Basic-Enhanced-Basic-Basic-Enhanced etc. If the number of sessions is large enough, you cluster standard errors at the session level and proceed.

But running different sessions with different rules in a random order is complicated. Plus, you might get an idea about what rules to run after running a few sessions. So what you do is run a few Basic Rules sessions, and then run a few Enhanced Rules sessions. This is where randomization has failed. Your subject population could be changing over time (perhaps early subjects are more eager, or have lower value of time). Or news events could change beliefs and preferences. The list of potential stories can be long; some can be ruled out, others cannot. But because your session order is not random, you are not guaranteed to have your treatment be orthogonal to everything else. As a result, you’ve missed out on the benefit of randomized experiments, and it’s unclear what to conclude from comparing your treatments.