Andrew Gelman’s post over at The Monkey Cage, in which he treats an argument about how the threat of meta-analysis should induce more disciplined empirical work in forward-thinking scholars, got me thinking more about the importance of replicability…but in the context of theory rather than empirical work.
Specifically, on that first day of an intro game theory class for grad students, you find yourself explaining to what might be a skeptical crowd the value of both modeling strategic interaction (a rather easy sell) and doing it formally (a tougher sell). The two are, of course, very different ideas—and they’re all too often conflated—but on the latter point, there’s a good argument to be made here that has nothing to do with game theory or assumptions of rationality but everything to do with replicability.
In short, formalizing one’s argument—apart from making it easier to get the logic right—is also a good way (and, in fairness not the only way) to make sure that said argument is replicable. Sentences and words can be sloppy; equations and operators are precise. And by virtue of that precision, tracing an author’s logic becomes easy when s/he provides proofs of how the conclusion was reached. These proofs can be mathematical (that’s the way I happen to do it), but if we’re just trying to prove the validity of an argument, then it can be done syllogistically or in whatever mode one likes. When one formalizes an argument, though, we can flip to the back of the article (or, sometimes, read just beyond the proposition) and trace exactly the logical path to verify that, yes, the hypotheses really do follow from the premises. That’s powerful stuff. Yes, one might contend that there are some nontrivial startup costs to being able to reproduce and verify the logic inside someone else’s formal proofs, but that’s also the case when it comes to advanced statistical analyses that we’d like to replicate as well. So I’m not terribly sympathetic to that objection.
Ultimately, formalizing our arguments allows us to create very clear replication files for our theories, rendering them transparent, reproducible, extendable, and—this is key—open to a greater degree of scrutiny. When our logic (whether good, bad, or absent) can’t hide behind verbiage, we’re better off as a discipline. We can scrutinize, correct, refine, refute, and improve in a way that we can’t when readers have to work too hard to back out the logic of our argumentation. Again, this doesn’t have to be formal, but formality does make complex logical structures with lots of moving parts easier to handle. (Hell, I need that mathematical crutch when the moving parts become too many, and I’m happy to admit it.)
Think of it this way: we’d be justifiably skeptical of empirical work that didn’t provide replication materials, and I’d argue that we should be equally skeptical of work that obfuscates its logic—intentionally or not—by not providing the reader some kind of transparent recipe for tracing their path from premises to conclusion. Yes, if you provide the details of your logic, you’re perhaps more likely to be firmly refuted, but—like the scholars addressed in Gelman’s post—that’s all the more reason to make sure you get the logic right the first time around.
The Wolf Den 