When theories meet critiques…and how to handle them

Note. This is aimed, for the most part, at game theory students, but it’s important to note that this is important for theories of all stripes, whether formal or verbal. So, whatever your inclinations for developing explanations, read on.

Theories, in their basic form, consist of assumptions (premises), some logic, and implications (hypotheses, conclusions, etc.), and there are any number of ways to critique them, but today we’re going to set aside the question of the logic of theories and assume that you’ve got a logically consistent, valid argument. (How you get this is another story for another time.) But, assuming that the logic is right, one thing that any scholar will run into when others see their theory, whether in a paper or at a conference, is the question of what gets left out of the model. Granted, given the infinitude of things that could be in any model, the correct answer to “what have you left out?” is, strictly speaking, “nearly everything.” But very often, folks will ask, “But what about factor x? Shouldn’t that also affect the outcome variable? And, if so, why is it not in your model?” Sometimes, that’s a useful critique of your theory, and sometimes it’s not, and the key is identifying when it is and when it isn’t. Of course, as we’ll see, even when it’s not useful for the theory, it often turns out to be good for thinking about controls for testing its implications…but we’ll get there after the jump.

I suspect that a lot of these critiques come from thinking empirically about a question, considering potential control variables or other variables that might explain a particular outcome. There’s nothing wrong with that at all, but it’s almost like putting the cart before the horse, and it can produce theoretical critiques of differing use for the builder of a theory. I’m going to argue here that some of these questions imply that you think about adding to or relaxing some assumptions in your theory, while others are more important for testing its hypotheses, entering as controls we might add when it comes to recovering more or less accurate estimates of empirical relationships in the data.

So when does a question about a missing factor in a theory mean that the theory should be reconsidered, an assumption relaxed or a parameter introduced? I generally tell my grad students that they should worry about the theory when a proposed addition to their model would be a “sign-changer,” but not when it’s a “size-changer,” as that would imply worrying about proper control variables. What do I mean by that? Let’s say you’ve got a theory that applies to a wide range of cases empirically—say, states—but one can make a reasonable argument that the effect, whatever it is, should be greater in democracies than in autocracies. This is the perfect example of what I mean by a size-changer, because accounting for regime type in a regression here would lead to different effects across the regime types: democracies would have a bigger coefficient, for example, but the relationship would be positive for both sets of states. So while the sign of the coefficient wouldn’t change, the size would. But since the sign of the predicted effect is the same, it’s probably more useful—to the extent that you’re interested in differences across regime types—to control for that empirically rather than complicate your theoretical model with it, because the dynamic is only mediated by regime type, not fundamentally changed. In other words, you’d get the same comparative statics across both regime types in your theoretical model, so that complication wouldn’t be doing any theoretical lifting: it would just be a complication.

But let’s say that someone can tell a story that the coefficient would change sign, from positive to negative, if you accounted for regime type. For instance, let’s say that the relationship between your two factors, which your theory predicts to be positive, might be negative for democracies and positive for autocracies. If that’s the case, and your model predicts only the positive, then you’ve got a model of autocracies and not one of democracies, because your assumptions reflect only those conditions that make the relationship positive. (See how powerful—and therefore useful—assumptions are? “Assumption” is not a four-letter word (in fact it’s a ten-letter one). But more on that in a future post.) If you want a theory that applies to all states, you’ve got a problem, because the proposed omitted factor—regime type—turns out to be a sign-changer. You’d get a differently-signed coefficient if across both regime types, and you’d likely be unable to recover any meaningful empirical relationship if you didn’t control for it. In fact, you’d be right to decide that controlling for it empirically isn’t enough, unless you do want to restrict yourself to autocracies (provided you can tell a story as to why your assumptions reflect only autocracies), but if you don’t (and the fact that you tried to build a theory unconditional on regime type indicates that you don’t), you’d need to think hard about how to incorporate that into the theory, so it can tell how how the interaction of this factor will produce outcomes in the data. So including another factor, in this case, doesn’t merely mean differently-sized but similarly-signed coefficients when you move to testing; it means differently-signed, interactive effects, that may turn out to be critical for explaining the puzzle that motivated you to write the theory in the first place…and to subjecting it to a test that can actually falsify its predictions.

So, to sum up, if the critique is a plausible size-changer, worry about controlling for it in your empirical tests, but if it’s a sign-changer, think hard about adjusting your model to reflect it, especially if you want to move to testing what may be an important interactive effect. Granted, there’s probably an unlimited number of potential sources of sign-changers, but thinking carefully about when changing a theory to reflect them will allow you to (a) handle critiques in a productive way and (b) develop better explanations of politics in general…which, presumably, is what this enterprise is all about.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s