Read on Twitter
Were you taught that you cannot interpret a main effect in the presence of an interaction? That interactions "supersede" main effects? That you have to use hedging language like, "the effect of A depends on B"?
Then I've got a little provocation for you. Thread...
Then I've got a little provocation for you. Thread...
Imagine the following study: People with depression are randomly assigned to get either drugs or psychotherapy. In addition, they are asked which they believe is more effective: half say drugs, half say therapy. Outcome is functioning after treatment (0-100). So it's a 2x2
The PI asks two grad students to each run an ANOVA. (What can I say, she likes redundancy.) She wants to know if psychotherapy is more effective than drugs. Both grad students go off and do their thing, and report back at the next lab meeting
Grad student 1 goes first. He shows this slide and says, "There is a significant crossover interaction. So we cannot clearly say that psychotherapy is more effective than drugs. It depends on people's beliefs."
Then it's grad student 2's turn. She presents her analysis. "There is a significant main effect of psychotherapy. As you can see there is also a main effect of treatment-belief matching. But there is no interaction. So we can clearly say that psychotherapy is better than drugs."
Friends, they analyzed the same data
What is going on? Well, there were 4 groups of participants. Grad student 1 and grad student 2 made different arbitrary decisions about how to code those groups into a 2x2. Both made a code for "got drugs vs. psychotherapy", but they differed on the rest
Which is correct? Both. Neither. They're arbitrary. It doesn't matter.
This is, in fact, always true. I picked an example where either coding could make sense. But the math doesn't care what makes sense to humans. You can always recode interactions as main effects and vice versa
This is, in fact, always true. I picked an example where either coding could make sense. But the math doesn't care what makes sense to humans. You can always recode interactions as main effects and vice versa
This is a problem if you think interactions supersede main effects. Fortunately, that folk wisdom is wrong. ANOVA is an additive model with 4 terms: intercept + main effect + main effect + interactions. No "depends" in there
Rosnow and Rosenthal wrote a set of papers in the 80s and 90s about how many empirical articles present incorrect interpretations of interactions. This one has the delightful title, "Some Things you Learn Aren't So" https://journals.sagepub.com/doi/abs/10.1111/j.1467-9280.1995.tb00297.x
As R&R wrote, researchers often interpret an interaction by looking at the cell means. But that's incorrect. The cell means depend on all 4 terms of the ANOVA. To interpret the interaction, you have to just look at interaction part of the model
The folk wisdom about interactions superseding main effects is an extension of that. And the example I gave is a demo of how following it can lead you to weird places, like letting an arbitrary coding decision dictate whether/how you're allowed to interpret something
But if you are interpreting each term as it's actually defined in the model, you don't run into that. Psychotherapy is better than drugs by 10 points. Believing in the treatment you're getting is better than not, also by 10 points. You can draw both of those from either analysis
As R&R note, sometimes you want to interpret the cell means. That's fine! But then don't call them main effects and interactions. Your hypothesis is probably better represented by a custom contrast, which you could test instead. (They wrote a whole book about that)
And the next time someone tells you you can't interpret your main effects because there's an interaction, pause for a moment and make sure what you're interpreting is actually the main effect. If you are, proceed with confidence and send 'em a copy of R&R /end
