Read on Twitter
This paper is a must read for all comm scholars conducting mediation analyses.
TLDR: mediation analyses, in Comm, are increasing exponentially; are difficult to conduct; & in most cases inadequate for causal inferences.
Thread
for further thoughts.
https://journals.sagepub.com/doi/abs/10.1177/1077699020961519
TLDR: mediation analyses, in Comm, are increasing exponentially; are difficult to conduct; & in most cases inadequate for causal inferences.
Thread
for further thoughts.https://journals.sagepub.com/doi/abs/10.1177/1077699020961519
Outset: There's a clear increase of studies reporting mediation analyses in Comm.
Notably, there's a strong prevalence of (bias toward?) significance.
„Of the 387 studies in the sample, 382 reported one or more statistically significant mediation models.“
„Of the 387 studies in the sample, 382 reported one or more statistically significant mediation models.“
However, mediation is difficult:
"Commenting on the general state of mediation analysis, Kline (2015) [stated] that most studies used research designs 'that are inadequate to establish mediation, so relatively little of the extant literature on mediation is actually worthwhile'"
"Commenting on the general state of mediation analysis, Kline (2015) [stated] that most studies used research designs 'that are inadequate to establish mediation, so relatively little of the extant literature on mediation is actually worthwhile'"
What's the problem?
Omitted variable bias: A potential infinite number of other variables/mediators could also explain effect.
Plus: What about temporal order?
Causal claims are limited to factors that are manipulated. Often there's no manipulation or long. design!
Omitted variable bias: A potential infinite number of other variables/mediators could also explain effect.
Plus: What about temporal order?
Causal claims are limited to factors that are manipulated. Often there's no manipulation or long. design!
To establish causality, ideally each var/med should be manipulated.
"[Like Bullock et al 2010] we view “mediation analysis as a cumulative enterprise” & that discovery & confirmation of mechanisms in Comm require a systematic series of studies rather than one standalone study."
"[Like Bullock et al 2010] we view “mediation analysis as a cumulative enterprise” & that discovery & confirmation of mechanisms in Comm require a systematic series of studies rather than one standalone study."
Papers's conclusion for Comm ...
„An analysis of 387 articles found that many studies of mediation are based on inadequate designs that can lead to unwarranted causal inferences.“
„An analysis of 387 articles found that many studies of mediation are based on inadequate designs that can lead to unwarranted causal inferences.“
FWIW, other fields struggle as well:
„On organizational research, Saylors and Trafimow (2020) pointed out that the field’s propensity for complex mediation models meant that “much of the knowledge generated in top journals is likely false” (p. 1).“
„On organizational research, Saylors and Trafimow (2020) pointed out that the field’s propensity for complex mediation models meant that “much of the knowledge generated in top journals is likely false” (p. 1).“
Comm too focused on "mechanisms":
„The first is the pervading indirect effects paradigm in the field that places emphasis on explicating mechanisms that underlie media effects (Neuman & Guggenheim, 2011; Valkenburg et al., 2016). This creates a mutually reinforcing cycle ...
„The first is the pervading indirect effects paradigm in the field that places emphasis on explicating mechanisms that underlie media effects (Neuman & Guggenheim, 2011; Valkenburg et al., 2016). This creates a mutually reinforcing cycle ...
... where journal editors and reviewers privilege “mechanisms” and “processes” that underlie theories while authors observe the propensity for elaborate models in the extant literature and conclude that this is what is required for publication.“
Very much agree. Less is more.
Very much agree. Less is more.
The authors conclude that above all we should temper claims regarding causality and mention limitations.
Bc such mediation models can still be valuable, but should rather be a starting point for future research, which then addresses each relation specifically.
Bc such mediation models can still be valuable, but should rather be a starting point for future research, which then addresses each relation specifically.
I really like the paper. It's actually not all that negative, but written in a positive, nice, and constructive way.
Plus: it offers several relevant and practical solutions.
Plus: it offers several relevant and practical solutions.
