Read on Twitter
***HARKing***
A THREAD
HARKing stands for Hypothesizing After the Results are Known. It occurs when researchers present their post hoc hypotheses as if they are a priori hypotheses. Kerr (1998) wrote the seminal article on HARKing… https://doi.org/10.1207/s15327957pspr0203_4
A THREAD
HARKing stands for Hypothesizing After the Results are Known. It occurs when researchers present their post hoc hypotheses as if they are a priori hypotheses. Kerr (1998) wrote the seminal article on HARKing… https://doi.org/10.1207/s15327957pspr0203_4
HARKing has been described as one of the four horsemen of the replication apocalypse (Bishop, 2019). But is it really that bad?
In my view (Rubin, 2017, 2019), HARKing is often seen as problematic because it is bundled together with other "questionable research practices,” such as p-hacking. But what happens if we consider HARKing on its own, separate from these other issues?
In other words, if readers can rule out other questionable research practices (e.g., by checking publicly available research materials and data and sensitivity/robustness analyses), and they can clearly see how the current hypothesis has been…
…deduced from a pre-existing theory, rather than induced atheoretically from the current results, then does it matter if they mistakenly believe that the researcher deduced the hypothesis before, rather than after, they knew their result?
Here's Kerr (1998, p. 209):
"HARKing can entail concealment. The question then becomes whether what is concealed in HARKing can be a useful part of the “truth”...or is instead basically uninformative (and may, therefore, be safely ignored at an author’s discretion)."
"HARKing can entail concealment. The question then becomes whether what is concealed in HARKing can be a useful part of the “truth”...or is instead basically uninformative (and may, therefore, be safely ignored at an author’s discretion)."
Addressing this sort of question, some have argued that the timing of the deduction of a hypothesis (i.e., pre- vs. post-result) does not form a “useful part of the ‘truth’” (e.g., Brush, 2015; Keynes, 1921; Mill, 1843; Oberauer & Lewandowsky, 2019; Szollosi & Donkin, 2019).
For example, Lewandowsky (2019) recently summed up this point as follows:
“If a hypothesis is entailed by an independently formulated theory it does not matter when the theorist becomes aware of the supporting results.” https://featuredcontent.psychonomic.org/avoiding-nimitz-hill-with-more-than-a-little-red-book-summing-up-psprereg/
“If a hypothesis is entailed by an independently formulated theory it does not matter when the theorist becomes aware of the supporting results.” https://featuredcontent.psychonomic.org/avoiding-nimitz-hill-with-more-than-a-little-red-book-summing-up-psprereg/
To be clear, I’m not suggesting that it’s OK to lie in research reports or elsewhere because, in general, lying is wrong. Instead, my point is that the information concealed by HARKing may not be problematic for scientific progress as long researchers…
provide (a) a clear deduction (theoretical rationale) from a pre-existing theory to the current hypothesis and (b) open research materials, data, and robustness analyses that allow readers to check for problematic research practices.
In this case, even if readers are unaware that a hypothesis has been HARKed, they are still able to criticize (a) the theoretical quality of the HARKed hypothesis, (b) the methodology for testing that hypothesis, and (c) the statistical analyses.
In summary, on its own, HARKing may not be quite as problematic for scientific progress as the apocalyptic horseman imagery implies, and we may need to turn to other explanations for failed replications.
In particular, and like many others, I think we should be more concerned about the quality of the theory and theoretical deduction rather than the timing at which the deduction takes place.
Putting aside “questionable research practices,” it is possible that replication attempts fail because researchers make a theoretical error about the conditions necessary for replication rather than because their theorizing is secretly post hoc (i.e., HARKing).
My most recent paper on HARKing is here: https://drive.google.com/file/d/1bGIUjHSEAoJYJke6RWtBphXJjZLr1UeX/view?usp=sharing
You may also be interested in:
Hollenbeck and Wright (2017): https://doi.org/10.1177/0149206316679487
Rubin (2017): https://drive.google.com/file/d/1AFAGymkp6o4ui_1N2TBs1t5WiYavjSA2/view?usp=sharing
Vancouver (2018): https://doi.org/10.1017/iop.2017.89
Prosperi (2019): https://doi.org/10.1057/s41599-019-0340-8
Mohseni (2020): http://philsci-archive.pitt.edu/18523/
END
Hollenbeck and Wright (2017): https://doi.org/10.1177/0149206316679487
Rubin (2017): https://drive.google.com/file/d/1AFAGymkp6o4ui_1N2TBs1t5WiYavjSA2/view?usp=sharing
Vancouver (2018): https://doi.org/10.1017/iop.2017.89
Prosperi (2019): https://doi.org/10.1057/s41599-019-0340-8
Mohseni (2020): http://philsci-archive.pitt.edu/18523/
END
