Now reading: "The Role of Replication Studies in Theory Building" (2021, Perspectives on Psychological Science), by Elizabeth Irvine 1/n https://journals.sagepub.com/doi/full/10.1177/1745691620970558

"Replication studies have been presented as a way of directly supporting theory development by enabling researchers to more confidently and precisely test and update theoretical claims." 2/n

"I use contemporary work from philosophy of science to make explicit and emphasize just how much theory development is required before “good” replication studies can be carried out and show just how little theoretical payoff even good conceptual replications offer." 3/n

(Risky (or not so risky) guess: Liz will argue this route: https://twitter.com/IrisVanRooij/status/1306300102722621441?s=20 ... curious to find out whether I am right or wrong) 4/n

"I suggest that in many areas of psychology aiming at replication is misplaced and that instead replication attempts are better seen as exploratory studies that can be used in the cumulative development of theory and measurement procedures." 5/n

"There is not, as far as I can tell, a huge amount of discussion about the level of theoretical development required to design a good replication." 6/n

"What counts as a good replication will be determined by local factors, but there are some very general areas of theory that are required to build good replications." 7/n

"To be clear, these areas of theory do not have to be finalized or entirely correct, but they do at least need to be fairly plausible and fairly well worked out. I outline these areas here and briefly discuss their likely availability." 8/n

"First ...one needs to have a pretty good idea of how differences in, e.g., social or historical context, or demographic, educational, or cultural differences ... might affect whether an original set of stimuli will trigger the same psychological response in a new population" 9/n

"The wealth of possible factors that might affect whether psychometric invariance is achieved, particularly in social psychology, is vast. As a result, there is often no well-developed & widely accepted set of background theory [to] inform the design of good replications" 10/n

"In principle, a procedure that incorporates changes to all of the design facets of the original study may still be fairly similar in terms of basic causal structure. Indeed, implementing all of these changes may sometimes be necessary for preserving psychometric invariance" 11/

"In contrast, a replication study that changes only 1 design facet [may do] so in a way that changes the causal structure of the experimental intervention in a major way ... Assessing causal independence is therefore messier than counting mere changes in experimental design" 12/

"In sum then, to do a good direct replication, one needs to have a fairly well-developed and widely accepted set of background theory about the extended causal profile of the target phenomenon: enough to adapt experimental stimuli to new populations and contexts." 13/

(Preregistered prediction in tweet 4 was borne out. That's the power of theoretical thinking )

"Without this information, a researcher cannot claim to have performed a good replication. And without this claim, the apparent success or failure of the replication is not obviously informative about whether the original findings can, in fact, be replicated." 15/

"In the next section I argue that the power of replication studies to inform theory is also (sometimes) overstated in the psychological literature. That is, even where one can perform good replications, they do not offer a huge theoretical payoff." 16/

"A useful example to illustrate this is from Chang’s (2004) work on the history of thermometry." 17/

(Interesting! Irvine also builds her argument (in part) on Chang's work. See other relevant threads, here: https://twitter.com/IrisVanRooij/status/1341869233093931010?s=20. Scroll up and down for context)

"[Experimentalist Henri Victor Regnault] found that air thermometers were in fact the most repeatable. Mercury, air, and the glass used to make the thermometers all expand when hot, and when glass expands, it affects thermometer readings." 18/

" however, Regnault “never strayed from the recognition that comparability [repeatability] did not imply truth” (p. 83); that is, the repeatability of air thermometers was no guarantee that they were capable of accurately measuring temperature." 19/

"A potentially bigger concern then is whether good conceptual replications are all that powerful. In fact, there is an emerging consensus in philosophy of measurement that even good conceptual replications do not offer a significant theoretical payoff. " 20/