This preprint provides an important critical evaluation of metascientific approaches to replications and how the underlying motives differ from other approaches to across multiple fields. Should be a key reading for metascientists. Some notes below. https://osf.io/preprints/socarxiv/96qxv/

“At odds with its inclusive rhetoric, […], the current activism around replication has emerged from a narrow category of scientist.”

“does this [metascience] activism represent a political movement as much as an epistemic one, threatening the autonomy of fields across science while redirecting resources and intellectual attention toward metascientists?”

“Metascientists […] have virtually no engagement with the recent social studies of science despite its history of research on experimental practice, replication, communication in science, and the effects of changing guidelines.

The lack of engagement with this literature has resulted in a constrained, distorted theory of replication that has, in turn, motivated policies that are likely to produce unintended outcomes.”

The authors present detailed interviews (N = 60) with the board of editors at Science and uncover a multitude of views on replications from different fields. Then they categorize different approaches to replications and different underlying motives.

Some editors comment that much of the recent replication talk has been monopolized by a type of antagonistic replication the authors label “diagnostic”. Basically replications done for public tests of truth. It’s concerned with evaluating the truth value of a claim.

The authors contrast the diagnostic motive for replication with an “integrative” motive in which scientists replicate in order to adopt and extend rather than verify. Many interviewees comment that they wouldn’t do replications for its own sake unless they want to build on it.

Interviewees offer a different, competing version of self-correction. “This suggests that the activism currently transforming scientific practice is driven by a specific ideology of science rather than universally accepted principles of ‘good science’.”

Integrative replication begins in trust. Diagnostic replication, on the other hand, is based on skepticism and distrust.

Researcher from many fields indicate that rather than explicit checks for veracity, replication is conducted with an eye towards developing new capacities or pursuing new questions.

“For a finding to become what Latour (1987) referred to as an “obligatory passage point,” other research groups must travel through it. They must adopt the technique, import the technology, or base future studies on some purported relationship.

If the original finding is wrong or is too difficult to recreate, it will languish—a dead end rather than a passage.”

Is there anything specific that characterizes these fields with an integrative motive for replications?

Yes! Low task uncertainty >> Where variables are known and controlled and experimental techniques and technologies are standardized and predictable. In such experimental contexts, standardization is expected to produce replicable outcomes. Researchers trust published results.

High task uncertainty, otoh, is characteristic of experimental contexts in which significant variables are either unknown or uncontrollable and/or experimental techniques and technologies are either unstandardized or unstandardizable.

In such contexts, as respondents reported, lack of standardization, routinization, and formalization created difficulties in the communication and evaluation of research findings.

In fields where task uncertainty is higher, replication is both more challenging and more difficult to evaluate and hence, while these fields have higher distrust in their results, diagnostic replications are less likely to be useful. >> A bit of a dilemma for metascience

“When task uncertainty is high, faithful replications require special effort. They demand investments of time and money to gain the requisite skill and acquire the needed research objects and experimental technologies.

And, still, failure—even uninterpretable failure—remains an option. In fields characterized by these types of uncertainties, researchers treat replication not primarily as process of uncovering truth or falsity.”

In such cases, an alternative path to self-correction is pursued, an organic one. Rather than spend time, money, and good will trying to officially correct the record, the errors would be corrected through the normal course of scientific investigation.

This organic self-correction “happens largely through the unpublished backchannels of a field. Professional networks spread information about the relative reliability of claims gleaned during attempts to integrate them.”

“Formal self-correction remembers wrongness; Organic self-correction forgets that which is not useful.”

While metascientists claim that organic self-correction is weak, this article suggests that there are many reasons why it’s preferable or superior to formal self-correction via diagnostic replications.

Political solution to replication problems >> Those who exert control over a field can mandate data, methods, and analytic standards. Yet, the push for more technically exacting science can be done as the cost of research that is more exploratory, uncertain, and meaningful.

Although metascience activism is seeking changes across the scientific landscape, many of the origins of the modern “crisis” stem from [social psychology]. The interpretation dominating replication discourse has been pushed by a small group of activist social psychologists.

“What needs to be scrutinized is the way the problems and solutions of social psychology have been extrapolated to the rest of the sciences. Although framed as a purely epistemic movement, reforms inevitably privilege some forms of knowledge and empower particular actors.”

Given this analysis, policies to push for diagnostic replications would be especially effective in fields with low task uncertainty, where replication is interpreted to be most diagnostic. Yet in such fields, researchers are less likely to pursue them.

Conversely, in fields higher in perceived task uncertainty, replication initiatives are likely to have less impact because replications in these fields are more susceptible to falling into the experimenters’ regress due to uncontrolled or unknown variability.

Diagnostic replications undertaken for unknown motivations “can create a culture of paranoia which, while in line with the abstract ideal of ‘organized skepticism,’ reflects a mistrust that is actually quite unusual in the history of science.”

My conclusion: The normalization of this state of paranoia and mistrust is exactly what I find most disconcerting and dangerous in the current metascience movement.