Read on Twitter
The @The_JOP released a new policy on pre-registration for experimental work. The policy has created a stir. Some think it holds experimental work to a higher standard. Others argue it signals a preference for experimental work. 1/n
One concern many have raised is that requiring pre-analysis plans reinforces inequality: those with robust research funds will run a pilot study, register hypotheses, and then field another study. Most researchers don’t have the resources to take this approach. 2/n
A related concern (and one that I haven’t seen discussed) is inequality in how pre-analysis plans are used in practice and taught across departments. Do quant methods classes teach pre-analysis plans? 3/n
It’s one thing to teach that a pre-analysis plan—like IRB – is a necessary box to tick before conducting research. 4/n
But in the same way that learning how to navigate IRB doesn’t necessarily mean one is conducting ethical research, knowing you need to register a pre-analysis doesn’t teach you how to use it. Part of this challenge is that it’s not clear what counts as a pre-analysis plan. 5/n
Do you include theory? Do you include model specifications? Do you include DeclareDesign code/simulations? If you include multiple hypotheses in your pre-analysis plan, do you report all of them in your paper?
If not, why? If so, why? 6/n
If not, why? If so, why? 6/n
After answering these questions, how do we navigate between reviewer comments, our pre-analysis plans, and the final submission? 7/n
My concern is that even in departments that have strong quant methods focus there is tremendous variation in how or if pre-analysis plans are taught/modeled. I worry that this will only further increase inequalities across the discipline. 8/n
I have found pre-analysis plans to be a useful tool to think though the questions I am asking and the strengths and limitations of the design I am employing to answer them. 9/n
A pre-analysis plan can be a helpful way to get clarity about the research process, setting up the contours of the research design – making sure the building blocks are in place. Even if journals do not require them, some grants do. 10/n
But you can register expectations while still leaving room for surprise and exploration. But when they are required, it opens the door for them to be used as a gatekeeping tool. I’ve had a paper rejected because I deviated from a pre-analysis plan, even though I didn’t. 11/n
Even when/if this criticism is merited, a revision explaining how results are exploratory rather than confirmatory is more helpful than using deviation from a pre-analysis plan as justification for rejection. 12/n
Again, knowing we need to pre-register an experimental study is one thing. It is something else altogether to know how to navigate across the building blocks of research/the politics of publication – research question, pre-analysis plan, analyses, and peer-review process. 13/n
I’d like to see the “rules of the game” more clearly delineated before we formally require pre-analysis plans for publication. 14/n
Here are a just a few responses to the @The_JOP decision that I have found useful.
