Read on Twitter
This was a labour of love.
Epstein and King said almost 20 years ago that almost all legal research was irreproducible (and I think that's still true). But, how can we go about doing a reproducible legal analysis, using modern tools?
1/11
https://osf.io/preprints/lawarxiv/2j3ye
Epstein and King said almost 20 years ago that almost all legal research was irreproducible (and I think that's still true). But, how can we go about doing a reproducible legal analysis, using modern tools?
1/11
https://osf.io/preprints/lawarxiv/2j3ye
One example of irreproducibility is saying "Courts tend/often/usually decide X" without giving any background about the decisions they looked at to come to this conclusion.
This happens ALL the time. Epstein & King give examples, but here are two from Melbourne LR.
2/11
This happens ALL the time. Epstein & King give examples, but here are two from Melbourne LR.
2/11
My target was IMM v The Queen, a controversial 2016 HCA case that seemed remove judges' ability to exclude unreliable evidence, the kind implicated in wrongful convictions (forensic science, eyewitness IDs). But it was a confusing decision and seemed to leave some outs.
3/11
3/11
So, I wanted to see if subsequent judges read this broadly, focusing on the parts that removed their discretion to exclude unreliable evidence. Did they apply it beyond the facts of IMM itself? Are apparent exceptions to the rule being applied and made out?
4/11
4/11
How did I make this reproducible? I preregistered the scope and coding. All the code for describing the findings is public. And so is the data. I even made an app to let readers browse the cases (and read them for free on @austlii ). A PRISMA diagram shows excluded cases.
5/11
5/11
Here's the app: https://openlaw.shinyapps.io/imm-app/
All the data, code, etc is here, and maybe others can use it to do similar projects: https://osf.io/j3xhg/
6/11
All the data, code, etc is here, and maybe others can use it to do similar projects: https://osf.io/j3xhg/
6/11
What did I find?
First, the part of the judgment telling judges to assume evidence is max. reliable has been cited the most, and the exceptions have almost never been made out. NSW is most aggressive about trusting jurors with unreliable evidence:
7/11
First, the part of the judgment telling judges to assume evidence is max. reliable has been cited the most, and the exceptions have almost never been made out. NSW is most aggressive about trusting jurors with unreliable evidence:
7/11
And, as some worried, the decision has not been limited to the evidence at issue in IMM (that's the plurality of decisions though) - it's been applied to save all sorts of evidence.
8/11
8/11
As you can see from that last table, exclusion of "simply unconvincing" evidence has been considered most with eyewitnesses. This is probably b/c the maj. used an eyewitness ex to explain what unconvincing meant. But there's no sound reason to limit it to that example.
9/11
9/11
So, I think it is possible to do legal research and analysis in a reproducible way. Are the lengths I went to necessary in all cases? Definitely not. But I think a lot of this can be borrowed for smaller and more discrete projects to make them more useful and credible.
10/11
10/11
Most importantly:
11/11
11/11
