1/n Alright, recent discussions re the Dunning-Kruger effect sparked my interest, and I have put together a blog on building a generative model of the classic effect! (the start of what I hope to be a series)

Link: http://haines-lab.com/post/2021-01-10-modeling-classic-effects-dunning-kruger/ https://twitter.com/Nate__Haines/status/1346325849541369856

2/n We start with the controversy, much of which boils down to whether or not the effect results from regression to the mean, versus a true psychological bias. We walk through some simulations showing how exactly regression to the mean can re-produce the classic D-K effect:

3/n Next, we discuss some issues with interpreting the effect as entirely driven by regression to the mean. The core problem is that regression to the mean is not an explanation—it results from a non-perfect correlation between variables. But what causes the correlation?

4/n We then discuss a simple noise + bias model that can generate the Dunning-Kruger effect, and demonstrate how a combination of perception noise and bias (i.e. over-confidence) can produce the D-K effect (h/t @inferencelab, who demonstrated this too: https://drbenvincent.medium.com/the-dunning-kruger-effect-probably-is-real-9c778ffd9d1b):

5/n One issue w the noise+bias model is that, although it does model generative psych mechanisms, it loses the classic interpretation of the D-K effect where those w low "skill" over-estimate a lot, and those w high skill under-estimate a little. We develop a new model that can:

6/n This "new" model is actually very inspired by signal detection theory and related models (so not so new :D), which grounds the model in other math psych/cog sci literature. Further, the proposed psych mechanisms alone can produce the traditional interpretation of the effect:

7/n We then develop a second model inspired by work in the confidence and risky decision-making literature, which can similarly generate the D-K effect using psychological mechanisms alone:

8/n Next, we fit the models to real data! (from Pennycook et al., 2017). In this dataset, participants answer 8 items on an objective measure, and then guess how many they got correct. Some items are as follows:

9/n The raw data in this task indeed show the traditional D-K effect, and the objective and perceived ratings are only moderately correlated:

10/n Upon fitting the models to the data, the models are readily able to capture important patterns in the observed data, and they re-produce the D-K effect:

11/n Most importantly, the parameter estimates from both models support a psychologically-driven D-K effect! The locations of the posterior distributions are where we would expect them for generating a psychological D-K effect:

12/n More intuitively, we can plot model-predicted relationship between the underlying objective skill and perceived skill parameters for each participant (here, averaged across the posterior distributions, so each point indicates one person). The D-K effect is very clear!

13/n In sum, I think the classic interpretation of the Dunning-Kruger effect is encoded well by these models, and when fit to real data, they show strong evidence for the idea that the D-K effect is driven by psychological biases—not just measurement error.

14/14 As always, all @mcmc_stan and #rstats code is included in the post! Feedback welcomed. Hopefully this inspires more generative modeling

15/14 I should make a final note here that the model leaves open the possibility that the Dunning-Kruger is domain specific—that what I am under-avg at, I assume I'm better than I am, and vice-versa. We would need to sample the same people across level of skill to check that.