Read on Twitter
Brief thread (1/n): New paper by Grice & colleagues, who've been making this simple but profound point for several years: It's theoretically informative to analyze/report how many Ss show the predicted empirical pattern:
https://journals.sagepub.com/doi/abs/10.1177/2515245920922982
https://journals.sagepub.com/doi/abs/10.1177/2515245920922982
2/n: An example from my lab: We're currently revising our multi-class/dual-site study of task-unrelated thoughts (TUTs) in university classrooms. A reviewer asked a good question... https://psyarxiv.com/3ds2y
3/n: ...what did our data show RE timecourse of TUTs within a class session? Relevant lab work consistently shows TUTs rise with time-on-task but classroom studies show mixed results (most show increases, but some don't). So, we looked.
4/n: For each student, we split thought probes into 1st half vs 2nd half of class session, & computed TUT rate for each half. We found a modest increase, ON AVG, from 1st half M=.213 to 2nd half M=.265 (+.052), t(850)=6.26, d=.214 [.146,.282].
5/n: So, conventionally we'd say that time-in-class showed a "small effect" on student TUT rate. Makes sense, right? Boredom/fatigue should rise during class, making concentration harder "for the avg student." This implies, tho, that for most students there was a small increase.
6/n: Did most of our students show a small TUT increase from 1st to 2nd half of class. NOPE! More subjects showed either no TUT change (n=267; 31%) or a TUT decrease (n=223; 26%) than showed the M-implied pattern of TUT increase (n=361; 42%).
7/n: Here's a Sankey/alluvial plot (made by @MattWelhaf) that illustrates our findings. Here we rounded each S's TUT rate to nearest .10 and plotted change from 1st to 2nd half. Only the students in gold showed the implied pattern.
8/n: What does this mean? Hard to say, but it's more f*cking complicated than we would appreciate from looking only at the aggregate stats. Are these diffs robust & meaningful? We see some weak correlations b/w this chg score & course outcomes, but only suggestive at this point.
9/9: Wherever the field goes with this question, tho, we can't (or shouldn't) ignore what our actual, individual subjects are doing and showing. Fin. :)
