Read on Twitter
Striking article from @ProfRubega et al: Designed intensive, 15-wk sci comm course for grad students (for credit). External evaluators gave similar communication competence ratings to trainees & matched, untrained controls.
https://journals.sagepub.com/doi/10.1177/1075547020971639
https://journals.sagepub.com/doi/10.1177/1075547020971639
Authors note the small sample size (18 trainees) but conclude: "We are confident that if our sample size obscures a real training effect, it is likely so small as to be of little practical difference with respect to the impact of the training on trainees."
This article highlights a major issue with a lot of reports on training effectiveness: They almost exclusively rely on trainee self-assessments—what they think they learned/improved—without any sort of external validation.
External evaluation is often time & effort intensive. Layer on top of this, the method for evaluation of communication competence is likely rather subjective. The authors found substantial variance in assessments, even for the highest & lowest overall scores.
"This suggests that even a relatively narrow audience does not agree on what constitutes 'good communication,' a significant problem for the goal of establishing a rigorous assessment framework that allows us to compare the relative value of different training approaches."
Still the authors still advocate for "the importance of direct, external assessment of science communication training models through measurement of the impact on an audience, rather than self-assessments by trainees, or by personnel administering the training."
This team includes (science) communications professionals who clearly invested significant time in developing & teaching the course—3 times. They walk through potential explanations/alternatives, but I never get the sense they're trying to spin this into a 'win'.
At the close: "If we are serious about helping scientists succeed at communicating information that is crucial to informed policy and public welfare, we will need to reconsider how training is assessed, and quite possibly the nature of the training itself."
