Read on Twitter
Since everyone seems to have lost their damn minds over the "more tweets = more citations" paper, and I'm moderately qualified to talk about it, I thought I'd share a few thoughts (1/n).
https://pubmed.ncbi.nlm.nih.gov/32504611/ https://twitter.com/JSheltzer/status/1272574976454012930
https://pubmed.ncbi.nlm.nih.gov/32504611/ https://twitter.com/JSheltzer/status/1272574976454012930
To refresh your memory, the study randomized 112 papers into 2 groups, 1 group which gets tweeted about by the Thoracic Surgery Social Media Network (TSSMN), and a control group that gets no tweets :( (2/n)
The average number of tweets received by the "tweeted" group is 15. However, a crucial part of the study design is that each designated article gets an initial tweet from a designated delegate, and 11 other delegates are required to retweet it. (3/n)
I cannot stress this enough, so I'll resort to the caps lock: THIS IS NOT REPRESENTATIVE OF HOW PEOPLE USE TWITTER. When I tweet about an article I find interesting, I do not require my 11 friends to retweet it. (4/n)
Essentially, of the ~15 tweets per paper, only ~3 are "organic" engagement, the other 12 are manufactured by the TSSMN intervention. Compare this to the non-tweeted group, which received an average of 0 tweets after the intervention. (5/n)
Put another way, this means these 12 TSSMN delegates account for roughly 80% of the Twitter activity FOR THE ENTIRE FIELD. (6/n)
This is troubling because it suggests that the TSSMN delegates have a tremendously outsized influence over which papers in the field get any attention at all on social media. (7/n)
Imagine doing this experiment in your own field--how many colleagues would be necessary to control *eighty percent* of the tweets about papers in your field? 10? 100? 1000? It is highly unlikely that 12 extra tweets will translate to 4x as many citations across the board (8/n)
The study design is outlined in a previous paper ( https://www.sciencedirect.com/science/article/pii/S0003497519311622), but I'll point out that the "tweeting" intervention was done over a 2-week period in June, 2018. (9/n)
Since most papers in the field are only receiving ~1-2 tweets pre-intervention, this means there was a massive uptick in social media attention for the 56 papers in the tweeted group (10/n)
Imagine if the papers in your field are suddenly receiving a 10-15-fold increase in attention on twitter over a 2-week period--of course you pay special attention to them, they look incredibly important relative to the rest of the literature! (11/n)
Given the apparent influence of TSSMN, I assume thoracic surgery is a small, tight-knit community, so it's even possible that the extra citations for the "tweeted" papers are coming from the delegates themselves, precisely because they were randomized to notice them. (12/n)
To properly assess the causal relationship between tweeting and journal citations, we'd need to take a deep dive into the citation networks for each of the 112 papers, and ignore any citations from papers authored by any of the 12 TSSMN delegates. (13/n)
The potential effects of outliers are also obscured--citation counts typically follow a lognormal distribution, so differences could be explained by a few papers that accrued a lot of citations--perhaps the "tweeted" group included 2-3 highly-cited reviews, just by chance (14/n)
Criticisms aside, I think it's a cool paper and it's great to see so many people actively engaging with these metaresearch questions (>6,000 retweets on @JSheltzer's tweet so far???) (15/n) https://twitter.com/JSheltzer/status/1272574976454012930
Can you game your h-index by setting up a Twitter botnet to promote your work? Probably not. Can you engage your colleagues by enthusiastically sharing your own research? Absolutely. (16/n)
I'll conclude with a plug for my own work about the crucially important task of interpreting altmetrics in context--there are many facets of social media engagement beyond simply self-promotion and academic prestige (END) https://www.biorxiv.org/content/10.1101/2020.03.06.981589v1
