This morning, @_akpiper is presenting on "Generalization, Data, and the Problem of Textual Evidence in Literary Studies" as part of #SpectrumsofDH. Live-tweeting to follow!

. @_akpiper Begins by noting ongoing work by @CecilyRaynor and others to continue building DH at McGill and honor the legacy of @sgsinclair. #SpectrumsofDH

. @_akpiper Talk comes from new book, "Can We Be Wrong? The Problem of Textual Evidence in a Time of Data". Problems and issues hold across the humanities. Collaborating w/ students & faculty across disciplines on this project. #SpectrumsofDH

. @_akpiper "Road to Damascus" moment came reading Fredric Jameson, "almost the perfect negative example" of problematic discourse of generalization. Third world literature as "national allegories". Discursive markers: "all", "are necessarily". #SpectrumsofDH

. @_akpiper Generalization is important, has positive affordances. Can't do without it. Enables action, allows processing and making decisions w/ less information. But can also be dangerous or hurtful. Have to mitigate risks of generalization going awry. #SpectrumsofDH

. @_akpiper Generalization is widespread practice in the humanities. Jameson positing a construct of "third-world literature" that is bounded. He's engaged w/ a small portion, will make a claim about all of it. But what are selection/definition criteria? #SpectrumsofDH

. @_akpiper Realistically, Jameson has probably only read maybe at most 1 in 1,000 pieces of "third world literature". Incredibly small samples leading to incredibly large conclusions. Need to engage with that discrepancy. #SpectrumsofDH

. @_akpiper More data isn't necessarily better; you could've gotten your data from what turns out to be a weird quadrant of the whole. There may also be latent structures. Humanists sometimes claim they don't generalize, "particularism hypothesis". #SpectrumsofDH

. @_akpiper Trying to use machine learning to study complex phenomenon of generalization. Want to generalize about use of generalization in the field. 5 steps of model building. #SpectrumsofDH

. @_akpiper Defining generalization as a "statement-level phenomenon"; things that people say or write can be classified as a generalization. Identifying positive and negative examples. Subject and verb structures are important here. #SpectrumsofDH

. @_akpiper Negative examples can explain what a group of people did, rather than making a claim about an entire class of people. Use of paradigmatic examples. Next: select data. Argue for a framework for looking at it. Can't look at all articles ever. #SpectrumsofDH

. @_akpiper Have to draw a line somewhere and argue that what you've chosen is representative of the whole. Open Science Collaboration work talk about how to look at previous work, assess them fairly about replicability, and fairly evaluate where to go from here. #SpectrumsofDH

. @_akpiper 230 articles, about 8,000 words apiece. Have to annotate those to make explicit our judgements as experts. Piper did large-scale annotation, then validated against judgments of team on smaller set. But agreement was pretty low, lots of ambiguity. #SpectrumsofDH

. @_akpiper Wanted to involve students to generalize our insight about generalization. Would this be intuitable by non-specialists? Specialists would probably have had higher agreement around annotation. #SpectrumsofDH

. @_akpiper Implementation; F1 score using CNN and ELMo was .769. Room for improvement, probably, but it's an ambiguous space. Lit studies has less, 1 out of every 2 sentences in intro to literary studies article could be classified as generalization. #SpectrumsofDH

. @_akpiper Differences across journals; PMLA has less generalization in introduction than others. It's fair to reject null hypothesis that literary studies doesn't generalize (at all, or seldomly). It's a fairly regular occurrence. Now we have evidence for that. #SpectrumsofDH

. @_akpiper How do we make more credible generalizations? Open Generalization framework. Drawing on work of OSF, replication studies. Openness as important epistemological criteria for making claims about any domain. #SpectrumsofDH

. @_akpiper Telos of generalization: be explicit that you're going to generalize, and what you're generalizing about. Also be clear and explicit about empirical claims, and set of data you'll look at. #SpectrumsofDH

. @_akpiper Signaling and foregrounding that the phenomenon of generalization is taking place. Doing so leads to caveats as you realize the limitations. Selection criteria: include negative examples (look at where it isn't true). If there's no neg example, worry. #SpectrumsofDH

. @_akpiper Create clear and consistent criteria for judging the passages, and articulate that. Measure level of agreement with others around those criteria. Other people have to agree with it (beyond your subjective interpretation) for better generalizations. #SpectrumsofDH

. @_akpiper Embrace uncertainty: don't rush to (in)validate things. Conclusion can just be "these are the things we could replicate, and these are the things we couldn't". Concept of confidence. Literary studies: no limitation, hyperbolic is rewarded. #SpectrumsofDH

. @_akpiper Process of confidence building and embracing uncertainty lets you find conclusions within certain conditions. Change the conditions: does the conclusion still hold? Can estimate within ranges. X might not be untrue, maybe we're just uncertain. #SpectrumsofDH

. @_akpiper Embracing a culture of limitation. Differences between when algorithm is confident vs. not that something is a generalization. Can see limits of what we can detect, how we can distinguish categories, build that into argument to foreground transparency. #SpectrumsofDH

. @_akpiper Goal is a community consensus around issues, rather than claims put forth by just one person. Supports a more. collaborative process. Imagine reworking Jameson's claim along these lines. #SpectrumsofDH