While survey studies often feel informative bc they give us the feeling of having a general overview by aggregating empirical data ...it’s important to remember that they have limitations and biases that are often glossed over in favor of quick data soundbytes particularly... 1/

...when circulated in media spaces. Sometimes, these quick data bytes resonate with pre-existing perceptions, calcifying ideas.

Survey designers are biased; they have an interpretation of the context within which they are constructing definitions, information & questions. 2/

If they operate axiomatically from those assumptions w/o transparency or awareness, this is a huge red flag.

This 2nd report in a 3-part series is a GREAT example of this. 3/ https://carnegieendowment.org/2021/02/09/how-do-indian-americans-view-india-results-from-2020-indian-american-attitudes-survey-pub-83800

Survey design is of particular importance. Questions can be value- and assumption- laden, but respondents may be precluded from disagreeing with how the question is framed, from disrupting the definitions or assumptions, and be forced to choose the “best” response. 4/

Alternately, the respondent may not know that the surveyor is using a different definition or attributing or interpreting a different meaning to their response than they would. In other words, the data collector is making meaning FOR the respondent instead of asking them. 5/

For instance, this survey adheres to the application of the liberal/conservative binary across both country’s political landscapes, when we know that that conflation is questionable, in terms of meaning. 6/

"Conventional wisdom reflected in much of the commentary on the diaspora’s political leanings underscores the diaspora’s simultaneous support for the left-of-center Democratic Party in the United States and the right-of-center BJP in India." 7/

In reality, the entire political landscape of the US is shifted to the right. The "center" is not the same. The study goes on to reinscribe this by creating questions that axiomatically conflate "same issues". 8/

"This raises the empirical question of whether such a contrast actually exists: Could the same person have divergent views on the same issues when considered in different contexts? Or do respondents’ policy attitudes remain stable across countries and contexts?" 9/

In other words, rather than asking if the respondents believe that these issues are, indeed, the same, they assume that they are. (e.g., conflating the 2017 travel ban with the CAA). The study then assigns each a correlated political grouping that is (tacitly) value-laden. 10/

There is also an element of “we all know what that means” in how the results are presented. This means that the dots will be connected according to people’s pre-existing biases. For instance, the stated correlation b/w BJP supporters who also support the US Republican Party...11/

...might suggest that it’s because the parties are similar (as per popular media convention), but people might support both for very diff. reasons. The survey only offers that there appeared to be contradictions.

Actually, it does much worse than that... 12/

"Indian Am's, iow, believe that white supremacy is a greater threat to minorities in the U.S., a country where they are a minority, than Hindu majoritarianism is to minorities in India, a country where Hindus (the most common faith of Indian Am's) are in the majority” 13/

This gives the clear impression that that Hindus are only concerned for our own well-being – whether in India or the US. This isn't just simple reporting of empirical data. The study axiomatically conflates white supremacy with Hindu majoritarianism. 14/

"Among Indian Americans, the BBC emerges as the most trusted source for news about India."

What now? There is NO interrogation about what that means, OR about how Indian Americans favor English-language Indian news sources.

*The language matters*. THIS NOT NEUTRAL. 15/

The same goes for the section on “free media”. There is no critical lens on how un-free "free media" actually is. It just assumes that free media means not controlled by government. Never mind that one of the greatest critiques of "free media" is about *corporate* control. 16/

Obviously, I could keep going. I think I've made my point.

One huge disclaimer: I am not an expert on survey study methodology. (I have some insight, as I used design survey studies for a nonprofit. But I wouldn't call myself credentialed.) 17/

This analysis is a critical reading of how biased, value-laden qualitative studies can appear as "neutral, empirical data" because they nicely aggregate data and present us with numbers and graphs, which we like and trust because they feel science-y.

Hope this is helpful.