Wednesday, August 21, 2019

An Information Retrieval Researcher’s Peer Review of Recent Studies of Search Engine Influence on Voting Behavior

A good part of my informatics research work over three decades has focused on the evaluation of search, also called information retrieval or IR. I have been amazed as the reach of search systems has become a mainstream part of our society, especially given that when I started, IR systems were only used by those who had computers with accounts from companies offering subscription search services.

Now, however, searching is ubiquitous. Indeed, it is almost impossible not to search, as it is offered in the address bar of most Web browsers. In addition, the name of one famous search engine, Google, has become a verb that synonymous with searching, i.e., Googling. Few of us can imagine a world without information on almost any topic being available nearly instantaneously.

It was therefore of interest this week when the President of the United States latched on to some research purporting that manipulation of Google was responsible for shifting three or more million votes to Hillary Clinton, which happens to be the amount of popular votes that she received over Donald Trump in the 2016 election (despite his narrow victory in the Electoral College).

This research has been put forth by Robert Epstein, PhD, who claims to be a liberal Democrat, as if that somehow indicates his analysis is not biased. Of course, one’s political views should not have any influence over the outcomes of their research.

Let’s look at Epstein’s multifaceted claims and the evidence supporting them from the standpoint of an IR researcher. First is the “finding” that Google manipulated search results to retrieve information “biased” toward Clinton. And second is that the retrieval of this information resulted in shifting of votes from Trump to Clinton.

The finding of manipulated search results comes from the paper posted as a PDF to Epstein’s Web site. As such, it is not peer-reviewed. The paper claims to show that in the run-up to the 2016 election and afterwards, 95 individuals, 21 of them of whom designated themselves as “undecided,” had their Google searches tracked and sent to a crowdsourcing site, Mechanical Turk, for rating as to whether they were biased toward Clinton or Trump. They eliminated searches from people who had Gmail accounts due to an unsubstantiated assertion that Google provided such users different results (which the company denies).

If I were sent this paper for peer review by an IR journal, I would ask the following: How did the researchers choose the individuals for the study? What evidence supports excluding those who had Gmail accounts? Who were the people on Mechanical Turk who did the ratings for the study? How were they instructed by the researchers to determine “bias?” I would certainly demand answers to questions like these before I would recommend acceptance for publication. The Methods section of the paper would need to be substantially expanded.

Let’s say, however, that the authors came back with acceptable answers to my questions, and the study were published. What about the second claim that this bias could lead to “manipulating” anywhere from 2.6-10.4 million votes in Clinton’s favor? The evidence for this comes from a paper that was published in a peer-reviewed journal, a prestigious one at that, the Proceedings of the National Academy of Sciences (PNAS). That study, published in 2015, looked at five randomized trials assessing the “search engine manipulation effect” (SEME).

These studies may be credible, but it is dubious whether they can be used to claim biased search results may have impacted voting in the US 2016 election. The first three experiments in the PNAS paper recruited individuals in the San Diego, CA area to rate who they might vote for the two candidates in the Australian Prime Minister election (chosen because most in San Diego would be unlikely to have prior knowledge). A fourth experiment replicated the first three with a national audience of individuals recruited from Mechanical Turk, while a fifth experiment recruited undecided voters to assess information about candidates in a local election in India. There should be no question that any kind of exposure to information can influence one’s decision about voting, although it would be questionable whether these sorts of results could be applied to a national US election where these same people would be bombarded by articles, reports, advertising, and other sorts of information, perhaps even Fake News promulgated by foreign entities on Facebook or Twitter.

Epstein fused the results of this research together to claim that biased search results moved several million votes in the direction of Clinton in the 2016 election. He took this conclusion to a receptive audience of Republicans in the US Senate. The outcome was predictable, with no skepticism whatsoever. And then came the crowning glory of it all, a Presidential tweet.

The mainstream fact checkers had a field day with these claims. Clearly one incompletely reported and probably highly flawed study, fused with another one showing that in some instances, search results can influence voting behavior, is hardly evidence that alleged bias by Google moved votes to Clinton in 2016. Here are some of their assessments:
The results of all this remind of the famous joke by comedian Stephen Colbert, who once noted, reality has a liberal bias. I do believe it is important and I strive to keep political biases out of our research. Even in my teaching, I aim to present opposing points of view, although not aiming to give equivalence to all points of view. But research like this needs to be called out for its thinly veiled political goals, and I suppose on that front, its “results" can be called successful.
at

No comments:

Post a Comment