Wednesday, August 21, 2019

An Information Retrieval Researcher’s Peer Review of Recent Studies of Search Engine Influence on Voting Behavior

A good part of my informatics research work over three decades has focused on the evaluation of search, also called information retrieval or IR. I have been amazed as the reach of search systems has become a mainstream part of our society, especially given that when I started, IR systems were only used by those who had computers with accounts from companies offering subscription search services.

Now, however, searching is ubiquitous. Indeed, it is almost impossible not to search, as it is offered in the address bar of most Web browsers. In addition, the name of one famous search engine, Google, has become a verb that synonymous with searching, i.e., Googling. Few of us can imagine a world without information on almost any topic being available nearly instantaneously.

It was therefore of interest this week when the President of the United States latched on to some research purporting that manipulation of Google was responsible for shifting three or more million votes to Hillary Clinton, which happens to be the amount of popular votes that she received over Donald Trump in the 2016 election (despite his narrow victory in the Electoral College).

This research has been put forth by Robert Epstein, PhD, who claims to be a liberal Democrat, as if that somehow indicates his analysis is not biased. Of course, one’s political views should not have any influence over the outcomes of their research.

Let’s look at Epstein’s multifaceted claims and the evidence supporting them from the standpoint of an IR researcher. First is the “finding” that Google manipulated search results to retrieve information “biased” toward Clinton. And second is that the retrieval of this information resulted in shifting of votes from Trump to Clinton.

The finding of manipulated search results comes from the paper posted as a PDF to Epstein’s Web site. As such, it is not peer-reviewed. The paper claims to show that in the run-up to the 2016 election and afterwards, 95 individuals, 21 of them of whom designated themselves as “undecided,” had their Google searches tracked and sent to a crowdsourcing site, Mechanical Turk, for rating as to whether they were biased toward Clinton or Trump. They eliminated searches from people who had Gmail accounts due to an unsubstantiated assertion that Google provided such users different results (which the company denies).

If I were sent this paper for peer review by an IR journal, I would ask the following: How did the researchers choose the individuals for the study? What evidence supports excluding those who had Gmail accounts? Who were the people on Mechanical Turk who did the ratings for the study? How were they instructed by the researchers to determine “bias?” I would certainly demand answers to questions like these before I would recommend acceptance for publication. The Methods section of the paper would need to be substantially expanded.

Let’s say, however, that the authors came back with acceptable answers to my questions, and the study were published. What about the second claim that this bias could lead to “manipulating” anywhere from 2.6-10.4 million votes in Clinton’s favor? The evidence for this comes from a paper that was published in a peer-reviewed journal, a prestigious one at that, the Proceedings of the National Academy of Sciences (PNAS). That study, published in 2015, looked at five randomized trials assessing the “search engine manipulation effect” (SEME).

These studies may be credible, but it is dubious whether they can be used to claim biased search results may have impacted voting in the US 2016 election. The first three experiments in the PNAS paper recruited individuals in the San Diego, CA area to rate who they might vote for the two candidates in the Australian Prime Minister election (chosen because most in San Diego would be unlikely to have prior knowledge). A fourth experiment replicated the first three with a national audience of individuals recruited from Mechanical Turk, while a fifth experiment recruited undecided voters to assess information about candidates in a local election in India. There should be no question that any kind of exposure to information can influence one’s decision about voting, although it would be questionable whether these sorts of results could be applied to a national US election where these same people would be bombarded by articles, reports, advertising, and other sorts of information, perhaps even Fake News promulgated by foreign entities on Facebook or Twitter.

Epstein fused the results of this research together to claim that biased search results moved several million votes in the direction of Clinton in the 2016 election. He took this conclusion to a receptive audience of Republicans in the US Senate. The outcome was predictable, with no skepticism whatsoever. And then came the crowning glory of it all, a Presidential tweet.

The mainstream fact checkers had a field day with these claims. Clearly one incompletely reported and probably highly flawed study, fused with another one showing that in some instances, search results can influence voting behavior, is hardly evidence that alleged bias by Google moved votes to Clinton in 2016. Here are some of their assessments:
The results of all this remind of the famous joke by comedian Stephen Colbert, who once noted, reality has a liberal bias. I do believe it is important and I strive to keep political biases out of our research. Even in my teaching, I aim to present opposing points of view, although not aiming to give equivalence to all points of view. But research like this needs to be called out for its thinly veiled political goals, and I suppose on that front, its “results" can be called successful.

Sunday, August 4, 2019

Living the Asynchronous Life

I have always been highly productive in my academic career, finding time to teach, carry out research, write, manage my department, and mentor faculty and students. I believe that one of the reasons for my productivity is my leading a relatively asynchronous life.

What do I mean by asynchronous? In essence, most of my work is done by myself, wherever and whenever. And if I hit a block (or get tried of something), I can leave one task and move to another.

A good part of the asynchronous nature of my work comes from my teaching, which these days is mostly online. Most of my courses involve recorded (though frequently updated) lectures delivered online, followed by online discussion through the Oregon Health & Science University (OHSU) learning management system. I also interact with students by email and even sometimes by phone.

This mostly asynchronous work is also amenable to my lifestyle. Being a (very early) morning person, I can get most of my important and creative work done early in the day. A typical day for me involves 2-4 hours of work, exercise, and arriving at the office between 9-10 am. (Leading some to believe I just roll into the office late, when I have in fact have already been working for several hours.)

Another advantage of my mostly asynchronous work is that it allows me to pursue other aspects of my job that I enjoy, such as travel. I have often noted that I get some of my best work done in hotel rooms during travels. In almost every corner of the world, I can get access to the Internet, all my internal OHSU resources, and OHSU's learning management system. Having 3-4 hours uninterrupted in a hotel room can result in completing a day’s worth of work. And while not all of my work is amendable to being done on airplanes, I can choose work that is when flying. (It does not hurt to have status that gives me more roomy seating (and even sometimes upgrades).

There are some downsides to the asynchronous life. One is that the capacity for switching across tasks is not infinite. As with multitasking computers, too many tasks, and switching between them too frequently, can lead to so-called deadlock.

Another downside is that it can sometimes seem that work never ends. Even when a project is completed, there are others still demanding attention.

Nonetheless, given a combination of my satisfaction of creating academic papers, talks, and hire, plus my enjoyment of getting to see the world, I will always enjoy living the asynchronous life.