AoIR 2024

Polarisation via Search? Assessing the Political Spectrum of Google News Recommendations

Axel Bruns, Arjun Srinivas, Abdul Karim Obeid, James Meese, Daniel Angus, Timothy Graham, and Jean Burgess

• 2 Nov. 2024 – AoIR 2024 conference, Sheffield

Presentation Slides

Abstract

Introduction

For the past decade, the predominant concern relating to the recommendations returned by search engines has been about their potential to create ‘filter bubbles’ (Pariser, 2011): that is, to personalise the information sources recommended for a given search term to such an extent that different searchers may use them to form widely divergent views of the world. Such concerns have been debunked (Bruns, 2019): a series of studies undertaken using varying methodological approaches and addressing several national contexts (e.g. Krafft et al., 2019; Haim et al., 2018; Nechushtai & Lewis, 2018; Meese et al., 2023) have documented an overall lack of search result personalisation at the individual user level. Instead, these studies show that the information sources commonly recommended by leading search engines like Google Search and Google News are in fact highly consistent and privilege a small handful of major news outlets, reinforcing the market dominance of those news brands (Nechushtai & Lewis, 2018: 302).

However, while concerns about the algorithmic creation of ‘filter bubbles’ must therefore be dismissed, this does not imply that even the largely mainstream sources recommended by leading search engines are inherently unproblematic. In a new study that updates the earlier findings by Nechushtai & Lewis (2019), for instance, Nechushtai et al. (2023: 17) document the rise to prominence of the far-right news channel Fox News amongst the sources most recommended by Google News, Google Search, and YouTube. Not only prominent, Fox News has also been prioritised in YouTube recommendations as an “authoritative” source – a category that appears to default to traditional media outlets, regardless of the quality or trustworthiness of their coverage (YouTube, 2019). Its Australian NewsCorp counterpart Sky News Australia similarly appears frequently amongst search results in Australia – even though its video content was temporarily removed from YouTube for spreading misinformation during the height of the COVID-19 pandemic (Meade, 2021).

Even in the absence of per-user personalisation, search results may therefore produce problematic effects if they recommend sources providing fringe and highly biased content, and thereby point users to material that promotes increased issue, interpretive, and ideological polarisation. Drawing on a unique dataset of hundreds of millions of search results provided as data donations by citizen participants over the course of a 10-month period in 2021 and 2022, this paper investigates the range of news sources recommended by Google News for a selection of queries on political, COVID-related, and otherwise controversial topics. We examine the political spectrum of the sources that are most commonly recommended for these queries; assess whether the breadth of this spectrum varies between queries (and whether this may point to human intervention in query results for some topics that could be considered by platforms to be both controversial and societally significant); and examine whether this breadth grew or shrunk over the course of the study period (which included part of the COVID pandemic).

Dataset and Methods

For this analysis we build on the data gathered by the Australian Search Experience (ASE) project in the ARC Centre of Excellence for Automated Decision-Making and Society (Bruns 2022). ASE recruited several hundred volunteer Australian users who installed a browser plugin that automatically ran queries for a set of approximately 40 search terms across Google Search, Google News, Google Video, and YouTube every four hours, if their computers were switched on, from September 2021 to July 2022. The plugin reported the first page of search results for each query to a central database, resulting in a dataset with several hundred million individual search results as well as additional metadata on their ranking on the results page, query terms and timestamps, and other ancillary information. Search terms used in our queries included the names of leading Australian politicians and parties, COVID-related terms such as ‘COVID’, ‘vaccine’, ‘quarantine’, and ‘lockdown’, as well as potentially polarised topics related to social issues, such as ‘Critical Race Theory’ and ‘feminism’. Data were processed to account for demographic skews in the ASE user population.

In this paper, and in keeping with the overall theme of the panel within which it is located, we focus on the search results obtained from Google News. For each of the search terms, we establish the 30 most commonly recommended news sources over the course of the entire study period, and assess fluctuations in their relative prominence over time; our past analyses of these data (Authors, redacted) have already shown that a small number of sources generally account for a very large proportion of all search results (thus also echoing similar observations made for the US by Nechushtai & Lewis), and these top 30 sources will therefore usually cover the vast majority of news recommendations.

We then draw on available secondary data to assess the political positioning and trustworthiness of these sources; such sources include the Australian edition of the Digital News Report (Park et al., 2023), which provides survey-based information on the political leaning of the online and offline audiences of major Australian news brands, as well as equivalent reports covering any international news sources (BBC, CNN, etc.) that Google News may also recommend to Australian users, and trustworthiness assessments such as those produced by NewsGuard (NewsGuard Technologies, 2024) or the Global Disinformation Index (Glazunova et al., 2021). Where required we also consult the scholarly literature as well as government, industry, and NGO reports to assess the positioning and quality of news sources which are not covered in these reports.

Analysis and Implications

This critical assessment and annotation of the dominant news sources recommended by Google News for each search term then enables us to investigate the political spectrum represented by these search results, and its potential fluctuation over time. Where we identify a notable expansion or contraction of the breadth of that spectrum during the period investigated here, we will explore the relevant contexts – for instance, government initiatives to reduce the circulation of COVID disinformation, or publicly announced changes to Google’s own ratings of the quality of news sources, could both result in the systematic up- or down-ranking of specific news sources in search results.

We interpret the results of this study through the lens of the literature on polarisation. A variety of perspectives on a given topic is generally desirable in search results, but the persistent recommendation of vastly divergent news sources that present mutually incompatible worldviews can also foment further interpretive polarisation (Kligler-Vilenchik et al., 2020), as search users from all backgrounds are provided with material that supports and further entrenches their pre-existing biases about a given issue. (If search results recommend only sources located towards the extremes of the political spectrum, without also including prominent balanced and centrist voices, this would be exacerbated.)

Similarly, differences – between individual search terms – in the breadth of the political spectrum of recommendations may serve as an indicator of greater or lesser issue polarisation relating to the specific issues that these search terms address; conversely, an absence of such search term-specific differences in the dataset would mean that such issue polarisation is perhaps less likely than more general patterns of ideological polarisation that transcend specific issues (Lelkes, 2016).

We note here, however, that the patterns in the breadth of the political spectrum represented by Google News search results that our study observes point only to the potential for polarisation, rather than to a definitive diagnosis: it is up to search users to select from, engage with, and critically interpret the resources that Google News and other search engines recommend to them. The mere availability of different interpretations of a given issue or event does not necessarily lead to interpretive polarisation amongst the users who engaged with these interpretations (or at least not to a point where such interpretive differences become destructive).

We also do not intend to propose an ‘ideal breadth’ for the political spectrum represented by these search results. While there may be principled justifications for such a normative postulation, our principal aim for the present paper is to document and analyse the current status quo, and thereby to enable further debate about whether that status quo appears appropriate in the current political context.

References

Bruns, A. (2019). Are Filter Bubbles Real? Polity.

Bruns, A. (2022). Australian Search Experience Project: Background Paper. Working Paper 001. ARC Centre of Excellence for Automated Decision-Making and Society. https://doi.org/10.25916/k7py-t320

Glazunova, S., Dehghan, E., & FitzGerald, K. M. (2021). Disinformation Risk Assessment: The Online News Market in Australia. Global Disinformation Index and QUT Digital Media Research Centre. https://disinformationindex.org/wp-content/uploads/2021/09/GDI_QUT-Australia-Disinformation-Risk-Assessment-Report-21.pdf

Haim, M., Graefe, A., & Brosius, H.-B. (2018). Burst of the Filter Bubble? Effects of Personalization on the Diversity of Google News. Digital Journalism, 6(3), 330–343. https://doi.org/10.1080/21670811.2017.1338145

Kligler-Vilenchik, N., Baden, C., & Yarchi, M. (2020). Interpretative Polarization across Platforms: How Political Disagreement Develops over Time on Facebook, Twitter, and WhatsApp. Social Media + Society, 6(3). https://doi.org/10.1177/2056305120944393

Krafft, T. D., Gamer, M., & Zweig, K. A. (2019). What Did You See? A Study to Measure Personalization in Google’s Search Engine. EPJ Data Science, 8(1), Article 1. https://doi.org/10.1140/epjds/s13688-019-0217-5

Lelkes, Y. (2016). Mass Polarization: Manifestations and Measurements. Public Opinion Quarterly, 80(S1), 392–410. https://doi.org/10.1093/poq/nfw005

Meade, A. (2021, 1 Aug.). Sky News Australia Banned from YouTube for Seven Days over Covid Misinformation. The Guardian. https://www.theguardian.com/media/2021/aug/01/sky-news-australia-banned-from-youtube-for-seven-days-over-covid-misinformation

Meese, J., Obeid, A. K., Angus, D., & Bruns, A. (2023). Measuring Intermediary News Diversity: Google News in Australia. Working Paper 007. ARC Centre of Excellence for Automated Decision-Making and Society. https://doi.org/10.25916/xk6y-a642

Nechushtai, E., & Lewis, S. C. (2019). What Kind of News Gatekeepers Do We Want Machines to Be? Filter Bubbles, Fragmentation, and the Normative Dimensions of Algorithmic Recommendations. Computers in Human Behavior, 90, 298–307. https://doi.org/10.1016/j.chb.2018.07.043

NewsGuard Technologies. (2024). Website Reliability Ratings. NewsGuard. https://www.newsguardtech.com/solutions/newsguard/

Pariser, E. (2011). The Filter Bubble: What the Internet Is Hiding from You. Penguin.

Park, S., McGuinness, K., Fisher, C., Lee, J. Y., McCallum, K., Cai, X., Chatskin, M., Mardjianto, F. X. L. D., & Yao, S. (Pinker). (2023). Digital News Report: Australia 2023. News and Media Research Centre. https://apo.org.au/node/322606

YouTube Team, The. (2019, 3 Dec.). The Four Rs of Responsibility, Part 2: Raising Authoritative Content and Reducing Borderline Content and Harmful Misinformation. YouTube Blog. https://blog.youtube/inside-youtube/the-four-rs-of-responsibility-raise-and-reduce/