ICA 2022

Assessing the Personalisation of Australian Google News Results

Axel Bruns, Abdul Karim Obeid, Daniel Angus, and James Meese

• 30 May 2022 – Paper presented at the International Communication Association conference, Paris

Prerecorded Video

Presentation Slides

Abstract

Introduction

This project contributes to our understanding of how recommender systems and search personalisation intersect with news distribution. We present a preliminary analysis from a major Australian study of the personalisation of Google News results, building on a large dataset of the search recommendations encountered by more than 1,000 users. Preliminary findings show that the news and information sources recommended for particular searches are predominantly influenced by the search topic, and can differ widely with the thematic context of the search; by contrast, the specific personal interests and attributes of the search engine user, such as their political orientation, appear to have very little influence over the selection of sources recommended to participants. These early observations offer important initial insights into the extent and the drivers of personalisation on Google News, and the extent to which it affects the presentation of news sources to users.

Background

Several studies have questioned whether recommender systems and personalisation algorithms present new challenges to core democratic functions associated with the news media (e.g. Sunstein, 2001; 2009; 2017). Scholars have also noted that these new systems may present challenges for normative concepts traditionally associated with media diversity, such as generating a marketplace of ideas, encouraging active citizenship, and boosting the representation of minority voices (Helberger, 2019). Recent work has debunked the moral panic over the societal impact of filter bubbles and echo chambers (Bruns, 2019). However, research on the actual function and impact of recommender systems on the diversity of the news content is still ongoing.

Recent studies have found that personalisation occurs much less than we may think. Using a small number of artificial Google accounts that were given unique personas or specific interests (Haim et al., 2018), an exploratory study in Germany found that there was very little evidence of personalisation in Google News. A similar study from the United States also found little variance in news diversity between study participants who self-reported the first five Google News results from searches for political figures (Nechushtai and Lewis, 2019). However, both studies note that Google News generally presents news from a limited selection of sources (Haim et al., 2018; Nechushtai and Lewis, 2019).

While they point to an overestimation of the impact of personalisation on the results provided by Google News, these studies remain limited by their reliance on small datasets produced from artificial user personas established for research purposes (Haim et al., 2018), or self-reporting by limited participant groups focussed on specific searches (Nechushtai and Lewis, 2019). Our study deploys a different methodology to study Google News search results at a considerably larger scale, and thereby generates more robust insights into the extent and nature of news personalisation in Australia.

Method

This project uses a browser plugin that queries a number of Google platforms (Google Search, Google News, Google Video, and YouTube) every 4 hours. This extends a previous study, conducted in Germany by partner investigators at AlgorithmWatch (Krafft et al., 2019). Once installed, the plugin triggers a sign-up process to obtain ethical consent and gathers basic demographic details. It does not collect any personally identifying information. We promoted this project via traditional and social media, and have attracted more than 1,000 users at time of writing in early November 2021. These citizen scientists have donated nearly 375 million individual search results since August 2021.

Available for the desktop version of Chrome, Firefox, and Edge, our plugin borrows the user’s online persona by periodically opening a new browser window, if the participant’s computer is switched on and the browser is already running. The plugin then initiates a series of queries to platforms, cycling through a set of search terms that can be varied over the lifetime of the project, and sends the search results back to our central database. These searches will produce organic search results that reflect the extent to which those results are personalised to the individual user. Our initial search queries included the names of key politicians and parties, and terms relating to current Australian and world events (COVID-19, climate change). This produces a database of search results over time that can be cross-referenced against specific demographic details in order to determine whether and how search results differ between specific user cohorts.

While this dataset lends itself to a range of analyses, our immediate focus in this paper is specifically to assess the diversity of Google News results with respect to the news sources being recommended to Australian users for particular fields of interest. In doing so we explore whether different user demographics are directed to different news sources (e.g. based on assumed geographic location or political leaning); whether such patterns vary with the theme of the search; and whether and to what extent the mix of recommended sources changes over time, and possibly especially in breaking news contexts when the informational situation remains unsettled and less reliable sources may temporarily receive greater attention.

Preliminary Results

We illustrate this approach here with a set of sample analyses that focus on the impact of our participants’ stated political (party) preferences on the news sources curated for them by Google News, for three selected search topics. In the Australian political context, key political ideologies are represented by the permanent Coalition of the Liberal and National Parties (conservative), the Labor Party (centrist), and the Greens (progressive). For the purposes of this analysis we have grouped our participants based on their stated preferences, retained a separate category for participants who chose not to state their political orientation and have excluded minor parties. For the purposes of this preliminary illustration, we provide raw figures.

Figure 1 shows that for the keyword 'COVID', the top news sources returned are the politically neutral national public broadcaster, ABC, the Australian edition of the progressive newspaper The Guardian, US cable news channel CNN. Search results are dominated by major, mainstream news outlets from Australia, the US, and the UK, with some official US and UK health advice also featured. There are few substantial differences between the sources recommended to participants with different party preferences; Labor supporters are somewhat more likely to encounter New York Times content and less likely to see BBC content, while non-aligned participants see more Guardian than ABC recommendations. Coalition supporters’ recommendations diverge somewhat more strongly from overall patterns: but rather than receiving disproportionately more recommendations of conservative news sources, they are largely directed to The Guardian and BBC content instead.

Fig. 1: most common news sources for keyword ‘COVID’, by party preference

For the search topic ‘Critical Race Theory’, which has been controversial in the US context but is also the subject of growing discussion in Australia, US news sources ranging from the centrist New York Times to the arch-conservative Fox News are prominent. The Australian-led global scholarly news outlet The Conversation also features notably here. For this topic, we see virtually no significant divergences in the sources recommended to participants of different party affinity: the ranking of major news sources is largely identical.

Fig. 2: most common news sources for keyword ‘Critical Race Theory’, by party preference

The leading news sources for this and for our final example indicates the substantial contextualisation of recommendations by topic. Figure 3 shows patterns for the search term ‘Uluru Statement’, a major initiative by Australian Indigenous leaders that calls for better engagement between Australian lawmakers and Indigenous groups. While The Guardian, The Sydney Morning Herald, and the ABC remain highly ranked in this context, news outlets that speak to Aboriginal and Torres Strait Islander achievement, advocacy, and reconciliation are prominent here. These include the Indigenous-run National Indigenous Times and The Conversation. The presence of the Australian Academy of Science (science.org.au) and the University of New South Wales, the latter again also demonstrates the importance of Indigenous and non-Indigenous scholarly voices in this debate.

Fig. 3: most common news sources for keyword ‘Uluru Statement, by party preference

These are indications of the patterns in our data, rather than definitive results. Our existing community of citizen scientist participants also underrepresent certain demographic groups. Future analyses will normalise these data against overall Australian population demographics in order to produce more representative analyses of recommendation patterns for different demographic groups. Future work will systematically develop these observations of source recommendations across the more than 40 search terms and phrases we have deployed so far, and for the full range of demographic features rather than only for political preferences.

Conclusion

This analysis has demonstrated significant differences in the news sources recommended by Google News sources depending on the topic. Such patterns diverge from the substantial overall uniformity that has been observed by previous smaller-scale studies such as those by Haim et al. (2018) or Nechushtai & Lewis (2019). Like these studies, however, so far we do not observe any major divergences in the recommendations presented to different demographic groups. Except for the presence of a number of topic-specific sources, the generalist news sources recommended by Google News predominantly represent mainstream Australian and international news outlets.

References

Australian Bureau of Statistics. (2016). 2016 Census Of Population and Housing. https://www.abs.gov.au/websitedbs/censushome.nsf/home/2016

Bruns, A. (2019) Are Filter Bubbles Real? Polity.

Galtung, J., & Ruge, M.H. (1965). The structure of foreign news: The presentation of the Congo, Cuba and Cyprus crises in four Norwegian newspapers. Journal of Peace Research, 2(1), 64-90.

Haim, M., Graefe, A., & Brosius, H.B. (2018). Burst of the filter bubble? Effects of personalization on the diversity of Google News. Digital Journalism, 6(3), 330-343.

Helberger, N. (2012). Exposure diversity as a policy goal. Journal of Media Law, 4(1), 65-92.

Helberger, N. (2019). On the democratic role of news recommenders. Digital Journalism, 7(8), 993-1012.

Krafft, T.D., Gamer, M., & Zweig, K.A. (2019). What did you see? A study to measure personalization in Google’s search engine. EPJ Data Science, 8(1), 1–23. https://doi.org/10.1140/epjds/s13688-019-0217-5

Napoli, P.M. (2011). Exposure diversity reconsidered. Journal of Information Policy, 1, 246-259.

Nechushtai, E., & Lewis, S.C. (2019). What kind of news gatekeepers do we want machines to be? Filter bubbles, fragmentation, and the normative dimensions of algorithmic recommendations. Computers in Human Behavior, 90, 298-307.

O'Hara, K., & Stevens, D. (2015). Echo chambers and online radicalism: Assessing the Internet's complicity in violent extremism. Policy & Internet, 7(4), 401-422.

Pariser, E. (2011). The Filter Bubble: What the Internet Is Hiding from You. Penguin.

Sunstein, C.R. (2001). Republic.com. Princeton University Press.

Sunstein, C.R. (2009). Republic.com 2.0. Princeton University Press.

Sunstein, C.R. (2017). #Republic: Divided Democracy in the Age of Social Media. Princeton University Press.

Thorson, K., & Wells, C. (2015). How gatekeeping still matters: Understanding media effects in an era of curated flows. In T.P. Vos & F. Heinderyckx (eds.), Gatekeeping in Transition (pp. 25–44). Routledge.