SEASON 2025

Assessing Recommendation Diversity in Search Results: Approaches Using Data Donations and Artificial Personas

Axel Bruns, Daniel Angus, Ashwin Nagappa, Kateryna Kasianenko, Abdul Obeid, Shir Weinbrand, and Brett Tweedie

• 25 Sep. 2025 – Paper presented at the SEASON 2025 conference, Hamburg

Presentation Slides

Abstract

1. Introduction: The Australian Search Experience

This paper reports on two stages of the Australian Search Experience (ASE) research project, conducted under the auspices of the ARC Centre of Excellence for Automated Decision-Making and Society (ADM+S). The project responds to persistent concerns about the existence of ‘filter bubbles’ (Pariser, 2011) as caused by a personalisation of search results to the specific interests of search engine users, which would lead them to encounter widely divergent representations of reality and thereby threaten the shared informational basis that is fundamentally required for functioning societies. It addresses this idea by conducting systematic large-scale empirical research into the search results actually encountered by ordinary Australian users of the search functionality provided by Google Search, Google News, Google Video, and YouTube, for searches relating to key political figures, organisations, and topics, as well as other current events and issues.

The project operates in two phases. Phase 1, conducted in 2021 and 2022, built on and extended an approach pioneered by AlgorithmWatch in Germany (Krafft et al., 2019): both projects invited ordinary users to become involved in the research by installing a browser plugin that would regularly query relevant search engines for a number of preset search terms (such as the names of politicians and parties, and in Australia also current terms like COVID, vaccine, lockdown, etc.). In Australia, ASE managed to attract nearly 1,000 users as participants, and over the course of 10 months received some 4.85 million search engine result data donations from these participants, containing nearly 42 million individual results.

As this paper will show, analyses of this Phase 1 dataset reveal a considerable uniformity in search results even across a diverse range of participants (cf. Bruns, 2022; Meese et al., 2024); search results personalisation was encountered only in a number of highly specific contexts – searches for ‘vaccine’ would produce localised information on vaccination centres close to the searcher’s location, for instance. While search engines clearly have the ability to personalise results to the presumed identity of the searcher, then, other than in a distinct set of circumstances they appear not to exercise that ability, counter to the ‘filter bubble’ supposition (cf. Bruns, 2019).

Phase 1 of the Australian Search Experience project also remains limited by its data donation approach, however: for practical and ethical reasons, much like the German AlgorithmWatch project it could use only a limited range of highly generic search terms which do not cover the full breadth of possible approaches to information searching that ordinary users would employ. For instance, as search terms deployed by the browser plugin would end up in participants’ search histories, we could not search for more controversial topics like COVID-19 conspiracies or anti-vaccination content. Having determined in Phase 1 that search result personalisation for identical keywords is largely absent, therefore, ASE Phase 2 – which commenced in 2024 – takes a markedly different approach to the study of search result diversity: here, we use a combination of human input and artificial intelligence to generate a wide variety of potential search queries for broadly similar topics (e.g. from “are vaccines harmful?” to “how often should I get vaccinated?”), and deploy these queries regularly from a fleet of virtual machines whose browsing histories have been designed to represent a variety of distinct personas that might trigger any residual search results personalisation deployed by the search platforms.

2. Analytical Approach

This paper presents a detailed analysis of the results from Phase 1 of the Australian Search Experience, and of emerging results from Phase 2. To begin with, we focus on the organic search results displayed by Google Search: taking the first page of search results in ranked order, as reported by our data donors, we convert this information into a results vector, and engage in a systematic pairwise comparison of these results vectors per search term, participant, and day. This enables us to generate an overall results similarity rating per search term and day, and trace this similarity rating over time (identifying key points at which the relative uniformity of results is disrupted by external events or internal platform changes); as well as to cluster participants and their reported results by similarity, and identify any outlier groups with distinctly different results (e.g. participants who have set their default browser environment to a language other than English). 

Figure 1: Average search results similarity rating per day for the search term ‘Scott Morrison’ in October 2021

Fig. 1 illustrates this for the search term “Scott Morrison” (Australia’s then Prime Minister), during October 2021: it shows the average cosine similarity value for all ranked lists of search results donated on each day in October, fluctuating around the monthly average of 0.91 on a scale from 0 (entirely dissimilar) to 1 (same results in the same order). Notable drops in similarity are experienced on 15 October, when various news reports of a major press conference that day intrude into the otherwise static list of generic background information on Morrison, and for several days from 23 October, when the number of organic search results displayed by Google is temporarily reduced to make way for other information boxes. Overall, too, for more than two thirds of all daily pairwise comparisons between our data donors, the average pairwise similarity rating is above 0.90.

For Phase 2 of the project, which uses a broader range of distinct search queries, we are then able to build on this approach in order to assess the similarities and divergences in search results for related but non-identical queries on the same topic. Here, we are especially interested in determining at which point and how strongly broad results similarity declines: while “Scott Morrison achievements” and “Scott Morrison legacy” might produce similar responses, for instance, “Scott Morrison controversies” and “Scott Morrison failures” are likely to diverge more markedly. Differences here point to the continuing centrality of user literacy and agency: while the techno-determinist thesis that search engines place their users into individualised ‘filter bubbles’ has by now been thoroughly disproven (also see Haim et al., 2018; Nechushtai & Lewis, 2019; Nechushtai et al., 2023), the ability to effectively query search engines for relevant content remains unevenly distributed, and users’ own pre-existing preferences and attitudes can still result in substantive preferential attachment to and/or selective avoidance of information that aligns or disagrees with their own worldviews. Phase 2 is now underway and will have results to present by the time of the SEASON 2025 conference.

3. Acknowledgements

This research is supported by the Australian Research Council through the ARC Centre of Excellence for Automated Decision-Making and Society.

4. References

Bruns, A. (2019). Are Filter Bubbles Real? Polity.

Bruns, A. (2022). Australian Search Experience Project: Background Paper. Working Paper No. 1. ARC Centre of Excellence for Automated Decision-Making and Society. https://doi.org/10.25916/k7py-t320

Haim, M., Graefe, A., & Brosius, H.-B. (2018). Burst of the Filter Bubble? Effects of Personalization on the Diversity of Google News. Digital Journalism, 6(3), 330–343. https://doi.org/10.1080/21670811.2017.1338145

Krafft, T. D., Gamer, M., & Zweig, K. A. (2019). What Did You See? A Study to Measure Personalization in Google’s Search Engine. EPJ Data Science, 8(1), Article 1. https://doi.org/10.1140/epjds/s13688-019-0217-5

Meese, J., Obeid, A. K., Angus, D., Bruns, A., & Srinivas, A. (2024). Examining Exposure Diversity on Google News in Australia. Journal of Quantitative Description: Digital Media, 4. https://doi.org/10.51685/jqd.2024.019

Nechushtai, E., & Lewis, S. C. (2019). What Kind of News Gatekeepers Do We Want Machines to Be? Filter Bubbles, Fragmentation, and the Normative Dimensions of Algorithmic Recommendations. Computers in Human Behavior, 90, 298–307. https://doi.org/10.1016/j.chb.2018.07.043

Nechushtai, E., Zamith, R., & Lewis, S. C. (2023). More of the Same? Homogenization in News Recommendations When Users Search on Google, YouTube, Facebook, and Twitter. Mass Communication and Society, 27(6), 1309–1335. https://doi.org/10.1080/15205436.2023.2173609

Pariser, E. (2011). The Filter Bubble: What the Internet Is Hiding from You. Penguin.