Introducing the ADM+S Australian Search Experience Project

I’ve not yet had the chance to write much about one of the major new projects I’m involved with: the ARC Centre of Excellence for Automated Decision-Making and Society (ADM+S), a large-scale, multi-institutional, seven-year research centre that investigates the impact of automated decision-making technologies (including algorithms, artificial intelligence, and other such technologies) on all aspects of our personal and professional lives. In particular, for the first year of the Centre I’ve led the News & Media Focus Area, which recently held its inaugural symposium to take stock of current research projects and plan for the future. (This was also the time for me to hand that leadership over to my colleagues Jean Burgess (QUT) and James Meese (RMIT), as I step back from that role to concentrate on another major project – more on this in a future update.)

Within News & Media, I’ve also led a major research project which we launched publicly in late July, and which is now producing first research outcomes: the Australian Search Experience. Inspired by an earlier project by our ADM+S partner organisation AlgorithmWatch in Germany, this project investigates the extent to which the search results Australian users encounter as they query search engines like Google are personalised and therefore differ from user to user; if they are, this would leave open the possibility of user being placed in so-called ‘filter bubbles’ – a concept which I’ve questioned in my recent book Are Filter Bubbles Real? We even have a promo video:

Investigating such personalisation is difficult: since every user is assumed to see a personalised set of search results, we need to compare these results across a large number of users in order to determine whether there is any significant personalisation, and what aspects of these users’ identities might drive such personalisation. While some studies approach this challenge by setting up a large number of ‘fake’ user accounts that are given a particular user persona by making them search repeatedly for specific topics that are expected to contribute to the search engine’s profile for the account, AlgorithmWatch’s earlier, German study took a different approach and invited a large number of real users to contribute as citizen scientists to the study. To do so, they were asked to install a browser plugin that regularly searched for a predefined set of keywords and reported the results back to AlgorithmWatch’s server.

Our ADM+S project uses this same data donation approach, but extends it further: we query four major search engines (Google Search, Google News, Google Video, and YouTube), and we are able to vary our search terms over the duration of the project. Like the earlier project, we also ask users to provide some basic demographic information (in order to link any systemic personalisation patterns we may encounter with those demographics), but never access any of our participants’ own search histories. Our browser plugin is available for the desktop versions of Google Chrome, Mozilla Firefox, and Microsoft Edge, and I’m pleased to say that more than 1,000 citizen scientists have now installed the plugin.

If you’re based in Australia, and you’d like to contribute to the project, please go to the project Website to install the browser plugin. We’d love to get to 1,500 citizen scientists before the end of 2021.

For more background, I spoke to QUT’s Real World News earlier this year to explain the approach we’ve taken in developing this project:

Thanks to the data donations made by our community of citizen scientists so far, the project has been generating data since late July, and we took the News & Media Symposium in late September as an opportunity to review some of the patterns that are emerging from this dataset. Like the earlier AlgorithmWatch project in Germany, we are not seeing any evidence of detailed personalisation for search engines like Google Search or Google News at this stage; any personalisation in the results our participants have encountered seems limited to their geographic location, which makes sense especially for highly state-specific information (for instance on COVID-19 vaccination sites, lockdowns, or quarantine rules). The story seems somewhat different for YouTube results, though, where the first few results appear to be relatively uniform, but content further down the results page appear to differ more markedly. I’m reluctant to read too much into these results until we’ve had the chance to do considerable further analysis, but they point to considerably different patterns even between the nominally similar Google Video and YouTube search engines.

For more, see my presentation in our session on the Australian Search Experience project at the ADM+S News & Media Symposium on 30 September 2021 (which begins with a discussion of the broader logic behind such data donation projects by AlgorithmWatch co-founder Matthias Spielkamp):

And if you’re curious about the rest of the News & Media Symposium, video recordings of the full programme of public presentations from 30 September and 1 October 2021 are now also available from the ADM+S Website.