AoIR 2024

Challenges in Acquiring and Analysing News Data at Scale: A Case Study of News Polarisation in Australian Climate Change Coverage

Katharina Esau, Axel Bruns, Laura Vodden, Michelle Riedlinger, Samantha Vilkins, Laura Lefevre, and Carly Lubicz-Zaorski

• 2 Nov. 2024 – AoIR 2024 conference, Sheffield

Presentation Slides

Abstract

Introduction

At a time when the threat of climate change and the urgency of action are unmatched, the role of the news media industry in shaping public discourse and policy cannot be overstated. Sustainable solutions must, by definition, address the broader interests of humanity, not just of those with substantial resources (Malekpour et al., 2023). However, news media coverage is influenced not only by the newsworthiness of issues and claims, but also by who has the necessary political and financial standing to be mentioned in the news, reflecting networks of power and the profit motives of media organisations (Bennett, 1990; Splendore, 2020; Wichgers et al., 2021). Furthermore, news reporting can be partisan and a driver of societal polarisation (e.g., Feldman et al., 2017; Müller et al., 2017). News is not purely a representation of reality; the voices that get to speak and the ways their stories are told in the news impact on our lived realities. News shapes our perception of social reality, and news content is thus crucial for understanding current levels of societal polarisation.

Against this background, this study proposes and evaluates a framework to analyse news polarisation within national media landscapes, specifically applying it to climate change coverage by Australian professional news media. Our study recognises Australia's unique challenges: patterns of extreme weather (Bradshaw et al., 2024) and corresponding public attention (Crellin & MacNeil, 2023), vulnerability to global warming across environmental (Legge et al., 2020) and social impacts (e.g., on health: Lansbury et al., 2020; and social justice: Porter et al., 2020), an increasingly concentrated media landscape dominated by Rupert Murdoch's News Corporation (Environment and Communications References Committee, 2021), and the presence of outright climate change denialism among political and corporate elites (Lucas, 2021). Despite 87% of Australians believing that climate change should be a governmental priority (Bradley et al., 2022), the topic remains ensnared in a highly polarised public discourse, marked by political disagreement and disengagement. Political leanings influence Australians’ expectations of news coverage on climate change, with left-leaning audiences favouring activist journalism and right-leaning ones seeking diverse viewpoints (Park et al., 2022). This leads to our central research question:

How polarised is the Australian news media landscape in its coverage of climate change?

Types of News Media Polarisation

Despite its relevance in public debate, the concept of polarisation is poorly defined in media and communication research. Different forms or types of polarisation (e.g., political, ideological, issue, positional, policy, affective) have been identified in media contexts (e.g., Arguedas et al., 2022; Hart et al., 2020), but these sometimes overlap or are used interchangeably. Building on theoretical approaches from political science and media and communication studies, we examine four types of polarisation that we suggest are critical to determining news media polarisation: source polarisation (which actors get to speak?), positional polarisation (what positions are represented?), value polarisation (what societal and/or individual values are explicitly or implicitly privileged?), and affective polarisation (what emotions are attached to different actors and perspectives?).

In order to address these questions, the empirical component of this paper also addresses the profound challenge of news data availability at scale, a significant obstacle for news research well beyond our specific area of interest. It tackles this issue through a systematic comparison of leading digital news databases—GDELT, MediaCloud, GNews, NewsAPI, NewsDataIO, Factiva, and ProQuest—that evaluates key attributes such as the diversity of media outlet coverage, wildcard search capabilities, access to historical archives, full-text data export capabilities for computational analysis, associated costs, and quality metrics, such as data completeness.

This represents the first comprehensive effort to address the issue of fluid news content in today's digital media landscapes (e.g., Karlsson, 2012). Our findings reveal that—despite the fulsome claims of commercial news data vendors—no single database can provide a comprehensive news content sample representative of a national media landscape like Australia's, a limitation that extends to other Western democracies. This difficulty in obtaining a reliable news sample is a fundamental barrier to the systematic assessment of news polarisation, as well as for other studies that require large-scale, full-text news data access. The potential consequences of this are clear in the Australian case, where the combination of limited media diversity and both restricted and obscured news access could lead to disproportionately impacted findings, particularly when the balance between public-access and paywalled journalism is unevenly distributed across a polarised political spectrum.

Standardised Manual and LLM-Supported Content Analysis

Using both algorithmic and manual methods, we have compiled a hybrid dataset from NewsDataIO, Factiva, and ProQuest, and conducted a content analysis of climate change coverage across major mainstream and alternative Australian media outlets. Through this content analysis, we introduce and assess a multi-dimensional framework for the study of news polarisation, covering source, positional, value, and affective polarisation. This framework provides an innovative, in-depth understanding of how news content varies across political and financial power structures, ideological positions, emotional responses, and social and individual value systems.

We operationalise this framework through an innovative mixed-methods approach that builds on standardised manual content coding of a subset of the data and extends it to a much larger dataset by training a Large Language Model (LLM) to code the remainder of the dataset. In taking this approach we also evaluate the capability of current LLMs to augment human content coding efforts. In doing so we test for and iteratively improve intercoder reliability both between human and LLM coding as well as between multiple distinct repetitions of the LLM coding process, in order to reach acceptable levels of coding agreement. The paper documents this process, and thereby also contributes to advancing this new frontier in mixed-methods content analysis.

What results from this effort is a coded dataset of news content, at scale, across four dimensions of polarisation in news content. This enables us to identify the patterns of commonalities and cleavages in the coverage of a given issue (here, climate change) between and across different news outlets in Australia, and to trace the evolution of such patterns over time. This approach is also translatable to the study of news polarisation on other societal issues beyond climate change.

Different outlets (or groups of outlets, defined for instance by their organisational affiliations with the same media group) may turn out to favour distinct approaches to the coverage of climate change: variously minimising or exaggerating its impact; supporting action at the domestic or international level; or promoting industrial or societal responses. They may also feature distinct sets of sources (politicians; activists; scientists; industry) in their coverage. We interpret these patterns against the backdrop of past research into the Australian media landscape and its economic and political structure (e.g., Cunningham & Turnbull, 2020), and assess whether they serve as evidence of polarisation in the outlets’ approaches to their reporting, in order to document and evaluate the Australian news media’s performance on the coverage of climate change and its impacts.

Our study thus conceptualises and measures polarisation in news content across a selection of major mainstream and alternative Australian news outlets, thereby assessing the ability of contemporary news to inform, represent, and serve its intended publics. The case study of climate change coverage contributes to the broader discourse on news media polarisation by offering empirical evidence on whether and how news content, influenced by economic and political pressures, may perpetuate and exacerbate divisions within society.

References

Arguedas, R. Craig, A., Robertson, T., Fletcher, R., & Kleis Nielsen, R. (2022). Echo chambers, filter bubbles, and polarisation: A literature review. Reuters Institute for the Study of Journalism. https://reutersinstitute.politics.ox.ac.uk/sites/default/files/2022-01/Echo_Chambers_Filter_Bubbles_and_Polarisation_A_Literature_Review.pdf

Bennett, W. L. (1990). Toward a theory of press-state relations in the United States. Journal of Communication, 40(2), 103–127.

Bradley, G., Deshpande, S., Foxwell-Norton, K., Hennessey, N., & Jackson, M. (2022). Climate action survey summary for policy and decision making. Griffith Climate Action Beacon. https://www.griffith.edu.au/__data/assets/pdf_file/0024/1538304/Climate-Action-Survey-Summary-for-Policy-and-Decision-Making.pdf

Bradshaw, S., Hughes, L., Mullins, G., Karoly, D., Gergis, J., & Arndt, D. (2024) Climate whiplash: Wild swings between weather extremes. Climate Council. https://www.climatecouncil.org.au/wp-content/uploads/2024/02/CC_MVSA0392-CC-Report-Mid-Summer-2024_V9-FA-Screen-Single.pdf

Crellin, C. & MacNeil, R. (2023). Extreme weather events and public attention to climate change in Australia. Climatic Change, 176(121), 1–7.

Cunningham, S., & Turnbull, S. (Eds.). (2020). The Media and Communications in Australia (4th ed.). Routledge.

Feldman, L., Hart, P. S., & Milosevic, T. (2017). Polarizing news? Representations of threat and efficacy in leading US newspapers’ coverage of climate change. Public Understanding of Science, 26(4), 481–497.

Hart, P. S., Chinn, S., & Soroka, S. (2020). Politicization and polarization in COVID-19 news coverage. Science Communication, 42(5), 679–697.

Karlsson, M. (2012). Charting the liquidity of online news: Moving towards a method for content analysis of online mews. International Communication Gazette 74(4), 385–402.

Lansbury Hall, N., & Crosby, L. (2022). Climate change impacts on health in remote Indigenous communities in Australia. International Journal of Environmental Health Research, 32(3), 487–502.

Legge, S., Rumpff, L., Garnett, S. T., & Woinarski, J. (2023). (2023). Loss of terrestrial biodiversity in Australia: Magnitude, causation, and response. Science, 381(6658), 622–631.

Lucas, A. (2021). Investigating networks of corporate influence on government decision-making: The case of Australia’s climate change and energy policies. Energy Research & Social Science, 81, 1–26.

Malekpour, S., Allen, C., Sagar, A., Scholz, I., Persson, Å., Miranda, J. J., Bennich, T., Dube, O. P., Kanie, N., Madise, N., Shackell, N., Montoya, J. C., Pan, J., Hathie, I., Bobylev, S. N., Agard, J., ... Al-Ghanim, K. (2023). What scientists need to do to accelerate progress on the SDGs. Nature, 621(7978), 250–254.

Müller, P., Schemer, C., Wettstein, M., Schulz, A., Wirz, D. S., Engesser, S., & Wirth, W. (2017). The polarizing impact of news coverage on populist attitudes in the public: Evidence from a panel study in four European democracies: Polarizing news effects on populist attitudes. Journal of Communication, 67(6), 968–992.

Park, S., McGuinness, K., Fisher, C., Lee, J. Y., McCallum, K & Nolan, D. (2022). Digital news report Australia 2022. News & Media Research Centre. https://apo.org.au/sites/default/files/resource-files/2022-06/apo-nid317946_0.pdf

Porter, L., Rickards, L., Verlie, B., Bosomworth, K., Moloney, S., Lay, B., Latham, B., Anguelovski, I., & Pellow, D. (2020). Climate justice in a climate changed world. Planning Theory & Practice, 21(2), 293–321.

Environment and Communications References Committee (2021). Media diversity in Australia.https://parlinfo.aph.gov.au/parlInfo/download/committees/reportsen/024602/toc_pdf/MediadiversityinAustralia.pdf

Splendore, S. (2020). The dominance of institutional sources and the establishment of non-elite ones: The case of Italian online local journalism. Journalism, 21(7), 990–1006.

Wichgers, L., Jacobs, L., & van Spanje, J. (2021). The battle of frame building: The reciprocal relationship between journalists and frame sponsors. The International Journal of Press/Politics, 26(3), 674–698.