AoIR 2025
Extending Our Capabilities: Towards LLM-Assisted Frame Analysis of Australian Climate Movement News Coverage
Laura Vodden, Katharina Esau, Axel Bruns, Tariq Choucair
- 16 Oct. 2025 – Paper presented at the 2025 Association of Internet Researchers conference, Niterói, Rio de Janeiro
Presentation Slides
Abstract
Introduction
Frame analysis is a content analysis method that seeks to understand how narratives are shaped and presented to audiences, via their framing in communication. Manual frame analysis is a time- and labour-intensive task, which limits the scope of empirical research (Kuang et al., 2024; Walter & Ophir, 2019). These constraints along with the increasing volume of text-based data have prompted researchers to employ computational methods to analyse data at scale (Kermani et al., 2023; Kroon et al., 2023). Computational methods offer the advantage of processing large volumes of media content, allowing for broader and more comprehensive analyses (Matthes & Kohring, 2008), but currently do not sufficiently capture media frames (Ali & Hassan, 2022), which is a core concept in political, media and communication studies.
Large Language Models (LLMs) have the potential to bridge this gap and expand the scope of content analysis in general and frame analysis specifically. Several authors (e.g. Alizadeh et al., 2025) have identified the potential value of LLMs in frame analysis, but presently there is no established methodology available for applying LLMs to analyse framing in the news. Our pilot study develops a methodology to incorporate LLMs to aid researchers in analysing the framing of climate movements in news coverage. We find that LLMs can be used to inductively build frames from news content, and, additionally, enhance manual approaches to frame analysis.
We selected 12 mainstream and alternative Australian media outlets with national reach. Articles were sourced from ProQuest between January 2019 and October 2023, containing at least one of the following key terms: “Fridays for Future” OR “School Strike 4 Climate” OR “School Strike for Climate” OR “Extinction Rebellion”. This search yielded a corpus of 3,183 articles. From this corpus, we randomly selected 12 articles (one per media outlet), ensuring coverage across different time periods, for manual coding, while setting the remainder aside for LLM-assisted coding.
Five human coders applied an established qualitative frame analysis approach to the sampled articles, using Entman’s (1993) frame elements and an existing coding scheme from Lichtenstein and Esau (2016). Coders analysed each statement in the articles to identify the speaker (journalist, cited person, or organisation) and recorded any problem definitions, causes, blame attributions, proposed solutions, and solution addressees. The coders then discussed and synthesised their findings to produce a coherent list of the frames present in the articles and used by particular speakers. These synthesised frames were further refined into a distinct set of frame descriptions.
We then applied Meta’s Llama-3 language model to replicate this analysis. Following a framework for LLM-assisted content analysis developed by Chew et al. (2023), we iteratively refined the prompt, ran the model, qualitatively compared its results to our manually identified frames, and adjusted the prompt as needed (see Figure 1).
In keeping with the process undertaken by the human coders, we divided the process into two stages, requiring two separate prompts. We structured the first prompt to extract the frame elements from the news text by outlining the overall task and providing a summary of frame analysis and a small number of examples (few-shot prompting), then breaking the task further into subtasks (e.g., identifying all problem definitions and corresponding causes and blame attributions from the text). The second prompt contained instructions to synthesise the output of the first prompt, to arrive at a list of frames. We instructed the LLM to withhold responses when uncertain and to provide supporting evidence by extracting relevant excerpts directly from the news articles, fostering traceability throughout the process. Finally, we specified a standardised output format—JSON—to facilitate structured analysis. We verified the existence of each LLM-generated frame within the relevant news article, and compared the frames against those produced by the human codes.
While the human experts identified 13 distinct frames in total, the LLM identified 12. Of these, nine common frames were identified by both the humans and the LLMs. The human coders identified four frames that the LLM did not, and the LLM identified three frames that the human coders did not. The LLM was generally more succinct in its construction of frames than the human experts, but the LLM struggled to identify frames that were specific to a particular issue, and as such the LLM-generated frames were more general in their scope than those identified by the human experts. The human experts did not all construct frames consistently; the style of language and depth of the frames varied widely between human coders, and further work on this project will see our human researchers benefit from a second round of coder training, to address possible inconsistencies in defining a frame and possibly allow for improvement of LLM-generated results as a result of refining the overall approach to this frame analysis.
Table 1: Frames identified by human coders, and by the LLM, showing where responses overlapped and diverged.
Despite these teething issues, our results indicate the future utility of LLMs in inductively generating frames from news articles. The next step is to refine our master list of frames using the LLM to analyse a much larger corpus using these frames as a guide—offering benefits in terms of increased scale and reduced time that would otherwise be spent on manual coding. We will also conduct further systematic comparisons of frame detection performance between different LLM systems and models, and between zero- and fewshot prompting approaches.
While this study deals with climate change, we also plan to extend this methodology to other topics that are part of our ongoing research project (e.g., transgender communities rights, indigenous rights, abortion). Regardless of topic, LLMs may identify frames within a text that researchers, due to their own biases and preconceptions, or the limitations of coder training, may overlook or misinterpret, taking the use of LLMs in research beyond simply extracting information from content, and towards improving human-led research by causing us to question our own approaches to doing research.
References
Ali, M., & Hassan, N. (2022). A survey of computational framing analysis approaches. In Y. Goldberg, Z. Kozareva, & Y. Zhang (Eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (pp. 9335–9348). Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.emnlp-main.633
Alizadeh, M., Kubli, M., Samei, Z. et al. (2025). Open-Source LLMs for Text Annotation: A Practical Guide for Model Setting and Fine-Tuning. Journal of Computational Social Science, 8(17). https://doi.org/10.1007/s42001-024-00345-9
Chew, R., Bollenbacher, J., Wenger, M., Speer, J., & Kim, A. (2023). LLM-assisted content analysis: Using large language models to support deductive coding. arXiv. http://arxiv.org/abs/2306.14924
Entman, R. M. (1993). Framing: Toward clarification of a fractured paradigm. Journal of Communication, 43(4), 51–58. https://doi.org/10.1111/j.1460-2466.1993.tb01304.x
Kermani, H., Makou, A. B., Tafreshi, A., Ghodsi, A. M., Atashzar, A., & Nojoumi, A. (2023). Computational vs. qualitative: Analyzing different approaches in identifying networked frames during the Covid-19 crisis. International Journal of Social Research Methodology. https://doi.org/10.1080/13645579.2023.2186566
Kroon, A., Welbers, K., Trilling, D., & Van Atteveldt, W. (2023). Advancing automated content analysis for a new era of media effects research: The key role of transfer learning. Communication Methods and Measures. https://doi.org/10.1080/19312458.2023.2261372
Kuang, X., Liu, J., Zhang, H., & Schweighofer, S. (2024). Towards Algorithmic Framing Analysis: Expanding the Scope by Using LLMs. Research Square. https://doi.org/10.21203/rs.3.rs-5032053/v1
Matthes, J., & Kohring, M. (2008). The content analysis of media frames: Toward improving reliability and validity. Journal of Communication, 58(2), 258–279. https://doi.org/10.1111/j.1460-2466.2008.00384.x
Walter, D., & Ophir, Y. (2019). News frame analysis: An inductive mixed-method computational approach. Communication Methods and Measures, 13(4), 248–266. https://doi.org/10.1080/19312458.2019.1639145
