The next speaker in this session at the AoIR 2025 conference is my QUT colleague Tariq Choucair, presenting a comprehensive review of the state of the art in the use of LLMs in content analysis research. This review focusses on the use of LLMs to analyse human-generated text, understood as a social phenomenon. This extends past reviews of computational text analysis methods that were published in previous years: the considerable growth in LLM use has made it necessary to return to the recent literature to examine how computational methodologies have evolved since then.
Past methods have enabled the analysis of large-scale human-generated textual data (e.g. from social media platforms) with computational tools, and thereby to track changes in communication patterns and enable the use of new and large data sources in media and communication research. But such methods also needed to be queried for the validity of their results, their fit for purpose against the needs of such research, and their overemphasis on English-language data.
The emergence of Large Language Models over the past few years has substantially changed this field; what, then, are the current trends in LLM-assisted computational research? How and where are LLMs being integrated into such research efforts? This review examines the type of content analysis tasks, their integration into analytical workflows, the interdisciplinary innovations in the field, and the questions around reliability, validity, interpretability, agency, and epistemic authority.
The study captured relevant articles from five scholarly publication databases (including arXiv preprints) and manually validated their relevance to the topic. Of some 34,000 articles that were initially identified, some 17,000 were relevant and published after 2017, when attention first began to turn to the potentials of Large Language Models. This was further refined to filter only for articles that were highly relevant to the topic, and used generative AI systems – this left some 1,434 articles that were of central relevance here.
The coding scheme for these articles was developed through an iterative process. It examines the content analysed using LLMs, the variables applied to this content, and the analytical pipeline; future work will also example the validation processes, and authors’ critical engagement with LLM uses. These articles appeared in a broad range of journals, variously covering computer science, neural computing, data science, linguistics, educational technology, public health, and business, with computational social science at the centre of the network of topics. There was a substantial growth in such articles from the early 2020s onwards; publication volumes took off in 2023 and was substantially larger still in 2024.
Content analysed via LLMs was very diverse: it included news, academic content, benchmark datasets, interviews, and especially social media content, but also many other smaller categories. Variables examined included task and quality assessment, content, sentiment, rhetoric style, and many other aspects. LLMs were used variously as preprocessing engines in a multi-stage pipeline; as a subject of evaluation (to see how well they performed in assessment tasks); as a stand-alone analytical tool; as assistants in human-led workflows; and as generators for synthetic data to be analysed.
