Skip to main content
Home
Snurblog — Axel Bruns

Main navigation

  • Home
  • Information
  • Blog
  • Research
  • Publications
  • Presentations
  • Press
  • Creative
  • Search Site

LLMs in Content Coding: The 'Expertise Paradox' and Other Challenges

Snurb — Saturday 2 November 2024 22:34
Elections | Polarisation | Journalism | Industrial Journalism | Internet Technologies | 'Big Data' | Artificial Intelligence | AoIR 2024 |

And the final speaker in this final AoIR 2024 conference session is the excellent Fabio Giglietto, whose focus is on coding Italian news data using Large Language Models. This worked with some 85,000 news articles shared on Facebook during the 2018 and 2022 Italian elections, and first classified such URLs as political or non-political; it then produced and clustered text embeddings for these articles, and used GPT-4-turbo to classify the dominant topics in these clusters.

This required considerable prompt crafting, especially also to ensure that prompts remained within the LLM’s token limits. Key challenges here included the choice of LLM, the approaches to clustering articles into topics, the validation of the coherence of clustering results, and the process for the LLM-based labelling of clusters and its validation.

These challenges group into three broad categories: first, LLMs are general-purpose tools, and this complicates the validation of the results they produce. Second, LLMs can produce anything from very general to very granular results, and to instruct them to produce results at a specific level of granularity is a non-trivial challenge. Third, because of their training on very large datasets LLMs can surpass human coders in their level of expertise – this ‘expertise paradox’ might mean that human-coded data do not always represent a gold-standard ground truth.

And that’s the end of the final session. Thanks so much as always especially to the conference hosts and organisers, who did an awesome job wrangling this large and diverse conference on the occasion of its 25th anniversary! Clearly, AoIR is in good hands.

  • 383 views
INFORMATION
BLOG
RESEARCH
PUBLICATIONS
PRESENTATIONS
PRESS
CREATIVE

Recent Work

Presentations and Talks

Beyond Interaction Networks: An Introduction to Practice Mapping (ACSPRI 2024)

» more

Books, Papers, Articles

Untangling the Furball: A Practice Mapping Approach to the Analysis of Multimodal Interactions in Social Networks (Social Media + Society)

» more

Opinion and Press

Inside the Moral Panic at Australia's 'First of Its Kind' Summit about Kids on Social Media (Crikey)

» more

Creative Work

Brightest before Dawn (CD, 2011)

» more

Lecture Series


Gatewatching and News Curation: The Lecture Series

Bluesky profile

Mastodon profile

Queensland University of Technology (QUT) profile

Google Scholar profile

Mixcloud profile

[Creative Commons Attribution-NonCommercial-ShareAlike 4.0 Licence]

Except where otherwise noted, this work is licensed under a Creative Commons BY-NC-SA 4.0 Licence.