LLMs in Content Coding: The 'Expertise Paradox' and Other Challenges

And the final speaker in this final AoIR 2024 conference session is the excellent Fabio Giglietto, whose focus is on coding Italian news data using Large Language Models. This worked with some 85,000 news articles shared on Facebook during the 2018 and 2022 Italian elections, and first classified such URLs as political or non-political; it then produced and clustered text embeddings for these articles, and used GPT-4-turbo to classify the dominant topics in these clusters.

This required considerable prompt crafting, especially also to ensure that prompts remained within the LLM’s token limits. Key challenges here included the choice of LLM, the approaches to clustering articles into topics, the validation of the coherence of clustering results, and the process for the LLM-based labelling of clusters and its validation.

These challenges group into three broad categories: first, LLMs are general-purpose tools, and this complicates the validation of the results they produce. Second, LLMs can produce anything from very general to very granular results, and to instruct them to produce results at a specific level of granularity is a non-trivial challenge. Third, because of their training on very large datasets LLMs can surpass human coders in their level of expertise – this ‘expertise paradox’ might mean that human-coded data do not always represent a gold-standard ground truth.

And that’s the end of the final session. Thanks so much as always especially to the conference hosts and organisers, who did an awesome job wrangling this large and diverse conference on the occasion of its 25th anniversary! Clearly, AoIR is in good hands.