Exploring Effective Persuasion Using LLMs

The next speaker in this ACSPRI 2024 conference is Gia Bao Hoang, whose interest is in the use of LLMs for detecting efficient persuasion in online discourse. Such an understanding of effective persuasion could then be used for productive and prosocial purposes, or alternatively to identify problematic uses of persuasion by bad actors.

For this analysis, Bao is using data from the Change My View subreddit, where users clearly indicate whether the arguments made have actually changed their minds; and the Truth Wins dataset, which stems from human experiments and contains human-labelled data from persuasion and attention games. He used term frequency vectorisation to develop a logistic regression baseline and applied a Llama3 zero-shot classifier to predict the change in belief in the original poster in the Change My View dataset. (There’s more here, but I’m not completely following the process.)

Accuracy of this analysis is decent enough, at 70%; the analysis shows up a number of key features (shareable, interesting if true, positive, persuasive, and interesting) that appear to aid persuasion, and their combination potentially further amplifies the effect; actual truthfulness appears to matter much less. Positive emotions enhance shareability; interest and surprise increase shareability and perceived credibility. There are promising signs for future analysis here, therefore.