Human vs. LLM Coding of Australian Charities’ Civic Activities

The final speaker in this ACSPRI 2024 conference session is Aaron Willcox, presenting work with the Scanlon Research Institute to explore local government-level civic opportunities. For organisations, such opportunities include hosting events, offering memberships, involving individuals through volunteering, and taking action through advocacy and campaigns.

The project used the Australian Charities and Not-for-Profits Commission database, which contains valid information on the activities of some 30,000 charities around the country; it used Web scraping and human coding to identify civic activities, and utilised Large Language Models to directly code the data as well as to emulate the coding process of human coders. In each case, coding produced a true/false rating for each of the four types of civic opportunities.

This showed that taking action was the least common activity; events were most prominent, and memberships and volunteering were also frequent. Overall, though, LLM results were limited in quality; human coders didn’t always agree with each other either, though.

The analysis also showed a substantial number of charities in Australia dedicated to advancing religion as well as social and public welfare; by contrast, areas such as reconciliation have far fewer charities operating in the country. Human rights and legal advocacy charities have much higher civic index ratings, however, while religious charities rate far lower here.