Using GPT-4 for Text Analysis: Insights from English and German Language News Classification Tasks
Journal
REAL-Info 2024: First Workshop on Reliable Evaluation of LLMs for Factual Information
Type
conference contribution
Date Issued
2024-06-01
Author(s)
Abstract
Large language models are rapidly becoming an essential tool for social scientists. In particular, they have the potential to completely change the way researchers approach text analysis. In this study, we use the GPT-4 model to classify the content of newspaper articles and assess their sentiment. To do this, we collect headlines and leads from U.S. and German newspapers (n = 1,629) on how generative AI is represented in major news media outlets in both countries and inductively develop coding instructions based on this data. We then feed the data and instructions to GPT-4 and a human coder to compare their outputs and assess validity. We find that the coding procedure is highly reliable, with substantial to near perfect agreement between the human coder and GPT-4. We also find that reliability decreases for more complex constructs and is modestly lower for classification tasks performed in German than in English. Based on this analysis, we argue that LLMs offer powerful new approaches to text analysis that cross methodological divides between qualitative and quantitative approaches to empirical text analysis.
Language
English (United States)
HSG Classification
contribution to scientific community
Event Title
The 18th International AAAI Conference on Web and Social Media
Event Location
Jacobs School Of Medicine And Biomedical Sciences, Buffalo, NY
Event Date
June 3rd – 6th 2024