The Promise of Evaluative Algorithmic Advice: A Field Experiment on Writing Improvement
Journal
Academy of Management (AOM) Meeting
Type
conference paper
Date Issued
2024-08-09
Author(s)
Research Team
IWI6
Abstract
The design and impact of algorithmic advice has become more important than ever with generative artificial intelligence’s (AI) diffusion in organizational and private realms. Unfortunately, challenges associated with the computational nature of AI-based systems and human sensemaking can hinder augmentation of human work. Prevalent forms of algorithmic advice commonly provide a user with ‘one best’ solution that can induce the user to fixate on the advice and neglect own critical reasoning. To overcome such interaction challenges, we explore the potential of evaluative algorithmic advice that provides users with more open-ended and engaging feedback. As part of two controlled experiments, we find that humans prefer evaluative over contrastive advice for writing feedback. We then conducted a field experiment in the context of an educational business pitch writing task with two conditions: (i) contrastive algorithmic advice improving users’ writing without any further feedback; and (ii) evaluative algorithmic advice providing feedback in form of open-ended questions and pro and contra arguments. Users’ writing could be improved regardless of the type of algorithmic advice, yet we show that users exposed to evaluative advice engaged significantly more with the task and the advice. Our study explores how algorithmic advice may serve as a critical stimulator rather than attenuating human agency in AI augmentation.
Language
English
Keywords
Algorithmic Advice
Human Sensemaking
Writing Field Experiment
Randomized Controlled Trial
HSG Classification
contribution to scientific community
Refereed
Yes
Publisher place
Chicago, IL, USA
Event Title
Academy of Management (AOM) Meeting
Event Location
Chicago, IL, USA
Event Date
09.08.2024
Subject(s)
Division(s)