LLM-Augmentation for Idea Evaluation: Developing a Reference Model for Evaluation Pipelines
Journal
Lecture Notes in Computer Science
ISSN
978-3-031-93975-4
ISSN-Digital
978-3-031-93976-1
Type
book section
Date Issued
2025-05-27
Author(s)
Editor(s)
Samir Chatterjee
Vom Brocke Jan
Anderson, Ricardo
Abstract
Automated approaches to idea evaluation increasingly leverage generative artificial intelligence to support decision-makers. However, contextualizing evaluations within specific domains remains challenging, particularly at varying levels of large language model (LLM) augmentation. Existing research employs embeddings to derive semantic insights, yet these representations often lack domain-specific contextualization. Recent advancements, such as chat-based LLMs, present new opportunities to incorporate context through prompting. To address these challenges, we propose a structured evaluation pipeline that integrates embeddings with feature engineering to enhance the contextualization of chat-based LLM evaluations. Using a real-world innovation challenge, we instantiate this pipeline and assess its predictive performance across different levels of augmentation. Our findings reveal that incorporating contextual information improves predictive accuracy but depends on fine-grained idea quality dimensions. By codifying our approach into a reference model, we provide a transferable framework that generalizes across various evaluation contexts employing LLMs.
Keywords
Idea Evaluation
LLM-Augmentation
Generative AI
Reference Pipeline
Design Science Research
Book title
Local Solutions for Global Challenges
Publisher
Springer Nature Switzerland
Volume
15703
Start page
151
End page
164
Event Title
DESRIST 2025
Event Location
Montego Bay, Jamaica
Event Date
02.-04.06.2025
Division(s)