Generating Synthetic Multi-Turn Conversations for Scalable Functional Testing of Conversational AI Systems
Journal
European Conference on Information Systems (ECIS)
Type
conference paper
Date Issued
2026
Author(s)
Abstract
Artificial intelligence (AI) systems with conversational interfaces are increasingly used to augment humans and even automate complex workflows. Yet systematic functional testing remains difficult because of open-ended multi-turn input. Existing evaluation benchmarks focus on isolated single-turn performance and provide limited insight into how systems behave in realistic interaction scenarios. This paper investigates how synthetic multi-turn conversations can be generated to support scalable functional testing of conversational AI systems. Using a design science research approach, we develop an artifact that employs LLM-based role play to generate synthetic conversations for testing an insurance claim decision system. The generation process combines software testing techniques such as equivalence partitioning and boundary value analysis with persona-based user simulation. We report preliminary results from the first design cycle and derive six design principles for generating diverse, faithful, and diagnostically useful conversational test data.
HSG Classification
contribution to scientific community
Refereed
Yes
Subject(s)
Division(s)
File(s)![Thumbnail Image]()
Name
Generating Synthetic Multi-Turn Conversations_ECIS2026.pdf
Size
1.5 MB
Format
Adobe PDF
Checksum (MD5)
11a52c6ab3918af2edba3cc91b6dfca1