Art der Publikation: Beitrag in Sammelwerk

Synthetic Conversations, Real Insights: Towards Realistic User Simulations with Generative Agents

Autor(en):

Holldack, Florian; Banh, Leonardo; Strobel, Gero

Herausgeber:

Ais

Titel des Sammelbands:

Proceedings of the 46th International Conference on Information Systems (ICIS)

Ort(e):

Nashville, Tennessee, USA

Veröffentlichung:

2025

Link zum Volltext:

https://aisel.aisnet.org/icis2025/isdesign/isdesign/16/

Zitation:

Download BibTeX

Kurzfassung

The rise of generative AI and large language models (LLMs) has sparked interest in applying generative agents to simulate users in conversational recommender system (CRS) evaluations. While CRS rely on user perception for success, traditional user studies are costly and time-intensive. Existing simulation approaches lack interaction depth and user-centric evaluation. This study addresses these gaps by leveraging LLM-based generative agents to conduct scalable, subjective CRS assessments. We present a comparative study where 200 generative agents and 50 human participants interact with a prototypical agentic CRS and evaluate their experience using a structured questionnaire. Results indicate that generative agents approximate human-like behavior and subjective assessments at the macro-level, despite granular precision limitations, offering an alternative to traditional user studies. Our findings advance research on CRS evaluation by demonstrating how agentic simulations can support human-aligned assessments of socio-technical systems and open new avenues for applying Generative AI in user-centered decision support.