The rise of generative AI and large language models (LLMs) has sparked interest in applying generative agents to simulate users in conversational recommender system (CRS) evaluations. While CRS rely on user perception for success, traditional user studies are costly and time-intensive. Existing simulation approaches lack interaction depth and user-centric evaluation. This study addresses these gaps by leveraging LLM-based generative agents to conduct scalable, subjective CRS assessments. We present a comparative study where 200 generative agents and 50 human participants interact with a prototypical agentic CRS and evaluate their experience using a structured questionnaire. Results indicate that generative agents approximate human-like behavior and subjective assessments at the macro-level, despite granular precision limitations, offering an alternative to traditional user studies. Our findings advance research on CRS evaluation by demonstrating how agentic simulations can support human-aligned assessments of socio-technical systems and open new avenues for applying Generative AI in user-centered decision support.

Context

With rapid technological advancements, artificial intelligence (AI) has become integral to various sectors. Generative AI (GenAI) tools like ChatGPT or GitHub Copilot, with their unique content creation capabilities, pose transformative potential in Software Engineering by offering new ways to optimize software development processes. However, the integration into current processes also presents challenges that require a sociotechnical analysis to effectively realize GenAI's potential.

Objective

This study investigates how GenAI can be leveraged in the domain of Software Engineering, exploring its action potentials and challenges to help businesses and developers optimize the adoption of this technology in their workflows.

Method

We performed a qualitative study and collected data from expert interviews with eighteen professionals working in Software Engineering-related roles. Data analysis followed the principles of Grounded Theory to analyze how GenAI supports developers' goals, aligns with organizational practices, and facilitates integration into existing routines.

Results

The findings demonstrate several opportunities of GenAI in Software Engineering to increase productivity in development teams. However, several key barriers were also identified, that should be accounted for in successful integrations. We synthesize the results in a grounded conceptual framework for GenAI adoption in Software Engineering.

Conclusions

This study contributes to the discourse on GenAI in Software Engineering by providing a conceptual framework that aids in understanding the opportunities and challenges of GenAI. It offers practical guidelines for businesses and developers to enhance GenAI integration and lays the groundwork for future research on its impact in software development.