Managing the Algorithm: Prompt Engineering for AI-based Systems as an Emerging Business Skill

Type
fundamental research project
Start Date
April 2024
End Date
May 2025
URI
Description
Today, artificial intelligence (AI) is pervasive as it influences a lot of different areas in our private and working lives, for instance through AI-based conversational agents, which are applied in various usage scenarios. Enabled through the advancements of natural language processing (NLP) achieved by large language models (LLMs), AI mimics our human natural intelligence. In this context, it can lead an intelligent conversation with a human counterpart as recent examples such as ChatGPT show very impactful. Interacting with these intelligent AI-based agents through conversational dialogs is nowadays an omnipresent foundation of everyday life. LLMs offer a unique opportunity for individuals to develop and enhance their modern competencies, such as communication, reasoning, creativity, and problem-solving, by interacting with these AI systems through natural language dialogs. On top of that, creating new value with AI is a transformative competency for the future (OECD 2018). Nonetheless, generative AI and LLMs also face difficulties, such as the need for technical skills, the risk of producing wrong or meaningless outputs (oftentimes called hallucination), and the lack of reasoning needed for approaching complex problems. Thus, the improvement of communicating with AI-based systems is a critical endeavor. By using prompts, which are textual instructions and examples of the desired interactions, users can enhance the quality and relevance of the AI outputs. However, engineering prompts comes with several challenges. First, prompting in itself is a challenging task that involves active research on the direct and indirect effects of prompts on generative AI models, and lacks established workflows for prompt engineering, which requires extensive experimentation and evaluation by NLP experts to avoid undesirable AI outputs. Second, understanding the process of how AI-based systems produce outcomes based on user inputs is critical for generative AI outputs but not well understood. Third, although managing algorithms through effective prompts is an important skill, there is a critical lack of what defines prompt engineering as a business skill. Prompt engineering not only relates to understanding the technical aspects of AI-based systems but also involves the ability to apply this skill effectively and creatively in a variety of situations such as analytical thinking, strategic planning, decision-making, and negotiation in collaboration with an algorithm. The goal of this project departs from prior techno-centric research and introduces a skill-based approach to prompt engineering as the critical key for enabling individuals to manage algorithms. Thus, the main research question (RQ) is based on the question “How can prompt engineering for business skills in generative AI-based tool environments be conceptualized, designed, and evaluated?” Guided by this RQ, the project is built upon three work packages, all aiming at publications in top-tier journals and conferences.Conceptually, we draw upon a skill development perspective to overcome the aforementioned challenges and answer the main RQ. We argue that effective prompt engineering strategies allow for improved skills in the business domain. Understanding how to conceptualize prompt engineering from a skill perspective and how to design effective learning processes is a time-critical endeavor for research to provide scientific evidence in this novel research area. The project has three overall deliverables: First, we provide a conceptualization of prompt engineering skills in the business domain. Second, we design a learning process based on worked examples to embed prompt engineering strategies into higher education business teaching. Third, the project provides empirical insights into the effects of prompt strategies and contributes to research related to worked examples when considering the interaction with emerging generative AI technologies.
Leader contributor(s)
Member contributor(s)
-
Funder

SNF – Third Party

Topic(s)
digital learning
generative artificial intelligence
management education
business skills
Division(s)

IWI - Institute of In...

Filters

Settings

Now showing 1 - 3 of 3
  • Thumbnail Image
    Some of the metrics are blocked by your 
    consent settings
    Item type:Publication,
    Learning How to Manage the AI: Incorporating Generative AI into Management Learning and Education (PDW)
    (2025)
    Andreas Janson  
    ;
    Schöbel, Sofia
    ;
    Trinh, Mai
    ;
    Tolzin, Antonia
    ;
    Kirmse, Rosemarie
    Artificial intelligence (AI) is reshaping private and professional spheres. Generative tools such as ChatGPT, powered by large language models (LLMs), present unique opportunities for augmenting competencies like communication, reasoning, creativity, and problem-solving. Despite these benefits, challenges such as technical complexity, risks of inaccurate outputs, limited reasoning, and deskilling persist. The integration of generative AI tools like ChatGPT in management education demands a rethinking of how we cultivate critical AI skills like prompt engineering to navigate both opportunities and risks effectively. This PDW introduces a skill-based approach on learning to manage the AI, addressing the need to proactively incorporate fostering AI skills such as prompt engineering for management students. The session focuses on (1) making a case for AI skills in management education, (2) how we proactively shape learning processes with AI, and (3) how we adapt our own management teaching to provide learning opportunities with AI. This 90-minute PDW includes two parts: (1) An expert panel discussion on integrating Generative AI into research and teaching, and (2) interactive roundtable sessions to develop actionable strategies for participants to apply in their own contexts. The session is designed to be inclusive, providing foundational knowledge for AI novices while offering advanced strategies for AI-experienced educators.
    Type:conference contribution
    URI:https://www.alexandria.unisg.ch/handle/20.500.14171/122335
  • Thumbnail Image
    Some of the metrics are blocked by your 
    consent settings
    Item type:Publication,
    AI literacy and its implications for prompt engineering strategies
    (2024)
    Nils Knoth
    ;
    Antonia Tolzin
    ;
    Andreas Janson  
    ;
    Jan Marco Leimeister  
    Artificial intelligence technologies are rapidly advancing. As part of this development, large language models (LLMs) are increasingly being used when humans interact with systems based on artificial intelligence (AI), posing both new opportunities and challenges. When interacting with LLM-based AI system in a goal-directed manner, prompt engineering has evolved as a skill of formulating precise and well-structured instructions to elicit desired responses or information from the LLM, optimizing the effectiveness of the interaction. However, research on the perspectives of non-experts using LLM-based AI systems through prompt engineering and on how AI literacy affects prompting behavior is lacking. This aspect is particularly important when considering the implications of LLMs in the context of higher education. In this present study, we address this issue, introduce a skill-based approach to prompt engineering, and explicitly consider the role of non-experts' AI literacy (students) in their prompt engineering skills. We also provide qualitative insights into students’ intuitive behaviors towards LLM-based AI systems. The results show that higher-quality prompt engineering skills predict the quality of LLM output, suggesting that prompt engineering is indeed a required skill for the goal-directed use of generative AI tools. In addition, the results show that certain aspects of AI literacy can play a role in higher quality prompt engineering and targeted adaptation of LLMs within education. We, therefore, argue for the integration of AI educational content into current curricula to enable a hybrid intelligent society in which students can effectively use generative AI tools such as ChatGPT.
    Type:journal article
    Journal:Computers and Education: Artificial Intelligence
    Volume:6
    Issue:June
    DOI:10.1016/j.caeai.2024.100225
    URI:https://www.alexandria.unisg.ch/handle/20.500.14171/119942
    Scopus© Citations 376
  • Thumbnail Image
    Some of the metrics are blocked by your 
    consent settings
    Item type:Publication,
    Worked Examples to Facilitate the Development of Prompt Engineering Skills
    (2024)
    Antonia Tolzin
    ;
    Nils Knoth
    ;
    Andreas Janson  
    This paper explores the evolving field of prompt engineering in Artificial Intelligence (AI), with a focus on Large Language Models (LLMs). As LLMs exhibit remarkable potential in various educational domains, their effective use requires adept prompt engineering skills. We introduce a skill-based approach to prompt engineering and explicitly investigate the impact of using worked examples to facilitate prompt engineering skills among students interacting with LLMs. We propose hypotheses linking prompt engineering, worked examples, and perceived anthropomorphism to the quality of LLM output. Our initial findings support the critical relationship between proficient prompt engineering and the resulting output quality of LLMs. Subsequent phases will further explore the role of worked examples in prompt engineering, aiming to provide practical recommendations for educational improvement and industry application. Additionally, this research aims to shed light on the responsible utilization of LLMs in education and contribute insights to educational practice, research, and organizational development.
    Type:conference paper
    Journal:Thirty-Second European Conference on Information Systems (ECIS 2024)
    DOI:10.1016/j.caeai.2024.100225
    URI:https://www.alexandria.unisg.ch/handle/20.500.14171/119943
    Scopus© Citations 376