Dagstuhl Seminar 24272
A Game of Shadows: Effective Mastery Learning in the Age of Ubiquitous AI
( Jun 30 – Jul 05, 2024 )
(Click in the middle of the image to enlarge)
Permalink
Please use the following short url to reference this page:
https://www.dagstuhl.de/24272
Organizers
- Nick Falkner (University of Adelaide, AU)
- Juho Leinonen (Aalto University, FI)
- Miranda C. Parker (San Diego State University, US)
- Andrew Petersen (University of Toronto Mississauga, CA)
- Claudia Szabo (University of Adelaide, AU)
Contact
- Marsha Kleinbauer (for scientific matters)
- Christina Schwarz (for administrative matters)
Shared Documents
- Dagstuhl Materials Page (Use personal credentials as created in DOOR to log in)
Impacts
- Goodbye Hello World : Research Questions for a Future CS1 Curriculum : article in Koli Calling '24: Proceedings of the 24th Koli Calling International Conference on Computing Education Research - Keuning. Hieke; Luxton-Reilly, Andrew; Ott, Claudia; Petersen, Andrew; Kiesler, Natalie - New York : ACM, 2024. - pp. 1-2.
- The Role of Generative AI in Software Student CollaborAItion - Kiesler, Natalie; Smith, Jacqueline; Leinonen, Fox, Armando; Juho; MacNeil, Stephen; Ihantola, Petri - Cornell University : arXiv.org, 2025. - 7 pp..
Schedule
The integration of generative AI (GenAI) into education raises significant issues and opportunities, particularly concerning mastery learning and programming education. A primary concern is that students may bypass deep engagement with their learning tasks by relying on AI tools or search engines, which leads to a superficial understanding of the material. This tendency forces instructors to focus more on monitoring for academic dishonesty rather than on effective teaching. To address this, innovative approaches to presenting curricula and materials could foster greater student engagement and reduce the inclination to rely on external aids.
The transformative potential of AI in education is likened in Armando Fox's talk to the early use of movie cameras, which initially focused on replicating existing practices rather than exploring new possibilities. The emphasis is on avoiding mere substitution of traditional methods with AI, and instead, leveraging AI to create entirely new modes of learning. The goal is to integrate AI in a way that complements foundational educational concepts and develops the necessary intellectual frameworks to utilize these new tools effectively.
Mastery learning remains an effective approach in the age of AI, with a deep understanding of the importance of clearly defined learning goals and assessments designed to resist cheating. One proposed method involves students setting their own learning goals, which are then approved by instructors. This approach not only personalizes learning but also utilizes GenAI to assist in developing and refining assignments. GenAI can also serve as a brainstorming partner for educators, providing valuable support in creating innovative class activities and assessments.
AI tools like code co-pilots are nowadays becoming integral for understanding and implementing code. This shift presents challenges for traditional assessment methods, as AI can outperform students in specific coding tasks. It is essential for students to develop strong communication skills about code, as these will become increasingly crucial in collaborative programming environments where AI tools are prevalent. This suggests that the skill of discussing code should be as rigorously taught as technical coding skills. In addition, providing accurate feedback and identifying essential skills for effective software development remain critical. While LLMs can offer valuable feedback, ensuring that this feedback is accurate and relevant remains a challenge. Additionally, defining the skills necessary for students to develop software with the support of LLMs, such as program specification and refactoring, is crucial for leveraging these tools effectively.
The integration of mastery learning with GenAI presents both significant potential and challenges. While GenAI can enhance personalized learning, assessment creation, and feedback, effective implementation requires careful consideration of the tools used, their alignment with educational goals, and their impact on learning outcomes. Ensuring that these tools are suitable for diverse educational contexts and measuring their effectiveness will be key to successfully adopting mastery learning supported by GenAI, ultimately aiming to improve educational outcomes and better prepare students for future professional challenges.
The seminar was structured into three main sections: lightning and keynote talks, brainstorming, and workshop groups. At the beginning of the seminar, each attendee delivered a lightning talk. Two keynote talks were delivered, as follows:
- Prof Andrew Luxton-Reilly, University of Auckland: “It’s the end of the world as we know it, but I feel fine! Teaching and learning with GenAI”
- Dr Claudia Ott, University of Otago: “A Decade of Mastery Learning at Otago – Pitfalls, Challenges & Opportunities”
The lightning talk sessions were followed by a brainstorming session to identify existing challenges and opportunities. This session served a dual purpose, in that it also allowed us to identify the three main working groups of the seminar. These groups focused on (i) curriculum and pedagogy of mastery learning in the era of GenAI, (ii) university and organisation structures that facilitate the delivery and operationalisation of mastery learning in the era of GenAI, and on (iii) designing courses in a curriculum that is GenAI focused.
Publications to Date
At the date of the submission of this report, the following papers and posters had been accepted for publication:
- “Models of Mastery Learning for Computing Education”, by Claudia Szabo, Miranda Parker, Michelle Friend, Johan Jeuring, Tobias Kohn, Lauri Malmi, Judithe Sheard was accepted for publication at SIGCSE 2025 as a position paper.
- “Goodbye HelloWorld – Research Questions for a Future CS1 Curriculum”, Hieke Keuning, Andrew Luxton-Reilly, Claudia Ott, Andrew Petersen and Natalie Kiesler was accepted at Koli Calling 2024 as a poster.
Creative Commons BY 4.0
Nick Falkner, Juho Leinonen, Miranda C. Parker, Andrew Petersen, and Claudia Szabo
This Dagstuhl Seminar will identify challenges and potential solutions for the practice of mastery learning within tertiary computing education in the context of the recent rapid advancement of highly scalable large language models (LLM). While an understanding of the capabilities and weaknesses of LLMs allows for its guarded use by academics, its use by students warrants in-depth attention. When considering the need for repeated practice within mastery learning and the lack of in-depth knowledge specific to novice programmers, it is critical to understand how to effectively incorporate LLMs into the mastery learning process.
Mastery learning is an educational approach where students master a specific set of skills before being allowed to move on to a different set of skills, often more advanced than the previous. A differentiating aspect of mastery learning is that students are only allowed to move to the next set of skills after they have demonstrated that they are proficient with the current set. Students are given ample opportunities to practice, to receive timely, supportive feedback, and to demonstrate mastery. Instructors using a mastery approach must have a deep understanding of student progress and structure support accordingly.
Many computer science topics, including programming, appear to naturally lend themselves to a mastery learning approach. However, adoption of mastery learning is limited beyond the first year of tertiary education, where traditionally students learn how to program. A typical implementation of mastery learning in a programming course features a pass/fail assessment based on a sequence of several mastery tests.
The wide availability of LLMs provides both opportunities and challenges for mastery learning. For example, the duality of the mark, the pressure of the tests, and the availability of LLMs can lead students to overreliance for both simple and complex problems, and in some cases, plagiarism and collusion. However, LLMs may also be used to provide timely feedback, if framed appropriately, or to create new resources for practice, which would reduce the cost of adopting a mastery learning approach.
In light of this, our questions are:
- What learning theories and pedagogies support implementation of mastery learning in the era of LLMs?
- To what extent can we co-implement mastery learning with AI? If so, how can we use LLMs to encourage adoption of mastery learning more broadly, beyond CS1 and across all years of a degree?
- How can we implement mastery learning at scale?
- How can mastery learning be designed to reduce the risk of plagiarism?
- How can LLM-based pedagogical tools be designed to support mastery learning?
The timeliness of this seminar is critical: in the last twelve months, LLMs have revolutionized many commercial fields and their use in education continues to grow. This use is mostly by students who use LLMs to generate answers to school work, while universities are grappling with the impact of LLMs on assessment policies and pedagogy.
Nick Falkner, Juho Leinonen, Miranda C. Parker, Andrew Petersen, and Claudia Szabo
Please log in to DOOR to see more details.
- Claudio Álvarez Gómez (Universidad de los Andes - Santiago, CL) [dblp]
- Nick Falkner (University of Adelaide, AU) [dblp]
- Armando Fox (University of California - Berkeley, US) [dblp]
- Michelle Friend (University of Nebraska - Omaha, US) [dblp]
- Petri Ihantola (University of Jyväskylä, FI) [dblp]
- Johan Jeuring (Utrecht University, NL) [dblp]
- Hieke Keuning (Utrecht University, NL) [dblp]
- Natalie Kiesler (Technische Hochschule Nürnberg, DE) [dblp]
- Päivi Kinnunen (University of Helsinki, FI) [dblp]
- Tobias Kohn (KIT - Karlsruher Institut für Technologie, DE) [dblp]
- Juho Leinonen (Aalto University, FI) [dblp]
- Andrew James Luxton-Reilly (University of Auckland, NZ) [dblp]
- Stephen MacNeil (Temple University - Philadelphia, US) [dblp]
- Lauri Malmi (Aalto University, FI) [dblp]
- Claudia Ott (University of Otago, NZ) [dblp]
- Miranda C. Parker (San Diego State University, US) [dblp]
- Andrew Petersen (University of Toronto Mississauga, CA) [dblp]
- Jaromír Šavelka (Carnegie Mellon University - Pittsburgh, US) [dblp]
- Judithe Sheard (Monash University - Clayton, AU) [dblp]
- Jacqueline Smith (University of Toronto, CA) [dblp]
- Claudia Szabo (University of Adelaide, AU) [dblp]
- Lisa Zhang (University of Toronto Mississauga, CA) [dblp]
Classification
- Other Computer Science
Keywords
- Computing education
- Mastery learning
- LLMs
- ChatGPT
- Machine learning
