Blogs (3) >>
Fri 22 Mar 2024 11:10 - 11:35 at Oregon Ballroom 204 - LLM - tools Chair(s): Geoffrey Herman

In this paper, we detail the successful deployment of a machine learning autograder that significantly decreases the grading labor required in the Breakout computer science assignment. This assignment — which tasks students with programming a game consisting of a controllable paddle and a ball that bounces off the paddle to break bricks — is popular for engaging students with introductory computer science concepts, but creates a large grading burden. Due to the game’s interactive nature, grading defies traditional unit tests and instead typically requires 8+ minutes of manually playing each student’s game to search for bugs. This amounts to 45+ hours of grading in a standard course offering and prevents further widespread adoption of the assignment. Our autograder alleviates this burden by playing each student’s game with a reinforcement learning agent and providing videos of discovered bugs to instructors. In an A/B test with manual grading, we find that our autograder reduces grading time by 44%, while slightly improving grading accuracy by 6%, ultimately saving roughly 30 hours over our deployment in two offerings of the assignment. Our results further suggest the practicality of grading other interactive assignments (e.g., other games or building websites) via similar machine learning techniques.

Fri 22 Mar

Displayed time zone: Pacific Time (US & Canada) change

10:45 - 12:00
LLM - toolsPapers at Oregon Ballroom 204
Chair(s): Geoffrey Herman University of Illinois at Urbana-Champaign
10:45
25m
Talk
Evaluating Automatically Generated Contextualised Programming ExercisesGlobal
Papers
Andre del Carpio Gutierrez The University of Auckland, Paul Denny The University of Auckland, Andrew Luxton-Reilly The University of Auckland
DOI
11:10
25m
Talk
A Fast and Accurate Machine Learning Autograder for the Breakout Assignment
Papers
Evan Liu Stanford University, David Yuan Stanford University, Syed Ahmed Oakland University, Elyse Cornwall Stanford University, Juliette Woodrow Stanford University, Kaylee Burns Stanford University, Allen Nie Stanford University, Emma Brunskill Stanford University, Chris Piech Stanford University, Chelsea Finn Stanford University
DOI
11:35
25m
Talk
Beyond Traditional Teaching: Designing a virtual teaching assistant using LLMs for CS educationGlobal
Papers
Mengqi Liu Mcgill university, Faten M'Hiri Mcgill university
DOI