Blogs (4) >>
Fri 22 Mar 2024 14:35 - 15:00 at Meeting Rooms B115-116 - LLMs, Debugging, and Detection Chair(s): John Edwards

Automated assistance for detecting cheating on programs has long been investigated by CS educators, especially with the rise of “homework help” websites over the past decade, and recently with AI tools like ChatGPT. The main detection approach has long been flagging similar submission pairs. Modern cheating, like hiring contractors or using ChatGPT, may not yield such similarity. And, cases based on similarity alone may be weak. Thus, over the past several years, building on logs from an online program auto-grader (zyBooks), we developed additional “cheating concern metrics”: points rate, style anomalies, style inconsistencies, IP address anomalies, code replacements, and initial copying. Most are defined not only for one programming assignment but also across a set of assignments. The metrics can help catch more kinds of cheating, provide more compelling evidence of cheating, reduce false cheating accusations based on similarity alone, and help instructors focus their limited cheat-detection time on the most egregious cases. We describe the techniques, and our experiences (via our own Python scripts and a commercial tool) for several terms, showing benefits of having more metrics than just similarity. Of 30 cheating cases over 3 terms and 300 students, most were based on metrics beyond similarity, all students admitted, none later contested, and time per student was only 1-2 hours (far less than previously). Our goal is to prevent cheating in the first place, by reducing opportunity via strong detection tools, as part of a multi-faceted approach to having students truly learn and stay out of trouble.

Fri 22 Mar

Displayed time zone: Pacific Time (US & Canada) change

13:45 - 15:00
LLMs, Debugging, and DetectionPapers at Meeting Rooms B115-116
Chair(s): John Edwards Utah State University
13:45
25m
Talk
Can Language Models Employ the Socratic Method? Experiments with Code DebuggingGlobalCC
Papers
Erfan Al-Hossami UNC Charlotte, Razvan Bunescu UNC Charlotte, Justin Smith UNC Charlotte, Ryan Teehan New York University
DOI
14:10
25m
Talk
Detecting ChatGPT-Generated Code Submissions in a CS1 Course Using Machine Learning ModelsCC
Papers
Muntasir Hoq North Carolina State University, Yang Shi North Carolina State University, Juho Leinonen Aalto University, Damilola Babalola North Carolina State University, Collin Lynch North Carolina State University, Thomas Price North Carolina State University, Bita Akram North Carolina State University
DOI
14:35
25m
Talk
Towards Comprehensive Metrics for Programming Cheat DetectionCC
Papers
Frank Vahid UC Riverside / zyBooks, Ashley Pang UC Riverside, Benjamin Denzler University of California, Riverside
DOI