Blogs (3) >>
Fri 22 Mar 2024 13:45 - 14:10 at Meeting Rooms B115-116 - LLMs, Debugging, and Detection Chair(s): John Edwards

When employing the Socratic method of teaching, instructors guide students toward solving a problem on their own rather than providing the solution directly. While this strategy can substantially improve learning outcomes, it is usually time-consuming and cognitively demanding. Automated Socratic conversational agents can augment human instruction and provide the necessary scale, however their development is hampered by the lack of suitable data for training and evaluation. In this paper, we introduce a manually created dataset of multi-turn Socratic advice aimed at helping a novice programmer fix buggy solutions to simple computational problems. The dataset is then used for benchmarking the Socratic debugging abilities of a number of language models, ranging from fine-tuning the instruction-based text-to-text transformer Flan-T5 to zero-shot and chain of thought prompting of the much larger GPT-4.

Fri 22 Mar

Displayed time zone: Pacific Time (US & Canada) change

13:45 - 15:00
LLMs, Debugging, and DetectionPapers at Meeting Rooms B115-116
Chair(s): John Edwards Utah State University
13:45
25m
Talk
Can Language Models Employ the Socratic Method? Experiments with Code DebuggingGlobalCC
Papers
Erfan Al-Hossami UNC Charlotte, Razvan Bunescu UNC Charlotte, Justin Smith UNC Charlotte, Ryan Teehan New York University
DOI
14:10
25m
Talk
Detecting ChatGPT-Generated Code Submissions in a CS1 Course Using Machine Learning ModelsCC
Papers
Muntasir Hoq North Carolina State University, Yang Shi North Carolina State University, Juho Leinonen Aalto University, Damilola Babalola North Carolina State University, Collin Lynch North Carolina State University, Thomas Price North Carolina State University, Bita Akram North Carolina State University
DOI
14:35
25m
Talk
Towards Comprehensive Metrics for Programming Cheat DetectionCC
Papers
Frank Vahid UC Riverside / zyBooks, Ashley Pang UC Riverside, Benjamin Denzler University of California, Riverside
DOI