Fri 25 Oct 2024 17:20 - 17:40 at IBR West - Education and Language Usability Chair(s): Arjun Guha

This paper documents a year-long experiment to “profile” the process of learning a programming language: gathering data to understand what makes a language hard to learn, and using that data to improve the learning process. We added interactive quizzes to The Rust Programming Language, the official textbook for learning Rust. Over 13 month, 62,526 readers answered questions 1,140,202 times. First, we analyze the trajectories of readers. We find that many readers drop-out of the book early when faced with difficult language concepts like Rust’s ownership types. Second, we use classical test theory and item response theory to analyze the characteristics of quiz questions. We find that better questions are more conceptual in nature, such as asking why a program does not compile vs. whether a program compiles. Third, we performed 12 interventions into the book to help readers with difficult questions. We find that on average, interventions improved quiz scores on the targeted questions by 20pp. Fourth, we show that our technique can likely generalize to languages with smaller user bases by simulating our statistical inferences on small N. These results demonstrate that quizzes are a simple and useful technique for understanding language learning at all scales.

Fri 25 Oct

Displayed time zone: Pacific Time (US & Canada) change

16:00 - 17:40
Education and Language UsabilityOOPSLA 2024 at IBR West
Chair(s): Arjun Guha Northeastern University; Roblox
16:00
20m
Talk
Forge: A Tool and Language for Teaching Formal Methods
OOPSLA 2024
Tim Nelson Brown University, Ben Greenman University of Utah, Siddhartha Prasad Brown University, Tristan Dyer Brown University, Ethan Bove Brown University, Qianfan Chen Brown University, Charlie Cutting Brown University, Thomas Del Vecchio Brown University, Sidney LeVine Brown University, Julianne Rudner Brown University, Benjamin Ryjikov Brown University, Alexander Varga Brown University, Andrew Wagner Northeastern University, Luke West Brown University, Shriram Krishnamurthi Brown University
DOI
16:20
20m
Talk
HiPy: Extracting High-Level Semantics From Python Code For Data Processing
OOPSLA 2024
Michael Jungmair Technical University of Munich, Alexis Engelke TU Munich, Jana Giceva Technical University of Munich
DOI
16:40
20m
Talk
Identifying and Correcting Programming Language Behavior MisconceptionsOOPSLA 2024 Distinguished Paper Award
OOPSLA 2024
Kuang-Chen Lu Brown University, USA, Shriram Krishnamurthi Brown University
DOI
17:00
20m
Talk
PyDex: Repairing Bugs in Introductory Python Assignments using LLMs
OOPSLA 2024
Jialu Zhang University of Waterloo, José Pablo Cambronero Google, USA, Sumit Gulwani Microsoft, Vu Le Microsoft, Ruzica Piskac Yale University, Gustavo Soares Microsoft, Gust Verbruggen Microsoft
DOI
17:20
20m
Talk
Profiling Programming Language LearningOOPSLA 2024 Distinguished Paper Award
OOPSLA 2024
Will Crichton Brown University, Shriram Krishnamurthi Brown University
DOI Pre-print