Debugging and testing compilers has been an strenuous task. Developers need to create a large number of test cases, invoke the compiler on them, and then verify the results against the expected output. Since compilers have many stages, developers might find it difficult to observe the behavior of a certain stage, let alone debugging the stage. Moreover, introducing new features to the compiler may affect existing components intrinsically in ways that traditional tests cannot reveal.

We propose a new prespective on incorporating interactivity into tests. Each test file is divided into smaller blocks by empty lines. The test program sequentially executes each block and appends the inferred type, evaluation results, diagnostic messages, and debugging information as comments to each block. Because each block inherits the context of the previous block, developers can incrementally compose test files in an interactive manner. Additionally, developers can selectively inspect debugging information from different stages by prepending flags to each test block. By monitoring file changes, the test program reruns modified tests and update comments in situ. Those updates are displayed in editors in real time, which makes composing tests as smooth as using an interactive notebook.

These comments are committed to the version control system and can thus track changes made to the compiler. It relies on the diff operation of version control systems, thus we name this testing method “diff-based testing” because. We have implemented this testing system on the MLscript compiler and developed an interactive editor plugin for exploring debugging information.