Evaluating the Impact of Scaffolding and Visualizations for Mutation Testing Exercises in Software Engineering Education

Mutation testing is an effective testing technique for improving how well a test suite can detect small changes to a program under test. This testing technique is seeing increased industry adoption. This paper aims to study the use of mutation testing in an educational setting and understand students’ technical and conceptual challenges in applying mutation testing concepts. We report on two case studies of incorporating mutation testing into software engineering curricula. The Scaffolding Study explores the impact of using different mutation analysis tools directly or indirectly via a uniform interface provided by an educational infrastructure. We observe that scaffolding (indirect tool use) improved the consistency of student performance for those using the same mutation analysis tool on the same code as well as helping students perform more effective mutation testing. The Visualization Study explores the impact of different forms of output of a mutation analysis tool. Specifically, it ssesses to what extent visualizations support students in reasoning about mutants and writing tests to detect them. We observe that like scaffolding, visualizations helped students perform more effective mutation testing, with lower-performing students seeing a boost in particular. We further explore challenges around automatic assessment of mutation testing exercises. For example, we observe that even with assignment scaffolding, 18-21% of student submissions required manual modifications to successfully execute.