Fault
URL
Change Log
| When | What |
|---|---|
| January 29th, 2016 | Donated by Rene Just |
Reference
Studies who have been using the data (in any form) are required to include the following reference:
@INPROCEEDINGS{7372009,
author={Shamshiri, Sina and Just, Rene and Rojas, Jose Miguel and Fraser, Gordon and McMinn, Phil and Arcuri, Andrea},
booktitle={Automated Software Engineering (ASE), 2015 30th IEEE/ACM International Conference on},
title={Do Automatically Generated Unit Tests Find Real Faults? An Empirical Study of Effectiveness and Challenges (T)},
year={2015},
pages={201-211},
keywords={Generators;Java;Manuals;Software;Testing;Writing;automated test generation;empirical study;regression testing;test effectiveness;unit testing},
doi={10.1109/ASE.2015.86},
month={Nov},}
About the Data
Overview of Data
Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs
Paper Abstract
Rather than tediously writing unit tests manually, tools can be used to generate them automatically — sometimes even resulting in higher code coverage than manual testing. But how good are these tests at actually finding faults? To answer this question, we applied three state-of-the-art unit test generation tools for Java (Randoop, EvoSuite, and Agitar) to the 357 real faults in the Defects4J dataset and investigated how well the generated test suites perform at detecting these faults. Although the automatically generated test suites detected 55.7% of the faults overall, only 19.9% of all the individual test suites detected a fault. By studying the effectiveness and problems of the individual tools and the tests they generate, we derive insights to support the development of automated unit test generators that achieve a higher fault detection rate. These insights include 1) improving the obtained code coverage so that faulty statements are executed in the first instance, 2) improving the propagation of faulty program states to an observable output, coupled with the generation of more sensitive assertions, and 3) improving the simulation of the execution environment to detect faults that are dependent on external factors such as date and time.