• repo
• defect
• mccabehalsted
• jm1.html

jm1

URL

• Data Link (DOI)
• Link to NASA.

Change Log

Notes from the Author

Title/Topic: JM1/software defect prediction

• Sources:
  • Creators: NASA, then the NASA Metrics Data Program
  • Contacts:
    • Mike Chapman, Galaxy Global Corporation (Robert.Chapman@ivv.nasa.gov) +1-304-367-8341;
    • Pat Callis, NASA, NASA project manager for MDP (Patrick.E.Callis@ivv.nasa.gov) +1-304-367-8309
• Past usage:
  • How Good is Your Blind Spot Sampling Policy?; 2003; Tim Menzies and Justin S. Di Stefano; 2004 IEEE Conference on High Assurance Software Engineering (http://menzies.us/pdf/03blind.pdf).
    • Results:
      • Very simple learners (ROCKY) perform as well in this domain as more sophisticated methods (e.g. J48, model trees, model trees) for predicting detects
      • Many learners have very low false alarm rates.
      • Probability of detection (PD) rises with effort and rarely rises above it.
      • High PDs are associated with high PFs (probability of failure)
      • PD, PF, effort can change significantly while accuracy remains essentially stable
      • With two notable exceptions, detectors learned from one data set (e.g. KC2) have nearly they same properties when applied to another (e.g. PC2, KC2). Exceptions:
        • LinesOfCode measures generate wider inter-data-set variances;
        • Precision’s inter-data-set variances vary wildly
  • “Assessing Predictors of Software Defects”, T. Menzies and J. DiStefano and A. Orrego and R. Chapman, 2004, Proceedings, workshop on Predictive Software Models, Chicago, Available from http://menzies.us/pdf/04psm.pdf
    • Results:
      • From JM1, Naive Bayes generated PDs of 25% with PF of 20%
      • Naive Bayes out-performs J48 for defect detection
      • When learning on more and more data, little improvement is seen after processing 300 examples.
      • PDs are much higher from data collected below the sub-sub-system level.
      • Accuracy is a surprisingly uninformative measure of success for a defect detector. Two detectors with the same accuracy can have widely varying PDs and PFs.
• Relevant information:
  • JM1 is written in “C” and is a real-time predictive ground system: Uses simulations to generate predictions
  • Data comes from McCabe and Halstead features extractors of source code. These features were defined in the 70s in an attempt to objectively characterize code features that are associated with software quality. The nature of association is under dispute. Notes on McCabe can be found here and notes on Halstead can be found here. These metrics are widely used for defect prediction purposes, a quick tutorial on defect prediction is given here.
• Number of instances: 10885
• Number of attributes: 22 (5 different lines of code measure, 3 McCabe metrics, 4 base Halstead measures, 8 derived Halstead measures, a branch-count, and 1 goal field).
• Attribute Information:
  • loc : numeric, McCabe’s line count of code
  • v(g) : numeric, McCabe “cyclomatic complexity”
  • ev(g) : numeric, McCabe “essential complexity”
  • iv(g) : numeric, McCabe “design complexity”
  • n : numeric, Halstead total operators + operands
  • v : numeric, Halstead “volume”
  • l : numeric, Halstead “program length”
  • d : numeric, Halstead “difficulty”
  • i : numeric, Halstead “intelligence”
  • e : numeric, Halstead “effort”
  • b : numeric, Halstead
  • t : numeric, Halstead’s time estimator
  • lOCode : numeric, Halstead’s line count
  • lOComment : numeric, Halstead’s count of lines of comments
  • lOBlank : numeric, Halstead’s count of blank lines
  • lOCodeAndComment: numeric
  • uniq_Op : numeric, unique operators
  • uniq_Opnd : numeric, unique operands
  • total_Op : numeric, total operators
  • total_Opnd : numeric, total operands
  • branchCount : numeric, the flow graph
  • defects : {false,true}, module has/has not one or more reported defects
• Missing attributes: none
• Class Distribution: the class value (defects) is discrete
  • false: 2106 = 19.35%
  • true: 8779 = 80.65%

Comments on the Data

Martin Shepperd | August 19th, 2011 at 10:52 am I’ve realised this version of the dataset differs from the version available from the NASA MDP website. The following tabulates the differences for all 13 NASA datasets: {{{

Dataset n cases p features CM1 — NASA 505 40

CM1 — Promise 498 21

JM1 — NASA 10878 21

JM1 — Promise 10885 21

KC1 — NASA 2107 21

KC1 — Promise 2109 21

KC3 — NASA 458 40

KC3 — Promise 458 39

MC1 — NASA 9466 39

MC1 — Promise 9466 38

MC2 — NASA 161 40

MC2 — Promise 161 39

MW1 — NASA 403 40

MW1 — Promise 403 37

PC1 — NASA 1107 40

PC1 — Promise 1109 21

PC2 — NASA 5589 40

PC2 — Promise 5589 36

PC3 — NASA 1563 40

PC3 — Promise 1563 37

PC4 — NASA 1458 40

PC4 — Promise 1458 37

PC5- NASA 17186 39

PC5 — Promise 17186 38

KC4- NASA 125 40

KC2 — Promise 522 22 }}}

Therefore researchers are strongly advised to indicate which version of these files they are using!

{{{

Version, Cases(n), Features(p) JM1 — NASA, 10878, 24 JM1 — Promise, 10885, 22

}}}

Reference

• M. Shepperd, Q. Song, Z. Sun and C. Mair, Data Quality: Some Comments on the NASA Software Defect Data Sets (under review), 2011

People

• Tim Menzies
• Justin Di Stefano
• Mike Chapman
• Pat Callis
• Andres Orrego
• Robert (Mike) Chapman

Papers

“How Good is Your Blind Spot Sampling Policy?” Available from http://menzies.us/pdf/03blind.pdf.

@INPROCEEDINGS{1281737,
author={Menzies, T. and Di Stefano, J.S.},
booktitle={High Assurance Systems Engineering, 2004. Proceedings. Eighth IEEE International Symposium on},
title={How good is your blind spot sampling policy},
year={2004},
pages={129-138},
keywords={formal specification;program verification;sampling methods;software metrics;automatic formal methods;black box probing;blind spot sampling;
            defect detectors;formal specification;public domain defect data;software assessment;Aerospace engineering;Computer science;Costs;
            Detectors;Mission critical systems;NASA;Project management;Proposals;Sampling methods;Systems engineering and theory},
doi={10.1109/HASE.2004.1281737},
ISSN={1530-2059},
month={March},}

“Assessing Predictors of Software Defects” Available from http://menzies.us/pdf/04psm.pdf .

@INPROCEEDINGS{Menzies04assessingpredictors,
    author = {Tim Menzies and Justin Distefano and Andres Orrego S and Robert (mike Chapman},
    title = {Assessing predictors of software defects},
    booktitle = {in Proceedings, workshop on Predictive Software Models},
    year = {2004}
}