Impact of the Distribution Parameter of Data Sampling Approaches on Software Defect Prediction Models

Kwabena Ebo Bennin, Jacky Keung, Akito Monden

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Citations (Scopus)

Abstract

Sampling methods are known to impact defect prediction performance. These sampling methods have configurable parameters that can significantly affect the prediction performance. It is however, impractical to assess the effect of all the possible different settings in the parameter space for all the several existing sampling methods. A constant and easy to tweak parameter present in all sampling methods is the distribution of the defective and non-defective modules in the dataset known as Pfp (% of fault-prone modules). In this paper, we investigate and assess the performance of defect prediction models where the Pfp parameter of sampling methods are tweaked. An empirical experiment and assessment of seven sampling methods on five prediction models over 20 releases of 10 static metric projects indicate that (1) Area Under the Receiver Operating Characteristics Curve (AUC) performance is not improved after tweaking the Pfp parameter, (2) pf (false alarms) performance degrades as the Pfp is increased. (3) a stable predictor is difficult to achieve across different Pfp rates. Hence, we conclude that the Pfp parameter setting can have a large impact on the performance (except AUC) of defect prediction models. We thus recommend researchers experiment with the Pfp parameter of the sampling method since the distribution of training datasets vary.

Original languageEnglish
Title of host publicationProceedings - 24th Asia-Pacific Software Engineering Conference, APSEC 2017
PublisherIEEE Computer Society
Pages630-635
Number of pages6
Volume2017-December
ISBN (Electronic)9781538636817
DOIs
Publication statusPublished - Mar 1 2018
Event24th Asia-Pacific Software Engineering Conference, APSEC 2017 - Nanjing, Jiangsu, China
Duration: Dec 4 2017Dec 8 2017

Other

Other24th Asia-Pacific Software Engineering Conference, APSEC 2017
CountryChina
CityNanjing, Jiangsu
Period12/4/1712/8/17

Keywords

  • Defect prediction
  • Empirical software engineering
  • Imbalanced Data
  • Preprocessing
  • Sampling methods
  • Search based SE

ASJC Scopus subject areas

  • Software

Fingerprint Dive into the research topics of 'Impact of the Distribution Parameter of Data Sampling Approaches on Software Defect Prediction Models'. Together they form a unique fingerprint.

  • Cite this

    Bennin, K. E., Keung, J., & Monden, A. (2018). Impact of the Distribution Parameter of Data Sampling Approaches on Software Defect Prediction Models. In Proceedings - 24th Asia-Pacific Software Engineering Conference, APSEC 2017 (Vol. 2017-December, pp. 630-635). IEEE Computer Society. https://doi.org/10.1109/APSEC.2017.76