Assessing the cost effectiveness of fault prediction in acceptance testing

Akito Monden, Takuma Hayashi, Shoji Shinoda, Kumiko Shirai, Junichi Yoshida, Mike Barker, Kenichi Matsumoto

Research output: Contribution to journalArticle

39 Citations (Scopus)

Abstract

Until now, various techniques for predicting fault-prone modules have been proposed and evaluated in terms of their prediction performance; however, their actual contribution to business objectives such as quality improvement and cost reduction has rarely been assessed. This paper proposes using a simulation model of software testing to assess the cost effectiveness of test effort allocation strategies based on fault prediction results. The simulation model estimates the number of discoverable faults with respect to the giventest resources, the resource allocation strategy, a set of modules to be tested, and the fault prediction results. In a case study applying fault prediction of a small system to acceptance testing in the telecommunication industry, results from our simulation model showed that the best strategy was to let the test effort be proportional to 'the number of expected faults in a module x log(module size).' By using this strategy with our best fault prediction model, the test effort could be reduced by 25 percent while still detecting as many faults as were normally discovered in testing, although the company required about 6 percent of the test effort for metrics collection, data cleansing, and modeling. The simulation results also indicate that the lower bound of acceptable prediction accuracy is around 0.78 in terms of an effort-aware measure, (Norm(P-{opt})).The results indicate that reduction of the test effort can be achieved by fault prediction only if the appropriate test strategy is employed with high enough fault prediction accuracy. Based on these preliminary results, we expect further research to assess their general validity with larger systems.

Original languageEnglish
Article number6497441
Pages (from-to)1345-1357
Number of pages13
JournalIEEE Transactions on Software Engineering
Volume39
Issue number10
DOIs
Publication statusPublished - 2013
Externally publishedYes

Fingerprint

Cost effectiveness
Testing
Telecommunication industry
Software testing
Cost reduction
Resource allocation
Industry

Keywords

  • Complexity measures
  • fault prediction
  • quality assurance
  • resource allocation
  • simulation

ASJC Scopus subject areas

  • Software

Cite this

Monden, A., Hayashi, T., Shinoda, S., Shirai, K., Yoshida, J., Barker, M., & Matsumoto, K. (2013). Assessing the cost effectiveness of fault prediction in acceptance testing. IEEE Transactions on Software Engineering, 39(10), 1345-1357. [6497441]. https://doi.org/10.1109/TSE.2013.21

Assessing the cost effectiveness of fault prediction in acceptance testing. / Monden, Akito; Hayashi, Takuma; Shinoda, Shoji; Shirai, Kumiko; Yoshida, Junichi; Barker, Mike; Matsumoto, Kenichi.

In: IEEE Transactions on Software Engineering, Vol. 39, No. 10, 6497441, 2013, p. 1345-1357.

Research output: Contribution to journalArticle

Monden, A, Hayashi, T, Shinoda, S, Shirai, K, Yoshida, J, Barker, M & Matsumoto, K 2013, 'Assessing the cost effectiveness of fault prediction in acceptance testing', IEEE Transactions on Software Engineering, vol. 39, no. 10, 6497441, pp. 1345-1357. https://doi.org/10.1109/TSE.2013.21
Monden, Akito ; Hayashi, Takuma ; Shinoda, Shoji ; Shirai, Kumiko ; Yoshida, Junichi ; Barker, Mike ; Matsumoto, Kenichi. / Assessing the cost effectiveness of fault prediction in acceptance testing. In: IEEE Transactions on Software Engineering. 2013 ; Vol. 39, No. 10. pp. 1345-1357.
@article{64a483e04ea8430ea760441bff77c7e7,
title = "Assessing the cost effectiveness of fault prediction in acceptance testing",
abstract = "Until now, various techniques for predicting fault-prone modules have been proposed and evaluated in terms of their prediction performance; however, their actual contribution to business objectives such as quality improvement and cost reduction has rarely been assessed. This paper proposes using a simulation model of software testing to assess the cost effectiveness of test effort allocation strategies based on fault prediction results. The simulation model estimates the number of discoverable faults with respect to the giventest resources, the resource allocation strategy, a set of modules to be tested, and the fault prediction results. In a case study applying fault prediction of a small system to acceptance testing in the telecommunication industry, results from our simulation model showed that the best strategy was to let the test effort be proportional to 'the number of expected faults in a module x log(module size).' By using this strategy with our best fault prediction model, the test effort could be reduced by 25 percent while still detecting as many faults as were normally discovered in testing, although the company required about 6 percent of the test effort for metrics collection, data cleansing, and modeling. The simulation results also indicate that the lower bound of acceptable prediction accuracy is around 0.78 in terms of an effort-aware measure, (Norm(P-{opt})).The results indicate that reduction of the test effort can be achieved by fault prediction only if the appropriate test strategy is employed with high enough fault prediction accuracy. Based on these preliminary results, we expect further research to assess their general validity with larger systems.",
keywords = "Complexity measures, fault prediction, quality assurance, resource allocation, simulation",
author = "Akito Monden and Takuma Hayashi and Shoji Shinoda and Kumiko Shirai and Junichi Yoshida and Mike Barker and Kenichi Matsumoto",
year = "2013",
doi = "10.1109/TSE.2013.21",
language = "English",
volume = "39",
pages = "1345--1357",
journal = "IEEE Transactions on Software Engineering",
issn = "0098-5589",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
number = "10",

}

TY - JOUR

T1 - Assessing the cost effectiveness of fault prediction in acceptance testing

AU - Monden, Akito

AU - Hayashi, Takuma

AU - Shinoda, Shoji

AU - Shirai, Kumiko

AU - Yoshida, Junichi

AU - Barker, Mike

AU - Matsumoto, Kenichi

PY - 2013

Y1 - 2013

N2 - Until now, various techniques for predicting fault-prone modules have been proposed and evaluated in terms of their prediction performance; however, their actual contribution to business objectives such as quality improvement and cost reduction has rarely been assessed. This paper proposes using a simulation model of software testing to assess the cost effectiveness of test effort allocation strategies based on fault prediction results. The simulation model estimates the number of discoverable faults with respect to the giventest resources, the resource allocation strategy, a set of modules to be tested, and the fault prediction results. In a case study applying fault prediction of a small system to acceptance testing in the telecommunication industry, results from our simulation model showed that the best strategy was to let the test effort be proportional to 'the number of expected faults in a module x log(module size).' By using this strategy with our best fault prediction model, the test effort could be reduced by 25 percent while still detecting as many faults as were normally discovered in testing, although the company required about 6 percent of the test effort for metrics collection, data cleansing, and modeling. The simulation results also indicate that the lower bound of acceptable prediction accuracy is around 0.78 in terms of an effort-aware measure, (Norm(P-{opt})).The results indicate that reduction of the test effort can be achieved by fault prediction only if the appropriate test strategy is employed with high enough fault prediction accuracy. Based on these preliminary results, we expect further research to assess their general validity with larger systems.

AB - Until now, various techniques for predicting fault-prone modules have been proposed and evaluated in terms of their prediction performance; however, their actual contribution to business objectives such as quality improvement and cost reduction has rarely been assessed. This paper proposes using a simulation model of software testing to assess the cost effectiveness of test effort allocation strategies based on fault prediction results. The simulation model estimates the number of discoverable faults with respect to the giventest resources, the resource allocation strategy, a set of modules to be tested, and the fault prediction results. In a case study applying fault prediction of a small system to acceptance testing in the telecommunication industry, results from our simulation model showed that the best strategy was to let the test effort be proportional to 'the number of expected faults in a module x log(module size).' By using this strategy with our best fault prediction model, the test effort could be reduced by 25 percent while still detecting as many faults as were normally discovered in testing, although the company required about 6 percent of the test effort for metrics collection, data cleansing, and modeling. The simulation results also indicate that the lower bound of acceptable prediction accuracy is around 0.78 in terms of an effort-aware measure, (Norm(P-{opt})).The results indicate that reduction of the test effort can be achieved by fault prediction only if the appropriate test strategy is employed with high enough fault prediction accuracy. Based on these preliminary results, we expect further research to assess their general validity with larger systems.

KW - Complexity measures

KW - fault prediction

KW - quality assurance

KW - resource allocation

KW - simulation

UR - http://www.scopus.com/inward/record.url?scp=84884841320&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84884841320&partnerID=8YFLogxK

U2 - 10.1109/TSE.2013.21

DO - 10.1109/TSE.2013.21

M3 - Article

AN - SCOPUS:84884841320

VL - 39

SP - 1345

EP - 1357

JO - IEEE Transactions on Software Engineering

JF - IEEE Transactions on Software Engineering

SN - 0098-5589

IS - 10

M1 - 6497441

ER -