Arguing Practical Significance in Empirical Software Engineering

Richard Torkar, Carlo A. Furia, Robert Feldt, Francisco Gomes de Oliveira Neto, Lucas Gren, Per Lenberg, Neil A. Ernst

A key goal of empirical research is assessing practical significance, which answers the question of whether the observed effects of some compared treatments show a difference that is relevant in practice in realistic scenarios. Even though plenty of standard techniques exist to assess statistical significance, connecting it to practical significance is something that is not straightforward or routinely done; indeed, only a few empirical studies in software engineering assess practical significance in a principled and systematic way. In this paper, we argue that Bayesian data analysis provides suitable tools to rigorously assess practical significance. We demonstrate our claims in a case study comparing different test techniques. The case study's data was previously analyzed (Afzal et al., 2015) using standard techniques focusing on statistical significance. We build a multilevel model of the same data, which we fit and validate using Bayesian techniques. By applying cumulative prospect theory on top of the statistical model, we are able to quantitatively connect the output of our statistical analysis to a practically meaningful context, thus assessing practical significance. Our study demonstrates that Bayesian analysis provides a technically rigorous, yet practical framework, for empirical software engineering. In combination with cumulative prospect theory, Bayesian analysis supports seamlessly assessing practical significance in an empirical software engineering context, thus potentially extending the relevance of research for practitioners.

Knowledge Graph



Sign up or login to leave a comment