Combining Operational and Debug Testing for Improving Reliability

This paper addresses the challenge of reliability-driven testing, i.e., of testing software systems with the specific ob- jective of increasing its operational reliability. We first examined the most relevant approach oriented toward this goal, namely operational testing. The main issues that in the past hindered its wide-scale adoption and practical application are first discussed, followed by the analysis of its performance under different conditions and configurations. Then, a new approach conceived to overcome the limits of operational testing in delivering high reliability is proposed. The two testing strategies are evaluated probabilistically and by simulation. Results report on the performance of operational testing when several involved parameters are taken into account, and on the effectiveness of the new proposed approach in achieving better reliability. At higher level, the findings of the paper also suggest that a new way of thinking the concept of testing for reliability improvement may help in making up with novel testing approaches for high-reliability demanding systems.