Improving testability of safety-critical software

Nowadays software controls large majority of the systems that humankind use. Systems that software is used widely, such as transportation, military, medicine and avionics must be safe during the operation. Failure in these critical systems may cause catastrophic results (i.e. loss of human life, loss or severe damage to environment or equipment etc.). In order to avoid failure on safety critical software or at least mitigate the risks, elaborated testing is required. Safety critical software systems present more testing challenges as compared to general-purpose systems. Safety critical software's dependency on hardware, the limited user interface and lack of tools makes the testability of this software more difficult. Quality in safety critical software is generally tied to platform-specific testing tools geared towards debugging. Engineers usually develop safety critical software systems with sequential phases and test with test at the end approach for certification purposes. Although effective testing helps to correct the functioning of software systems, the goal of testing is not proving that no errors exist. The goal of testing is to prove that software may have some bugs. No matter how much time and resources allocated for testing, it is impossible to test all inputs and corresponding outputs of software because of time and budget constraints of the project. These general testing principles are also valid for safety critical domain. However, designing for testability through hardware abstraction in conjunction with low complexity, few lines of code and simplicity of the code help improve testability. Test at the end approach is not sufficient for any software system when requirements are unclear or change frequently. To overcome weaknesses of test at the end, software systems can be tested starting from the early stages. To test in the early stages, software must be testable. Testability of software is defined as whether or not software supports testing activities. Software testability is considered under software quality. Software quality is defined as degree to which the software product satisfies stated and implied needs when used under specified condition. To evaluate software quality first we need a way to measure it. Correct measurement of software quality can help project team to understand which parts of software are testable and which are not. Measurement is the assignment of numbers to object or event properties. Suitable metrics should be defined for object or event before the measurement. Measurement in software development plays an important role in every phase. For measuring software quality, suitable software metrics should be chosen. With the help of software metrics it is possible to obtain some clues about software quality. The important point here is that software metrics should be analyzed together. If metrics are analyzed on their own, they will not tell too much. This thesis investigates testability from the perspective of metrics used in an object-oriented system. The main idea is giving an overview of software metrics with the prioritization of testability as the overall goal. Several metrics have been proposed to identify testability weaknesses. However, it is sometimes difficult to be convinced that those metrics are really related to testability. For the purpose of this study, testability of safety critical software and its relation to object oriented patterns, principles and practices were investigated. In this work testability is understood as unit level testability. A case study was conducted for railway interlocking management software component. Studied software component was first developed under National Railway Signalization Project by the collaboration of The Scientific and Technological Research Council of Turkey and Istanbul Technical University. The original railway interlocking management software was developed using object oriented design and programming. The component was analyzed to measure its existing testability. Testability of software component was determined by using object oriented and non-object oriented software metrics. Well known Chidamber and Kemerer (CK) metrics suite was chosen for object oriented metrics. With the widespread use of object-oriented technologies, CK metrics have proved to be very useful. Complexity and lines of code (LOC) metrics were chosen for non-object oriented metrics. The measurement results showed that most of the classes inside railway interlocking management software are very complex according to cyclomatic complexity, weighted methods per class and response for a class metrics. Their cohesion is low according to lack of cohesion of methods metric and couplings between them are high according to coupling between object classes metric. Complexity, cohesion and coupling are three important factors that affect testability of software. Testing software becomes a tedious task as the size and complexity of software increases. In addition, lack of cohesion inside software system is directly related with complexity. Likewise software system that has huge numbers of couplings is hard to test. Because these couplings must be supplied by, providing real or fake objects before, testing begins. In terms of complexity, cohesion and couplings existing testability of railway interlocking management software is low. To prove the correlation between good design and software testability, second version of the interlocking management software was developed. For second version, test driven software design and development approach was used. The primary benefit of using test driven development is that it improves the design of the code. In addition, some well-known object oriented principles and patterns were applied to increase the software testability. After completion of development, the second version is analyzed using same metrics and results were compared with previous results. In addition, some code coverage metrics like statement, function and branch coverage were measured. The measurement results showed that testability of second version is considerably improved in terms of complexity, cohesion and coupling. This argument is supported by code coverage metrics results. However, several threats to validity of study were also identified. First, this study's sample size was quite small. This was due primarily safety critical software systems are usually developed by using closed source software technologies. Therefore, the study sample is chosen from a closed sourced safety critical software project that author previously worked. A small size of study makes it easy to argue that the numbers in this study may not reach a suitable level of statistical significance. In addition, it is easy to argue that metric analysis is insufficient for software testability improvement. Various researchers took different approaches to the software testability measurement in their studies. Generally, software testability is not measured by using software metrics in those studies. However, in this study the chosen metrics are used for measuring complexity, cohesion and coupling in software systems. Therefore, improvement in that trio should affect software testability in positive way. In addition, one can argue that validation of test driven development for improving software testability is insufficient. However, test driven development effort in this study is supported by code coverage metrics like statement, function and branch coverage.