Sertifikasyon testi için test senaryolarının önceliklendirilmesi ve paralel yürütülmesi

Systems that are developed in the consumer electronics domain are subject to testing for certification regularly. For instance, each Smart TV product must go through a certification process that is regulated by application vendors like Netflix, YouTube and Amazon. There exist a separate test suite to be used as part of the certification process pertaining to each application. Each test suite includes hundreds of test cases. Many of these test cases have to be executed manually and it might take several days to complete the execution of certification test suites. There also exists a high variation among the execution times of test cases and severities of faults detected by these test cases. Therefore, prioritization of test cases and parallel test execution can significantly impact the duration and effectiveness of the process. We measure test effectiveness by calculating the average of the percentage of faults detected. In this thesis, we introduce variants of existing metrics to take parallel test execution, varying test execution times, and fault severity levels in ordinal scale into account. We also propose an approach for prioritizing such test cases that can be executed on multiple test stations in parallel. We employ integer linear programming for optimizing the allocation of test execution tasks on a set of available test stations. The goal of this allocation is to minimize the overall test execution time. Test cases that are allocated to a particular stations are executed in increasing order of their execution times. We conducted an industrial case study in the context of certification testing for Smart TV software. We used certification test suites of 3 Smart TV applications applied on 3 TV software projects as real experimental objects. We repeated our measurements for two test cycles for each project, adding up to 6 tests performed for each of the 3 test suites. So, our evaluation involves 18 test sessions. We evaluated 3 scenarios for each of these sessions, where the number of available test stations are 1, 3 and 5. We compared the effectiveness of our approach with respect to a greedy approach as the baseline. We observed that the overall test execution time can be reduced by up to 16\% even when only 3 test stations are available. Test effectiveness is also improved as a result of optimal scheduling of test cases.