Scalable Model-based Robustness Testing: Novel Methodologies and Industrial Application
Appears in the following Collection
- Institutt for informatikk 
AbstractEmbedded systems, as for example communication and control systems, are being increasingly used in our daily lives and hence require thorough and systematic testing before their actual use. Many of these systems interact with their environment and, therefore, their functionality is largely dependent on this environment whose behavior can be unpredictable. Robustness testing aims at testing the behavior of a system in the presence of faulty situations in its operating environment (e.g., sensors and actuators). In such situations, the system should gracefully degrade its performance instead of abruptly stopping execution. To systematically perform robustness testing, one option is to resort to Model-Based Robustness Testing (MBRT), which is a systematic, rigorous, and automated way of conducting robustness testing. However, to successfully apply MBRT in industrial contexts, new technologies need to be developed to scale to the complexity of real industrial systems. This thesis presents a solution for MBRT on industrial systems, including scalable robustness modeling and executable test case generation. One important contribution of this thesis is a scalable RobUstness Modeling Methodology (RUMM), which is achieved using Aspect-Oriented Modeling (AOM). It is a complete, automated, and practical methodology that covers all features of state machines and aspect concepts necessary for MBRT. Such methodology, relying on a standard (Unified Modeling Language or UML) and using the target notation as the basis to model the aspects themselves, is expected to make the practical adoption of robustness modeling easier in industrial contexts. The applicability of the methodology is demonstrated using an industrial case study. Results showed that the approach significantly reduced modeling effort (98% on average), improved separation of concerns, and eased model evolution. The approach is further empirically evaluated using two controlled experiments involving human subjects and results showed that the proposed methodology significantly improves the readability of models as compared to modeling using standard UML notations. Another important contribution of this thesis is an efficient approach for solving constraints (written in Objects Constraint Language (OCL)) on the operating environment of a system, which is mandatory for emulating faulty situation in the environment for the purpose of MBRT. A set of novel heuristics is devised for various OCL constructs, which are required for the application of search algorithms. The heuristics have been empirically evaluated on an industrial case study for robustness testing and the results showed to be very promising and significantly better than the existing works in the literature on OCL constraint solvers. A final contribution of the thesis is robustness test case generation from the models developed using RUMM. Test case generation also includes scripts generation for environment emulation, which is mandatory for automated robustness testing again using an industrial case study. In preliminary experiments, the execution of test cases found one critical, robustness fault in a deployed industrial system.
List of papers.
Paper 1 S. Ali, L. Briand, H. Hemmati, and R. K. Panesar-Walawege A Systematic Review of the Application and Empirical Investigation of Search-based Test-Case Generation © ACM, 2010. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in IEEE Transactions on Software Engineering (TSE), 2010, vol 36, no 6, pp. 742-762, 2010 doi:10.1109/TSE.2009.52
Paper 2 S. Ali, L. Briand, and H. Hemmati Modeling Robustness Behavior Using Aspect-Oriented Modeling to Support Robustness Testing of Industrial Systems Journal of Software and Systems Modeling (SOSYM), 2011. Online First June 15 2011 The original publication is available at www.springerlink.com doi:10.1007/s10270-011-0206-z
Paper 3 S. Ali, T. Yue, and L. Briand Does Aspect-Oriented Modeling Help Improve the Readability of UML State Machines? Submitted version, published in: Software & Systems Modeling, 2012, Online First November 16 2012 The original publication is available at www.springerlink.com doi:10.1007/s10270-012-0293-5
Paper 4 S. Ali, M. Z. Iqbal, A. Arcuri, and L. Briand. Solving OCL Constraints for Test Data Generation in Industrial Systems with Search Techniques Submitted version, published as: Generating Test Data from OCL Constraints with Search Techniques © ACM, 2013. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in IEEE Transactions on Software Engineering, Early acccess April 1 2013 doi:10.1109/TSE.2013.17
Paper 5. S. Ali, L. Briand, A. Arcuri, and S. Walawege. An Industrial Application of Robustness Testing using Aspect-Oriented Modeling, UML/MARTE, and Search Algorithms The original publication is available at www.springerlink.com Lecture Notes in Computer Science, 2011, Volume 6981/2011, 108-122. doi:10.1007/978-3-642-24485-8_9