Miika Lehto
Oct 25, 2011
Posted in category: 2011 Member Articles

The concept of risk-based testing and its advantages and disadvantages

In general,  the objective of testing is to detect defects. This means that tests are performed in order to show a lack of quality as revealed by defects. In general, this means establishing the difference between the product and the requirements. Therefore, testing always involves comparing; it requires an object that is to be tested and terms of reference with which the object should comply. (Koomen and Pol 1999)
A defect in the system may lead to undesirable consequences for both the development organization and the system users. The potential for such undesirable consequences is usually referred to as risk. Higuera (1996) introduces that risk always involves uncertainty and loss. Uncertainty implies that the risk may or may not happen and there are no 100 percent probable risks. Loss, in turn, implies that if risk becomes a reality, undesired consequences or losses will occur.
A test approach that takes into account a risk is called risk-based testing. By identifying and analyzing the risks related to the system, it is possible to focus the test effort on the most critical areas of the system. However, one could say that testers have always utilized risk-based testing. For instance, Amland (2000) presents that testers have always used risk-based testing but they have done it in ad-hoc fashion since they have not usually utilized formal risk assessment methods but based their decisions what to test on their personal expertise. Hence, the fundamental difference between traditional testing and risk-based testing approach is that risk-based testing brings formal risk assessment methods to the testing process.
According to Claesson (2001), risk-based testing can be used to give directions for which test strategies to use; select test objects and the most important test cases; explore the risks when performing the test execution; evaluate the current level of system quality and risk; report test results related to the risks that were evaluated; and provide input for the decision to continue or stop testing.
The usage of risk-based testing brings several advantages to the testing organization. Firstly, it provides a method to prioritize tests against deadlines (Schaefer 1998). Hence, testers can show management that they have made the best use of the resources invested to testing by using risk-based testing (Bach 1999). Secondly, risk-based testing enables to find suitable test coverage for the assessed risk (Ottevanger 1999). However, this may not happen immediately since it requires experience and expertise to assess suitable test coverage reliably. Consequently, it is likely that the more experience the test organization is, the better they can assess the sufficient test coverage for each risk. In addition to these advantages, Claesson (2006) specifies the following benefits of risk-based testing:

  • Projects are able to define when to stop testing.
  • Test cases can be reduced and focused on the most critical areas.
  • Less but more efficient test cases can be specified.
  • A better and more focused tests and risk analysis are performed.
  • Problem areas are discovered early. Preventive activities can be started immediately.
  • Better strategies and test objects/cases can be selected.
  • Overall test goals, strategies, and directions for testing can be focused and continuously adjusted against problem areas.
  • Risks can be continuously monitored to know the status of the project and its quality.

Although risk-based testing has several advantages, it also includes some disadvantages. One disadvantage is that there may be unrecognized risks involved and risks that are assessed to be too low (Pyhäjärvi and Pöyhönen 2006). However, this only causes problems if the risks will become a reality. This disadvantage also emphasizes the importance of risk identification and analysis processes as a basis of risk-based testing approach. Another weakness is that risk assessment can be based on too subjective criteria (Pyhäjärvi and Pöyhönen 2006). The reason for that is simply the lack of reliable objective criteria and in that case it is quite common to trust to experts’ judgments. This fluently leads to the next weakness which is the difficulty to identify and select the right stakeholders for risk assessment. For example, if a customer is asked to participate in risk assessment, it can be quite a surprise for the customer in some cases that the product is not tested fully. Hence, in the worst case, the attendance of a customer in the risk assessment sessions may have a negative effect on the customer satisfaction and how the customer appreciates the supplier.
Finally, it may also be difficult to attach a test to an identified risk (Pyhäjärvi and Pöyhönen 2006). This is the case if the risks are described too abstractly. Therefore, it is essential to describe the risks as concretely as possible and to assess the testability of risks during risk identification.

References

Amland S., “Risk-based testing: Risk analysis fundamentals and metrics for software testing including a financial application case study”, The Journal of Systems and Software, Vol. 53, No. 3, 2000, pp. 287-295.

Bach J., “Risk Based Testing”, Software Testing and Quality Engineering Magazine, Vol. 1, No. 6, 1999.

Claesson A., “A Risk Based Testing Process”, Enea Realtime AB, 2001.

Claesson A. “Efficient Test Techniques for Function and System Testing”, Enea Realtime AB, 2006.

Higuera, R.P., Haimes, Y.Y., “Software Risk Management , Software Engineering Institute, CMU/SEI-96-TR-012, June 1996

Koomen T., and Pol M., “Test process improvement: a practical step-by-step guide to structured testing”, Addison Wesley, 1999, pp 5-6.

Ottevanger I. B., “A Risk-Based Test Strategy”, IQUIP Informatica B.V, 1999.

Pyhäjärvi M., and Pöyhönen E., ”Ohjelmistotestaus, osa 2”, 2006.

Schaefer H., “Strategies for Prioritizing Test Against Deadlines”, STAR WEST, 1998.

Contact us Download BT Standard as PDF

 

Looking for answers to your questions about the BT Standard? Get in touch!