Software Testing Methodologies Dichotomies

• Testing consumes atleast half of the time and work required to produce a functional program
• History reveals that even well written programs still have 1-3 bugs per hundred statements

Productivity and Quality in Software: 

• In production of consumer goods and other products, every manufacturing stage is subjected to quality control and testing from component to final stage
• If flaws are discovered at any stage, the product is either discarded or cycled back for rework and correction

• Productivity is measured by the sum of the costs of the material, the rework and the discarded components and the cost of quality assurance and testing

• There is a trade-off between quality assurance costs and manufacturing costs
• If sufficient time is not spent in quality assurance, the reject rare is high and so is the net cost
• If inspection is good and all errors are caught as they occur, inspection costs dominates and again the net cost suffers

• Testing and Quality assurance costs for manufactured items can be as low as 2% in consumer products or as high as 80% in products such as spaceships, nuclear reactiors and aircrafts, where failures threaten life. Where as the amnufacturing cost of a software is trivial
• The biggest part of software cost is the cost of bugs, the cost of detecting them, the cost of correcting them, the cost of desinging tests that discover them and the cost of running those tests
• For software, quality and productivity are indistinguishable, because the cost of a software copy is trivial

Goals for Testing:

• Testing and test design are parts of quality assurance that should also focus on bug prevention
• A prevented bug is better than a detected and corrected bug

Phases in testers mental life can be categorized into the following 5 phases:
Phase 0: Until 1956: (Debugging Oriented)
Phase 1: 1957-1978: (Demonstration Oriented)
Phase 2: 1979-1982: (Destruction Oriented)
Phase 3: 1983-1987: (Evaluation Oriented)
Phase 4: 1988-2000: (Prevention Oriented)

Phase 0: Until 1956: (Debugging Oriented)

• There is no difference between testing and debugging
• Phase 0 thinking was the norm in early days of software development till testing emerged as a discipline

Phase 1: 1957-1978: (Demonstration Oriented)

• The purpose of testing here is to show that software works
• Highlighted during the late 1970's
• This failed because the probability of showing that software work 'decreases' as testing increases. i.e. the more the test, the more likely a bug is found

Phase 2: 1979-1982: (Destruction Oriented)

• The purpose of testing is to show that software doesn't work
• This phase also failed because the software is never released, as we find one bug or the other. Also a bug corrected may also lead to another bug

Phase 3: 1983-1987: (Evaluation Oriented)

• The purpose of testing is not to prove anything but to reduce the perceived risk of not working to an acceptable value (Statistical Quality Control)
• Notion is that testing does improve the product, to the extent that testing catches bugs, and to the extent that those bugs are fixed
• The product is released when the confidence on that product is high enough ( Note: This is applied to large software products with millions of code years of use)

Phase 4: 1988-2000: (Prevention Oriented)\

• Testability is the factor considered here, Once reason is to reduce the labour of testing
• Other reason is to check the testable and non-testable code
• Testable code has fewer bugs than the code that's hard to test
• Identifying the testing techniques to test the code is the main key here

Test Design

• Software design is the process of implementing software solutions to one or more set of problems, One of the important parts of software design is the Software Requirements Analysis(SRA)
• We know that the software code must be designed and tested, but many appear to be unaware that tests themselves must be designed and tested
• Tests should be properly designed and tested before applying it to the actual code
• Testing is not everything. There are approaches other than testing to create better software
• Methods other than testing include:

Inspection Methods: Methods like walk throughs, desk checking, formal inspections and code reading appear to be as effective as testing, but the bugs caught do not completely overlap
Design Style: While designing the software itself, adopting stylistic objectives such as testability, openness and clarity can do much, to prevent bugs
Static Analysis Methods: It includes formal analysis of source code during compilation, In earlier days, this is a routine job of the programmer, Now, the compilers have taken over this job
Languages: The source language can help reduce certain kinds of bugs. Programmers find new bugs while using new languages
Development Methodologies and Development Environment: The development process and the environment in which that methodology is embedded, can prevent many kinds of bugs