In 2005, I worked for a company, a leader in the mobile phone and communication space. They hired me just as they launched a very stylish-looking, expensive, flip phone model (the kind people like to flash around quite a bit!).
While testing for a software update on the flip phone, I discovered that when a missed call is given to a number that’s switched off and again when it is switched on, the flip phone started vibrating and continues to vibrate till the battery runs out! The ‘switch off’ button (or for that matter any other button) on the phone fails to work in such an event. The only way to cut off power cycle is to pull out the battery and put it back!
Can you imagine how the neurons in my brain vibrated when I discovered this bug? I was excited, and typically, all testers who detect a bug are excited!
Birth of the software tester
Software testing happened even before someone was designated that way; curious programmers tested their code but weren’t specialised to do it. Gerald M Weinberg is known to have started the first test team in IBM that built specialised testers in the 1950s. The growth of this breed was slow, but consistent till 1980.
Since then hundreds and thousands of software test teams have sprouted across companies that produce software. The business value of software testing as per Gartner Research of 2008 crosses more than 14 billion US dollars. Of this, a chunk of the testing business is outsourced and India acts as hub for such testing work. If you add businesses that thrive around software testing, you can add another billion dollars to it!
What testers do
They provide quality-related information to the company management to help them take better-informed decisions about their products. Being a software tester is sometimes like being able to handle a situation such as the Chakravyuha, a war strategy, first cited in the epic Mahabharatha. Chakravyuha was a strategy to encircle the enemy in 360 degrees, making it nearly impossible for the enemy to escape. Imagine being at the batting crease when 6 runs is needed of one ball to win an important match. That’s the modern Chakravyuha. Let’s break down the job further:
Understand the specification or requirements of the software and question them. A defect found at the requirement level is much cheaper to fix than once you have finished developing the product. So testers interact with business analysts and review the specifications of the software.
They contribute to design of the product by testing the design and reporting the problems with it and later reviewing the fixes.
Most people think that a programme is the same as a software. It’s not! A tester must be able to work on software ranging from that of a mobile phone to that of a Merc!
Dealing with testing: There are more than a hundred types of test techniques, tens of approaches to test, thousands of test tools, dozens and dozens of technologies and domains and millions of possible tests.
The challenge here is: what approaches do we apply? What techniques do we apply? How do we know that it is a right approach and technique, what tools can help us, what if there are 100 tools that do the same thing, what suits our context, how do we know we have tested enough, how much time do we have to test, how many of those millions of possible tests can we do and how do we know if we made the right sampling from those millions of possible tests?!
The choices are many and the quality of work depends on the wisdom of our choices. Hence, good testing is a context-driven activity.
Top testing myths In the IT boom from 2003 onwards, several IT companies hired in hoards. Those who could not evolve into good testers quit their jobs or moved to a different role, and this phenomenon led to several myths about the profession.
Myth 1: It’s a boring job!
Every profession has some level of boring stuff and I admit that software testing is no exception. An example of what is boring in software testing is : executing test cases. But the good news is people have devised methodologies, approaches and techniques to get away from executing test cases and those ideas are gaining huge popularity. Some of the prominent ones are Rapid Software Testing and Exploratory Testing.
Myth 2: For engineering grads, only!
Interestingly, one of the finest testers Jon Bach (James Bach’s brother) has a journalism degree! In India, organisations such as Infosys, Wipro, CTS and TCS, recruit BSc, BCom, and other graduates to test software. Many product companies have also provided opportunities for people in support to move to testing since being in support is a wonderful way to know how the customer thinks. I think it’s a field open to the finest thinkers.
Myth 3: I can be a great tester without knowing programming.
Though knowledge of programming is not a mandatory skill, in the long run one realises it does help . Moreover, it is also an easily learnable skill. Some people need 100 hours and others need 200. There are many jobs that don’t demand testers to know programming but you could be a better tester if you knew how to write and read code. I have personally found lots of problems in the software I was testing because I read the code and understood the problem in it.
Myth 4: Software tester salaries not as good as developers’.
Before 2004, software testers were paid less as compared to developers, but from 2005 onwards most parts of the industry started treating developers and testers on par. Today in 2009, I know testers who are paid more salary than developers. But by saying that I don’t mean that testing has overtaken development in terms of money.
The skilled and the vocal shall always be paid a lot in this industry. Be it a tester, or a developer or a network administrator or a support executive. If you want to be an average Joe, no problem, you will still be paid on par with all other average Joes of the industry.
Myth 5: Anyone can be a tester.
The question I have to people who say that is:
Anyone can play cricket but the question is; who plays well?
Anyone can write books but the question is; who writes well?
Anyone can test but the question is; who tests well?
Myth 6: Software testing doesn’t have a career growth Visit the Naukri or Monster website and search for openings related to software testing. You’d discover roles and designations such as the following (See Box)plus the proposed salary packages of that order.
In addition to these, you may want to choose to be an independent consultant or a freelance consultant. The money you earn as a freelance consultant depends on your reputation and credibility. Bigger the reputation, up goes the moolah!
How to find a job
Campus recruitment: Many companies pick recruits for the position of developers, testers, support and networking. If you are about to be picked from one such campus recruitment you might want to specifically state your request and choice to be a tester. After one joins an organisation straight out of college through campus recruitment, they usually provide an intensive training on testing and other technologies before being put into a project.
Off campus recruitment: Off-campus recruitment happens either directly by a walk-in or through a model called active sourcing. In active sourcing, leading testing institutes are hired to train and deploy entry-level and mid-level testers based on requests.
Selecting an institute
Typically, institutes offer a diploma or certificate programme in software testing, which ranges from one to six months. They cost anywhere between, Rs. 10,000 to 40,000 and some institutes have an installment scheme. You must be a graduate to be eligible. Typically those with a BBA, BSc, BCom, BE, MTech or MBA, apply.
Smart tips
Ask for hands-on testing training or complete practical testing training. Don’t go to training institutes that merely run a few hundred slides and certify you as a tester. It’s like someone teaching you how to drive a car by running a thousand slides of how to do it!
Also, make sure that campus placement assistance is included.
Onsite opportunities
Scenario 1: Many companies in North America, Europe, and the United Kingdom and other parts of the world who outsource their testing work also need onsite staff for better coordination between developers and testers and hence prefer to have anywhere ranging from half to entire test team on site. There are lots of testers who are onsite, working in several locations in the world.
Scenario 2: In product organisations, there are onsite opportunities as well. Products developed in India for the rest of the world need to be tested at several locations. For instance, in 2004, I was a part of a team that was testing mobile phones that were supposed to work in 3G environments.
In 2004 India didn’t have a 3G network (sadly, barring MTNL and BSNL not in 2009, too) and some of our team members had to fly to Paris for Inter Operability Tests and United States for 3G network tests and others had to travel to Japan and Korea for other kinds of tests.
Message for tomorrow’s testers
I was invited by a large services company to address an audience of 100+ fresh graduates, who were hired on campus and were put into testing division. I am using the word ‘put’ because most of the graduates there appeared to be misguided about the prospects in testing.
Post the talk, several graduates said to me, “Why did none of your seniors from this company talk to us about how much fun and what a challenge it is to test software and the way they have been growing in it?”
My response to them was, “Your seniors didn’t. Now that you have asked this question, will you spend time for your juniors, guiding them properly?” I heard a “Yes, Sir” in unison, similar to the ones seen in military.
As I travelled back home, I felt that the generation that is going to reshape the world and reshape the way software is built and tested is today’s generation that is passing out from colleges and universities. I am glad that through this article, I could reach out to a few and hope they don’t do mistakes like the ones of my generation are doing. I wish, I was of your batch!
Dealing with people
There are several people looking for information from software testers: Senior Management, Business Analysts, Designers, Architects, Developers, Customers, Support Staff, End users, IT Admin. Besides domain knowledge, soft skills include:
Communication skills – writing, reading and speaking,People management skills,Body language, Issue-handling skills to solve problems and to avoid traps,Multi-tasking,Note-taking ,Your credibility with all people shall depend on the above mentioned skills, your technical skills and your thinking skills.
Monday, March 22, 2010
Tuesday, November 11, 2008
Using Fault Tree Analysis to Improve Software Testing
Testing a software product to remove hidden defects is an integral part of the software development life cycle (SDLC). Yet it is well accepted that running a software product through every possible scenario to check for defects is not just difficult, but usually impossible. The enormous cost and huge effort required is simply too much. Thus, more limited testing remains a major part of the software development effort as do the challenges faced in software testing.
The application of process improvement tools to the software development life cycle is becoming popular in the software community. These techniques have already been successfully leveraged by manufacturers which have encouraged software professionals to apply such tools to the SDLC. Using fault tree analysis (FTA) is one good way to improve the effectiveness of software testing. It can help identify the potential causes of a problem, suggest suitable corrective action and offer insight into preparing test case scenarios.
Challenges in Software Testing
1. Inherent challenge: It is next to impossible to test a software product of average complexity to all of its specifications and features. The number of test cases required to test every aspect of a software application would be so large that it would be economically impossible to prepare and execute. For example, a simple program to analyze a string of 10 alphabetic characters could have 2610 combinations. At a rate of one combination per microsecond, testing the string would take 4.5 million years, according to author Watts S. Humphrey in his book Managing the Software Process.
2. Laborious process of test case preparation and documentation: Test case preparation is labor intensive and has to fit into what is normally an already tight schedule. Often project teams are tempted to pay less attention to this activity. Considering the large number of test cases to be developed, it takes considerable effort to document and maintain the documentation. The project team seldom documents all the test cases and has to conduct testing with additional undocumented test cases.
3. Effectiveness of test cases: Identifying a test case is as important as writing a line of code. Lack of proper methods makes this task more challenging. Software engineering researchers, such as Glenford J. Myers in his book Software Reliability, Principles and Practices, have observed that it is impossible to test one's own programs. Test cases for a module created by the software developer tend to have an ingrained bias toward an application's functionality. Such test cases often are prepared to prove what is being developed, instead of to reveal defects – the proper objective of test cases.
Figure 1
4. Resource crunch: More effort is spent in software testing than in any other phase of software development. Figure 1 shows the distribution of efforts in the software development life cycle. The percent distributions are typical industry averages. While specific amounts of effort are scheduled for testing, projects often end up with less testing time than planned because the design and construction phases consume more effort than estimated. Tools that may increase effectiveness of testing are unavailable or unnoticed. Even if tools are available, a project team faced with a new learning curve may not be inclined to use them.
How Can the Challenges Be Met?
An analysis of the testing process reveals that one of the root causes of ineffectiveness is the process of test case creation. A test case is considered effective when it can reveal a defect. With good test cases, most latent defects can be identified and fixed before a product is shipped. Hence improving the test case creation process will help make the software testing process more effective.
The complexity of conventional test case documents often tends to become a bottleneck to improving effectiveness. The way out is to deploy the right tools to design useful test cases, ones which can reveal defects. It may not be necessary to test every possible combination since many of them could be redundant. The focus must be on those tests which can accurately tell about the health of the software.
Fault tree analysis may help simplify designing better test cases to improve effectiveness of the test process. The FTA preparation process brings in a variety of ideas, broadens the scope of thinking and adds creativity to the process.
What Is Fault Tree Analysis?
Fault tree analysis is a top-down approach to identify all potential causes leading to a defect. Each cause is further broken down into least possible events or faults. The analysis begins with a major defect. All the potential events – individual or in combination – that may cause the defect are identified. Potential events are further traced down in a similar way to the lowest possible level.
Table 1
Two logic symbols – known as logic gates – And and Or are used to represent the sequencing of events. The And symbol indicates that all preceding events must exist simultaneously for a defect to occur. The Or symbol indicates that either of the preceding events may lead to said defect. Table 1, known as a truth table, illustrates how the logic gates behave. Let's consider the And gate. The output will exist only when both inputs are present simultaneously. With reference to fault tree analysis, the fault condition exists only if the preceding events exist simultaneously. In the case of the Or gate, either of the inputs is required to produce the output condition. That is, either input state may result in a fault condition.
Figure 2: Demonstration of FTA
Figure 2 illustrates how FTA could be used for a typical situation of troubleshooting, e.g., computer not starting. The fault tree is shown with Levels 1 through 3 tracing the fault conditions, Level 1 being the highest.
There are at least two situations (faults) that may result in a computer not starting. Since either of the situations – power failure or booting failure – is capable of producing the Level 1 fault, the Or gate is used to represent their combination. The Level 2 fault, power failure, may result if the primary power source fails and at the same time the uninterruptible power supply (UPS) is down. An And gate is used to represent this situation. The event, "UPS down," may be further traced to faults like battery failure, hardware failure and so on.
Deciding the scope of FTA at the beginning is essential to limit the analysis to the required level. For example, if the focus is on problems with a computer, there is no need to analyze the failures of a UPS as it may not be an integral part of a computer. However, the failure of a motherboard associated with a booting problem may be discussed further as it is very much part of a computer system.
Advantages of Applying FTA
FTA can be advantageous to software projects in at least three ways:
Value addition: FTA has the potential to serve as a defect-prevention tool. If FTA is performed before baselining the design, it can provide valuable information on application failures and their mechanisms. This information could be utilized to improve the design by preventing the potential defects or by introducing fault-tolerating abilities. FTA is most effective for more complex functions but may not be adding much value when applied to the simple functions of a software application. FTA utilizes the potential of teamwork to bring in a variety of ideas and broaden thinking.
Simplicity: FTA is very simple and can be prepared by project teams with minimum training. Its graphical presentation improves readability and makes it easy to maintain in the event of changes.
Traceability: Some of the conventional test case tools provide a unique identification to individual test cases. Such traceability could be added to FTA by appropriately identifying the individual scenario.
An FTA Case Study
Here is a common example of improving the security of software application by using controlled access. A weakness in choosing an appropriate login name or password may result in weaker application security (user ID and password are focused on more in this example than other factors, such as network or other interfaces). Figure 3 illustrates how this is represented.
The user ID and the password are considered further to see what could lead to a defect, i.e., poor security. The short length, non-use of digits or special characters, and validity not bounded by time, etc., could make a password weak. Similarly such situations could be listed for user IDs and other primary concerns.
Each scenario is identified with a unique number to establish traceability. Such traceability helps test cases to be related to other project artifacts like requirements, design or program specifications. The valid and invalid conditions for respective scenarios also can be noted for quick reference during testing.
The application of process improvement tools to the software development life cycle is becoming popular in the software community. These techniques have already been successfully leveraged by manufacturers which have encouraged software professionals to apply such tools to the SDLC. Using fault tree analysis (FTA) is one good way to improve the effectiveness of software testing. It can help identify the potential causes of a problem, suggest suitable corrective action and offer insight into preparing test case scenarios.
Challenges in Software Testing
1. Inherent challenge: It is next to impossible to test a software product of average complexity to all of its specifications and features. The number of test cases required to test every aspect of a software application would be so large that it would be economically impossible to prepare and execute. For example, a simple program to analyze a string of 10 alphabetic characters could have 2610 combinations. At a rate of one combination per microsecond, testing the string would take 4.5 million years, according to author Watts S. Humphrey in his book Managing the Software Process.
2. Laborious process of test case preparation and documentation: Test case preparation is labor intensive and has to fit into what is normally an already tight schedule. Often project teams are tempted to pay less attention to this activity. Considering the large number of test cases to be developed, it takes considerable effort to document and maintain the documentation. The project team seldom documents all the test cases and has to conduct testing with additional undocumented test cases.
3. Effectiveness of test cases: Identifying a test case is as important as writing a line of code. Lack of proper methods makes this task more challenging. Software engineering researchers, such as Glenford J. Myers in his book Software Reliability, Principles and Practices, have observed that it is impossible to test one's own programs. Test cases for a module created by the software developer tend to have an ingrained bias toward an application's functionality. Such test cases often are prepared to prove what is being developed, instead of to reveal defects – the proper objective of test cases.
Figure 1
4. Resource crunch: More effort is spent in software testing than in any other phase of software development. Figure 1 shows the distribution of efforts in the software development life cycle. The percent distributions are typical industry averages. While specific amounts of effort are scheduled for testing, projects often end up with less testing time than planned because the design and construction phases consume more effort than estimated. Tools that may increase effectiveness of testing are unavailable or unnoticed. Even if tools are available, a project team faced with a new learning curve may not be inclined to use them.
How Can the Challenges Be Met?
An analysis of the testing process reveals that one of the root causes of ineffectiveness is the process of test case creation. A test case is considered effective when it can reveal a defect. With good test cases, most latent defects can be identified and fixed before a product is shipped. Hence improving the test case creation process will help make the software testing process more effective.
The complexity of conventional test case documents often tends to become a bottleneck to improving effectiveness. The way out is to deploy the right tools to design useful test cases, ones which can reveal defects. It may not be necessary to test every possible combination since many of them could be redundant. The focus must be on those tests which can accurately tell about the health of the software.
Fault tree analysis may help simplify designing better test cases to improve effectiveness of the test process. The FTA preparation process brings in a variety of ideas, broadens the scope of thinking and adds creativity to the process.
What Is Fault Tree Analysis?
Fault tree analysis is a top-down approach to identify all potential causes leading to a defect. Each cause is further broken down into least possible events or faults. The analysis begins with a major defect. All the potential events – individual or in combination – that may cause the defect are identified. Potential events are further traced down in a similar way to the lowest possible level.
Table 1
Two logic symbols – known as logic gates – And and Or are used to represent the sequencing of events. The And symbol indicates that all preceding events must exist simultaneously for a defect to occur. The Or symbol indicates that either of the preceding events may lead to said defect. Table 1, known as a truth table, illustrates how the logic gates behave. Let's consider the And gate. The output will exist only when both inputs are present simultaneously. With reference to fault tree analysis, the fault condition exists only if the preceding events exist simultaneously. In the case of the Or gate, either of the inputs is required to produce the output condition. That is, either input state may result in a fault condition.
Figure 2: Demonstration of FTA
Figure 2 illustrates how FTA could be used for a typical situation of troubleshooting, e.g., computer not starting. The fault tree is shown with Levels 1 through 3 tracing the fault conditions, Level 1 being the highest.
There are at least two situations (faults) that may result in a computer not starting. Since either of the situations – power failure or booting failure – is capable of producing the Level 1 fault, the Or gate is used to represent their combination. The Level 2 fault, power failure, may result if the primary power source fails and at the same time the uninterruptible power supply (UPS) is down. An And gate is used to represent this situation. The event, "UPS down," may be further traced to faults like battery failure, hardware failure and so on.
Deciding the scope of FTA at the beginning is essential to limit the analysis to the required level. For example, if the focus is on problems with a computer, there is no need to analyze the failures of a UPS as it may not be an integral part of a computer. However, the failure of a motherboard associated with a booting problem may be discussed further as it is very much part of a computer system.
Advantages of Applying FTA
FTA can be advantageous to software projects in at least three ways:
Value addition: FTA has the potential to serve as a defect-prevention tool. If FTA is performed before baselining the design, it can provide valuable information on application failures and their mechanisms. This information could be utilized to improve the design by preventing the potential defects or by introducing fault-tolerating abilities. FTA is most effective for more complex functions but may not be adding much value when applied to the simple functions of a software application. FTA utilizes the potential of teamwork to bring in a variety of ideas and broaden thinking.
Simplicity: FTA is very simple and can be prepared by project teams with minimum training. Its graphical presentation improves readability and makes it easy to maintain in the event of changes.
Traceability: Some of the conventional test case tools provide a unique identification to individual test cases. Such traceability could be added to FTA by appropriately identifying the individual scenario.
An FTA Case Study
Here is a common example of improving the security of software application by using controlled access. A weakness in choosing an appropriate login name or password may result in weaker application security (user ID and password are focused on more in this example than other factors, such as network or other interfaces). Figure 3 illustrates how this is represented.
The user ID and the password are considered further to see what could lead to a defect, i.e., poor security. The short length, non-use of digits or special characters, and validity not bounded by time, etc., could make a password weak. Similarly such situations could be listed for user IDs and other primary concerns.
Each scenario is identified with a unique number to establish traceability. Such traceability helps test cases to be related to other project artifacts like requirements, design or program specifications. The valid and invalid conditions for respective scenarios also can be noted for quick reference during testing.
Well Known Software Failures
Software systems are pervasive in all aspects of society. From electronic voting to online shopping, a significant part of our daily life is mediated by software. In this page, I collect a list of well-known software failures. I will start with a study of economic cost of software bugs.
Contents
Economic Cost of Software Bugs
Air-Traffic Control System in LA Airport *****
Northeast Blackout **
NASA Mars Climate Orbiter ****
Denver Airport Baggage-handling System *
The number of *s is the ironic factor I assign to each story. The one with most *s is the most ironic one. Why? You will find out.
Economic Cost of Software Bugs
Report Date: 2/2002 Price Tag: $60 Billion Annually
WASHINGTON (COMPUTERWORLD) - Software bugs are costing the U.S. economy an estimated $59.5 billion each year, with more than half of the cost borne by end users and the remainder by developers and vendors, according to a new federal study.
Improvements in testing could reduce this cost by about a third, or $22.5 billion, but it won't eliminate all software errors, the study said. Of the total $59.5 billion cost, users incurred 64% of the cost and developers 36%.
Out of curiosity of how the study calculated the cost, I skimmed through the report. The following is a summary of their methodology.
It divided software developing process into stages: Requirement Gathering and Analysis, Architectural Design, Coding, Unit Test, Integration and Component, RAISE System Test, Early Customer Feedback, Beta Test Programs, and Post-product Release.
Bugs are generated at each stage of the software development process. The later in the production process that a bug is discovered, the more costly it is to repair the bug. Then impact estimates were developed relative to two counterfactual scenarios. The first scenario investigates the cost reductions if all bugs and errors could be found in the same development stage in which they are introduced. This is inferred to as the cost of an inadequate software testing infrastructure. The second scenario investigates the cost reductions associated with finding an increased percentage (but not 100 percent) of bugs and errors closer to the development stages where they are introduced. This is referred to as a cost reduction from feasible infrastructure improvements.
The study examined the impact of buggy software in several major industries -- automotive, aerospace and financial services -- and then extrapolated the results for the U.S. economy. It then concluded software bugs are costing (the first scenario) the U.S. economy an estimated $59.5 billion each year. Improvements in testing (the second scenario) could reduce this cost by about a third, or $22.5 billion
The report also included interesting tables that show the frequency of which stages errors are found, and relative cost to repair defects when found at different stages.
Air-Traffic Control System in LA Airport
Incident Date: 9/14/2004
(IEEE Spectrum) -- It was an air traffic controller's worst nightmare. Without warning, on Tuesday, 14 September, at about 5 p.m. Pacific daylight time, air traffic controllers lost voice contact with 400 airplanes they were tracking over the southwestern United States. Planes started to head toward one another, something that occurs routinely under careful control of the air traffic controllers, who keep airplanes safely apart. But now the controllers had no way to redirect the planes' courses.
The controllers lost contact with the planes when the main voice communications system shut down unexpectedly. To make matters worse, a backup system that was supposed to take over in such an event crashed within a minute after it was turned on. The outage disrupted about 800 flights across the country.
Inside the control system unit is a countdown timer that ticks off time in milliseconds. The VCSU uses the timer as a pulse to send out periodic queries to the VSCS. It starts out at the highest possible number that the system's server and its software can handle—232. It's a number just over 4 billion milliseconds. When the counter reaches zero, the system runs out of ticks and can no longer time itself. So it shuts down.
Counting down from 232 to zero in milliseconds takes just under 50 days. The FAA procedure of having a technician reboot the VSCS every 30 days resets the timer to 232 almost three weeks before it runs out of digits.
Northeast Blackout
Incident Date: 8/14/2003 Price Tag: $6 - $10 Billion
NEW YORK (AP) - A programming error has been identified as the cause of alarm failures that might have contributed to the scope of last summer's Northeast blackout, industry officials said Thursday.
The failures occurred when multiple systems trying to access the same information at once got the equivalent of busy signals, he said. The software should have given one system precedent.
With the software not functioning properly at that point, data that should have been deleted were instead retained, slowing performance, he said. Similar troubles affected the backup systems.
NASA Mars Climate Orbiter
Incident Date: 9/23/1999 Price Tag: $125 million
WASHINGTON (AP) -- For nine months, the Mars Climate Orbiter was speeding through space and speaking to NASA in metric. But the engineers on the ground were replying in non-metric English.
It was a mathematical mismatch that was not caught until after the $125-million spacecraft, a key part of NASA's Mars exploration program, was sent crashing too low and too fast into the Martian atmosphere. The craft has not been heard from since.
Noel Henners of Lockheed Martin Astronautics, the prime contractor for the Mars craft, said at a news conference it was up to his company's engineers to assure the metric systems used in one computer program were compatible with the English system used in another program. The simple conversion check was not done, he said.
Denver Airport Baggage-handling System
Incident Date: 11/1993 - 6/1994 Price Tag: > $200 million
(Scientific America) -- Scheduled for takeoff by last Halloween (1993), the airport's grand opening was postponed until December to allow BAE Automated Systems time to flush the gremlins out of its $193-million system. December yielded to March. March slipped to May. In June the airport's planners, their bond rating demoted to junk and their budget hemorrhaging red ink at the rate of $1.1 million a day in interest and operating costs, conceded that they could not predict when the baggage system would stabilize enough for the airport to open.
Contents
Economic Cost of Software Bugs
Air-Traffic Control System in LA Airport *****
Northeast Blackout **
NASA Mars Climate Orbiter ****
Denver Airport Baggage-handling System *
The number of *s is the ironic factor I assign to each story. The one with most *s is the most ironic one. Why? You will find out.
Economic Cost of Software Bugs
Report Date: 2/2002 Price Tag: $60 Billion Annually
WASHINGTON (COMPUTERWORLD) - Software bugs are costing the U.S. economy an estimated $59.5 billion each year, with more than half of the cost borne by end users and the remainder by developers and vendors, according to a new federal study.
Improvements in testing could reduce this cost by about a third, or $22.5 billion, but it won't eliminate all software errors, the study said. Of the total $59.5 billion cost, users incurred 64% of the cost and developers 36%.
Out of curiosity of how the study calculated the cost, I skimmed through the report. The following is a summary of their methodology.
It divided software developing process into stages: Requirement Gathering and Analysis, Architectural Design, Coding, Unit Test, Integration and Component, RAISE System Test, Early Customer Feedback, Beta Test Programs, and Post-product Release.
Bugs are generated at each stage of the software development process. The later in the production process that a bug is discovered, the more costly it is to repair the bug. Then impact estimates were developed relative to two counterfactual scenarios. The first scenario investigates the cost reductions if all bugs and errors could be found in the same development stage in which they are introduced. This is inferred to as the cost of an inadequate software testing infrastructure. The second scenario investigates the cost reductions associated with finding an increased percentage (but not 100 percent) of bugs and errors closer to the development stages where they are introduced. This is referred to as a cost reduction from feasible infrastructure improvements.
The study examined the impact of buggy software in several major industries -- automotive, aerospace and financial services -- and then extrapolated the results for the U.S. economy. It then concluded software bugs are costing (the first scenario) the U.S. economy an estimated $59.5 billion each year. Improvements in testing (the second scenario) could reduce this cost by about a third, or $22.5 billion
The report also included interesting tables that show the frequency of which stages errors are found, and relative cost to repair defects when found at different stages.
Air-Traffic Control System in LA Airport
Incident Date: 9/14/2004
(IEEE Spectrum) -- It was an air traffic controller's worst nightmare. Without warning, on Tuesday, 14 September, at about 5 p.m. Pacific daylight time, air traffic controllers lost voice contact with 400 airplanes they were tracking over the southwestern United States. Planes started to head toward one another, something that occurs routinely under careful control of the air traffic controllers, who keep airplanes safely apart. But now the controllers had no way to redirect the planes' courses.
The controllers lost contact with the planes when the main voice communications system shut down unexpectedly. To make matters worse, a backup system that was supposed to take over in such an event crashed within a minute after it was turned on. The outage disrupted about 800 flights across the country.
Inside the control system unit is a countdown timer that ticks off time in milliseconds. The VCSU uses the timer as a pulse to send out periodic queries to the VSCS. It starts out at the highest possible number that the system's server and its software can handle—232. It's a number just over 4 billion milliseconds. When the counter reaches zero, the system runs out of ticks and can no longer time itself. So it shuts down.
Counting down from 232 to zero in milliseconds takes just under 50 days. The FAA procedure of having a technician reboot the VSCS every 30 days resets the timer to 232 almost three weeks before it runs out of digits.
Northeast Blackout
Incident Date: 8/14/2003 Price Tag: $6 - $10 Billion
NEW YORK (AP) - A programming error has been identified as the cause of alarm failures that might have contributed to the scope of last summer's Northeast blackout, industry officials said Thursday.
The failures occurred when multiple systems trying to access the same information at once got the equivalent of busy signals, he said. The software should have given one system precedent.
With the software not functioning properly at that point, data that should have been deleted were instead retained, slowing performance, he said. Similar troubles affected the backup systems.
NASA Mars Climate Orbiter
Incident Date: 9/23/1999 Price Tag: $125 million
WASHINGTON (AP) -- For nine months, the Mars Climate Orbiter was speeding through space and speaking to NASA in metric. But the engineers on the ground were replying in non-metric English.
It was a mathematical mismatch that was not caught until after the $125-million spacecraft, a key part of NASA's Mars exploration program, was sent crashing too low and too fast into the Martian atmosphere. The craft has not been heard from since.
Noel Henners of Lockheed Martin Astronautics, the prime contractor for the Mars craft, said at a news conference it was up to his company's engineers to assure the metric systems used in one computer program were compatible with the English system used in another program. The simple conversion check was not done, he said.
Denver Airport Baggage-handling System
Incident Date: 11/1993 - 6/1994 Price Tag: > $200 million
(Scientific America) -- Scheduled for takeoff by last Halloween (1993), the airport's grand opening was postponed until December to allow BAE Automated Systems time to flush the gremlins out of its $193-million system. December yielded to March. March slipped to May. In June the airport's planners, their bond rating demoted to junk and their budget hemorrhaging red ink at the rate of $1.1 million a day in interest and operating costs, conceded that they could not predict when the baggage system would stabilize enough for the airport to open.
Saturday, October 25, 2008
Defect Management
Defects determine the effectiveness of the Testing what we do. If there are no defects, it directly implies that we don’t have our job. There are two points worth considering here, either the developer is so strong that there are no defects arising out, or the test engineer is weak. In many situations, the second is proving correct. This implies that we lack the knack. In this section, let us understand Defects.
What is a Defect?
For a test engineer, a defect is following: -
•Any deviation from specification
•Anything that causes user dissatisfaction
•Incorrect output
•Software does not do what it intended to do.
Bug / Defect / Error: -
•Software is said to have bug if it features deviates from specifications.
•Software is said to have defect if it has unwanted side effects.
•Software is said to have Error if it gives incorrect output.
But as for a test engineer all are same as the above definition is only for the purpose of documentation or indicative.
Defect Taxonomies
Categories of Defects:
All software defects can be broadly categorized into the below mentioned types:
•Errors of commission: something wrong is done
•Errors of omission: something left out by accident
•Errors of clarity and ambiguity: different interpretations
•Errors of speed and capacity
However, the above is a broad categorization; below we have for you a host of varied types of defects that can be identified in different software applications:
1.Conceptual bugs / Design bugs
2.Coding bugs
3.Integration bugs
4.User Interface Errors
5.Functionality
6.Communication
7.Command Structure
8.Missing Commands
9.Performance
10.Output
11.Error Handling Errors
12.Boundary-Related Errors
13.Calculation Errors
14.Initial and Later States
15.Control Flow Errors
16.Errors in Handling Data
17.Race Conditions Errors
18.Load Conditions Errors
19.Hardware Errors
20.Source and Version Control Errors
21.Documentation Errors
22.Testing Errors
Life Cycle of a Defect
The following self explanatory figure explains the life cycle of a defect:
What is a Defect?
For a test engineer, a defect is following: -
•Any deviation from specification
•Anything that causes user dissatisfaction
•Incorrect output
•Software does not do what it intended to do.
Bug / Defect / Error: -
•Software is said to have bug if it features deviates from specifications.
•Software is said to have defect if it has unwanted side effects.
•Software is said to have Error if it gives incorrect output.
But as for a test engineer all are same as the above definition is only for the purpose of documentation or indicative.
Defect Taxonomies
Categories of Defects:
All software defects can be broadly categorized into the below mentioned types:
•Errors of commission: something wrong is done
•Errors of omission: something left out by accident
•Errors of clarity and ambiguity: different interpretations
•Errors of speed and capacity
However, the above is a broad categorization; below we have for you a host of varied types of defects that can be identified in different software applications:
1.Conceptual bugs / Design bugs
2.Coding bugs
3.Integration bugs
4.User Interface Errors
5.Functionality
6.Communication
7.Command Structure
8.Missing Commands
9.Performance
10.Output
11.Error Handling Errors
12.Boundary-Related Errors
13.Calculation Errors
14.Initial and Later States
15.Control Flow Errors
16.Errors in Handling Data
17.Race Conditions Errors
18.Load Conditions Errors
19.Hardware Errors
20.Source and Version Control Errors
21.Documentation Errors
22.Testing Errors
Life Cycle of a Defect
The following self explanatory figure explains the life cycle of a defect:
Subscribe to:
Posts (Atom)
Posts
