Owner: Quality Frog - Questioning Software URL:http://www.questioningsoftware.com Join Date: Sat, 26 May 2007 14:43:36 -0500 Rating:0 Site Description: Questioning Software. Ben Simo's software quality and testing blog. Topics include: heuristics, test automation, exploratory testing, model-based testing, test education, and more. Site statistics:Click here
Don't Ignore The Little Bugs 2007-05-24 19:03:00 And that's how it happened.Believe me. It's true.Because . . .just because . . .a small bugwent KA-CHOO!One of my favorite children's books is also one of my favorite testing books. (Thanks go to Rob Sabourin* for alerting me to the testing connection.) Because a Little Bug went Ka-CHOO! tells the story of a multitude of problems that cascade from a little bug's sneeze. A worm gets mad. A turtle gets bopped. A bucket gets stuck. A policeman takes flight -- in a motorcycle sidecar. A boat nearly sinks. Pandemonium ensues.This book illustrates how things that seem to be insignificant can have substantial lasting impact on a larger system.Software bugs that appear to be trivial can be a sign of a larger problem. When we testers encounter bugs, we are usually looking at a symptom of a problem and not the underlying error that produced the bug. This requires that we do some investigation to determine if a bug is more serious than it first appears and if it is more wide-spread t Read more:Ignore
Driving for quality 2007-05-23 00:30:00 Years ago, I taught defensive driving classes to people cited for violating traffic law in Arizona. The central theme of my classes was that our attitudes behind the wheel often have a bigger impact on safe driving than our skill as drivers. I would start each class by having each student describe what they did to get in my class and what other drivers do that annoy them. We then reviewed the two lists (which usually ended up being identical) and the class discussed whether each item was mostly due to driver skill or attitude. Nearly everything on the lists could be traced to an attitude problem.We are likely guilty of the same faults that we find in others. I believe the attitudes of those involved in software projects can impact the quality of software more than the skills of the team. And when skill is lacking, the right attitude fosters learning. A little patience and a good attitude can go a long way.I used to teach the SIPDE mnemonic to my driving students to help improve safety Read more:Driving
Don't forget to think 2007-05-21 21:30:00 This past week at STAR East, James Bach presented a number of questions, magic tricks, games, and riddles to testers that volunteered to be tested. I feel like I did fairly well on some of them and failed miserably on others. James uses these tests to teach testers to think
. I thought I had learned some valuable lessons until I was presented with two riddles from children today. If only I could learn to think more like a child. :) I think that I sometimes let my search for hidden meanings keep me from seeing the obvious.A riddle from my children:You are blindfolded, placed at the start of a maze, and told to get to the other end. How do you navigate the maze?You can feel your way through.or a better answer isTake off the blindfold.It can be easy to blindly feel our way through though a challenge -- one obstacle at a time. At times it can be good to isolate problems but problems taken out of their context can be misleading. We need to look at problems in the context of their en
Model-Based Test Engine Benefit #2: Simplified test result analysis 2007-05-21 08:00:00 Automation is of little value if it does not report useful information that can be quickly reviewed by testers.Reported results should contain enough information to answer the following questions:What happened?What is the state of the application?How did the application get in that state?What automation code was executed?What automation data/parameters were used?Some failures reported by automated tests will be errors in the system under test and others will be errors in the automation model or code. It is important that results point the reader to both.I have found logging of the following information to be useful:Test (test configuration information)TitleStart TimeScript File(s)Model FilesTest SetSeverityEnvironmentObject MapAction Table(s)Oracle Table(s)Computer NameOperating SystemTesterActions (controlling the application)Source (where is the action defined?)TitleStart TimeAction DetailsDurationState TransitionAutomation CodeResult DetailsSnapshot (screen capture, saved files, etc Read more:Engine
STAR East 2007 Conferred 2007-05-19 15:01:00 I am sitting in the airport waiting to fly home from STAR-East. The conference was great. It was not great due to the many wonderful presentations. It was great because of what happened outside the scheduled activities. I got to confer with colleagues from around the world.The best part of conferences such as STAR-East is the opportunity to confer with peers and thought leaders in our industry. It is an opportunity to discover that we are experiencing common problems and share possible solutions. It is an opportunity to learn from the best. I often learn more over dinner and in the hallways than I learn in the presentations.I was amazed at how quickly the conference attendees disappeared once the scheduled activities were completed. I know that many of us computer geeks are introverts. We may not be the most social bunch of people, but I believe a conference without conferring is a wasted opportunity. See y'all at CAST.Ben Simo
http://QualityFrog.com
Model-Based Test Engine Benefit #1: Simplified automation creation and maintenance 2007-05-19 10:17:00 Model-Oriented DesignProcedural automated test scripts may be easy to record or script. However, they are difficult to maintain when applications change. They are also difficult to adapt to new test ideas. Maintenance is simplified by automating the procedure generation in addition to the execution. New actions, validations, and data can be added to existing tests. This allows testers to spend more time thinking up new test ideas instead of maintaining procedural scripts.Simplified GUI Interaction CodingMost GUI automation tools contain complex vocabularies for controlling objects and retrieving information from those objects. There are usually different methods for interacting with different classes of objects. This requires that toolsmiths learn a class-sensitive vocabulary and be aware of the class as they code tests. There is an easier way: create functions that automatically detect an object's class and apply the appropriate method. This allows for the same command to be Read more:Engine
2007 STAR East Presentation 2007-05-18 00:32:00 My STAR East Presentation slides may be viewed here.A good article that gives a high-level overview of Model-Based Test (MBT) automation is Harry Robinson's Intelligent Test Automation STQE (Better Software) article.The tool I referenced for creating visual images of models may be found at GraphViz.org.I will be posting some examples and details for getting started with the development of a Model-Based Test Engine over the next few weeks. Feel free to contact me with any questions. Your questions may help dictate how and what information I post.You may view some existing blog posts about MBT by clicking here.BenBen Simo
http://QualityFrog.com
Faking It 2007-05-16 22:36:00 Pradeep Soundararajan recently posted a podcast about fake experience on resumes. [listen] This reminded me of an experience I had with a fake resume.A colleague came to me, dropped a resume in my hand, and asked if I had worked for a company listed on the resume. I quickly scanned the resume and noticed a former employer listed in the experience. I checked the dates and discovered that they included a period that I worked for that company. I then read details that listed projects in which I had been intimately involved. However, I did not recognize the name at the top of the resume. I then called several people at that company and could not find anyone that knew this person.The experience listed on the resume was fake. It was a lie.Lying on your resume can come back to haunt you -- sometimes even many years down the road. Don't fall into that trap. - Sunder RamchandranResumes: Fake it and you break itThis blatant lie was easily caught. Even if I had not worked for the company
Automating outside the box 2007-05-16 00:36:00 testany standardized procedure for measuring sensitivity or memory or intelligence or aptitude or personality etc • the test was standardized on a large sample of studentsthe act of testing something the act of undergoing testing • he survived the great test of battle trying something to find out about itautomationthe act of implementing the control of equipment with advanced technology; usually involving electronic hardware • automation replaces human workers by machines the condition of being automatically operated or controlled • automation increases productivityequipment used to achieve automatic control or operation • this factory floor is a showcase for automation and robotic equipment What is test automation?I just read some marketing literature from some leading test automation tool vendors. According to one of the vendors, their tool supports the following: Novice testers can create robust and easily maintainable tests that mimic real-life use of an applica
Distracted by the machinery 2007-05-09 07:24:00 Let's not allow the machinery
of testing to distract us from the craft of testing.Over ten years, ago James Bach published his first version of Test Automation Snake Oil.In this article, James identified eight "reckless assumptions" of the classic arguments for test automation. If we aren't careful, it can be easy to start believing statements based on these assumptions.Testing is a "sequence of actions."Testing means repeating the same actions over and over.We can automate testing actions.An automated test is faster, because it needs no human intervention.Automation reduces human error. We can quantify the costs and benefits of manual vs. automated testing.Automation will lead to "significant labor cost savings."Automation will not harm the test project.Have you made any of these assumptions? Read the article for details. Then take a look at Sam Burgiss' great review.Ben Simo
http://QualityFrog.com
I'll be at STAR East in Orlando next week 2007-05-09 07:22:00 I'll be at STAR East next week I'll be in Orlando
from mid-day Tuesday through Saturday evening. Last I heard, there's still time to register.I will be a track speaker Thursday morning. My session is titled "Build a Model-Based Testing Framework for Dynamic Test Automation". Come learn how test automation can go beyond retracing paths already traced by manual testing. Learn how to send your automation out to explore an application and bring back useful information. Learn to model behavior instead of script test cases.Come find me sometime after the presentation to talk about details that won't fit in a 50-minute presentation.If you'd like to talk testing, send me an email. (If you don't get a response, my SPAM filter may be getting in the way. Try the blog comment feature.)I am looking forward to talking testing with great minds. I have found that I often learn as much -- or more -- during discussions over lunch and dinner at conferences than I learn during the scheduled conferen
Falsifiability Testing 2007-05-07 22:22:00 The best experiments deduce an effect from the hypothesis and then isolate it in the very context where it may be disproved.- Michael Kaplan & Ellen Kaplan, Chances Are: Adventures In ProbabilityIn Focusing on Falsifiability, Stuart Thompson writes the following about a tester-friend's statement that testers can add value from the start of a project if they understand the project and its direction."With a clear understanding of what the software was actually trying to do, his team was able to provide useful feedback to the developers even within the first couple of release cycles."When testers know the problem that software is trying to solve, they can first focus on the things that matter. They can test the assumptions. They can identify contexts in which the assumptions and the implementation can be disproved. Project managers and developers are usually focused on the solution. Testers can help identify the new problems created by proposed solutions and provide information to help t Read more:Testing
Coffee Break Machine Testing 2007-05-05 13:45:00 It's good to have some idea as to what something does before you start testing it -- or eating it.Cookie Monster does a little testing in a 1967 IBM training film.Thanks to UtterlyGeek.What do you call this kind of testing?Ben Simo
http://QualityFrog.com Read more:Testing
, Coffee
, Break
, Machine
Not Gonna Bow 2007-05-05 11:10:00 Individuals and interactionsover processes and toolsWorking softwareover comprehensive documentationCustomer collaboration over contract negotiationResponding to changeover following a plan- Agile ManifestoNeary 400 years ago, Francis Bacon challenged the status quo in scientific thought in “The New Organon”. James Bach recently pointed out some interesting quotes from this work that apply to software testers. I agree.Bacon argued that placing our preconceived beliefs over what we observe causes great harm. He went so far as to describe these harmful preconceived notions as “idols”. Bacon put these idols into four categories:Idols of the Tribe: Errors common to mankind.Idols of the Cave: Errors specific to each individual’s education and experience.Idols of the Market Place: Errors formed through association with others — often due to misunderstanding others.Idols of the Theater: Errors formed from dogma (institutionalized doctrine) and flawed demonstrations.All of these ex
Heuristics in Test Automation 2007-05-04 08:26:00 "A Requirement is a quality or condition that matters to someone who matters."- Cem Kaner, James Bach, and Bret PettichordLessons Learned In Software Testing Automated tests are usually coded to perform validations against written requirements. Computers are deterministic in that they need specific instructions regarding what to test, what counts as a passed test, and what counts as a failed test. This is one of the weaknesses of test automation. Many requirements exist beyond the hard written requirements. Test automation can be a great tool for measuring hard requirements. For example, automation can be great for validating mathematical calculations.Test automation can also be a great tool for testing the fuzzy requirements through the use of heuristics. In addition to coding validations (oracles) for hard facts, automation can be used to report things that require human attention. Automation
can report information to help direct the attention of human testers."Only weak bugs have lo
Hey Dad, when I grow up, I also want to be square. 2007-05-02 20:16:00 Erkan Yilmaz, a fellow tester blogger from Germany, recently pointed out that the slogans in my post "Slogans are models" may not transmit their message across languages and cultures. He attempted to guess at what some of the slogans meant without the context of American culture and advertising. These slogans that most Americans will instantly understand didn't work very well out of their context.We both saw this as an example of how recipients of information do not always have the full context in which the information originated. As testers, we need to admit when we don't understand and seek the answers (and context) from those that know. Sometimes we may need to bring subject matter experts into the conversation to fully understand what we are testing. If you don't know, ask questions. If you think you know, ask questions. You are bound to learn something.I recently sat in some presentations by people from the "business" (as in not IT) side of some projects in which I am involved.
Heuristics In Software Testing 2007-04-28 09:27:00 The point of philosophy is to start with something so simple as to seem not worth stating, and to end with something so paradoxical that no one will believe it."- Sir Bertrand RussellA heuristic is a commonsense guideline used to increase the probability of solving a problem by directing one's attention to things that are likely to matter. The word is derived from the Greek "heurisko" which simply means "I find". The exclamation "eureka", meaning "I found it!", shares roots with heuristic.Just as the "Eureka!" screaming forty-niners of California's Gold Rush did not have secret knowledge or tools to tell them exactly where all the gold was buried; software testers don't know exactly where the bugs are going to hide. However, both software testers and gold miners know where bugs and gold have been found before. We can use that knowledge of past discoveries and the nature of what we seek to create heuristics that help us narrow in on areas most likely to contain the treasure.Gold mine Read more:Testing
, Software
, Software Testing
How Doctors Think 2007-04-27 00:09:00 Michael Bolton recently brought Dr. Jerome Groopman’s newest book, “How Doctors
Think”, to my attention. Michael suggested to the software-testing list that this book contains information that is relevant to software testing. In this book, Dr Groopman explores how doctors diagnose medical problems and what can be done to reduce bad judgment.Michael Krasny recently interviewed Dr. Groopman on public radio about this book. You can listen to the interview here. Dr Groopman made a number of statements during this interview that I believe apply to us software testers’ search for and diagnosis of software bugs. I expect to find more in the book.Here are some gems from the radio interview:About computer-assisted diagnosis software.It’s a two-edged sword. On one hand, it can be useful if you feed in the most important and relevant symptoms, but the problem is that you need to be able to illicit them effectively from the patient and so you return to language and you return to some of
How many tests do you need? 2007-04-21 19:12:00 “Testing is potentially an infinite process.” – James BachNearly 30 years ago, Glenford Myers provided a self-assessment test to software testers in his book The Art of Software Testing. Much of this book is outdated; however, his test is still regularly used to illustrate the complexity of testing software. He asked readers to write down the test cases required to adequately test the following program.The program reads three integer values from a card. The three values are interpreted as representing the lengths of the sides of a triangle. The program prints a message that states whether the triangle is scalene, isosceles, or equilateral.Glenford Myers then fills the next page of the book with a list of 14 questions that should be answered by test cases for this simple program. He then makes the point that more complex programs are exponentially more difficult to test.Elizabeth Hendrickson recently created a modern version of the program (no punch cards are required) and challen Read more:tests
Performance Testing Lessons Learned 2007-04-16 18:10:00 Web and client/server load testing can easily become a complex task. Most people I've met got started in load testing with only minimal training in using the test tools. This is how I got started in load testing -- although I had an advantage in that I had been exposed to load testing of communications systems. I also had experience with automated single-user performance testing. I had led some small-scale manual load tests with multiple testers on a conference call hitting the same client-server application at once. (And we found some show-stopping bugs doing that manual testing.) I had watched others perform load tests. I had read numerous load test plans and reports. However, I had never directly participated in executing automated load tests... then I was asked to lead a load testing project. Through the years, I have made many mistakes designing, scripting, and executing load tests. Load testing easily becomes complex. Tool sales people sometime tell us that nearly anyone can cre Read more:Testing
, Performance
, Lessons
, Learned
, Lessons Learned
Model-Based Test Engine Benefits 2007-04-15 21:47:00 A Model-Based Test Engine
(MBTE) is a test automation framework that generates and executes tests based on a behavioral model. Instead of performing scripted test cases, a MBTE generates tests from the model during execution. Instead of implementing models in code, a MBTE can process models defined in tables. Both human testers and computers can understand models defined in tables. A MBTE can be built on top of most existing GUI test automation tools. Combining good automation framework practices with Model-Based Testing (MBT) can transform some common test automation pitfalls to benefits.Implementing a MBTE can produce the following:Simplified automation creation and maintenance.Simplified test result analysis.Automatic handling of application changes and bugs.Generate and execute new tests – and find new bugs.More to come...Ben Simo
http://QualityFrog.com Read more:Benefits
Why does test automation often fail to deliver? 2007-03-21 07:29:00 Software Is AutomationTesters are continually asked to test more in less time. Test automation is often considered to be a solution for testing more in less time. Software is automation. It makes sense to automate the testing of software. However, functional black box test automation rarely does more in less time. Why does automation seldom deliver faster, better, and cheaper testing?Common Test Automation PitfallsThe following problems are often encountered during test automation projects.1. Tests are difficult to maintain and manage.Many GUI test automation attempts to replace just a single step of the manual testing process: test execution. This can work well when there is value in repeatedly executing the same steps and the application is stable enough to run the same script multiple times. However, in the real word, applications change and doing the same thing over and over again is rarely beneficial. A thinking human tester still needs to create the tests, code the automation, re
Best Practices Aren’t 2007-02-12 22:03:00 The first two of the Seven Basic Principles of the Context-Driven of software testing are:1. The value of any practice depends on its context.2. There are good practices in context, but there are no best practices.As a former quality school gatekeeper, I understand the value of standards – in both products and processes. However, I am concerned by the current “best practices” trends in software development and testing. The rigidity that we demand in product standards can hurt in process standards. Even the CMM (which is often viewed as a rigid process) has “Optimizing” as the highest level of maturity. A mature process includes continuous learning and adjustment of the process. No process should lock us into one way of doing anything.Nearly 100 years ago, the industrial efficiency pioneer Frederick Taylor wrote “among the various methods and implements used in each element of each trade there is always one method and one implement which is quicker and better than any of the Read more:Best Practices
When is a bug a feature? 2007-02-08 01:42:00 Ever find a bug that "can't" be fixed?I am not referring to fixes prevented by technical or schedule limitations. I am referring to bugs that have become expected feature
s.Here are two...Experience #1I once found a bug in communications software that would cause most of the systems likely to be on the other end of a connection to crash. I don't mean the failure of a single process. This was a memory allocation issue that could take down entire systems that were essential to their owners. An argument could be made that the bug was really in all those other systems and not in the system that I was testing. However, most of the large number of systems that made up the pre-existing installed base could not create the condition that would make another system fail. Due to the large number of systems (from a variety of vendors) it was determined that the condition that caused the failure should be prevented by new systems (and future releases of existing systems) instead of immediately fixi
People, Monkeys, and Models 2007-02-07 20:05:00 Methods I have used for automating “black box” software testing…I have approached test automation in a number of different ways over the past 15 years. Some have worked well and others have not. Most have worked when applied in the appropriate context. Many would be inappropriate for contexts other than that in which they were successful.Below is a list of methods I’ve tried in the general order that I first implemented them.Notice that I did not start out with the record-playback test automation that is demonstrated by tool vendors. The first test automation tool I used professionally was the DOS version of Word Perfect. (Yes, a Word Processor as a test tool. Right now, Excel is probably the the one tool I find most useful.) Word Perfect had a great macro language that could be used for all kinds of automated data manipulation. I then moved to Pascal and C compilers. I even used a pre-HTML hyper-link system called First Class to create front ends for integrated computer-assist Read more:Monkeys
Finite State Machines 2007-03-24 17:54:00 Software behavior can be modeled using Finite State
Machines (FSMs). FSMs are composed of states, transitions, and actions. Each state is a possible condition of the modeled system. Transitions are the possible changes in states. Actions are the events that cause state transitions. For example, the following FSM shows the expected behavior of a car keyless entry system.Images like the above are great for human use, but not machines. State transitions and the actions that trigger them can also be defined in a table format that can be processed by a computer. The above FSM can be represented using the table below.The requirements for each state can also be defined using tables. The table below contains sample requirements for example keyless entry system.
Ben Simo
QualityFrog.com
Model-Based Test Engine Benefit #3: Automatic handling of application changes and bugs 2007-06-01 20:05:00 Automated tests based on models have one important feature that scripted testing cannot: automated handling of application changes
and bugs. I do not mean that model-based automation can think and make decisions like a human tester does when they discover something unexpected. Instead, the automated selection of test steps supports working around the unexpected without special exception handling code for each situation.For example: If there are two methods for logging into an application and one breaks the test engine can try the alternate option to get to the rest of the application. If a traditional scripted automated test encounters an unexpected problem it will not be able to complete.The model-based test engine (MBTE) can be coded to not try an action after a pre-defined number of failures. The MBTE's selection algorithm can then seek out other options that have not yet been found to fail. This also results in the MBTE reattempting failed actions and exposing failures that only o Read more:Engine
When testers create bugs 2007-05-31 00:38:00 How come dumb stuff seems so smart while you're doing it?- Dennis the MenaceDebasis Pradhan's blog entry Testers don't make Bugs. Oh Really? got me thinking about a time that I as a tester actually introduced a bug into a system. Debasis' post is about bugs that slip by testers and escape into the wild. This is not the case in my story. I asked developers to put a bug in the software and they followed my instructions.I was testing a data mastering system that assembled and converted data from a data repository's format to a variety of other formats for distribution to customers and inclusion in a variety of software products. I create
d a data validation tool that was used to inspect the huge volume of transformed data: comparing the actual output of the mastering system to the expected format and presentation. The validation tool also performed some heuristic-based tests that alerted testers and developers to data that may require manual inspection.Over the course of many months,
Where No Confabulation Goes Untested 2007-05-29 17:56:00 conferverbhave a conference in order to talk something overThe Conference of the Association for Software Testing (CAST) is coming this July.I missed last year's conference but have heard great things about it. Based on all the wonderful things I've heard from those that were there, I am looking forward to this year's conference.The CAST is different than most conferences where people sit and listen to someone present to an audience without public questioning of what is presented. AST encourages testers to test the presentations. Time is allowed for discussion at every presentation. Challenging ideas is encouraged. I could go on. However, I don't think I can push this conference any better than David Gilbert. Therefore, please take a look at David's blog post: CAST in stone.CAST early bird registration ends this week. Register athttp://associationforsoftwaretesting.org/conference/registration.htmlI hope to see you there.Ben
Ben Simo
QualityFrog.com
Poka-Yoke 2007-06-02 11:30:00 Poka-Yoke is not a dance. Its not an event at a rodeo. Its not what my kids do to each other in the back seat of the car. Poka-Yoke is Japanese for "mistake-proofing". Poka-Yoke was developed by Japanese industrial engineer Shigeo Shingo. He realized that people cannot be expected to work like machines and consistently do everything the same way every time they do it. People make mistakes and poorly designed processes can make it easier for people to err. Poka-Yoke's goal is to make it difficult for people to make mistakes through mistake prevention and detection.PreventionApplied poka-yoke gives users warnings about incorrect behavior and directs users towards the correct behavior. Computer PS/2 keyboards and mice share the same physical connector design but the connectors are usually color-coded to indicate which device goes into which port on a computer. Some computing hardware is shipped with warning stickers on top of connectors telling users to read a manual or install
Recommend Site To A Friend
