Skip to content

Being CEWT#6

September 27, 2018

Last weekend I had a chance to participate in CEWT#6 and to do a short talk on  good or good-enough testing.

I don’t have intention to replicate my talk here, but I’d like to point to some of the materials that influenced my thinking on that topic.

In no particular order:

Heuristic Test Planning: Context Model, James Bach

The Psychology of Computer Programming: Silver Anniversary, eBook Edition, Gerald M. Weinberg

“Most programmers feel that there is such a thing as good programming. Although this feeling may be valid, it does not follow that there is such a thing as a good program. Not, at least, in the sense that we can study the program out of the context in which it was developed and in which it will be used and declare it to have a goodness of 83.72 percent.”

“Just as we could not say in absolute terms what is a “good program,” we cannot say what is a “good programmer,” “a good programming manager,” or, for that matter, “a good piece of software.” As a result, we shall be forced to elevate the discussion of programming by lowering the level of discourse: we shall only be able to make conclusions of general validity if we get down to much more detail than is usually found in discussions of “what makes a good programmer.” And perhaps we shall find that what makes a good programmer is much like what makes a good friendship—a mutual recognition and encouragement of individuality.”

Test Design: A BBST Workbook, Context-Driven Press., Kaner, Cem; Fiedler, Rebecca

“For any nontrivial program, the set of possible tests is infinitely large. From the perspective of this challenge, test design is a sampling problem: How can we sample a small subset of tests, from this infinite pool of possibilities, that is good enough to teach us what we need to know about the program under test?”

Perfect Software and Other Illusions About Testing, Gerald M. Weinberg

“You may or may not agree with Hamlet that there is nothing either good or bad. For the sake of argument, though, let’s suppose there is such a thing as a “good” test and ask the question, How can one know whether a particular test (or set of tests) is, indeed, good? Let’s start with something even better than “good” by looking at a definition of “perfect.” A perfect set of tests would have the following characteristics: a. It would detect every bug in a system. b. It would never detect a non-bug as a bug. c. It would give us complete confidence that it has done a and b. d. It would accomplish a, b, and c quickly and cheaply enough for our needs.”

“Not ensuring that development is done well: Poorly developed code needs good testing but usually receives poor testing, thus compounding problems. What use are good tests of shoddy code?”

“You may or may not agree with Hamlet that there is nothing either good or bad. For the sake of argument, though, let’s suppose there is such a thing as a “good” test and ask the question, How can one know whether a particular test (or set of tests) is, indeed, good? Let’s start with something even better than “good” by looking at a definition of “perfect.” A perfect set of tests would have the following characteristics:
a. It would detect every bug in a system.
b. It would never detect a non-bug as a bug.
c. It would give us complete confidence that it has done a and b.
d. It would accomplish a, b, and c quickly and cheaply enough for our needs.”

“Expecting testing to produce quality: Quality is a product of the entire development process. Poor testing can lead to poor quality, but good testing won’t lead to good quality unless all other parts of the process are in place and performed properly.”

“Is there at least one question about your product that testing can help you answer? If there’s no question, then there’s no reason to test.”

“Not ensuring that development is done well: Poorly developed code needs good testing but usually receives poor testing, thus compounding problems. What use are good tests of shoddy code?”

“Expecting testing to produce quality:
Quality is a product of the entire development process. Poor testing can lead to poor quality, but good testing won’t lead to good quality unless all other parts of the process are in place and performed properly.”

Lessons Learned in Software Testing: A Context-Driven Approach. John Wiley and Sons., Kaner, Cem; Bach, James; Pettichord, Bret

“Good test strategies tell compelling stories that explain and justify the testing to be done. Just as there are many stories that can be told, there are many possible test strategies.”

“A good test strategy is:
* Product-specific. Whatever good a generic test strategy may be, a strategy that is specific to the product and technology at hand will be that much better.
* Risk-focused. Show how the test process will address the things that matter most. Connect the test process to your mission on this project.
* Diversified. In most situations, a diversified strategy is better than a monolithic strategy. A diversified strategy is one that includes a variety of different test techniques or approaches. Problems that sneak past one test approach may well be captured by another approach.
* Practical. You must be able to perform the strategy. Don’t suggest a test strategy that is far beyond the capabilities of the project.”

“If you do this skillfully and faithfully, any bad bug that arises after release will probably be for one of three reasons:
* You didn’t understand the risk dynamics as well as you thought. Now you know better.
* You made a mistake in the testing. You’ll do better next time.
* Your risk assessment was correct. But management elected to take the risk.
Stuff happens. Your ability to know what information is enough will improve as you gain experience with a product line. It’s not a sin in testing to miss a bug. It’s only a sin to be careless, thoughtless, or not to learn from your experience.”

“A less thorough, more diversified test strategy is better than a more thorough, less diverse strategy. In other words, it’s better to do more different kinds of testing to a pretty good level, than to do one or two kinds of testing perfectly. We call this the principle of diverse half-measures.”

“Furthermore, we think it’s better to live with uncertainty about testing than to embrace false certainty.”

““Enough testing” means “enough information for my clients to make good decisions.””

A Framework for Good Enough Testing, James Bach

“Good Enough testing is the process of developing a sufficient assessment of quality, at a reasonable cost, to enable wise and timely decisions to be made concerning the product.”

Investigating Bugs: A Testing Skills Study, James Bach

“Forget about bad testers, even good testers are notoriously bad at explaining what they do. Doing testing, describing testing, and teaching testing are all different things.”

“Skepticism.
Skepticism might be called the fear of certainty. It can be seen as central to the challenge of thinking scientifically; thinking like a tester. Good testers avoid sweeping claims about the product, because any claim may be refuted with the execution of the next test.”

“Consulting with developers or other testers.
Vital information needed to investigate problems is scattered among many minds. Good testers develop an understanding of the network of people who may be able to offer help, and know to  approach them and efficiently elicit the information they need. In the case of developers, testers need the ability to discuss and question software architecture.”

What Is a Good Test Case?, Cem Kaner

Test Framing, Michael Bolton

RST Appendices

“Good testing is often considered mysterious and unstructured. This leads people to panic and bury themselves in restrictive rules and templates. Stop that! You just need to know how to think about it and what to look for.”

“Good testing, like programming, is instead a challenging intellectual process.”

Basic skills of a good programmer?, Gerald Weinberg

“I also examine my strengths (long-comings?) because I know that my greatest strengths can quickly become my greatest weaknesses.”

“Many programmers do examine themselves critically, but then they work to improve their greatest strengths, to the exclusion of their weaknesses. That practice takes them a certain distance, but the nature of computers is to limit your ability, by highlighting your greatest weaknesses.
Good programmers make good use of their best tools, and you are your best tool, so sharpen yourself.”

And couple of tweets about our ability/posibility to learn enough before deadlines:

No comments yet

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: