Notes from Testing Without Testing talk

Earlier today I attended talk by James Bach called Testing Without Testing. I did not know anything about James before and for some reason I assumed that the talk would be about Software Engineers in Test vs. Test Engineers or something like that, testing vs. automated testing or something.

It wasn't. And I was not disappointed.

The talk could be divided to three parts:

1. Conventional testing books are bad and test engineer don't study their craft
2. Testing does not aim to improve quality
3. What testing is

Here's what I thought about these three things.

Books are bad and so are you

James opened the session by talking about how conventional testing books are superficial about the terms they use. Many books advice us to ask questions and use techniques like equivalence partitioning etc. So how does one know what are the questions one should ask? How do you know that all values in one equivalence class actually belong to that class? Do we just assume? In many cases we do, even though we should be able to reason why they belong to that class. And in order to know what questions to ask, we should have good domain knowledge on the thing we are testing.

At work my title is Senior Test Engineer, but I write code daily at work or as a hobby. The software engineer in me knows that there are some excellent books which cover their subjects exhaustively. For example Code Complete, Design Patterns, The Pragmatic Programmer and The Mythical Man-Month.

The test engineer can't find any good books.

Blogs? Joel on Software, Coding Horror, Programming in the twenty-first century, etc. versus nothing.

Conferences? Every popular programming language and framework has it's own annual conference, for testing there's Google Test Automation Conference. And that's pretty much it.

Interesting open source tools to follow? Interesting Repositories at GitHub versus Selenium and Robot Framework.

People? Methodologies?

You can't test quality in

James argued that test engineers goal is not to increase quality, just like soccer goalie does not try to make goals. Except in very rare cases.

I believe that James is right that better quality is not the goal of testing. It's not the goal because no matter how much you test, the quality of the product does not increase. Only way to improve the quality of the product is to write better code. Sure, testing might help one to write better code, just as continuous integration, better standards and faster build process can.

So what testing is?

For this topic James had a slide full of little text bubbles and in the center there was something he called "checking". That checking is the thing that people who are not test engineers think that testing is.

By checking he means things like filling a registration form, submitting it and checking that account was created. So things which are in many cases covered by test cases.

That checking is part of testing, but only a part of it. In addition to that, there were things like familiarizing with the product, project postmortems, and guiding helpers. And by helpers he meant people inside the house who use the software, but don't participate in creating it. Persons like help desk and manual writers.

I really do agree that this thing called checking is only part of the thing we do. Personally I feel that the checking part should be automated as much as possible. In our company intranet I've written that test automation should set us free from filling up that account creation form for the millionth time and it should give us superpowers to do things we humans normally cant; like do complex math and send million TCP packets inside tight time frame.

So aim for test automation that will give you time to do testing we humans excel; seeing, hearing and forming opinions.