Monday, March 27, 2017

The Myth of Automating without Exploring

I feel the need of calling out a mystical creature: a thinking tester who does not think. This creature is born because of *automation*. That somehow, because of the magic of automation, the smart, thinking tester dumbs down and forgets all other activities around and just writes mindless code.

This is what I feel I see when I see comparisons of what automation does to testing, most recently this one: Implication of Emphasis on Test Automation in CI.

To create test automation, one must explore. One must figure out what it is that we're automating, and how could we consistently check the same things again and again. And while one seeks for information for the purposes of automation, one tends to see problems in the design. Automation creation forces out focus in detail, and this focus in detail that comes naturally with automation sometimes needs a specific mechanism when freeform exploring. Or, the mechanism is the automation thinking mindset. 

I remember reading various experience reports of people explaining how all the problems their automation ever found were found while creating the automation. I've had that experience in various situations. I've missed bugs for choosing not to automate because the ways I chose to test drove my focus of detail to different areas or concerns. I've found bugs that leave my automated tests in "expected fail" state until things get fixed.

The discussion around automation is feeling weird. It's so black and white, so inhumane. Yet, at core of any great testing, automated or not, there is a smart person. It's the skills of that person that turn the activity into useful results. 

Only the worst of the automators I've met dismiss the bugs they find while building the automation. Saves them time, surely, but misses a relevant part of feedback they could be providing. 


  1. Maaret,
    I agree wholeheartedly with you on this. If you are not exploring and testing the behavior of the SUT as you begin (and continue as you work with the system) the automation work you are not doing your job. Blindly writing code is not going to cut it. Pure "Happy Path" testing/navigation of the system will only go so far. In order to understand how the system really works, and how best to have the automation interact with it, means doing your own "self testing". You have to get your own hands dirty in the functionality to see its behavior. Once you've learned how the system behaves then you can get the automation to act accordingly. Automation is only as good as the person who built it, and part of that is the front-end work of the person building it.

    Jim Hazen

  2. I completely agree. Much of what I see on test automation focuses entirely on the implementation or technology side of it (in part probably because vendors are out there selling), and seems to ignore the fact that creating good test automation inherently requires the same skills that a "manual" tester needs.

    One unfortunate reality is there are automation engineers out there who do not create or design tests, but simply take fully-specified test cases and implement them. This is such an inefficient waste of peoples' time (and runs into some of the same problems that Agile and XP exist to solve), I can't begin to describe how much I hate it. But it's real.

  3. Completely agree ... this dichotomy that is being propagated is creating an artificial chasm between the "exploratory" and "automated" practices of testing . Anyone who claims to have exercised both components of the Testing craft will tell you that both crafts have their roots in incisive critical analysis,focus and skill . In fact they are complimentary set of skills and mindset ... a contemporary tester can not be effective without either.

    1. There is a difference in which one starts with an artifact creation in mind, and the other starts with a performance for serendipitous information in mind. But they need to be more intertwined.

  4. I this post could mislead readers. The issue isn't that you can explore when creating automation. You can explore during any activity, e.g., cooking, gardening. The objectives of test automation are very different from exploratory testing. Test automation answers a very narrow question - does it *still* work? There may be a lot of work done to create test automation code. However, the outcome is very narrow - it can be very useful. Exploratory testing on the other hand attempts to answer, 'what do we not know that can be a risk'.

    1. Thanks for your comment. I find that exploratory testing is not one activity, but a group of activities where notetaking is one part of the activities. To me, when creating automation to run again later, it is a (tedious) form of notetaking.

      Also, test automation is not just regression automation. The automation we check in and decide to run regularly has the purpose of knowing if it still works. Other automation may be on purposes of just extending while exploring.

      I've seen many test automators do very good exploratory testing while automating. I've seen more test automators do bad exploratory testing while focusing on automation.

  5. but....
    1. Reality of the industry is different. Check the rails tutorial on testing: 'Testing is making sure it works correctly'. That's all there is.

    Is there any developer who talks about testing like you describe (I don't include people like Liz Keogh, dan north, godjko, I am talking about developers) in books, blogs, conferences? I know there are 2 instances.

    2. 'we can explore during automation' is an interesting idea. Not sure if we need to be more assertive?
    Sometime back there was an unnoticed spark in testing in the agile community: Check Liz's response to the question, 'So a lot of times...' and she says, 'I tend to trying and avoid having conversations about really blindly obvious stuff,....' When we take a stance like that - we can label that testing. The other stuff is automation.

    There have been others who have made similar strong statements in the past.

    I like Cem Kaner's idea that every test is a question. It's much more subtle, since no one asks questions for which the answers are known.

    3. Given that we can play fast and loose with the word 'test', the reality is that there is no compulsion for 'test' to mean anything. The world will keep spinning.

    In most companies, automation is very powerful. It has the king's ear. Testing will always be the ugly cousin. Therefore we can call anything 'testing' and no one will object.

    Nilanjan (