Tuesday, February 2, 2016

Thanks, but I still have test automation

There's another big buzz in the world of testing, as the allowed and acceptable words are being refined. This time, the word is automation.

I know testing cannot be automated for now. In its full features, it's a process of collecting empirical feedback on things from we know to expect to things we had no idea could exist. It's full of all kinds of wonderful activities, where a thinking human is in critical position. It's hard to talk about testing because your testing and my testing won't be exactly the same anyway. Why would it be such a bad thing when people talk about test automation meaning the parts they believe they can automate.

I also like to think of it this way. If programmers job in general is to automate things, to get started on that does not mean everything has to be automated. A robot that does heavy lifting for people is valuable, even if it only operates in the warehouse and does not deliver things at your front door.

Merriam-Webster defines automation as "the process of putting an apparation, operation or system under control of mechanical or electronic device". When we put parts of testing under control of electronic device (the tool that we need to attend to differently than if we did not have it), I'm doing test automation.

Having more words to explain the details would be helpful. What type of test automation are we talking about? Is it execution? Monitoring and reporting? Setting up data and environments? Which specific problem in the domain of testing we're trying to control removing humans at least partially? 

I've seen my share of wasteful test automation. I've seen it replace thinking where thinking is due. But I've seen that fixed by introducing experimentation culture and empirically assessing what worked for us. It can include failing with automation on the value it provides. We already have words to discuss that problem: value, opportunity cost, short-term and long-term. Many of us are having that conversation continuously.

A main principle I always remember of context-driven testing is that people are the most important aspect of any context. How about believing that people will look for solutions, and figure out what does good enough job for them without redefining anyone else's namespace. If using the word test automation makes me less of a context-driven tester, so be it. I just want to ensure the organizations I'm working with have the best chances of doing a good job. And good job looks different depending on organization and the day we're looking at it. Everyone is learning every day, including those of us who think they've already learned a thing or two.

** as for calling automators technical testers, no. There's technical testers who don't program but are intertwined into the system / environment technical aspects. Like saying people in IT departments wouldn't be technical.

7 comments:

  1. Hi Maaret

    It is not often we disagree but on this subject I feel there is a difference in perspectives.

    Using your same source, if we look up the definition of 'test' from Merriam-Webster we find the following:

    "a critical examination, observation, or evaluation : trial; specifically : the procedure of submitting a statement to such conditions or operations as will lead to its proof or disproof or to its acceptance or rejection ,a test of a statistical hypothesis> (2) : a basis for evaluation : criterion "

    Now when you add that to the definition of automation it becomes quite strange and not so straightforward as your article leads us to believe.

    "A critical examination, observation or evaluation as performed by a system under control of mechanical or electronic devices"

    To me this statement and definition now seems strange and generates many questions.

    Can a machine perform 'critical' evaluation or observations?

    What does 'critical' in this context mean?

    Hence my own issues with the use of the statement "test automation" rather than "tool assisted testing" or "check automation" . The execution of testing is far more than just a binary yes or no, as Michael and James point out in their article:

    "In testing, we design and perform experiments that help us develop our understanding of the status of the product."

    Taking this a step further in your article you make the following statement:

    "If programmers job in general is to automate things,"

    Why is this a programmer’s job? I and others have tools that can automate things it does not mean I need to hand it off to the programmer because it is their job!

    Automate what things? We can only automate what we know or can describe. What about the knowledge that cannot be described or coded? (Tacit knowledge - See Harry Collins or Polanyi). We have far more knowledge than any of us can describe or talk about never mind replicate in code.

    One final point I would like to address is the use of the word "value" James and Michael pick up on this in their article, what do you mean by value, value to who, to what? Value has numerous meanings that is unique to each individual. Therefore trying to tie value to introducing automation can led down a rabbit hole of manipulation to meet a particular persons definition of value. See value theory and philosophy for more on this topic.

    I appreciate your article since it gives a good starting point for such a discussion on this topic.

    ReplyDelete
    Replies
    1. Your response goes, in my reading, into the point of automating *all* testing. I try to say that I know it is a part. I know many other people know it is a part. It's a growing part. And renaming it doesn't do me that much good, so I choose to keep my "test automation".

      I'm well aware of Collins work and had the pleasure of meeting him while he was speaking at EuroSTAR. Nothing I say was intended to be against tacit knowledge.

      Value is many things, and different to different people. The value that the article seems to downplay is the value developers with extensive test automation experience: fast feedback, daring to change things without worrying so much about regression testing.

      I'm just against trying to use time on changing the spoken language of denying certain words. I'm all for adding more words to have a deeper discussion in the spirit of dialogue, in the hopes of both parties learning something.

      Delete
  2. Thank you for writing this, it needed saying.

    There is one thing I disagree with: I don't see James Bach's and Michael Bolton's article as redefining the context-driven namespace. They don't own that namespace, the context-driven community does. They do own the RST namespace, with which they can do whatever they want.
    However, it's also true that the RST namespace has a great influence over the CDT namespace. And I for one would like to see some more strong voices from within the community.

    In any case, I do not think that simply using the word 'test automation' makes anyone less (or more) of a context-driven tester. At most it can be the starting point for an honest conversation.

    ReplyDelete
  3. Thank you for writing this, it needed saying.

    There is one thing I disagree with: I don't see James Bach's and Michael Bolton's article as redefining the context-driven namespace. They don't own that namespace, the context-driven community does. They do own the RST namespace, with which they can do whatever they want.
    However, it's also true that the RST namespace has a great influence over the CDT namespace. And I for one would like to see some more strong voices from within the community.

    In any case, I do not think that simply using the word 'test automation' makes anyone less (or more) of a context-driven tester. At most it can be the starting point for an honest conversation.

    ReplyDelete
    Replies
    1. I needed to check if I talked about the name space and I didn't. All I say is that I still will use words that come easily to me and will not correct others on this. My perspective is we put too much effort on definitions and wordplay and not enough on conversations that seek mutual learning, believing people have best of intentions.

      Delete
    2. My comment on namespaces originated from "How about believing that people will look for solutions, and figure out what does good enough job for them without redefining anyone else's namespace. If using the word test automation makes me less of a context-driven tester, so be it."
      My apologies if I misinterpreted.

      Delete
    3. You extended it a lot from what I meant by it. But that is quite fine.

      Delete