Tuesday, May 24, 2016

Ways to get to continuous delivery

I listened in on a talk today about Test Automation and Continuous Delivery. Paraphrasing what I think the speaker said is that:

  • None she knows of does daily deliveries without test automation (and stays sane - we heard of an other example like that though where "expecting normal workdays would just not be appropriate") 
  • We need to stop coming up with excuses on avoiding the test automation
In the end of the talk, I offered a counter-experience to understand her ideas a little better. I work with a  team that has, for two years now, done daily deliveries and with ridiculously little test automation. I feel we are still very much sane, we deliver quite safely. We are working - slowly - on adding some test automation, but for a long time, there was none. It took us long to build up skills for the right kind of automation we would find useful.

What I believe I learned is that we had an interesting discussion about our beliefs in what to invest in.

In her case, the problem she was solving was that testers were very frustrated with finding simple bugs in manual testing. That is a problem we sort of shared.

In her case, her team sought for solutions in the testing space and went for test automation. She told the audience that their automated tests are valuable, and find bugs pretty much on a daily basis. 

In my case, my team sought for solutions in development and testing space, jointly. We had a limited amount of things we could do at a time, and conversations with my developers lead us to decide on investing primarily in cleaning up the code to make it more readable. 

I asked the speaker today how their code was, and she confirmed theirs was messy.

So my train of thought is: you can choose to invest in making things better for your testers. If you feel you can invest over silo limits (not within the the testing silo), these two might be your options. You could get to same or in my experience, a better place investing into clean code. But most people seem to go for test automation as the primary investment, even with unclean code. 

When your code is clean, your devs can have discussions around it easier. Devs share it easier, ask for feedback more eagerly. Devs make less mistakes as you go about changing it. With less mistakes, there's less of the simple bugs to be found in manual testing.

And when you top that with smart, adaptive exploratory testing like my team does, it is possible to do continuous delivery without automation. Automation would make us faster, but not necessarily safer. The cleanliness of code is what seems to have a connection in how we experience safety in customer feedback. 

I don't think this is an excuse to avoid test automation for us. The excuse is that there has been a lot of skills to build up, and we chose to build those skills clean code first, unit testing with asserts and approvals and selenium tests later. And we're just still taking baby steps in the automation front, but doing great on the clean code front, reaping the results in our ability to deliver without breaking production. 


  1. I think "clean code" always have good unit test coverage, and unit test's are part of test automation (If you run them in your CI and why would you not include them to CI).

    I agree that you can't have bad code if you deliver daily and fixing that should be first priority. But in complicated systems where are lot of people committing daily I think good automation is must (like exploratory testing).

    Automation should be build in layers (lot of low level testing and not so much high level E2E testing).

    Mika Heikkinen

    1. I don't have a lot of people, only 10. And when I say clean, I mean readable. To an extent that a non-programmer can have discussions about it.

      While we have been working on test automation, the payoff has been learning not testing done. So it is not a must, for us. It's definitely something we are working on.

      When you lack all sorts of skills and can start acquiring those skills somewhere but not everywhere at once, you have different choices. This was our choice. I just report that for us, it worked well.

      Then again, my team's devs test without automation too. They think. They talk. To me and to the end users.

  2. Thanks for your great post, I'm an QA automation lead, and my test only focus on E2E test, and the problem I got is my team (my team devs and manual QA) trust too much in Automation test they spend less effort on Exploratory testing and code refactor/review. But from another view, the problem when the project less automation is the morale of manual QA, when the project want release faster and faster

  3. Thanks for sharing the story :)

    Sometimes you need to work with the tools that you have and focus on what you can achieve with them.

    Too often we focus on a process or a practice, when we should really focus on the result. I think the fact that you do frequent, small deliveries is much more important than any methodology you choose to develop it actually. Its your foundation and doing this is what actually requires you to find ways to write reliable code. Without reliable code, you cannot do frequent deliveries.

    Now on test automation - it is just one way to achieve code reliability. Of course there are other and they frequently go hand to hand. Clean code in all cases is a must if you want to make quick progress and test automation is something that helps things move in that direction. This is why in my experience clean code and test automation usually go together. Disclaimer here - what I mean by test automation is mostly unit tests, while many people will think about system/end-to-end tests :)