How to treat a testing contractor

With EuroSTAR 2013 behind me, there's a story to share. I got to listen a great presentation 'Fooled by Unknown Unknowns' by Alexandra Casapu twice, as she was selected to do a do-over session as something you should have seen if you missed the first time.

Alexandra works with me (or for me), and inspired by her authentic story and great improvement in testing with our product, I have my own story to tell.

As a test manager / tester in a customer organization using services from Romania, I could go with a suspicious approach. I could require that the remote tester for visibility purposes creates test cases and runs marks them passed or failed. I could require detailed notes, detailed hour reports and detailed monitoring. I could be suspicious on how she uses her time, and focus on ensuring the time it took to prepare for this presentation was not in any way invoiced from the customer. I could give her the least interesting assignments in our testing that need to get done, and cherry pick the fun stuff for myself. I could give her ridiculously small timeframes to test large features, and then complain that she did not find all the problems. I could show her no support in thinking how to make the testing she does better, implicitly guiding her to shallowness. I could require compensation for missing the bugs she missed that I had to find in the feature she talked about.

I don't do any of that. And I wish there was more customers who buy testing services that would not do that either.

Instead, I guide her work like I'd like my work to be guided. I require thinking, and taking time to reflect and make sure we pay for those hours too. I provide feedback, by testing in same areas and by reviewing ideas of what to cover. I treat her as a colleague she is, with the idea that treating every day of testing as a learning opportunity, every day at work makes us a step better. And I strive for the same excellence in testing myself using most of my time on testing too. 

There are many customer organizations outsourcing testing without involving a skilled tester on their side. These customers may set unrealistic expectations and requirements on how the work should be done, and not have the necessary viewpoint to assessing whether the testing performed by the contractor was any good. The criteria of goodness may degenerate into numbers of passed test cases, instead of valuable testing done with a reasonable effort. With the trust missing, we end up building crazy approaches to manage the distrust.

Someone asked me on my way home from EuroSTAR if Alexandra asked permission to talk about our project. My reply was, that she did not ask - I requested her to do share a real story with real details, as that is a direction that I feel testing community needs. I'm fortunate to work for a company that shares my view on this to an extent that we're having researchers watch us work on testing, and hopefully soon enough publish results about what is it that skilled testers actually do.

Slow progress and patience - getting devs to test more

Torn between my two completely different teams, this is a story from the one that makes me impatient. When I joined - already 1,5 years ago - I negotiated on "test-fix-finalize" week where the whole team would do 'testing' - except we would not do testing, we would do code changes to make automated tests possible. We would fix bugs that we really would not want to document in our tests,. And we would add automated tests, unit-level first.

The allocation to "test-fix-finalize" has stayed, leaving a small window of exploring how the system works before it goes live. And we do need that window since we have not managed to get very good at small incremental changes or test automation on any level. Or, some might still argue that 2-3 weeks of developer work is small.

Quite recently we have moved into introducing Selenium tests. After the first examples and the discussion about "can't we just outsource this to some testers somewhere", the team has been adding those. There's been workshops, first one day and this month two days, where the whole team sits together and codes tests together. We don't do this for anything else except tests, so I'm kind of happy to contribute to that.

When adding tests, we've also been changing the product. And that again, to me, in some small scale, is a plus on the list of why I wanted the work to stay inhouse, with this team, to not get yet another round of tests that no one maintains, runs or feels any ownership of. Yes, we did that once or twice, before I joined.

Having people sit together has also other positive impacts. Like this week, while they were doing the tests and I was spending time on my other project, they started talking about how they want to do things. I get an email telling about their "decisions", where the most relevant to me was this: they decided to plan their effort so that it includes testing - well, their idea of testing - and do trade with other developers on tasks to get fresh pair of eyes on their own code. Kind of nice step, I just wish still the system testers would be part of that cycle too. But this did not exist before so its definitely progress. 

I was thinking about what actually brought this change: visibility and sharing the pain. A month ago we moved the system testing backlog into the same Jira-project where all developer work is tracked, because the time was right. And since then, the team had to see how the single tester was getting piled with three times the work that could be completed. And being the nice people they are, they are actively thinking how to help.

I wish it would help immediately, but I guess I need to work on my patience. Step at a time, never give up. If things don't feel right, there's a way to change them. Sometimes finding the way is just taking quite some time...

Reading specifications with thought

There's a rock I've hit hard too many times recently and the pain I feel is clearly telling me there's a need of changing. As part of me processing this, I thought I'd blog. After all, it's been too long.

Starting with an example, simplest out of the selection. We've been adding a feature to our product, which is about printouts. I've seen the spec, read it as thoughtfully as I did in that time, and waited a little to get the feature implemented to do some testing.

It turns out that the feature was nice in isolation of other surrounding similar features. It turns out it was nice on picture and text, but when put together with live scenarios, there's more than first meets the eye.

So I log a bug on that. We eagerly discuss it with the team and decide on a change to what we had thought previously. Implement and again comes testing...

It turns out that we missed yet another scenario. While the first issue was that printouts of two sorts worked differently on main view, the second one is that there's more than the main view and again inconsistencies that would lead us to a different design.

I'm frustrated, as I did not see these coming looking at the spec. Then again, anyone else did not either. But failing twice in a row, with very similar pattern, is just painful. And we keep failing with a similar pattern.

So thinking about what to change, I realize I'm puzzled. I could take more time to test the specifications, since I seem to have some touch to seeing the problems with the live product. Then again, someone else could do that too.

Positive side: no one in the team is initiation discussion whether these are bugs / should be fixed. Just frustrated with the inability to remove the root cause that keeps us stomping on similar issues again and again.

For anyone handling tester's issue reports - there's usually more than the one described symptom. Removing a symptom without addressing the underlying problem isn't taking us where we'd want to be.

Off to reading the next specification, with scenarios I just defined to support my thinking. Perhaps these will help the others see problems too, time will tell. Reading with thought is quite different from just reading. So many things between the lines that make us waste everyone's effort.

Types of events in Agile Finland community

I seem to be taking a habit of replying blog posts with blog posts, and here's the source of current inspiration to write: http://learninggamedev.wordpress.com/2013/08/01/join-me-in-re-imagining-the-agile-finland-coaching-circle/

I've been setting up a session to reimagine things in Agile Finland in a larger scale than just coaching circles, with 24 participant limit this time. Thus I've tried to also positioning the coaching circle in a bigger scheme of activity. The fact has been that for quite some time, coaching circles were pretty much the only activity, muddling them into "all things agile, from beginner to advanced". But it does not have to be so yet it can be.

With discussions around, I gathered the types of events we seem to bounce between when organizing:

• circles (like coaching circle) are for practice and deep knowledge though shared insights
• dojos and retreats and camps (like coding/testing) are for practice
• unfacilitated meetups are for meeting people interested in similar themes, but finding the right conversation is up to you
• facilitated meetups are for talking about a topic together in a group / subgroup, making finding the right conversation a little easier
• peer conferences (unconferences) are for sharing real experiences and debating them for insights
• conferences are for listening to ideas and experiences with possibility to discuss or gain experiences through exercises
• webinars are for listening to ideas and experiences in cases where you can't be onsite
• working parties are for creating something together. 
• courses are for teaching a topic with exercises to practice

I placed some of these types of events on a scale of Content vs. Group size to show how different event types are usually intended for different audiences.



For Agile Finland, the Agile Drinks/Dinners have been unfacilitated, with nice groups of people meeting up, sometimes with a specific theme and sometimes not. The Dinners we've organized within Finnish Association for Software Testing have been facilitated, meaning that there is one discussion ongoing for the whole group instead of everyone grouping up to a different theme.

We have the unfacilitated / facilitated Dinners. We have the dojos. We have the coaching circles. And we have the conferences. What we've been missing is something for a beginner/intermediate audience that allows a larger crowd and that happens more regularly - a mini-conference with a facilitated discussion twist. That's what Agile Breakfast will be about, and the first one will most likely be 6.9.2013 in Ruoholahti - first Friday morning of the month. Location still needs confirming.

For the events status, I collected this: https://docs.google.com/spreadsheet/ccc?key=0Ak7e3JZGe_hadDBscGF4NEFtZm5JdnVMZEE1ZXF0NUE&usp=sharing - just to see what there could be.

Choices, choices ... on how to use time

For about half-a-year, I've been thinking and preparing a webinar series on testing, with a few main motivations.

• Finland is not Helsinki. To do things with Finnish Association of Software Testing or Agile Finland, we can't do all sessions in Helsinki.  Some of the most inspiring Finns I want to hang out with are located in Oulu, Tampere and Turku. And most likely there are inspiring people I haven't had a chance of meeting, since they just don't come to sessions in Helsinki.
• Finland is not Europe or the whole world. I have a strong feeling that we need to be more connected in Europe as well as internationally, and doing EuroStar made me realize how few of our field's big names are from Europe outside UK. Just looking around has taught me about several people I'd like to know better, but I lack the chance of traveling like a consultant. Thus the webinar series would give people like me access to people and ideas that are not otherwise available. 
• Great people and great presentations are not online yet. Knowing from personal experiences of listening to a lot of presentations by Finnish colleagues, a lot of this stuff is not online. I can't give a link to listen to amazing presentations that others should hear, and I suspect my local community isn't only local community like this. Tapping into a wider range of presenters would be great. I'd love to show off my colleagues from Finland, and I'd love to see people I haven't seen before. 

I wrote a description document on this quite some time ago on this.

Since then, I've had chats on this with people. From someone running a lot of online content, I learned that webinars might be really dull way to deliver the contents. From another one running webinars I learned that the numbers on a scheduled session are surprisingly small. From our own technical trials, I learned that while it is possible, a lot of participants actually fade away during the webinar. For presenters, I learned that some people feel off talking all by themselves, and others find the talking to an invisible crowd comforting without the stress they get from live presentations.  From discussions, I learned that videotaping may be a key to what I want to achieve - new content over repeating the same old stuff. And that while we have these conferences where we actually just listen to the presentations, people find the breaks and the face-to-face connections and the forced presence of being there to listen really good. 

With the limited time available to organize something, all this talking is taking me to a conclusion different than the one I had half a year ago. I want to use my time slots on seeing people face to face, but videotaping the great contents to give others the chance of tapping into what we talk about. For the non-Finnish presenters, I want to help those who already have the webinars set up find the people and topics they should show.

Thus, I won't be organizing a series of webinars, but a series of events that happen in face-to-face settings, with a streaming possibility. And if anyone has contents they want to make available, I'm volunteering as happy to help to get that stuff out - in front of an audience or just a camera.

Refining agile to see what and why for Agile Finland

I'm organizing an aquarium discussion for August on finding a purpose for Agile Finland. I took the idea from someone at latest Agile Dinner / Drinks, and it should be fun hearing what people have to say on what we should be and do.

With the small group I talked with on the Agile Drinks session, we seemed to be quite well in synch with what we'd like. We'd like to shift focus from the adjective 'agile' into words that seem more meaningful like productive, results-oriented and happy. The things we value when we seek agile as opposed to not-agile.

Back with the purpose: nothing special, just save Finland, tweeted one of the discussion participants. The idea that valuable software is a key asset for the future here, we need to seek ways to collaborate and to do it better - for all roles in development and in touch with development.

I know I no longer want to waste my life into planning for an acceptance test for a year and a half, only to learn that either the plan fails (even though it was updated throughout the wait time) or that when the plan is successfully executed, the testing done is of no value really. I want to plan continuously, and do continuously. And learn from both our successes and failures.

Milestone for a test resistant team

Whenever someone ask me if we're "agile", I notice I start to feel the need to apologize. Sure, we're on that journey with some criteria, but not there on so many others.

For example, my team is somehow trying to have everyone do "testing", but their idea of testing easily could be "development buffer", "try it once" and "go for coffee, you can relax now". When they manage to find the motivation to pair up with another developer, it's a little better, but the enthusiasm never last long for repeating anything.

While agile teams could be test infected, this one is long on the route to be test resistant and the probabilities of a sudden infection are not getting better yet.

The team has rounds of "let's automate tests in some way" behind them. The most recent one on unit tests went the furthest, with some of the team managing to refactor the code to add tests (one out of nine) and others came up with all sorts of reasons why legacy code made things too difficult for them. The round before that was to take a summer intern and ask him to record tests based on test cases the team had done. Those tests were never run by anyone other than the intern.

For the unit testing effort, I negotiated 1/4 of the teams time over a 6 month period, so unlike usually, the external schedule pressure was really to the problem. We created structures where we could do relevant refactoring, but too  little of that happened. And we had an external coach to help us with the tests just a little, with the end result of us realizing the vastness of refactoring / architectural change that would be needed to get where we want to be.

With  the unit test automation gone and the problems of not enough testing happening, we agreed to do some group exploratory testing sessions. With a few of them under our belt, they seem to do the job for now, but with the late feedback and the repeating nature of checking similar situations are wearing out the developers. This will not last, not for this type of information need.

In one of our group exploratory testing sessions, I drifted off to look at my team mates and think about the next experiment in the efforts to make it better for us. We had failed with UI automation and none was interested in touching it. No one wanted to spend their summer on automating tests that would not be helpful. So I called a consulting colleague in testing, and we talked about the smallest possible thing and timeframe we could set up to do a selenium proof of concept. I could do that myself, if it did not include learning that I did not have time for right now. But I knew enough to explain to my boss why we'd invest on an external to do something like this right now.

The proof of concept was delivered this week, in a meeting with the whole team. The proof of concept showed developers a style of tests they would accept as part of their work. This time, the external did not set up extensive suite of fragile scripts to throw to the developers to maintain, but a few structured examples and a useful explanation of what doing these include for our software. The external did not introduce new tools or languages, but just a driver for the browser interface that could be called from the code as we know it now. Owning small scope from someone else created positive buzz, and if the time frame of setting up was longer, we'd have more implemented but less sense of ownership.

The tests were integrated right away into the local code repository the consultant had no access to. And they ended up being run with the build cycle to find a problem on their first day of existence, a side effect of common services change that did not work in this area quite as the idea was with a big visible error message - just the type of information we usually look for in the release testing. And the problem was quickly fixed to get the tests back running again.

For the first time, I think there is an actual chance this will fly. And next I'll need to find ways of showing what else there is in the lovely world of testing.