Time to update threat modeling

Working in a security company, there is an activity we try to routinely do, at least when anyone hints on not having done it. That activity is security threat modeling. Getting smart people together, supported by a heuristic (STRIDE) we explore what could possibly go wrong on security with the design we’ve made by now. Purpose of threat modeling is to learn and change when we learn. And for someone trying to drive forward better quality, there’s not a more powerful argument than connecting a bug somehow to security. 

Heuristics are used to keep the discussion both focused and multifaceted. People easily would focus on one type of problems, and heuristics like STRIDE help with thinking from a few more perspectives. It’s far from complete, but it has been the basic approach for good enough to get started with. The acronym opens up to words Spoofing, Tampering, Repudiation, Information disclosure, Denial of service and Elevation of privilege. 

Security is about not letting people do bad stuff with systems, hopefully while allowing the right people to do what they were trying to achieve with the use of the system. All of the perspectives easily map to the idea of the users and attackers. 

But with many modern systems, there is one often dismissed theme I would bundle with security, while I realize that for me as a tester it has long been a separate concern. That is one of abuse. I’m exploring extending STRIDE with an A. 

Abuse vectors are often unique. They are ideas of how we could unintentionally open up ways for targeting misbehaviors against a group with use of the systems. And thinking of the abuse threats is getting increasingly important.

Let’s explore a few ideas around this.

A prominent tech woman ended up with targeted abuse with Github. At a time when Github allowed people to be added to projects without their consent someone thought it was a fun thing to add her to projects she would by no means associate with. All those projects then end up being a part of her profile. We would want to make sure our features don’t enable high visibility cyber bullying. 

A group of people built a learning bot, which turned into a monster in a matter of hours. We would want to make sure that with the learning systems, we can control the quality of the data the system learns from.

A face recognition software was built, and it did not recognize faces of people of color. The sample set the system was built on did not do a good job at being representative, and the biases of the creators got coded into the functionality. We would want to make sure we don’t make a group of people invisible with systems intended for wide use. 

A phone had a feature of facial recognition for logging in. It worked perfectly for images of the owner’s face. We would want to make sure that if we use faces as credentials, gaining access to our personal data is not one picture away. 

Abuse as an attack vector is connected with STRIDE, but different. And we need to start actively thinking of it as we create more sophisticated systems that include our assumptions of privilege and biases. 

Mob Programming on a Robot Framework Test

A year ago as I joined my current place of work, I eagerly volunteered to share on Mob Programming in a session. As I shared, I told people across team lines that I would love to do more of this thing here. It took nearly a year for my call to action to sink in.

A few weeks back, a team lead from a completely different business line and product than I work on pinged me on our messaging system. They had heard that I did this thing called Mob Programming, and someone in the team was convinced enough that they should try it, so what then should be the practical steps? They knew they wanted to mob on test automation, and expressed that a challenge they were facing was that there was one who pretty much did all the stuff for that, and sharing the work in team was not as straightforward as one might wish for.  Talking with three members of the team including the team lead and whoever had drawn me in, we agreed on a time box of two hours. For setup, one would bring in the test setup (which was a little more than a computer as we were testing a Router Box) and ideas of some tests we could be adding.

It took a little convincing (not hard though) to get the original three people to trust me with the idea of not having first a session of teaching and introducing the test automation setup, and that we would just dive in. And even if we agreed on that in advance, the temptation of explaining bits that were not in the immediate focus of what the task drove us towards was too much to resist. 

The invitation for the team of 7 included “test automation workshop”, without a mention of mechanism we would be using on this. And as we got to work on adding test, I asked to figure out who knew the least and made them sit in front of the keyboard first, just saying the rule of “no thinking on the keyboard”. I also told them we’d be rotating on 3 minutes, so they would all get their chance on the keyboard and off, except for the one automation expert. Their rule was to refrain from navigating unless others did not know (count to three before you speak). 

Looking at the group work for 1,5 hours was a delight. I kept track of rotation, and stepped into facilitating only if the discussing got out of hand and selecting between options was too hard. I noticed myself unable to stop some of the lecture-like discussions where someone felt the need of explaining, but the balance of doing and talking was still good. People were engaged. And it was clear to an external person that the team had strong skills and knowledge that was not shared, and in the mob format insights and proposals of what was “definition of done” for the test case got everyone’s contributions.

I learned a bit more on Robot Framework (and I’m still not a fan of introducing a Robot language on top of Python when working with one still would seem a route of less pain). I learned of use of Emacs (that I had somewhat forgotten) and could still live without. I learned on different emphasis people had on naming and documentation on tests. I learned on their ideas of grouping tests into suites and tagging them. I learned of thinking in terms of separating tests when the Do is different, not when Verify needs a few checks for completeness. I learned to Google Robot Framework Library references. And I learned that this team, just like many other teams at the company, is amazing. 
Asking around in retro, here’s what I got from the team: This was engaging, entertaining. We expected to cover more. The result of what we did was different than the first idea of what the test would be, and the different means better in this case. 

My main takeaway was to do this with my team on our automation. If I booked a session, they would show up. 

What makes a test automation expert?

I was part of a working group that created an article called 125 Awesome Testers You Should Keep Your Eye on Always. It may not be obvious, but that list is a response to another article called 51 automated testing Experts You Should Keep Your Eye on Always. That list had only four women (at least it had four women!) and let me tell you a big public secret:

It is not because there aren't many awesome women in automation. It is because people don't look around and pay attention.

I could have many different criteria on what makes a test automation expert:

• Speaks about test automation in public (conferences, articles) in a way that others find valuable
• Does epic stuff on making automation work out and do real testing
• Is identified as a creator of a test automation framework or library
• Speaks only of automation and never in a manner that addresses its limits

The 125 awesome testers list does not identify automation separately, because I find that most people contribute to test automation in a significant way. Not all of people in either one of those lists have created an open source tool of their own. Not all people on either one of those lists write test automation code as their main thing.

We can be awesome at automation in so many ways. Writing code alone in a corner is not the only way. Many of us work in teams that collaborate: pair, or even mob. Coding is not the only way to do automation.

• Delivering insights that are directly transferable to useful test automation is a way of doing automation. 
• Working on the automation architecture, defining what we share is a way of doing automation.
• Helping see what we've done through lenses of value in testing is a way of doing automation. 
• Reading code without writing a line and commenting on what gets tested is a way of doing automation. 
• Pairing and mobbing are ways of doing automation.

We don't say coding is all there is to application development, why would coding be all there is to  test automation development?

There's been a particular experience that has shaped my experience around this a lot, which is working with mob programming.  After programming with 14 different programming languages, I still identified as a non-programmer because my interests were wider. I actively forgot the experience I had, and downplayed it for decades. What changes me was seeing people who are programmers in action. I did not change because I started coding more. I changed because I started seeing that everyone codes so little. 

The image below is from a presentation of Anssi Lehtelä, a fellow tester in Finland who has also now two years of mob programming with his team under his belt. A core insight I find we share is that in coding, there is surprisingly little of coding. It's thinking and discussions. And that's what we've always been great at too! And don't forget googling - they google like crazy!

Lists tell you who the list maker follows. Check if you have even a possibility to recognize the awesome women in automation using http://proporti.onl on your twitter feed. It can be brutal. Mine is 53 % women. In the numbers I can follow, there's easily a brilliant, inspirational woman to match every single man. In any topic, including automation. Start hearing more voices.

Announcing an Awesome Conference - European Testing Conference 2018

TL;DR: European Testing Conference 2018 in Amsterdam February 19-20. Be there! 

Two months of Skype calls with 120 people submitting to European Testing Conference 2018 in Amsterdam has now transformed into a program. We're delighted to announce people you get to hear from, and topics you get to learn in the 2018 conference edition! Each one of these have been hand-picked for practical applicability and diversity of topics and experiences in a process of pair-interview. Thank you for the awesome selection team of 2018: Maaret Pyhäjärvi, Franziska Sauerwein, Julia Duran and Llewellyn Falco.

We have four keynotes for you balancing testing as testers and programmers know it, cultivating cross-learning:

• Gojko Adzic will share on Painless Visual Testing
• Lanette Creamer teaches us on how to Test Like a Cat
• Jessica Kerr gives the programmer perspective with Coding is the easy part - Software Development is Mostly Testing
• Zeger van Hese Power of Doubt - Becoming a Software Sceptic

With practical lessons in mind, we reserve 90 minute sessions for the following hands-on workshops you get to choose to participate two, as we repeat the sessions twice during the conference:

• Lisa Crispin and Abby Bangser teach on Pipelines as Products Path to Production
• Seb Rose and Gaspar Nagy teach  on Writing Better BDD Scenarios
• Amber Race teaches on Exploratory Testing of REST APIs
• Vernon Richards teaches on Scripted and Non-Scripted Testing
• Alina Ionescu and Camil Braden teach on Use of Docker Containers

While workshops get your hands into learning, demo talks give you a view into looking someone experienced in doing something you would want to mimic. We wanted to do three of these side by side, but added an organizer bonus talk on something we felt strongly on. Our selection of Demo talks is:

• Alexandra Schladebeck lets you see Exploratory Testing in Action
• Dan Gilkerson shows you how to use Glance in making your GUI test code simpler and cleaner
• Matthew Butt shows how to Unit/Integration Test Things that Seem Hard to Test
• Llewellyn Falco builds a bridge for more complicated test oracles sharing on Property-Based Testing

Each of our normal talks introduces an actionable idea you can take back to your work. Our selection of these is:

• Lynoure Braakman shared on Test Driven Development with Art of Minimal Test
• Lisi Hocke and Toyer Mamoojee share on Finding a Learning Partner in Borderless Test Community
• Desmond Delissen shares on a growth story of Two Rounds of Test Automation Frameworks
• Linda Roy shares on API Testing Heuristics to teach Developers Better Testing
• Pooja Shah introduces Building Alice, a Chat Bot and a Test Team mate
• Amit Wertheimer teaches Structure of Test Automation Beyond just Page-Objects
• Emily Bache shares on Testing on a Microservices Architecture
• Ron Werner gets you into Mobile Crowdsourcing Experience
• Mirjana Kolarov shares on Monitoring in Production 
• Maaret Pyhäjärvi teaches How to Test A Text Field

In addition to all this, there's three collaborative sessions where everyone is a speaker. First there's a Speed Meet, where you  get to pick up topics of interest from others in fast rotation and make connections already before the first lunch. Later, there is a Lean Coffee which gives you a chance for deep discussions on testing and development topics of interest to the group you're discussing with. Finally, there's an Open Space where literally everyone can be a speaker, and bring out the topics and sessions we did not include in the program or where you want to deepen your understanding.

European Testing Conference is different. Don't miss out on the experience. Get your tickets now from http:/europeantestingconference.eu 

How Would You Test a Text Field?

I've been doing tester interviews recently. I don't feel fully in control there as there's an established way of asking things that is more chatty than actionable, and my bias for action is increasing. I'm not worried that we hired the wrong people, quite the opposite. But I am worried we did not hire all the right people, and some people would shine better given a chance of doing instead of talking. 

One of the questions we've been using where it is easy to make step from theory to practice is How would you test a text field? I asked it in all, around a whiteboard when not on my private computer with all sorts of practice exercises. And I realized that the exercise tells a lot more when done on a practice exercise.

In the basic format, the question talks of how people think of testing and how they generate ideas. The basic format as I'm categorizing things here is heavily based on years of thinking and observation by two of my amazing colleagues at F-Secure Tuula Posti and Petri Kuikka, and originally inspired by discussions on some of the online forums some decades ago. 

Shallow examples without labels - wannabe testers

There's a group of people who want to become testers but yet have little idea of what they're into, and they usually tend to go for shallow examples without labels. 

They would typically give a few examples of values, without any explanation of why that value is of relevance in their mind: mentioning things like text, numbers and special characters. They would often try showing their knowledge by saying that individual text fields should be tested in unit testing, and suggest easy automation without explaining anything else on how that automation could be done. They might go talking about hardware requirements, just to show they are aware of environment but go too far in their idea of what is connected. They might jump into talking about writing all this into test cases so that they can plan and execute separately, and generate metrics on how many things they tried. They might suggest this is a really big task and suggest to set up a project with several people around it. And they would have a strong predefined idea of their own of what the text field looks like on screen, like just showing text. 

Seeing the world around a text box - functional testers

This group of people have been testers and caught up some of the ideas and lingo, but also often over reliance on one way of doing things. They usually see there's more than entering text to a text box that could go wrong (pressing the button, trying enter to send the text) and talk of user interface more than just the examples. They can quickly list categories of examples, but also stop that list quit quickly as if it was irrelevant question. They may mention a more varied set of ideas, and list alphabetic, numeric, special characters, double-byte characters, filling up the field with long text, making the field empty, copy-pasting to the field, trying to figure out the length of the field, erasing, fitting text into the visible box vs. scrolling, and suggest code snippets of HTML or SQL, the go to answer for security. They've learned there's many things you can input, and not just basic input into the field, but it also has dimensions. 

This group of people often wants to show the depth of their existing experience by moving the question away from what it is (the text field) to processes and emphasize experiences around how relevant it is to report to developers through bug reports, how they may not fix things correctly and how a lot of time goes into retesting and regression testing. 

Tricks in the bag come with labels - more experienced functional testers

This group of testers have been looking around enough to realize that there are labels for all of the examples others just list. They start talking of equivalence partitioning and boundary values, testing positive and negative scenarios and can list a lot of different values and even say why they consider they're different. When the list starts growing, they start pointing out that priority matters and not everything can be tested, and may even approach the idea of asking why would anyone care of this text field, where is it? But the question isn't the first  thing, the mechanic of possible values is.  They prioritization focus takes them to address use of time into testing it, and they question if it is valuable enough to be tested more. Their approach is more diversified and they often are aware that some of this stuff could be tested on unit level and others require it integrated. They may even ask if seeing the code is available. And when they want to enter HTML and SQL, they frame those not just as inputs but as ideas around security testing. The answer can end up long, and show off quite much of knowledge. And they often mention they would talk to people to get more, and that different stakeholders may have different ideas. 

Question askers - experienced and brave

There's a group who seems to know more even though they show less. This group realizes that testing is a lot about asking questions, and mechanistic approach of listing values is not going to be what it takes to succeed. They answer back with questions, and want to understand typically the user domain but at best also the technical solution. They question everything, starting with their understanding of the problem at hand. What are they assuming, and can that be assumed? When not given a context of where the text field is, they may show a few clearly different ones to be able to highlight their choices. Or if the information isn't given, they try to figure out ways of getting to that information. 

The small group I had together started just with brainstorming the answer. But this level wasn't where we left of. 

After the listing of ideas (and assumptions, there was a lot of that), I opened a web page on my computer with a text field and an ok button and had the group mob to explore, asking them to apply their ideas on this. Many of the things they mentioned in the listing exercise just before immediately got dropped - the piece of software and possibility to use it took people with it.

The three exercises

The first exercise was a trick exercise. I had just had them spend 10 minutes thinking how they would test, and mostly they had not thought about the actual functionality associated with the text field. Facing one, they started entering values and looking at output. Over time, they came up with theories but did not follow up testing those and got quite confused. The application's text field had no functionality, only the button had. After a while, they realized to go into dev tools and the code. And were still confused with what the application did. And with a few rounds of three minutes each on the keyboard, I had us move on to the next example. 

The second exercise was text box in the context of a fairly simple editor application, but one where focusing on the text box alone without the functions immediately connected to the text box (unit test perspective) would miss a lot of information. The group was strong on ideas, but weaker on execution. When giving a value, what a tester has to do is to stop (very shortly) and look at what they learned. The learning wasn't articulated. They missed things that went wrong. Things where to me, an experienced exploratory tester, the application is almost shouting to tell how it is broken. But they also found things I did not remember, like the fact that copy pasting did not work. With hints and guidance through questions, I got them to realize where the text box was connected (software tends to save stuff somewhere) and eventually we were able to understand what we could do with the application and what with the file it connects to. We generated ideas around automation, not through the GUI but the file and discussed what kind of things that would enable us to test.  When asked to draw a conceptual picture of relevant pieces, they did good. There was more connections to be found, but that takes either a lot of practice on exploring or more time to learning layers. 

Again with the second exercise, I was left puzzled on what I observed. They had a lot of ideas as a group on what to try, but less of discipline in trying that out of following what they had tried. While they could talk of equivalence partitioning or boundaries, their actual approaches on thinking what values are equivalent and learning more as they used the application left something to hope for. The sources of actual data were interesting to see, "I want a long text" ended up as something they could measure but unawareness immediately of an online tool that would help with that. They knew some existed but did not go to get those. It could have been a priority call, but they also did not talk about doing a priority call. When the application revealed new functionality, I was making a mental note of new features of the text box I should test. And when that functionality (ellipsis shortening) changed into another (scroll bars), I had a bug in mind. Either they paid no attention, or I pointed that out too soon. Observation and reflection of the results was not as strong as idea generation. 

The third exercise was a text field in a learning API, and watching that testing unfold was one of the most interesting ones. One in the group quickly created categories of three outputs that could be investigated separately. This one was on my list because the works right / wrong is multifaceted, and in the perspective of where the functionality would be used and how reliable it would need to be. Interestingly in the short timeframe we stick with data we could easily generate, and this third one gave me a lot to think about as I made later one of my teams's developers test it, getting them even stronger into testing an output at a time, and insisting on never testing an algorithm without knowing what it includes. I regularly test their algorithms of assessing if the algorithm was good enough for the purpose of use, and found that the discussion was around "you shouldn't do that, that is not what testers are for". 

The session gave me a lot of food for thought. Enough so that I will turn this into a shorter session teaching some of the techniques of how to actually be valuable. And since my conference tracks planned are already full, I might just take an extra room for myself to try this out as the fourth track. 

Fixing Wishful Thinking

There's a major feature I've been testing for over a period of six months now.  Things like this are my favorite type of testing activity. We work together over longer period of time, delivering more layers as time progresses. We learn, and add insights, not only through external feedback but also because we have time to let our own research sink in. We can approach things with realism - the first versions will be iterated on, and whatever we are creating is indefinitely subject to change.

I remember when we started, and drew the first architectural diagrams on the wall. I remember when we discussed the feature in threat modeling and some things I felt I could not get through before got addressed with arguments around security. I remember how I collected dozens of claims from the threat modeling session as well as all discussions around the feature, and diligently used those to drive my learning while exploratory testing the product.

I'm pretty happy with the way that feature got tested. How every round of exploration stretched our assumptions a little more, giving people feedback that they could barely accept at that point.

Thinking of the pattern I've been through, I'm naming it Wishful Thinking. Time and time again with this feature and this developer, I've needed to address very specific types of bugs. Namely ones around design that isn't quite enough.

The most recent examples came from a learning that certain standard identifiers have different formats. I suggested this insight should lead to us testing the different formats. I did not feel much of support for it, not active resistance either. I heard how the API we're connecting to *must* already deal with it - one of the claims I write down and test against. So I tested two routes, through a provided UI and the API we connect with, only to learn that my hunch got confirmed - the API was missing functionality someone else's UI on top of it added, and we, connecting with the API missed.

A day later, we no longer missed it. And this has been a pattern throughout the six months.

My lesson: as a tester, sometimes you work to fix Wishful thinking. The services you are providing are a response to the developers you're helping out. And they might not understand at all what they need, but appreciate what they get - only in small digestible chunks, timed with enforcing the message into something actionable right now.

Learning through embarrassment

Through using one video from users to make a point, we created a habit of users sharing videos of problems. That is kind of awesome. But there's certain amount of pain watching people in pain, even if it wasn't always the software failing in software ways, but the software failing in people ways. 

There was one video that I got to watch that was particularly embarrassing. It was embarrassing, because the problem shown in that video did not need a video. It was simply a basic thing breaking in an easy way. One where the fix is probably as fast as the video - with only a bit of exaggeration. But what made it embarrassing was the fact that while I had tested that stuff earlier, I had not recently and felt personal ownership.

The sound track of that video was particularly painful. The discussion was around 'how do you test this stuff, you just can't test this stuff' - words I've heard so many times in my career. A lot of times the problems attributed to lack of testing are known issues incorrectly prioritized for fixing. But this one was very clearly just it - lack of testing. 

It again left me thinking of over reliance on test automation (I've written about that before) and how in my previous place of work lack of automation made us more careful. 

Beating oneself up isn't useful, but one heuristic I can come up with this learning: when all eyes are on you, the small and obvious becomes more relevant. The time dimension matters. 

I know how I will test next week. And that will include pairing on exploratory testing.