Monday, August 3, 2020

An Analysis of Exploratory Testing

Open space conferences like Socrates UK Digital Summer provide a great platform for making a little progress on finding ways to teach about exploratory testing in writing. For purposes of writing, I run an ensemble testing session to compare notes of what I did in prep alone vs. where the group ends up. Putting the two together could provide useful lessons for those who did not get to join.

For these sessions, I picked up a new test target. Eviltester posted some of his testing apps and games a while back, and EPrimer ended up as my choice as it promised 
  1. Not heavy on bugs - could actually focus on testing instead of bug reporting
  2. Completely unknown domain: proper English language writing style "eprime" I had never heard of. 
  3. WebUI with beautiful IDs
At this point, I encourage you to follow the link to the app and stop reading what I say before you tried it out yourself. If you did not follow my encouragement, I suggest that after reading this, pick up another of the eviltester test targets and apply what you learned here. 
Session Charter:  Explore EPrimer focusing on two kinds of documentation as output: test automation you can run (using e.g Robot Framework) and a mindmap. Time: 1 hour plus learning time for test automation tool if you have no experience and no expert available answering your questions in the moment. 
Two sessions, two results
As expected, the two session provided very different results that complement one another.

Session one produced ~30 tests one can run again, spread over 7 test suites, each named on the type of collection of data it was testing and a mindmap on realization that all tests were on single function while there were multiple but covered the domain description as specification well, identifying multiple problems against specification. 

Session two produced 5 tests one can run again, all in 1 test suite where a bit of commenting out is necessary to get the tests to run later. The coverage of functions was significantly better and the session identified 2 bugs. No mindmap was created and better function coverage came from choosing to understand everything a little and not diving systematically into specification. Single created test covered more ground. 

Breakdown of activities

Whenever we are doing exploratory testing, we get to make choices of where we use our limited time based on the best information available at the time of testing. We are expected to intertwine various activities, and when learning, it may be easier to learn one activity at a time before intertwining them.

If you think back to learning to drive (while stick gear was a thing), you probably have ended up in an intersection, about to move forward and your car shutting down as intertwining your actions with the gear and pedals were not quite as they should. You slowed down, made space for each activity and got the car moving again. Exploratory testing is like that, you control the pace and those who have practiced long will be intertwining activities in a way that appears magical. 

For this testing target, we had multiple activities we needed to intertwine (learn / design / execute):
  • Quickly acquiring domain knowledge: no one knew what eprime is, and we had our choice of reading about it.
  • Acquiring functional knowledge: using the application and figuring out what it does.
  • Creating simple scripts with multiple inputs and outputs: using same test as template for data-driven helps repeat similar cases in groups. 
  • Identifying css selectors: if you wanted test automation scripts, you needed to figure out what to click and verify and how to refer to those from the scripts. 
  • Controlling scope of tests: see it yourself, see it blink with automation, repeat all, repeat only the latest. 
  • Creating an invisible or visible model: Seeing SFDPOT (Structure, Function, Data, Platform, Operations, Time) to understand coverage in selected or multiple dimensions
  • Cleaning up test automation: Improving naming and structuring to make more sense than what was created in the moment.
  • Using the application: Making space for chances to see problems beyond the immediate test
  • Identifying problems: Recognizing problems with the application. 
  • Documenting problems: Writing down problems in either test automation or otherwise.
  • Working to systematic coverage: Pick a dimension, and systematically cover it learning more on it. 
  • Reading the code: We had the code and we could read it. That could add to our understanding. 
Taking another two hours on top of the two hours on cleaning up the results. I summarized final results like this:
The app isn't completely tested, as the exercise setting biased us towards documentation. 

Examples of activities

Quickly acquiring domain knowledge. Reading the specification of eprime. Focusing on examples of eprime. Refreshing knowledge of English grammar around the verb "to be" e.g. 5 basic forms of verbs and 6 different types of verbs, or 6 categories of verbs - all things I googled for as I am writing this. While the specification tells what knowledge was probably used to create the application, there is domain knowledge beyond what people choose to write down in specification. 

Acquiring functional knowledge. Using the application. Asking questions about what is visible, particularly the concepts that are no obvious: 'What is Possible Violations?".  Seeking data demonstrating it could work. Seeking large data to demonstrate functions through serendipity. 
Creating simple scripts with multiple inputs and outputs. Writing test automation that allows for giving multiple input and output values as parameters. Getting into the tool and into using the tool. 

Identifying css selectors. Getting to various values with code, understanding what is there in different functions to click and check. Feeling joy systematic ID use making the work easier.  Recognizing conflicts in UI language and selector language. 

Controlling scope of tests. Moving tests to separate files. Commenting out tests that already work. Running tests one by one. 

Creating an invisible or visible model. Ensuring we see all things work once before we dig deeper in any individually. Creating a map of learning in the last minutes of the session. Writing down notes of what we are seeing as either test automation or other types of documents. 

Cleaning up test automation. Rename everything named foo at time when we knew the least. Comment out things to focus on the next thing getting done efficiently. Using domain concepts as names of collections. 

Using the application. Spending time using the application to allow for serendipity. Observing look and feel. Observing selected terminology. 

Identifying problems. Seeing things that don't work. Like visual of it that is very bare-bones. Or line breaks turning valid positives into false negatives. 

Documenting problems. Writing these down in test automation. Figuring out it we want to leave it passing (documenting production behavior) or failing (documenting bugs). Remembering issues to mention. Writing them down as notes. Writing a proper bug report. 

Working to systematic coverage. Stopping to compare models to how well those are covered. Creating a visual model. Covering everything in the specification. Covering all visible functionality. 

Reading the code. Closing the window that has the code as it gets in the way of moving between windows. Reading the code to see how concepts were implemented.  

Some Reflections

Every time I teach exploratory testing, I feel I should find ways of teaching each activity separately. There is a lot going on at the same time, and part of its effectiveness is exactly that. 

In the group, someone suggested we could split the activity so that we first only document as test automation, not caring about any of the information other than what is true right now in the application. Then we could later review it against specifications and domain knowledge. That could work. It would definitely work as one of the many mixes when exploring - change is the only constant. This split is what approvaltesting is founded on, yet I find that I see different things when I use the application and create documentation intertwined, than receiving documentation that I could review. One night in between the actions is enough for me to turn into a different person. 

The Final Deliverables

In the last minutes of one of the sessions, I cooked up a mindmap of what was in my head on the application. I had only covered a small portion, focusing on counting Discouraged words. 

The robot tests from the two sessions combined with cleanup are available at: 

Wednesday, July 29, 2020

An Exploratory Tester's Zephyr

Zephyr, in case you did not know, is a Jira Test Management extension. I dislike Jira, and I dislike Zephyr. But what I like and don't like does not change (well, immediately) the whole organization, and I play within general bounds of organizational agreements. In this case, it means an agreement that tests are documented in Zephyr - for some definition of tests. 

This post is about how I play within those bounds, enabling exploratory testing. 

What Zephyr Brings In

Zephyr as a Jira plugin enables some very rudimentary test specific concepts:
  • Ticket reuse. When the jira ticket is a test, it can be run many times, like for example for each build we test. Normal Jira tickets are more straightforward in their lifecycle.
  • Steps. For some reason people still think tests have steps with expected values. If you don't know better, you might use these. DON'T. 
  • Mapping tests to releases. You can tell what test ticket connects with a particular Jira release. It shows the structure of how testing usually progresses in relation to changes. 
  • Grouping. You can group tests inside releases into test suites. You have many reasons you might want to group things. Zephyr calls mapping and grouping cycles. 
  • Run-time checklists. You can keep track of passes and fails, things in progress. You can do it either on level of a group of tests or on an individual test. You have a whole own view to making notes while testing on a particular test case, execution view. It seems to imagine all your test needs in one place: bug reporting, steps, notes. 
What I Bring In

When I document my plans of testing, I create a few kinds of tests:
  • [Explore] <write a one line summary here>
    These tests can be for the whole application like "Gap analysis exploration - learn all the problems they don't yet know", or a particular purpose like "Release", or an area of particular interest like "Use for people with disabilities". If I can get away with it, I have only one test case titled "[Explore] Release" and I only write notes on it at time of making a release. What this assumes though is that release is something more continuously flowing rather than one final act in the end - agile as if we meant it. 
  • [Scenario] <write a one line summary here>
    These tests are for very high level splitting of stakeholder perspectives I want to hold space for. They are almost like the ones I mark [Explore] expect that they all together try to summarize remembering the most important stakeholders and their perspective in the product lifecycle. These are in the system context, regardless of what my team thinks their component delivery responsibility has been limited to.  
  • [Feature] <write a one line summary here>
    These tests I use when I have bad or non-existent documentation on what we promise the software will do. These tests all together try to summarize what features we have and try to get to remain, but as a high level checklist, not going into details of it. These are in the context of the system, but more towards the application my team is responsible for. 
I use states of these tests to indicate scope ahead of me. 

If a test is Open (just like a regular Jira ticket), it is something I know we expect to deliver by a major milestone like a marketing release all the little releases work towards, but I have not seen a version in action we could consider for the major milestone scope. It reminds me to ask if we have changed our mind on these. 

If a test is Closed, it is still alive and used. but it is something where we have delivered all the way to production some version of it and we intend to keep it alive there. 

If I can get away with one test case, that is all I would do. There are many reasons for me not to be able to get away with it: a newer colleague we need a shared checklist with, me needing a checklist and creating it here with minimal extras, or auditing process that would not be fulfilled with just that one ticket of [Explore] Release. 

The updating of test status is part of release activities for me. Someone needs to create a release in Jira, which usually happens when the previous release is out. For that release, I add at most two Cycles:
  • Pre-Release Testing
  • Release Testing
Again, if I can get away with it, I have only one: Release Testing and within in, I have only one test: [Explore] Release that I mark passed and write notes if I have something useful to say. Usually the useful thing for me to say is "release notes, including scope of changes is available here <link>". 

The way testing works for me is that I see every pull request and nothing changes outside pull requests. I test selected bits and pieces of changes, assessing risk in the moment. I also have a set of test automation that is supposed to run blue/green (pick your color for 'pass') that hunts down need of attending to some detail. And I grow the set of automation. If you need 'proof' of passing for a particular release, we could in theory get that out of version control but why would you really want that?

The Pre-Release Testing Cycle, if it exists, I fill it when I think though what happened since last release and what still needs to happen before the next one and I drag in existing tests from all three categories [Explore], [Scenario] and [Feature] to be a checklist. What this cycle contains tells about themes and features I found myself limiting to. And when a Pass on the cycle isn't sufficient documentation, I can always comment the test ticket. 

My use of Zephyr is very different to my colleagues. Perhaps also to your use? 

Tuesday, July 21, 2020

Anchoring an idea while Exploratory Testing an API

One of the things we get to test is a customer oriented API. It's particularly lovely test target for multiple reasons:
  • Read-only: It only gets data, and does not allow us to change data. Makes it simpler! 
  • Time-constrained on API level: You can tell dates as input and it does freeze time for test automation purposes. You don't have to play with concepts of today() and now(). 
  • Limited and understandable UI level edits to data: There are some things we can change from GUI that impact the API but they are fairly straightforward. 
The main reason it brought us joy for testing today is that we found a bug on it a few weeks back where particular combination returns 500 error code (Server error) where it should not, and we got to start creating some tests back then to create a nice baseline for the time that bug would be fixed.

The long awaited message of bug fix arrived today, and the first thing we'd do is pull out the tests we had automated the last round (asserts and approvals, I wrote about those earlier as we set the project up). We ran the tests, expecting to see a fail for the assert for getting 500 for that bug. The results surprised us.

We still had that test passing, but now we also had another test failing with 500. Instead of going forward with the fix, we had momentarily gone backwards. 

Not long after, we got to try again with a new version. This time it was just as we expected. Within 30 seconds of realizing the version was available, we knew that on the level we automated our tests before, those were now matching today's expectations. 

For those of you concerned on the tests not running in CI, it is about the same time to go check they are blue as we did not place these tests as ones blocking the pipeline. These tests weren't designed for the pipeline, they were designed as an entry point for exploratory testing where we could leave some of them behind for the pipeline or for other purposes. 

We quickly drafted our idea of what we would test and change today:
  • Capturing and reviewing for correctness for the combination that we previously documented as receiving the 500 response for that bug
  • Ensuring we could see latest data after the most recent changes
  • Having easily configurable control over dates and times we had not needed in our tests before
  • Making some of the tests approval files smaller in size as long as they did not lose the idea of what we were testing with them
What turned out to be the most fun thing to test was the latest data. Starting with that idea, we found multiple other ideas of what to test, including things around changing more values on the data, and things around multiple overlapping limits. We needed to remind ourselves, multiple times, that we still have not seen our starting idea in action, even if we had seem many other ideas. 

As a conclusion of today, we came to the importance of anchor, and remembering that anchor. If writing it down helps, write it down. If having a pair that keeps you honest helps, have a pair. Whatever works for you. But a lot of times, when we do some testing, we end up forgetting what was the thing we set out to do in the first place. Anchoring an idea allows us to discover while we explore, and still stay true to what we originally set out to do. 

We ended up refactoring our test code a bit to make it more flexible for the ideas we had today, and we discovered one test we wanted to keep for future. It started off with one name and concept, yet though exploring we learned that what we wanted to keep for future was different to what we wanted and needed to do today. 
Truth is, we always throw some away, and that is where I recognize learning and thinking is going on. Can keep and should keep are two different things.  

Saturday, July 18, 2020

Dealing with Rejection in Teams

Have you ever come back from a conference, full of energy with the great ideas the speakers shared, gone to your team and suggested to try something new  to only hear that it's an idea that "would not work here", "now is isn't the time for that" or that "the idea is stupid" - implying you're stupid liking that idea. Surely your team isn't rejecting you, they are rejecting ideas you got really excited on. 

And it's not just the ideas that come from outside but ideas of doing something different, like having a cup of coffee with colleagues at least once a week. I'm often even commented down on naming variables in pull requests, only sometimes to something I agree is a better name but I stop fighting so that we don't get stuck. We get rejections of our ideas all the time. 

When you learn all your ideas are rejected, you move on to only dealing with ideas on a personal level and obeying ideas of powers to be. You take what others are offering, and within the box of them not seeing you do things, you do what you can do right inside that little box. 

I'm someone who counts. I count how many times my ideas get brushed down. I count how many times I brush down other people's ideas, and who are the people who reject other's ideas the most. I have been rejected a lot, yet I still keep trying because when I need to give up, I need to give up on the industry. 
Here are some ideas of how I deal with that rejection that we get in the teams:
  • repetion. Ask once, ask again. Kids have no shame in this, but adults get punished fairly soon. So if you repeat the same ask, be careful on how much annoyance you will add. A weekly repetition is probably better than repeating it many times in a row. But also, people end up liking things they've heard of many times better than those they hear 1st time. 
  • finding right time. Ask when they are more likely to say yes. Did you know that asking to get out of jail as a convict, you are more likely to get out if the time of your case happens just after lunch? Asking after completing a major milestone is likely to give different results than asking while you are in the middle of that worst crunch.
  • prioritize what you ask for. You'd like to see 10 changes,  so select one. It could be the one that means you the most. It could be the one they are most likely to accept. 
  • finding right words. It's not what you say, it's how you say it. Sometimes it is true, sometimes it's an excuse. Try to find a way of explaining it that makes the one listening to your proposal understand it. It could be logic. It could be financial benefits. It could be an appeal to your personal happiness ('yay, that worked for me on ensemble (mob) programming'). 
  • finding right messenger. Sometimes you will never be heard, so send someone else. I did this to get no estimates started a few jobs ago and too many times I could quote since. I like to say: "best ideas win if we care about work over credit" and feel sad how much of my credit I need to move away before reclaiming it through promoting results. 
  • finding right medium. Some people react better to verbal while others react better to written requests. Some people forget all things verbal and are only safe being asked in writing. Use one first, another later. 
  • convincing a subgroup. If you have a few people suggesting something, some folks here groups better than individuals. You may need to get buy in from people who are not making the decision to get through to those making the decision. 
  • make it sound temporary. Call it an experiment. Agree on a time you will do it, and when you give up, even when you are really thinking you should keep doing it. This worked great to get to having an agile team with no product owner and results that were improved significantly. 
  • confronting the rejection pattern. Tell people you've observed that your suggestions are rejected. Keep track of what ideas they did reject, and suggest a rule that  they must experiment with at least one every six months, or one out of ten you produce. This one is DANGER! 
  • visualizing of whose ideas we went with. Draw on a whiteboard a tally of names and label that as ideas proposed/implemented. See if seeing the pattern helps people realize they could work with it.
  • showing it works without the others. Just do it yourself. A lot of tech ideas only get traction if you show up with a prototype that works. You could also find others in community instead of your organization, and work like you wanted on your free time on learning projects. 
  • build a track record. Get some of your ideas through. Show you can try small and are willing to step away if they fail. Building that confidence may help them hear you better. 
  • create a patience raindance. Create a little routine that helps you through all this rejection so that you still can try again. My patience raindance routine is tweeting. It's a mysterious call for powers to be on granting me patience to try again until I succeed in getting to a happy place. 
  • amplify ideas of others. Don't be the person who shoots other people's ideas down. Try to approach them with the "let's try it out" attitude even when you hate it. You'd like them to do it for you, do it for them. 
Finally, dealing with rejection is a skill, and your need of developing that skill depends on your status in the organization and team. It is likely you will deal with this more if you are a tester and if you are a woman and even more if you are not white. Cope with rejection, but never ever give up. Only through the No you can get to a Yes. Protect yourself on the way there with versatile strategies of knowing when to give up and how to get to a yes. 

This blog post is brought to you by a twitter pile on of helpful agilist who seem to think getting rid of or changing contents of a daily meeting is something you just change in a team. They may come from a position of privilege where when they point out a change people just do it, but I find myself more often not in that situation and thus employing all the ways I have to be tenacious and get there anyway. I've been there too, and I recognize the difference. 

Friday, July 17, 2020

Where in Testing is Exploratory Testing?

When people start learning about testing, and agile testing in particular, they quickly get to a model of testing quadrants. Testing quadrants, popularized by Janet Gregory and Lisa Crispin with their Agile Testing book, place a collection of testing words in four quadrants. When you go look, you can find "exploratory testing" in the top right corner, meaning it is considered Business Facing Product Critique. 

As the term was coined 35 years ago to express a specific approach to testing, making it a technique in one corner of the quadrants was not the intent. It expressed a style of testing unfamiliar to the majority, that was observable in Silicon Valley product companies, a skilled multidisciplinary testing under product development constraints. 

The world moved on, and testing certifications had hard time placing a whole approach of testing into their views of the world. With introduction of ways to manage this style of testing (session- and thread-based exploratory testing), those seeking to place it in the technique box combined the management style and defined their idea of using only limited time - separately defined sessions - on this way of testing and everything it was born to be different from remained in the center. 

That means that in the modern world, exploratory testing is two things:
  1. a technique to fill gaps that all other testing leaves
  2. an approach that encapsulates all other testing 
As a technique, you can put it in the corner of quadrants. As a technique, you can put it on top of your test automation pyramid and make jokes about the pyramid turning into an ice cream cone with too much of exploratory testing on top. But as an approach, it exist for every quadrant, and for every layer. 

Due to the great confusion, questions about the other testing I do on top of exploratory testing are quite common. 

This response today inspired me to think about this a little more. 
Exploratory fills the gaps.
But for me, it does not fill the gaps. It is the frame in which all other testing exists. It is what encourages short loops to learn, challenges the limits of what I already have learned, makes me pay attention to what is real, and creates a sharp focus on opportunity cost. 

I scribbled an image on paper, that I recreated for purposes of this blog post. If all these shapes here are the other kinds of testing mentioned: feature testing, regression testing and non-functional testing, what is the shape of exploratory testing? 
The shape of exploratory testing is that it fills the gaps. But it also defines the shape of all the other tests. It's borders are by design fuzzy. We are both right: for me it is the frame from which all the other testing exists, even when it fills gaps. 

There is such thing as non-exploratory testing. It's the one where shape of other tests stay in place and are not actively challenged, and where particular artifacts are important over considering their value and opportunity cost. 

Where I worked, we had two teams doing great at testing. Both teams explored and left behind test automation as documentation. When asked what percentage they automated, their responses were very different. One automated 100%, and it was possible by not having as many ideas of what testing could be. The other automated 10%. Yet they had just as much automation as the first, but often found problems outside the automation box.  Easiest way to get get to 100% is by limiting your ideas of what testing could be. 

Seeing there's plenty of space in between the shapes and plenty of work in defining the shapes can be a make or break for great testing. 

Tuesday, July 14, 2020

Starter Project Overload - just tell me the steps

Today, I ended up spending a few hours setting up a javascript - jest - puppeteer environment, to enable comparison of the tests we do on a third framework (we did this before with record-playback within Datadog and Robot Framework). The two others served as stepping stones to understanding what you can and can't test and what maintenance of the scripts feel like, and continuing with either of the two in our team would mean that these tests belong to the tester alone. So Javascript is a definite priority for sharing with the rest of the team. 

Googling around did not make getting set up easy and fluent. There is too much instruction, unsurprisingly. The question from the intern is well warranted: "How the hell are you supposed to know which of these articles are worthwhile sources?"

My blog does not match the criteria of worthwhile sources I explained them: seek material as close to the original open source project as possible. But I thought I'd still write down for fun and benefit how simple it turned out to be.

To get the jest - puppeteer environment running to a point where you can start writing your own tests, here is what to do. 
  1. Install yarn (=> google "install yarn" and install as instructed)
    Yarn is a package manager. It pulls down packages someone else made available. 
    NPM is another package manager. You could use that too, the commands are just different then. 
    And yes, package manager is something you need on your development machine. The packages it brings down to your project are different in the sense that they are dependencies for what you build. The steps to install dependencies with package manager are setup, and you don't want all those files to be same for every single programming project you do on your computer. 

  2. Create a folder for your programming project to reside in and open the folder in VS Code
    This step is for familiarity and control. Having nothing and being in the empty folder when working with a new tool gives sense of control. 

  3. Run Terminal (from Terminal | New Terminal in VSCode) and install dependencies
    You will want to run 
    yarn add jest
    yarn add puppeteer
    yarn add jest-puppeteer
    that will create you a bunch of files under your empty folder under node_modules folder.
  4. Add jest to your path so that you can run it from command line
    yarn global add jest
  5. Initialize jest to create the configuration file
    jest --init
    This creates jest.config.js file. 

  6. Test that jest alone works for you

    Create a file sum.js
    function sum(ab) {
        return a + b;
      module.exports = sum;

    Create a file sum.test.js
    const sum = require('./sum');

    test('adds 1 + 2 to equal 3', () => {

    Straight out of Jest Getting Started! 

    Run the tests
  7. Add jest-puppeteer to jest.config.js file
    "preset": "jest-puppeteer"
    While at it, comment out 
    testEnvironment: "node",
  8. This step was the one that I was hunting for an hour! For everyone's benefit, I could have used the energy I use on this post to help correct the original jest documentation but I ended up adding to newbie confusion. 
  9. Test that jest puppeteer works for you

    create a file google.spec.js
    (NOTE! spec, not test or it worn't work - the second thing that was causing me pain in this flow) 

    describe('Google', () => {
        beforeAll(async () => {
          await page.goto('');
        it('should be titled "Google"'async () => {
          await expect(page.title()).resolves.toMatch('Google');

  10. Replace all tests/specs with whatever you want to work on. 
Next up is putting these in a container without using a container project starter. But that must be another blog post for my future reference. 

Thursday, July 2, 2020

Never tested an API? - A Python Primer from My Summer Trainee

With first of our release, I taught the most straightforward way I could to test an API for my summer trainee. I gave them a URL (explaining what a URL is), showed different part of it indicated where you connected and what you were asking for and ended up leaving office for four hours letting them test for the latest changes just as other people in the team wanted to get out of office for their summer vacation. They did great with just that in my absence, even if they felt the responsibility of releasing was weighing on them. 

No tools. No postman. Just a browser and an address. Kind of like this:

The API we were testing returned a lot more values. We were testing 20000 items as the built-in limit for that particular release, and it was clear that the approach to determine correctness was sampling. 

Two weeks later, today we returned to that API, with the idea that it was time to do something more than just looking at results in the browser. 

Python, in the interpreter

We started off by opening a command line, and starting python. 

As we were typing in import requests, I explained that we're taking a library into use. Similarly I explained print(requests.get("")), forgetting the closing parenthesis at first and adding it on a line after. 

With the 200 response, I explained the idea of this code meaning it was ok, but we'd need more to see the message we had earlier seen in a browser, and that while we could also use this for testing, we'd rather move to writing our code to a file in an IDE. 

Python like a script, in Pycharm

As we opened Pycharm and created a .py file to write things in, the very first lines were exactly the same ones we had been running from command line. We created two files. First requirements.txt in which we only wrote requests and second file ended up with name As the two lines were in, Pycharm suggested installing what requirements.txt defined and we ensured it was still running just the same. At first we found the Run menu in IDE, later the little green play buttons started to seem more appealing as well as the keyboard shortcut for doing this one often. 

We replaced the print with a variable that could keep our response to explore it further
response = requests.get("")
typing in response. and ctrl+space, we could see options of what to do with it and settled with 
At this point, we could see the same text we had seen before in browser, visually verify it just as much as with the browser and were ready to move on. 

Next we started working on the different pieces of the URL, as we wanted to test same things in different environments, and our API had a few more options than this one I use for educational purposes here. 

We pulled out the address into a variable, and the rest of it into another, and concatenated them together. for the call. 
import requests
address = ""
rest_of_it ="us/90210"
whole_thing = address + rest_of_it
response = requests.get(whole_thing)
The API we were playing with had a lot more pieces. With environments, names, id's, dates, limits and their suffixes in the call we had a few more moving parts to pull out with the very same pattern. 

As we were now able to run this for one set of values, our next step was to see it run for another set of values. On our API, we're working on a data-specific bug that ends up giving us a different status code of 500, we wanted to move for the idea of seeing that here. 

Making the status code visible with 
we started our work to have calls of the whole_thing where it wasn't what we started with but had multiple options. 
#rest_of_it ="us/90210"
rest_of_it = "fi/00780"
Every option we would try got documented, but the state of changing one into a comment and another into the one we would was not what we'd settle for. 

We wanted two things: 
  • a method that would take in the parts and form the whole_thing for us
  • a way of saving the results of calls 
We started with keeping a part of the results introducing pytest writing that into requirements.txt as second line. 
Again we clicked an ok adding what our environment was missing as Pycharm pinged us on that, and saved the response code codifying it into an assert. We remembered to try other values to see it fail to trust it in the first place. 
assert response.status_code == 200
Us still wanting the two things above, I interrupted our script creation to move us a step in a different direction. 

Python like a Class and Methods, in Pycharm

We googled for "pytest class example" under my instructions, and after not liking the first glance of the first hits, we ended up on a page:

We copied the example as file contents on our IDE. 

We hit a mutual momentary hiccup, to figure out three things: 
  1. We needed to set pytest as our default test runner from File | Settings | Tools | Python integrated tools | Default test runner. 
  2. The file must have Test in name for it to be recognized as tests
  3. We could run a single test from the green play button next to it
The original example to illustrate setup and teardown had a little bit too much noise, so we cleaned that up before starting to move our script in to the structure.
class TestClass():
def setup_class(self):

def teardown_class(self):

def setup_method(self):

def teardown_method(self):

def test_one(self):
assert True
We moved everything from the script we had created inside test_one() 
def test_one(self):
import requests
address = ""
# rest_of_it ="us/90210"
rest_of_it = "fi/00780"
whole_thing = address + rest_of_it
response = requests.get(whole_thing)
assert response.status_code == 200
And we moved the import from inside the test to beginning of the file to have it available for what we expected to be multiple tests. With every step, we run the tests to see they were still passing. 

Next, I asked the trainee to add a line right after def test_one(self) that would be like we imagined what we'd like to call to get our full address. We ended up with
define_address("foo", "bar")
representing us giving two pieces of text that would end up forming the changing parts of the address. 

A little red bulb emerged on the IDE next to our unimplemented method (interjecting TDD here!) and we selected Define function from the little menu of options on the light bulb. IDE created us a method frame.
def define_address(param, param1):
We had already been through the idea of Refactor | Rename coming up with even worse names and following the "let's rename every time we know a name that is better than what we have now" principle. I wouldn't allow just typing in a new name, but always go through Refactor to teach the discipline that would be benefiting from the tooling. Similarly, I would advice against typing whole words but allowing IDE to complete what it can. 

We moved the piece of concatenating two parts together into the method (ours had a little more parts than the example). 
def define_address(part1, part2):
whole_thing = part1 + part2
return whole_thing
and were left with a test case where we had to call the method with relevant parts of the address
def test_one(self):
# rest_of_it ="us/90210"
response = requests.get(define_address("", "fi/00780"))
assert response.status_code == 200
The second test we'd want as comment in the first became obvious, and we created a second test. 
def test_two(self):
response = requests.get(define_address("", "us/90210"))
assert response.status_code == 200
Verifying that response.text

Now that we had established the idea of test cases in a test class and structure of a class over writing just a script with a hint of TDD, we moved our attention to saving results of the calls we were making. Seeing "200 success" isn't quite what we'd look for. 

In the final step of the day, we introduced approvaltests into requirements.txt file.
We edited two line of our file, adding
from approvaltests.approvals import verify
and changing print to verify
We run the tests from terminal once to see them fail (as we saw them be ignored without this step on the usual run) 
pytest --approvaltests-use-reporter='PythonNative'
We saw a file TestClass.test_one.received.txt emerge in our files, and after visually verifying it captured what we had seen printed before, we renamed the file as TestClass.test_one.approved.txt. We run the tests again from the IDE to now see them pass, edited the approved-file to see it fail and corrected it back to verifying our results match. 

As finalization of the day, we added verification on our second test, again visually verifying and keeping the approved file around. 
def test_one(self):
response = requests.get(define_address("", "fi/00780"))
assert response.status_code == 200
And finally, we defined approvaltests_config.json file to include information where the files approvaltests create should go
"subdirectory": "approved_files"
These steps give us what we could do in a browser, and allow us to explore. They also help us save results for future with minimal effort, and introduce a baseline from which we can reuse things we've created. 

Looking forward to see what our testing takes us to next with the trainee.