Capture the Bugs - Feedback on learning how to test

You may have run into a popular learning tool in the security space, Capture the flag. These are essentially instrumented unsecure applications, where as you test, you get score for insighs and hints on what else could you try out as attack vectors.

Functional testing includes security testing, but in addition includes a whole lot of perspectives that may not have security implications. With that in mind, I paired with Ru Cindrea to co-create self-service version on a learning style I have used in classroom settings for decade(s) - learning testing by testing, and getting feedback founded on someone else going before you.

In this post, I want to provide you a link and current state description on work that will evolve further but would already be useful for your self-study, and talk about the background leading up to this.

Capture the Bug Current State

You can find the application at https://exploratory-testing-academy.github.io/capture-the-bugs/.

It now supports two features for one test target, e-primer:

1. input coverage - you can't find problems if you can't imagine things to try out in an application. If this feels hard to do, we also included a mode where it tells you classes, or even examples.
2. results coverage - you can list things you found, and it tries to compare them with in-browser AI model to a list of 65 known bugs. It tells you what bugs we know of when you think you're done.

I should change the UI to say "We expect you to try 35 kinds of input and report 65 issues we know of. Traditional professional testers find 18% of bugs on other comparable test targets, and learning testing (and using AI) raises the measured bar."

That is it. Test without the answer key, without us watching you test for now. We'll eventually add a choice of reporting your results to you, but it's not there now. You get feedback on input and results coverage. For input coverage, you can choose to update the level of guidance it provides you - from usual testing with your sources of imagination, to seeing classes we had in mind without the specific examples, to specific examples to make sense of how inputs are different. Results are evaluated after you call done, with an in-browser AI component trying to match your style of reporting to our style of baselining expectations.

We would love if you reached out in socials if you try this out, or if this style of learning is interesting to you. Maaret prefers DMs in linkedin or mastodon.

Backstory

Capture the bugs is really a step forward in a decade(s) long teaching of exploratory testing, learning to test by testing. Different test targets require different techniques, and this particular test target is heavy on inputs, while it has a lot of non-input related issues you could raise too. When hundreds of testers have tested this with me in job interviews, online pairing sessions and classrooms, I have built both an answer key to what is expected, and a course to teach you the theory around testing an application such as this. Course is available at https://qe-at-cgi-fi.github.io/material-portal/cetf/index.html for its latest edition.

While being aware of 65 issues on this application worth starting a conversation on, I've been doing benchmarking with other applications. For todo app, the discovery has been that people professionally paid to do testing, being put under the open schedule and required to test and report, find as little as 18% of bugs. This benchmarking has been both a motivation for working towards better, more accessible testing education on resultful testing where you decide actively on acceptable risks, rather than miss out on conversations you could be leading. The very same benchmark has been a sad realization on the testers at large on not delivering on our value proposition on being allowed to focus on quality.

At my 30th year on testing, I am even more focused on scale and legacy, and transforming from someone who loves to do testing to someone who grows more people who love to do testing. Testing is a great profession, but it is not an easy one. It is endless learning, collecting stories of bugs of relevance while walking a minefield of intentionally left behind issues. There is great hope for its future with task expansion, and the idea that to report a bug with AI is to fix it and leave behind test automation that tracks our expectations.

Capturing some more of the learning, one piece of action at a time.

How can our employees find information faster?

For first day post-vacation, two things happened that inspired this post:
1. I took a tool vendor course "OpenAI PartnerU Foundational Knowledge, July 2027"
2. I spent over two hours trying to order a mobile phone for a colleague joining next week

This combination left me with a sense of insight worth blogging about, on failure demand.

In this age of AI (= everything is AI), one of the core questions I noted down for the course I took was the idea of moving from discussing AI to moving to discuss what we do and then considering right tools to enhance it. One of the questions they highlighed was:

How can our employees find information faster?

I had just spent two hours on something that should have been five minutes. I wanted to select a specific phone from a short list of preapproved phones, and order one. But alas, things don't go as I want them to go. There had to be a bug.

The bug was a lovely combination of a new security feature in Chrome / Edge (shared engine for the architecture-aware) and the application handoff implementation between ordering systems of two different companies. The sense of powerlessness with a production bug was overwhelming. Multiple options of who could do something about it, hard to find routes to any and all of them, and a task I was on a schedule for but unable to process. A model example of cost of quality beyond appraisal (testing).

This idea of cost of quality, follows from the idea that quality if free. Someone could have tested and FIXED a bit more so that I would have to struggle a lot less.

I tested for workarounds quite a bit.

• I searched our internal boards for other people with the same kinds of problems to see if they had experienced it, only to discover a plethora of other problems in the same flows but not this
• I updated Chrome, in hopes of it being due to some internal state of not having updated my browser
• I tried restarting the browser, private browsing and more permissive browser settings, none of which helped
• I installed Edge on Mac in hopes of it working on Edge when it did not work on Chrome

And finally, I asked Edge Copilot. The Microsoft AI chat. Read the links about when a few security feature might have been introduced, and took the second option for workaround proposed: I used Firefox, and everything was fine again.

Going back to the question than directed my thinking:

How can our employees find information faster?

I could have asked AI immediately instead of first struggling through the easy options. But really, I should not have been in need of this information. My goal should not be to find this information faster, but to not need this information. I had run into an example of failure demand.

Failure demand is this idea in quality, where lack of it creates work we then consider relevant. It has been one of the most powerful heuristics on my learning journey to realize that we want less, not more of testing. Accepting failure demand is a result of powerlessness in the system. And we need to see the system, not just the work we can contribute.

I leave you with this. Always be asking not just how we can do a thing, but what causes the need of doing that thing. Maybe with the very same investment amount elsewhere (let the devs test more!) our organizations delivery systems are better off.

And this bug - not an easy fix. All I did was adding to the body of knowledge in the internal boards with whoever comes after me.

Appropriate speed for the conditions, deliberately

In 2001, I was working at F-Secure, going through a wide scale agile transformation. I was facilitating testing (and releases) for a product where we first got to weekly releases on a millions of personal computers scale, and was pretty proud on learning the ideas about controlling the leash of change to make scope of testing manageable.

Being pretty proud was founded on the idea of being completely opposed to CI/CD when I first learned about it some years before it. I still send a proverbial wave to a particular Petri from Reactor whose face I see whenever I realize how many things I have had strong opinions, loosely held, and completely wrong.

At that time and age, with that particular CI/CD change, I remember learning some skills that are invaluable now when AI is the thing. I learned to be deliberate about how I use my our time.

Before this whole agile thing, test environments rarely worked. We had this gated model, like a countdown to release, from letting developers build components, and them having quality engineers work with teams to package changed components into products, and test those. You tested, and things didn't work. You waited a lot.

Whenever we were waiting, a group of us got together to discuss how we could improve things. Well, we also used time on rating dark chocolate, a pastime I could not directly enjoy due to being allergic to chocolate, but the atmosphere was relaxed and lovely. I made so many lists of experiences, ideas and cross-pollinated those things both within the organization and outside. And because we were learning so much, we improved a lot too. For years, F-Secure Quality Engineering was known as the source of all relevant testers in Finland.

With CI/CD, the waiting for environments quietly vanished. The natural breaks the environment-related handoffs created for us were gone. And we were more productive than ever. But also, we started stagnating on improvement. With the learning culture we had in place, we called it out, and adapted. We started taking deliberate time for fun; for learning; for cross-pollinating our ideas as a community.

I see people on socials discussing now the sense of overwhelm that AI gives us. And I remember that I learned this early on. You have agency. You can choose the appropriate speed for the conditions at hand, deliberately. Even when someone in the world uses the Ferrari run run full speed on German autobahns, youe appropriate speed is one that is healthy for yourself and your team. If you are learning to drive and your attention is on the gear change, go slow to see if people are crossing.

We have choices to make, and the power to make them. Don't be the person telling yourself otherwise.

Faking plans is easier than fighting plans

Ever entered a project as tester early on into a project? At that time when you still hold the illusion that plans are about to be set and you are on time? And then, only to find out that the only early enough is in the time the project manager gets trained on their ideas.

Well, it's a pretty deflating experience, and unfortunately all too common. It appears there is no early enough without a lot of friction to include testing.

The source of friction is in the mental model. Having Plan - Implement - Test - Release is such a simple illustration. But when you actually overlay testing, it's the other side of everything, and if you wanted real tasks and control over it rather than long-running categories in which you report hours into, you would actually need a more finegrained breakdown of the testing tasks.

Context for an inspiration

I worked on a fixed-price project, where the fixed-price was 50% of real price. These are typical in consulting world, and what they effectively mean is that the project environment is born passively hostile with a continuous fight on showing there is a change request lurking somewhere in the communication of two or more parties.

The fixed price estimation included no test planning, because why would anyone plan from testing perspective separate from project. And the whole plan of testing was described in usual abstract terms of unit, integration, end to end -testing and UAT.

I got to plan testing before the project plan, so I outlined the testing tasks:

• Identifying testing not promised within 8 hours effort allocation, when promise was unit, integration, e2e, UAT and what of it was change to promised scope.

• Deciding on quality practices: reviews, pipelines, timing of implement & test in mutually supportive ways

• Reviews of requirements spec, solution spec

• Testing while implementing, collaboratively, unit and integration testing

• Testing after implementing while integrated with a 3rd party system, minimalized, e2e testing

• Testing while integrated with a 3rd party system, necessary scope, but could be e2e or UAT and thus done from two different budgets

• Staying updated on learnings in the project

While planning for testing and reviewing the documents, I found a bug worth 25 000 € / annually, and facilitated a confirming conversation with the client, and patted myself on the back for planning testing early on to avoid expensive problems down the road.

If it only was that simple. Weeks later, I take a look at the project plan and none of the things I planned for testing got integrated. While appearance of a plan is not the plan underneath the surface, I find it both sad and funny that this is how we still insist on doing project planning. Not with a twist: they would not plan for things with people, or read the things from people, they would have AI in between all of that.

There is a special sense of middle finger that comes with "I did not bother reading the idea of key milestones you were communicating with an image, I rather sent your image to AI and here's a wall of text it had to say about it that made me completely miss the point, with a twist of false claims I have no capability of detecting".

I took my wins of finding the bug, and decided appearance of plan and the plan that will be executed can coexist, because faking it is easier than fighting it.

Lessons from the inspiration

As always, doing the work teaches you a lot.

1. Changing other persons mental model was more possible at time before AI. AI allows for not making an effort in building own mental model.

2. Actual collaboration would win over co-creating and handing off artifacts. Two hours together on a plan wins two hours separately on the very same plan. That removes also the AI workslop in between that is worse than before.

3. Change management politics on fixed price projects remain an antipattern caused by competition tactics.

4. Feeling detached from appearance of plans is necessary for survival. You know, that sphere of influence, the cost of applying influence for change greatly outweighs the cost of faking it.

5. The things that matter like access to version control tool to collaborate on in implementation or pipeline so that tests are not in head of someone don't need to make it into plan to make it into action.

6. Use of AI splits us to two: developers and knowledge workers, and I find myself increasingly frustrated with the hurdles the latter crowd creates in the handoff.

The Numbers and the Stories

Some days, I specialize in testing. Well, that and AI, and both of these two anchors have been quite equally balanced in my day to day work for quite some time. Other days I feel like I specialize in close to obsessive counting, and storytelling.

I wanted to make a not of three numbers and stories today.

One: 30 % improvement in cost of testing services

Nine months ago, I was deep into defining SLAs (promises we keep, with numbers) and metrics (numbers that help us steer), and the work was not always smooth without friction. In addition to the intellectual challenge that setting goals of what we would collaboratively want to see with a client, many of the things I wanted to promise for visibility would require work to build up to.

Some of the motivation for needing a more wide range of perspectives through numbers came from the fact that I was driving a testing service transformation. Cutting 20 % and more from an established level of testing, but also moving it from management and process heavy towards lighter in management layers, and automation-centric.

Even if the frame of changes and the metrics that govern it are my design, the work has involved tens of people. And the numbers today tell us:

We cut down the testing by 30% (>4 FTE) and raised quality. We put through more improvements than before. We don't miss out on the features, not building automation for the changes. We are not done yet, but we are better on the way.

We've well scratched the surface of AI, but the cost savings are structural, not AI. And with the two put together in a learning investment that takes the people along the ride, I am convinced this client will see 50% cost cutting from the traditional and established baseline.
/Users/maaretp/Desktop/Screenshot 2026-06-24 at 6.11.38.png
Can I just anonymously celebrate getting here with a year of follow-up focus, after a design intensive of a few months?

Two: 25 000 € / year bug identified on time

On even more personal note, I have been allowed to volunteer as a tester in another client's project. The tester's work was scheduled a few months down the road in the project, but of course I would, true to any shift left ideals with sense of agency and power, show up early. And I tested the architecture, to find a bug in the design worth 25 000 € / year. While I feel bad for the few weeks of design work we have to redo for not being involved two weeks earlier, I am delighted on the sum of run-time cost savings as well as avoidance of down-stream work I can claim to have contributed on. Future costs are never certain, and no real barriers of doing this existed, because plans of timing my appearance are mere suggestions.

Can I just anonymously celebrate being worth my salary as a tester even against plans and wills?

Three: My people of the month: 154 individuals

Six weeks ago, I was feeling overwhelmed. Maybe the fact that things like One I start and support in scale, and at some point it really does not matter who the client is when there are enough of them. I've come to appreciate from a call I made with a recruiter that I would have to take 25% paycut to go improve testing with them. My unwillingness to do so since I can easily argue I'm worth every penny they pay me, with evidence from years of practicing this, is a fact I live with. That fact includes idea of letting me work on design a while, on steering longer, but creating a lot of ongoing threads. I was overwhelmed, and uncertain if my head could cope with the context switching, and I started to forget people's names.

I vibecoded a people coverage application for personal use. And I have been tracking names of people ever since. I have over the six weeks had actual conversations (they speak to me and I speak back) with 199 distinct individuals. 154 in the last four weeks. I make the worst ever salesperson, because only 19 of these people are clients. Meanwhile, I have had time to discuss things with 20 peers in community, and 110 of my CGI colleagues.

I might not recommend this amount of conversations, but that network of knowing a bit of these people is invaluable. And having that network grows the network, enabling us together to do something I could never do alone.

Just wanted to share you a few stories with numbers. Got any of your own?

Market changes our salaries

There was a job posting for what appeared to be a challenging quality engineering / testing role, and a clearly defined phone time to ask about the role. So I called to ask a few things.

• why was the position 1 year contract rather than continuous - because they wanted a longer trial period to see if the new-to-organization role would be valuable

• would the salary range of the position make it a pay cut if I considered - it indeed would, and I got the understanding that it would be about 25%.

The real reason I called is that it was from an organization that I got to consult with for a few weeks with my lovely current employer allowing me to work with probably half the invoicing per hour to create throughput metrics as my first assignment, and I have hoped to be allowed to focus on that particular organization ever since.

So I had to check. The google AI summary tells me today that "a principal quality engineer earns €6250 - €8750 per month". They were hoping for the lower end of that scale.

Today's phone call was a reminder that have, salary-wise, grown to a scale in which I rarely get to stay around full time for a longer timeframe, but I show up with similar topics in many organizations, driving changes that people hired for the lower end salary don't always get done. It reminded me that for an above expected salary, my last three positions have been tailor-made for me, rather than be ones where I would be able to apply for an open position. I have been lucky to be surrounded with multiple organizations already who have done that for me.

Moving from me to the phenomena this makes me investigate: we are living in a world where a lot of people acquiring attention of talent believe we can now buy that attention for a lot cheaper, and require more on all levels. For me this shows up as desperately seeking testers, but not finding the right kind of testers (who are also developers). It shows up as talent salary levels confusion, that I can't quite wrap my head around every day.

I realized that for this position they fill, they hire someone who proposes hiring me for a few days at a time to help. And I never get to focus on a single client if I want to stick with the higher end of salaries.

Where I work, we are all called consultant, but not all of us really do consulting. And some of us who do, hope we wouldn't need to. But I guess that is the market valuation of paying for a kind of work.

Demand increases when the price point goes down

A colleague at work had a broadband installed for their house this week, making the running joke of the week:

How many people does it take to install a broadband?

The empirically validated number appeared to be 5 distinct individuals showing up at site. For a end user price point of 99 €.

Evidently, if that is a great business, someone else paid most of the cost. It was also a great example of organizing work to be done by as many people as it felt possible, specializing in tasks towards the end result.

The running joke become the brain worm of the week for me, for the parallels that I struggle with in testing. Clients want to get testing done, for whatever their equivalent of 99 € is. If we need to send in separately project manager who allows for testing to happen, a test manager who organizes the frame, the tester who knows what to do in the frame, and the test automator who leaves something behind for repeating, not all these roles can get a professional level salary without someone invisible in the chain subventing the costs. And unlike with broadbands in Finland, we don't have one of those for testing in Finland.

On an individual level, the test lead colleagues who don't test themselves but hold space for testing, creating some of the best environments to test in with positive feedback and continous acknowlegment of the unique challenges of testing, they are worth a lot. On a structural level, clients are increasingly asking not to have to pay as much for management and taking control over the choices by choosing individuals excluding these.

There will always be some test leads, especially for end-to-end systems over multiple systems, teams and even organizations. Yet the reshaping trend, and declining numbers of this specialization are visible at large.

The demand drives the shapes of "testers". The demand comes often attached to low price points. Like this week, a tester deep into the agentic style of testing, with five years of test automation centric test experience, for 50 € / hour. Must live in Finland and speak fluent Finnish. While I would be qualified for the tasks, my price point is somewhat more than double to that, and as much as I would have been a great person for that project, I wasn't ready to cut my salary yet.

Now with AI in the picture, the shapes of our roles are changing even more. We'd rather pay 50 € / hour for someone who is seriously supported by the tool bringing in a new generation of testers that are AI native. The 80 € / hour AI native testers need to find words to describe their value-adding shape.

If the World Economic Forum Future of Work -report says 92 millions jobs are lost to AI by 2030, and 170 million new jobs are generated in the same timeframe, it says the position shapes are changing.

As an individual, I look at three dimensions in building myself up:

• Capability is what I can do, and I can always get better in all dimensions of capability. Some grow with new skills, some by honing existing skills to efficiency of execution. It took me 10 repeats of transforming organizations to continuous releases before I felt I had the foundation of repeating this properly in place.

• Position is what organization expects me to do, and more often than not, we may be more in some dimension than our position is. This is the conversation of "I can do more, shouldn't I be paid more" where you need a different position to be paid more. Organizations tend to set pay for position, not for person. In best cases the two match.

• Aspiration is what you would want to do given the choices. Like the fact that I would like to be a developer even when I am a director, and I get snappy when someone suggests that I need a "developer" to tell how to do a thing I am an expert on. If aspiration and position are misaligned, there is a chance the work of the position doesn't get done, so I have always found it imporant to own my responsibility to a position, while visualizing and working towards my aspiration.

It's now been 10 years since Geoffrey Hinton in 2016 said:

"It's just completely obvious that within five years, deep learning is going to do better than radiologists. ... We should stop training radiologists now." -- Geoffrey Hinton

10 years later, radiologist salaries are up by 40% and total amount of them is up by 15% (in USA). The demand went up. The shapes that gets compared are probably different in the tasks. This is an example of the Jevons paradox, where added efficiency increases demand.

We need to start doing better in talking about value and added efficiency in testing. However, counting "testers" might not work same as radiologists, because the trend last 10 years has shown that "testers" title gets merged with developers. We need to start getting counted as one relevant group.

Some also say that we have same amount of chefs but more kitchens, demand increasing but key productive resource not increasing. With AI through, it might be that task expansion brings in more chefs, and not all chefs need to be winning michelin stars with their cuisine.