Comparing Blue, Yellow, Red, Pink and Purple - How are they Different

As a tester, I have grown a practiced skill of classification. There are a lot of emergent criteria to classify, and with practice, I have added slices of perspectives that allow me to see things. Seeing what is said, I look for things that aren't said. And looking at testing tools, I look for options and differences.

And right now, AI for testing tools are overwhelming in options. With AI, everyone has a tool and discovery of value is harder than ever. I wanted to take a pick at calling out some of the patterns I might be seeing.

The Marketing Layer

Imagine there is a small tech company somewhere far away, and you are a business minded person setting up a product company some 30 years ago. You make a deal of taking their promising tech, and you add a facade layer, product name, a slidedeck and your access to people they didn't have access to. You take Blue (with a contract), you call it Yellow. Over the years you add things to Yellow, and eventually you remove Blue from your Yellow. A valid way of building a product business.

Imagine there are open source tools, and you are a technically savvy person who sees value on grouping them some 30 years ago. Each small piece is available with a license that allows for redistributing with constraints, but they aren't thematically packaged together. You take Red and Pink, and you call it Purple. Over the years, the thematic packing grows stronger in value, you add things that fits to Purple. Purple becames known, and majority of people don't even know there are Red and Pink inside.

Adding that marketing layer is today easier than it was some 30 years ago. And since it is easier, it's done in scale making understanding all the Yellows and Purples and Rainbows hard. Creating products with AI support changed the game.

Control over Long Term Value

Imagine a Finnish public sector company posting out a RFP for products in Accessibility Auditing and Monitoring. And ending up with both Blue and Yellow on their list. If Blue decides they can no longer build and maintain due to lack of financial resources, there is no Yellow. Paying for Yellow when the value is generated into both Yellow and Blue by Blue means you may end up eventially with a facade. Liking the facade of Yellow but needing the value of Blue is a real choice when making the purchase decision.

Architecture-awareness is not optional. For the Purples of the world, in worst case scenario, open source allows for Purple to maintain Pink2, when Pink actually is no longer around. For Yellows of the world, closed source is protected by win-win financial incentives.

If you are used to connecting things in world of accessibility Deque aXe is the Blue. There are a lot of tools that add little the what they make available as open source.

Architecture-aware

We look at a testing tool, and they have all these features. And they all have AI in them, a lot with the promise that you can use your already approved models* meaning the cost of what really is AI is outside the box they are selling. Their product does nothing without buying the other product. So how do we choose when comparing them all is not feasible, and architecture-awareness is needed for any comparisons of relevance beyond "I like this person and want to give them my attention, maybe even some of money".

Repeating this over and over again in being presented tools, I build and grow a classification frame, that allows me to get tool folks to tell me less marketing speak:

What we've built, I'd kept selling the "unique layer" angle, you'd have been right to call it blue-with-files-on-the-side.

Let's briefly touch these perspectives of classification for AI in testing -conversations.

First, I look for where the value add they invest in is:

• Packaging is when they have facade level value and tell their story uniquely
• Surface is when they provide value by having their own user interface they expect you to interact in
• Files is when they drop you things that make AI better for specific slice of work
• Harness is when you have a loop (agent) and some of their own built-in instructions akin to files to drop
• Model is the engine that does the work.

Second, I look at the time perspective of value:

• Constraints are the things that must be true (ideas about data most prominently) for any of the value to be available
• Lifecycle is the idea of what stays around when you no longer pay them money

A lot of the tools are like gym membership. Going to gym is required. And if you no longer pay, anything you built on is no longer available.

Third, I look for their idea of testing

• Task expansion is the idea that in age of AI, I don't want testing that reports bugs. I want testing that fixes the bugs. I don't want dedicated testers providing the service, I want to level up everyone in the teams.

My Yellow, my Choices

I would be hypocritical not calling out that I got my Yellow too. My Yellow is called, for now, CGI Test Intelligence Mesh, and it's files we drop in a particularly consumable way with a change of license mechanism.

If prompt engineering is an individual skill of how to talk to AI, context engineering is how your team and project talks to AI. And in my case, how your team brings in external influences impactfully.

The choices I make come from the belief that we need to distribute the control over long term value.

Capture the Bugs - Feedback on learning how to test

You may have run into a popular learning tool in the security space, Capture the flag. These are essentially instrumented unsecure applications, where as you test, you get score for insighs and hints on what else could you try out as attack vectors.

Functional testing includes security testing, but in addition includes a whole lot of perspectives that may not have security implications. With that in mind, I paired with Ru Cindrea to co-create self-service version on a learning style I have used in classroom settings for decade(s) - learning testing by testing, and getting feedback founded on someone else going before you.

In this post, I want to provide you a link and current state description on work that will evolve further but would already be useful for your self-study, and talk about the background leading up to this.

Capture the Bug Current State

You can find the application at https://exploratory-testing-academy.github.io/capture-the-bugs/.

It now supports two features for one test target, e-primer:

1. input coverage - you can't find problems if you can't imagine things to try out in an application. If this feels hard to do, we also included a mode where it tells you classes, or even examples.
2. results coverage - you can list things you found, and it tries to compare them with in-browser AI model to a list of 65 known bugs. It tells you what bugs we know of when you think you're done.

I should change the UI to say "We expect you to try 35 kinds of input and report 65 issues we know of. Traditional professional testers find 18% of bugs on other comparable test targets, and learning testing (and using AI) raises the measured bar."

That is it. Test without the answer key, without us watching you test for now. We'll eventually add a choice of reporting your results to you, but it's not there now. You get feedback on input and results coverage. For input coverage, you can choose to update the level of guidance it provides you - from usual testing with your sources of imagination, to seeing classes we had in mind without the specific examples, to specific examples to make sense of how inputs are different. Results are evaluated after you call done, with an in-browser AI component trying to match your style of reporting to our style of baselining expectations.

We would love if you reached out in socials if you try this out, or if this style of learning is interesting to you. Maaret prefers DMs in linkedin or mastodon.

Backstory

Capture the bugs is really a step forward in a decade(s) long teaching of exploratory testing, learning to test by testing. Different test targets require different techniques, and this particular test target is heavy on inputs, while it has a lot of non-input related issues you could raise too. When hundreds of testers have tested this with me in job interviews, online pairing sessions and classrooms, I have built both an answer key to what is expected, and a course to teach you the theory around testing an application such as this. Course is available at https://qe-at-cgi-fi.github.io/material-portal/cetf/index.html for its latest edition.

While being aware of 65 issues on this application worth starting a conversation on, I've been doing benchmarking with other applications. For todo app, the discovery has been that people professionally paid to do testing, being put under the open schedule and required to test and report, find as little as 18% of bugs. This benchmarking has been both a motivation for working towards better, more accessible testing education on resultful testing where you decide actively on acceptable risks, rather than miss out on conversations you could be leading. The very same benchmark has been a sad realization on the testers at large on not delivering on our value proposition on being allowed to focus on quality.

At my 30th year on testing, I am even more focused on scale and legacy, and transforming from someone who loves to do testing to someone who grows more people who love to do testing. Testing is a great profession, but it is not an easy one. It is endless learning, collecting stories of bugs of relevance while walking a minefield of intentionally left behind issues. There is great hope for its future with task expansion, and the idea that to report a bug with AI is to fix it and leave behind test automation that tracks our expectations.

Capturing some more of the learning, one piece of action at a time.

How can our employees find information faster?

For first day post-vacation, two things happened that inspired this post:
1. I took a tool vendor course "OpenAI PartnerU Foundational Knowledge, July 2027"
2. I spent over two hours trying to order a mobile phone for a colleague joining next week

This combination left me with a sense of insight worth blogging about, on failure demand.

In this age of AI (= everything is AI), one of the core questions I noted down for the course I took was the idea of moving from discussing AI to moving to discuss what we do and then considering right tools to enhance it. One of the questions they highlighed was:

How can our employees find information faster?

I had just spent two hours on something that should have been five minutes. I wanted to select a specific phone from a short list of preapproved phones, and order one. But alas, things don't go as I want them to go. There had to be a bug.

The bug was a lovely combination of a new security feature in Chrome / Edge (shared engine for the architecture-aware) and the application handoff implementation between ordering systems of two different companies. The sense of powerlessness with a production bug was overwhelming. Multiple options of who could do something about it, hard to find routes to any and all of them, and a task I was on a schedule for but unable to process. A model example of cost of quality beyond appraisal (testing).

This idea of cost of quality, follows from the idea that quality if free. Someone could have tested and FIXED a bit more so that I would have to struggle a lot less.

I tested for workarounds quite a bit.

• I searched our internal boards for other people with the same kinds of problems to see if they had experienced it, only to discover a plethora of other problems in the same flows but not this
• I updated Chrome, in hopes of it being due to some internal state of not having updated my browser
• I tried restarting the browser, private browsing and more permissive browser settings, none of which helped
• I installed Edge on Mac in hopes of it working on Edge when it did not work on Chrome

And finally, I asked Edge Copilot. The Microsoft AI chat. Read the links about when a few security feature might have been introduced, and took the second option for workaround proposed: I used Firefox, and everything was fine again.

Going back to the question than directed my thinking:

How can our employees find information faster?

I could have asked AI immediately instead of first struggling through the easy options. But really, I should not have been in need of this information. My goal should not be to find this information faster, but to not need this information. I had run into an example of failure demand.

Failure demand is this idea in quality, where lack of it creates work we then consider relevant. It has been one of the most powerful heuristics on my learning journey to realize that we want less, not more of testing. Accepting failure demand is a result of powerlessness in the system. And we need to see the system, not just the work we can contribute.

I leave you with this. Always be asking not just how we can do a thing, but what causes the need of doing that thing. Maybe with the very same investment amount elsewhere (let the devs test more!) our organizations delivery systems are better off.

And this bug - not an easy fix. All I did was adding to the body of knowledge in the internal boards with whoever comes after me.