Ideation and oracles - where AI does better than testers

As I was preparing for my PNSQC keynote 'Better together: Humans, AI and the next quality benchmark', I went back on my social media timelines on the public notes I had taken, and two stood out in particular.

The first one was note on research item in 2018 that was making rounds, that I had hard time digesting then.

The second one was few years later, when AI was new and I first time saw it doing better than my colleagues, in ideation for testing.

20.11.2023 "In this particular case, chatGPT did better than colleagues. People are so afraid to say what they think and make decisions."

I have been slowly coming to terms with two things:

1. Testers are not as good at testing as they think they are. Very few testers are really excellent at finding meaningful (or sufficient) results and targeting the testing they do for risk and architecture.

2. AI does much better now at testing than we would like to give it credit for.

However, for these two things, testing might not fit your common plan - specify - execute -frame, but would benefit from a different model to how the work is distributed. The frame we commonly recognize as exploratory testing.

Better, you say?

Today I wanted to write about two very specific ways of better I have been observing recently:

1. AI-enabled identification of properties, and use of properties effectively with extended automation capabilities of AI

2. AI-enabled ideation of what to consider testing, and use of AI for multi-perspective classification to choose most relevant ideas

Properties

To understand the fundamental shift in testing that AI is now enabling, we need to go back to 2012 and research by Cem Kaner. He had then been teaching BBST courses for a large number of students of testing, and observed that students at large numbers failed at a particular, very relevant exercise: identifying partial oracles that could be used for purposes of automation. The concept he was trying to teach would become essential for assessing systems in scale.

Kaner described the oracle problem as "one of the defining problems of software testing: one of the reasons that skilled testing is a complex cognitive activity rather than a routine activity". He pointed out that a popular notion of heuristic oracles of consistencies don't work: "consistencies don’t guide testers (not me, not the testers I know, not the students I teach) toward oracle ideas that are specific enough to be programmed and used by an automaton.

This is where we left off with BBST. While the course materials were redesigned to teach partial programmatic oracles, testers at large were particularly bad with these. Turns out generative AI is better at this.

The years brought me awareness of developer community solving this problem of designing programmatic oracles. They just used an entirely different word for it: properties. Properties were things that would be true to evaluate against for all or a signicant group of tests, so that you wouldn't have to hand-craft each example and assert - the approach the tester automation community at large adopted. The property-based testing tools, however, stayed in sidelines, and evolved further particularly in the Haskell community as Quickcheck. It has seen mainstream only in recent years, first by applying it in the much larger Python community as Hypothesis, then on APIs as Schemathesis and now language-agnostic for webUIs with Bombadil. The relevant piece of my puzzle however came with Antithesis property research skill, that does what students at Cem Kaner's course never learned to do: identify properties.

The Bombadil tool includes four default properties

• No HTTP error codes
• No uncaught exceptions
• No unhandled promise rejections
• No console errors

But the Antithesis research skill is capable of identifying domain-specific properties: statements that should hold true when we use them in our tests, beyond "it does not fail visibly or silently", as the defaults state.

The amount of relevant problems I have been able to identify that I was oblivious to since I caught up with this blows my mind. AI changed testing, for me.

Ideation

The other piece I wanted to offer as essentially different is ideation. At PNSQC last week, I participated a workshop facilitated by Ben Simo on "Collaborative Ideation". While generating ideas on an imagined chocolate wrapping machine might not be my favorite passtime, the steps through the exercise made it painfully obvious that humans are not a match for generating ideas in excess. Humans, however, do really well on asking critical questions about the generated ideas, and modeling dimensions of relevance. In the end, when we were asked to prioritize our ideas for the same target of testing for 10 essentially different project contexts, the overgeneration, classification and filtering approach that AI made straightforward really made sense.

It was not that you would ask AI to give you a perfect list. But you would ask for a list that you would classify towards your critical ideas of better towards perfect, allowing for learning to change the priorities.

You know what they say:

Do your testing so that at any time you run out of time, you have done the best testing you possibly can by that time.

The amount of perspectives to classify and consider that we could work through collaboratively in four hour workshop really solidified the experience: AI changed testing, for me.

Filling the Gaps

I am one of those people who can easily write something in an hour, except if I see that in my calendar the next meeting is in an hour. I've considered it one of my quirks to live with, and I compensate by doing my focused work very often outside office hours. With the flexibility of work, it does not matter when I do the lighter bits I wanted to do in my day like catching up on the socials and being available for people who often could use my attention, and that I might do some intensive work at weird hours.

Other people have more strict lines on work vs. free time. My thinking around work is more on the question on am I worth my salary this week than the hours I am available. I have weeks when I absolutely feel like the platform I have built in the past has to do the value creation for me. But mostly I find it possible to show where my value shows up.

In the last week, a few things happened that brought me to blog and speculate on this:

1. I realized that I favor spending my building time with AI over people, because people make me feel like they don't understand me. That was not a great realization that I have started to favor two forms of connections: 1:1 with AI, and broadcast with people. The more I let the distance in co-creation grow, the less I want to re-engage to bring others along.

2. This individual optimization of my time grows the gap between me and my colleagues. The gap is not a positive. The gap shows up in personal productivity and is not in the best interest of the organizations I serve.

I also made notes on bias in relationships. If I prefer talking to AI, and I prefer keeping my boss in the loop, who do I leave out? How I should design the communications? The more people belong to the team, the more AI in these relationships biases the way our communication works. And the way we use it currently, as individuals in isolation, how it grows the distance.

Then today I read an article in Finnish that made a few fairly interesting points that I illustrate with my own ideas around them:

• AI Pioneers may have made significantly different choices on where they invest the time saved up by use of these tools.

  • while my choice may be to invest it in learning more, the article dubbed an idea of "stealing from the boss with AI", meaning the personal productivity used in frame of was I worth my salary in a shared frame comparing to non-pioneers may also lead to the conclusion of more private time.

• Shared experiences build shared requirements. The more individual our experiences become with remote work and favoring AI as a collaborator, the more individualistic the requirements become. This impacts demand, and motivation for systems development. The personal productity differences are increasing the gap in experiences.

• When pioneers achieve a 4-day work week alone, they may not be inclined to bring everyone else along for the benefit. Individual rankings become the norm, and the wishes towards the great democratization of software development with AI are less achievable.

• Motivation and IT skills on individual level rather than company structures drive most of the benefits. Rediscovering individual workflows for each individual is the current state and individualism may keep us there.

Why blog about this?

I remember the times when builds were broken every day, and we could not test. We turned the wait time to improvement time. When we got away from the broken builds with CI/CD systems, we needed to be intentional for collaborative improvement time.

AI use, and the individual productivity gap in particular, is calling for making space for that intentionality for collaborative improvement time. We need targets that keep most of us along on the ride, rather than letting the individual productivity gap grow.