As I was preparing for my PNSQC keynote 'Better together: Humans, AI and the next quality benchmark', I went back on my social media timelines on the public notes I had taken, and two stood out in particular.
The first one was note on research item in 2018 that was making rounds, that I had hard time digesting then.
The second one was few years later, when AI was new and I first time saw it doing better than my colleagues, in ideation for testing.
20.11.2023 "In this particular case, chatGPT did better than colleagues. People are so afraid to say what they think and make decisions."
I have been slowly coming to terms with two things:
-
Testers are not as good at testing as they think they are. Very few testers are really excellent at finding meaningful (or sufficient) results and targeting the testing they do for risk and architecture.
-
AI does much better now at testing than we would like to give it credit for.
However, for these two things, testing might not fit your common plan - specify - execute -frame, but would benefit from a different model to how the work is distributed. The frame we commonly recognize as exploratory testing.
Better, you say?
Today I wanted to write about two very specific ways of better I have been observing recently:
-
AI-enabled identification of properties, and use of properties effectively with extended automation capabilities of AI
-
AI-enabled ideation of what to consider testing, and use of AI for multi-perspective classification to choose most relevant ideas
Properties
To understand the fundamental shift in testing that AI is now enabling, we need to go back to 2012 and research by Cem Kaner. He had then been teaching BBST courses for a large number of students of testing, and observed that students at large numbers failed at a particular, very relevant exercise: identifying partial oracles that could be used for purposes of automation. The concept he was trying to teach would become essential for assessing systems in scale.
Kaner described the oracle problem as "one of the defining problems of software testing: one of the reasons that skilled testing is a complex cognitive activity rather than a routine activity". He pointed out that a popular notion of heuristic oracles of consistencies don't work: "consistencies don’t guide testers (not me, not the testers I know, not the students I teach) toward oracle ideas that are specific enough to be programmed and used by an automaton.
This is where we left off with BBST. While the course materials were redesigned to teach partial programmatic oracles, testers at large were particularly bad with these. Turns out generative AI is better at this.
The years brought me awareness of developer community solving this problem of designing programmatic oracles. They just used an entirely different word for it: properties. Properties were things that would be true to evaluate against for all or a signicant group of tests, so that you wouldn't have to hand-craft each example and assert - the approach the tester automation community at large adopted. The property-based testing tools, however, stayed in sidelines, and evolved further particularly in the Haskell community as Quickcheck. It has seen mainstream only in recent years, first by applying it in the much larger Python community as Hypothesis, then on APIs as Schemathesis and now language-agnostic for webUIs with Bombadil. The relevant piece of my puzzle however came with Antithesis property research skill, that does what students at Cem Kaner's course never learned to do: identify properties.
The Bombadil tool includes four default properties
- No HTTP error codes
- No uncaught exceptions
- No unhandled promise rejections
- No console errors
But the Antithesis research skill is capable of identifying domain-specific properties: statements that should hold true when we use them in our tests, beyond "it does not fail visibly or silently", as the defaults state.
The amount of relevant problems I have been able to identify that I was oblivious to since I caught up with this blows my mind. AI changed testing, for me.
Ideation
The other piece I wanted to offer as essentially different is ideation. At PNSQC last week, I participated a workshop facilitated by Ben Simo on "Collaborative Ideation". While generating ideas on an imagined chocolate wrapping machine might not be my favorite passtime, the steps through the exercise made it painfully obvious that humans are not a match for generating ideas in excess. Humans, however, do really well on asking critical questions about the generated ideas, and modeling dimensions of relevance. In the end, when we were asked to prioritize our ideas for the same target of testing for 10 essentially different project contexts, the overgeneration, classification and filtering approach that AI made straightforward really made sense.
It was not that you would ask AI to give you a perfect list. But you would ask for a list that you would classify towards your critical ideas of better towards perfect, allowing for learning to change the priorities.
You know what they say:
Do your testing so that at any time you run out of time, you have done the best testing you possibly can by that time.
The amount of perspectives to classify and consider that we could work through collaboratively in four hour workshop really solidified the experience: AI changed testing, for me.
