A Seasoned Tester's Crystal Ball
This blog is about thinking of things past, present and future in testing. As much as I'd like to see clearly, my crystal ball is quite dim. Learning is essential and this is my tool for that. A sister blog in Finnish: http://testauskirja.blogspot.com
Saturday, January 3, 2026
The Words are a Trap
Wednesday, December 31, 2025
Routines of Reflection 2025
As I woke up to a vacation day 31.12.2025, a thought remained from sleep: I would need to rethink the strategies of how I use my time, and how I make my choices for the next year. I was trying to make sense into the year we are about to leave behind, and I knew that if there was a word I would use to describe it, it would most likely be consistent effort. On holidays and weekends, the consistent effort was into reading and I have been through more books in a year than I have read in the last ten combined (fiction, 51 titles on Kindle finished in 2025 and 73 in 2024, starting on the week I turned 50). On work, it was whatever was the theme of the week / month / quarter and I had adjusted direction learning so much throughout the year.
While efforts feel high and recognizable, I am not convinced with the strategies behind those efforts, and particularly the impact that I am experiencing or even aspiring. I am, after all, in a lovely unique career position where I have a lot of power over choices we make on testing, in an organization where I have a lot of learning to do on how to work on power with people, and particularly power with other organizations. Consulting, and my role in the AI-enhanced application testing transformation force every day to be one full of learning.
Describing the effort
As consultants, we track our hours used, leaving me with data of my year at work.
So know that I used 7% of my annual work hours on receiving visible training. This included:
- Participating in conferences I did not speak at: Agile Tampere, Oliopäivät Tampere, Krogerus Data Symposium
- Classroom training on Sales (did not like this), Delivery framework (liked this), start of Growth training (loving this).
- Ensemble learning for ISTQB Advanced Test Automation -certification and completion of full set of four Advanced certificates.
- Ensemble learning for CPACC Accessibility certification and completion of the certification, with start of accessibility advocacy that comes with holding the certificate without exam every five year.
- AI, particularly Agents in GitHub Copilot for non-automation use cases
- Python, teaching a 8-piece series of python for testing at work instead of complaining for same amount of time that some people did not know the basics - they do now.
- Contemporary exploratory testing, seeing versatile problems in target applications and combining automation into it
- Level of testing skill
- The controls at scale organization, allocation and targets
- Sense of agency with understanding of impacts
I'd like to think that some of the testing advice or inspirations I have provided this year have impacts that I only learn later on. Kind of like receiving a message this year from two people I worked with 10 years ago, one telling that I still impact their career on a regular basis due to the timing of when our professional paths crossed, and another telling me their organization has now better diversity mechanisms because our time together was one where I invested effort into letting people know I am not "guys" and that I would risk personal negative consequences for working for social justice.
So with all the reflection, I leave a call for myself and the community around me on finding out ways of fixing challenge 1 - skill of testing. While I have a sense of need of personal contribution in that space, I also know that the only way we solve problems in scale is democratization of knowledge and working together. So that is up next, going for 2026.
Closing off
I still think my reflection wins over what social-media-based AI tools can do. Top quote is my challenge 1.
Monday, December 15, 2025
Participant skills to retrospectives
I'm an avid admirer of retrospectives and the sentiment that if there was one agile practice we implement, let it be retrospectives. We need to think about how we are doing on a cadence, and we need to do it so that we ourselves get to enjoy the improvements. Thus retrospectives, not postmortems. Because let's face it: even if I learned today that we have a lessons learned from past projects database, for me to apply other people's lessons learned, it's likely that no amount of documentation on their context is sufficient to pass me the information. Retrospectives maintain a social context in which we are asked to learn.
Last week I argued that best of retros I have been in were due to great participants with an average (if any) facilitator. My experience spoken out loud resulted in a colleague writing a while blog post on facilitators skills: https://www.linkedin.com/pulse/retrospectives-why-facilitators-skills-matter-more-than-spiik-mvp--fmduf/?trackingId=yc302cSWzR0ZhHTSNoyVmQ%3D%3D
Success with retrospectives appears to be less of an issue with roles and responsibilities, than of a learning culture. When in a team where each gig is short and you could be out any moment (consulting!), it takes courage to open up about something that could be improved. The facilitator does not make it safe. The culture makes it safe. And facilitator, while they may hold space for that safety and point out lack of it, anyone can.
When someone speaks too much, we think it's the facilitator skills that matter in balancing the voices. Balancing could come from anyone in the team. Assuming the facilitator notices all things feels like too much on a single person.
Building cultures where work does not rely on a dedicated role is kind of what I'd like to see. Rotating the role on the way to such state tends to be a better version that having someone consistently run retros.
Having facilitation skills correlates with having participation skills too. At least it changes the dynamic from passive participant already afraid to express their ideas to an active contributor.
Friday, November 28, 2025
Observations of a habit transformation
A month ago, I gave a colleague an assignment. They were to create typescript playwright automation using Github Copilot and Playwright Agents. While making progress on the tests was important, learning to use agents to support with that work was just as important.
We had a scope for a test, which was one particular scenario previously created with a recording style automation tool. Recording took usually an hour, but there was no fixing the script. Whenever it would fail, a rerecording was the chosen form of maintenance. No one knew anymore if the thing that was recorded now matched what was recorded when the test was originally imagined. The format of the recording was an xml pudding where pulling out things to change took more effort than anyone had been willing to invest.
Halfway through the month, I checked with how the work was progressing to learn that it had seemed easier to work without agents due to familiarity. With a bit of direction that was no longer an option for continuing.
Three days before the deadline, I checked with how the work was progressing to learn the scope of the test had been forgotten and something new and shiny was being tested, mostly for playing with the Playwright Agents. With a bit of direction the scope was done by the review meeting.
Yes, I know I should be checking in more frequently. That option however was not a possibility.
Looking at what got done, I learned a few things though.
I learned that 134 LOC was added into 8 functions.
I learned three new significant capabilities (env configuration, data separation and parametrization, and fixtures) were added, and the scope of what the intended design of the original test had been had been captured.
I learned that making test reliable by adding verify for waiting to be at right place before proceeding had taken significant amount of work.
I learned that one type of element was never seen by the Playwright Record tool, and that required handcrafting the appropriate locators.
I learned that using agents comes with more context that I had not fully managed to pass on. If your agents out of the box are called planner, generator and healer, the idea that you might want to skip the planner or even write your own just following the existing as examples was not straightforward.
Seeing this unfold in hindsight from the pull requests, I modeled the process of how it was built.
First things were either recorded or prompted out after AI. Recording was clearly the preferred, controllable way of starting.
Then things were made work by adding things recording did not capture.
Then a lot of work was done on structure and naming.
There was a few iterations of making it work and making it pretty.
So I compared notes with some of the other assignments like this that I have given to people.
There were five essentially different ideas of how work like this would get done.Tuesday, November 25, 2025
Platform Product Testing
In the lovely world of consulting, I find myself classifying the types of engagements I work with. While Product / System Testing holds a special place in my heart with the close collaboration with teams building the product together, I come often to a table where the Product is platform product, the building is configuring and specially constrained programming, and the purpose of the platform product is to enable reuse for similar kinds of IT system needs. I often call this IT Testing.
With IT Testing, a firm belief on previous experience on the platform product at hand runs strong. That makes me particularly fascinated in modeling the similarities and differences, and insisting I can learn to test across platform products. So today I decided to take moment to explain what I have gathered so far on Platform Product Testing for Dynamics 365, SAP S4/Hana, Salesforce and Guidewire. The listing of platform products is more than these four, and I very intentionally excluded some of my lovely friends from work such as Infor and ServiceNow.
What makes testing of a platform product based system different is the experience of inability to tell what is a platform product problem (or feature), what is something you had control over in changing, and what comes from your rules combined with your data. For it to work for the business purpose it was acquired, the culprit does not seem like a priority. If it does not run your business the way your business needs running, there is a problem. Recognizing a problem starts then figuring out what to do with such problems. Acceptance testing with business experts is essential and critical, but also very disruptive to business as usual if it needs repeating regularly.
Since most of the functionality comes from the Platform Product, your integration project costs are usually optimized by focusing their testing on the things your contractor is changing and thus responsible for. This may mean that Acceptance testing sees an integrated end to end system, while other testing has been more isolated. Automation, if it exists, is the customers choice in investing in essentially multivendor feedback where some of the parts are the product that, theoretically, was tested before given to you - just not with your configurations, integrations and data that run your business.
Let's talk a bit about the platform products.
Dynamics 365, Power Platform, is a set of Microsoft Platform Products giving you ERP and CRM types of functionalities with lots of low-code promises.
Salesforce is primarily CRM-types of functionalities, and it's a cloud-based multi-tenant platform.
SAP S/4HANA is with ERP-types of functionalities and enough history so that the new and old mix.
Guidewire is insurance-focused platform product.
My curiosity with these started with noting vocabulary. A thing we know well in testing is a concept of a test environment. They come in two forms: long-running (production-like) and ephemeral. For Salesforce the environments are called sandbox and scratch org. For SAP matching concepts to get are testing environments, development environments and the supporting tooling of transport/ChaRM. For Dynamics 365 we talk about solution packages and expect understanding of containers. And for Guidewire we talk of bundles, patches and upgrades. While I recognize the dynamics of how things work in each, I get corrected a lot on use of wrong words.
Each of these lovely platform products comes with its own programming language. Salesforce gives us Apex. Guidewire introduces us to Gosu. Dynamics drives us to low code power apps components configurations. SAP gives us ABAB and configurations. For someone who holds dear the belief that sufficiently complex configuration is programming, I find these just fascinating.
My highlights so far are:
Dynamics 365
- Got to love containers as an approach and Azure DevOps makes this feel to me more like modern product development for deployment tooling side.
- UI automation requires understanding of user-specific settings, and I hear UI can be fragile. Locators for test automation aren't straightforward.
- Test from APIs and a bit from end to end in UI.
- Pay attention to solution layering discipline, and automate deployment and data seeding.
- Get started:
- store solutions in source control, build import/export pipelines via Power Platform Build Tools, prefer API tests and small UI smoke suites
SAP S/4HANA
- Automated change impact analysis relying on the structure of Transports/ChaRM is kind of cool given your tools of test management match the support. It is also generally not optional and trying other things can be trouble.
- Config changes have impacts across modules and business process chain testing is essential
- Get started:
- map transports to test suites, automate test runs on transport promotion, use S/4 HANA test automation tool where available and treat integration flows as first-class tests.
Salesforce
- Multi-tenant means quota limits. Stay within the limits. Testing too big is trouble.
- CI/CD and scratch orgs allow for lovely change-based test automation practice. Use mocks for integrations.
- Smart scoping of data for test purposes helps, plan for data subsetting and refresh cadence.
- Locators for test automation can be next level difficult for Shadow DOM and dynamic components.
- Get started:
- enforce Apex test coverage, minimize data creation in tests, use scratch orgs + CI for PR validation, monitor governor limits during pipeline runs.
Guidewire
- Product model–driven testing: insurance product model serves as testing anchor
- Collected open source toolset as 'Guidewire Test Framework' and enforced rules around sufficient use of them guide the ecosystem towards good practices like test automation and coverage.
- Limitations on some contracts on use of AI can significantly limit hopes for use of AI
- Get started:
- create a policy lifecycle regression pack; adopt Guidewire Testing Framework; run regression against each product model drop; negotiate test environment refresh cadence with vendor/ops.
For all reuse of test artifacts across clients is theoretically possible. For all, test data management is necessary but execution of its practice differs: Salesforce and D365 drive synthetic data and subsetting approaches, and SAP and Guidewire require larger production-like data sets; fast refresh capability and data masking is universal. All come with a CI/CD pipeline but each have a platform specific recommended one: Salesforce DX for Salesforce, Power Platform build tools + Azure Devops for Dynamics365, transport/ChaRM automation for SAP.
Universal truths, maybe:
- Need for strong regression testing due to vendor-driven releases
- Presence of custom code + configuration layer you must retest
- Requirement for representative test data
- Complexity of cross-module / cross-app business processes
- Integration-heavy test design (APIs, services, middleware)
- Organizational constraints around AI-generated artifacts
- Upgrade regression risk as a consistent pain point
Decided this could be helpful - Platform testing capabilities and constraints comparison. At least it helps me with learning coverage as I venture further into these.
| Capability / Constraint | Guidewire | Salesforce | Dynamics 365 / Power Platform | SAP (S/4HANA) |
|---|---|---|---|---|
| Metadata-driven development | (✔️) Config layers, product model | ✔️ Core concept | ✔️ Solutions + Dataverse | (✔️) Mostly configuration, less metadata-portable |
| Proprietary programming language | ✔️ Gosu | ✔️ Apex | — (PowerFx only for Canvas, but not core platform) | ✔️ ABAP |
| Strict platform resource limits | (✔️) Some internal limits | ✔️ Governor limits | (✔️) API limits & throttling | (✔️) Performance constraints by module |
| Vendor-controlled releases with required regression | ✔️ Product model upgrades & patches | ✔️ Seasonal releases | ✔️ Wave updates | ✔️ Transport-based releases & upgrade cycles |
| Automated test impact analysis supported | — | (✔️) Through metadata diffs + DX | (✔️) Via solution diffs & pipelines | ✔️ Transport-level impact analysis |
| Native test automation tooling | ✔️ Guidewire Testing Framework | (✔️) Apex tests + UI Test Builder (limited) | (✔️) EasyRepro / Playwright guidance but not “native” | ✔️ SAP S/4HANA Test Automation Tool |
| UI layer highly changeable / automation fragile | (✔️) Angular-based UI, moderate | ✔️ Lightning DOM changes often | ✔️ Model-driven apps update frequently | (✔️) Fiori stable but customizable |
| Complex cross-module business processes | ✔️ Policy ↔ Billing ↔ Claims | (✔️) Depends on org complexity | (✔️) Depends on app footprint | ✔️ Core ERP complexity across modules |
| Strong CI/CD support from vendor | (✔️) Limited compared to others | ✔️ Salesforce DX | ✔️ Azure DevOps + Build Tools | (✔️) SAP CI/CD + ChaRM |
| Easy ephemeral environment creation | — | ✔️ Scratch orgs | ✔️ Dev/Test environments via Admin Center | — (environments heavy, transports rely on fixed landscapes) |
| Heavy dependency on realistic test data | ✔️ Policy & claims data | (✔️) For integration flows | (✔️) For model-driven logic | ✔️ Mandatory for end-to-end flows |
| Contractual constraints on AI-generated code/config | (✔️) Vendor & client contracts commonly restrictive | (✔️) Org policies vary | (✔️) Varies by tenant/governance | (✔️) Strong compliance usually restricts |
| Complex upgrade regression risk | ✔️ High | (✔️) Medium | (✔️) Medium | ✔️ Very high |
| Platform-driven integration patterns (APIs, services) | ✔️ SOAP/REST internal services | ✔️ REST/Bulk/Messaging | ✔️ Dataverse APIs + Azure | ✔️ BAPIs/IDocs/OData |
| Stable API layer for automation | (✔️) Good internal APIs | ✔️ Strong API surface | ✔️ Dataverse APIs stable | ✔️ Strong API layer but complex |
Out of all the difference, this one is the most defining: ephemeral test environments: Salesforce, Dynamics 365 AND vendor-native automation tooling: SAP, Guidewire.
Habit of AI in Quality Engineering
It was June 2024, and I was preparing to meet a journalist for an interview due to my recent appointment as Director for AI-Driven Application Testing at CGI Finland. The journalist drove the interview with two requests:
- Demo and show!
- What is our approach at CGI on this
- CGI DigiShore is an AI solution for modernizing legacy applications. I found a lot of value in generating artifacts to understand yesterday's code for testing purposes, and building concepts towards DigiShore Coverage for Testing.
- CGI AppFactory is a delivery concept where we optimize delivery teams of humans and agents. While the official messaging might not so say, I learned from discussions with my fellow developers that regardless of titles, the delivery mode has a foundation in exploratory testing and understanding how we would continuously explore while documenting tests with modern automation.
- CGI Navi is an artifact generator that can be run in hosted mode when all your data should not leave and ship to someone further away. With it I learned more on contractual trust relationships between organizations, and driving genAI use forward in practice even when that trust for some sets of data cannot be in place.
- Resultful testing where the level of practice needs to be better than what we experience in scale
- Scaling habits by democratizing knowledge, and allowing time to learn in layers
- Open information while working towards business incentives that allow for sustainable work
- Learning by doing, learning by teaching
- Scaling by practices and tools
- Better quality signals with metrics
Friday, November 7, 2025
Dual-purpose Task Design
Dual-purpose thinking is big in the world right now. Dual-purpose technologies are innovations with application for both civilian and military uses, such as AI, drones and cybersecurity perspectives with any software we rely on. A few mentions of these in a conference I was listening in to, and my mind was building connections to using dual-purpose to explain exploratory testing.
In the last weeks, I have found myself explaining to my fellow consultants that our CVs do work for us when we sleep or work on something else. I started explaining this to help people see why they should update their CVs, and what I see in the background processes. We get requests to let our customers know the scale of certain experiences, and we collect that information from the CV collections. Those CVs have a dual purpose. They serve as your personal entry to introduction with interesting gigs. But they also serve as a corporate entry to things that are bigger than the individual. Usually without us having to actively work on them.
Today, as I was ensemble testing a message passed through an API and processed through stages, I realized that dual-use thinking is something core to how I think about exploratory testing too. I design my testing tasks so that instead of them being separate, they are dual use. I optimize time by actively seeing the overlap. And it saves me a ton of trouble we were facing today.
I described this in a post on LinkedIn:
An ensemble testing session today showed the difference between manual testing, automated testing and contemporary exploratory testing in a fairly simple and concrete way.
Imagine you are testing with a message to API, with the three approaches.
Manual testing is when you run the message through postman. You get confused on what values you changed last because you did not name your tests for whatever you tried just now, there was no version control and information on what value X means is held in your head. The baseline versions of these you carefully document in Jira Xray, but not with the message. That you create manually from templates whenever you need it.
Automated testing is when you spend significant time in creating the message with whatever keywords you have, and then verifying the right contents with whatever keywords you have. Because of the mess of keywords and rules around how carefully these must be crafted to qualify, from a combo of values in a message to automated testing, there's quite a distance.
Because the first is fast and the second is slow, you carefully keep the two of them separate by having separate tasks. Maybe even separate people, but even when they are both on you, intertwining the two is not an option.
I push for the third option that I keep calling contemporary exploratory testing. Edit your messages in version controlled files in VSCode. Send them over with simple or complicated key keywords. The main point is you can leave your new combinations behind in version control as you discover them. Structure them towards whatever has all the right checks in place when the time is right, but build ways of structuring and naming that help document the concepts your inputs and checks have. See that you are building something while exploring that helps you refresh your results.
This all summed up as a picture with texts on the ideas I was working through. I seek ideas that are beyond what I see in day to day, hands-on testing with people capable of "manual testing" and "test automation". I seek Contemporary Exploratory Testing with Programmatic Tests.







