Platform Product Testing

In the lovely world of consulting, I find myself classifying the types of engagements I work with. While Product / System Testing holds a special place in my heart with the close collaboration with teams building the product together, I come often to a table where the Product is platform product, the building is configuring and specially constrained programming, and the purpose of the platform product is to enable reuse for similar kinds of IT system needs. I often call this IT Testing. 

With IT Testing, a firm belief on previous experience on the platform product at hand runs strong. That makes me particularly fascinated in modeling the similarities and differences, and insisting I can learn to test across platform products. So today I decided to take moment to explain what I have gathered so far on Platform Product Testing for Dynamics 365, SAP S4/Hana, Salesforce and Guidewire. The listing of platform products is more than these four, and I very intentionally excluded some of my lovely friends from work such as Infor and ServiceNow. 

What makes testing of a platform product based system different is the experience of inability to tell what is a platform product problem (or feature), what is something you had control over in changing, and what comes from your rules combined with your data. For it to work for the business purpose it was acquired, the culprit does not seem like a priority. If it does not run your business the way your business needs running, there is a problem. Recognizing a problem starts then figuring out what to do with such problems. Acceptance testing with business experts is essential and critical, but also very disruptive to business as usual if it needs repeating regularly. 

Since most of the functionality comes from the Platform Product, your integration project costs are usually optimized by focusing their testing on the things your contractor is changing and thus responsible for. This may mean that Acceptance testing sees an integrated end to end system, while other testing has been more isolated. Automation, if it exists, is the customers choice in investing in essentially multivendor feedback where some of the parts are the product that, theoretically, was tested before given to you - just not with your configurations, integrations and data that run your business. 

Let's talk a bit about the platform products. 

Dynamics 365, Power Platform, is a set of Microsoft Platform Products giving you ERP and CRM types of functionalities with lots of low-code promises.

Salesforce is primarily CRM-types of functionalities, and it's a cloud-based multi-tenant platform. 

SAP S/4HANA is with ERP-types of functionalities and enough history so that the new and old mix.

Guidewire is insurance-focused platform product.   

My curiosity with these started with noting vocabulary. A thing we know well in testing is a concept of a test environment. They come in two forms: long-running (production-like) and ephemeral. For Salesforce  the environments are called sandbox and scratch org. For SAP matching concepts to get are testing environments, development environments and the supporting tooling of transport/ChaRM.  For Dynamics 365 we talk about solution packages and expect understanding of containers. And for Guidewire we talk of bundles, patches and upgrades. While I recognize the dynamics of how things work in each, I get corrected a lot on use of wrong words. 

Each of these lovely platform products comes with its own programming language.  Salesforce gives us Apex. Guidewire introduces us to Gosu. Dynamics drives us to low code power apps components configurations. SAP gives us ABAB and configurations. For someone who holds dear the belief that sufficiently complex configuration is programming, I find these just fascinating.

My highlights so far are:

Dynamics 365

• Got to love containers as an approach and Azure DevOps makes this feel to me more like modern product development for deployment tooling side.  
• UI automation requires understanding of user-specific settings, and I hear UI can be fragile. Locators for test automation aren't straightforward. 
• Test from APIs and a bit from end to end in UI. 
• Pay attention to solution layering discipline, and automate deployment and data seeding. 
• Get started:
• store solutions in source control, build import/export pipelines via Power Platform Build Tools, prefer API tests and small UI smoke suites

SAP S/4HANA

• Automated change impact analysis relying on the structure of Transports/ChaRM is kind of cool given your tools of test management match the support. It is also generally not optional and trying other things can be trouble. 
• Config changes have impacts across modules and business process chain testing is essential
• Get started: 
• map transports to test suites, automate test runs on transport promotion, use S/4 HANA test automation tool where available and treat integration flows as first-class tests.

Salesforce

• Multi-tenant means quota limits. Stay within the limits. Testing too big is trouble. 
• CI/CD and scratch orgs allow for lovely change-based test automation practice. Use mocks for integrations. 
• Smart scoping of data for test purposes helps, plan for data subsetting and refresh cadence. 
• Locators for test automation can be next level difficult for Shadow DOM and dynamic components.
• Get started: 
• enforce Apex test coverage, minimize data creation in tests, use scratch orgs + CI for PR validation, monitor governor limits during pipeline runs.

Guidewire

• Product model–driven testing: insurance product model serves as testing anchor
• Collected open source toolset as 'Guidewire Test Framework' and enforced rules around sufficient use of them guide the ecosystem towards good practices like test automation and coverage. 
• Limitations on some contracts on use of AI can significantly limit hopes for use of AI
• Get started: 
• create a policy lifecycle regression pack; adopt Guidewire Testing Framework; run regression against each product model drop; negotiate test environment refresh cadence with vendor/ops.

For all reuse of test artifacts across clients is theoretically possible. For all, test data management is necessary but execution of its practice differs: Salesforce and D365 drive synthetic data and subsetting approaches, and SAP and Guidewire require larger production-like data sets; fast refresh capability and data masking is universal. All come with a CI/CD pipeline but each have a platform specific recommended one: Salesforce DX for Salesforce, Power Platform build tools + Azure Devops for Dynamics365, transport/ChaRM automation for SAP. 

Universal truths, maybe: 

• Need for strong regression testing due to vendor-driven releases
• Presence of custom code + configuration layer you must retest
• Requirement for representative test data
• Complexity of cross-module / cross-app business processes
• Integration-heavy test design (APIs, services, middleware)
• Organizational constraints around AI-generated artifacts
• Upgrade regression risk as a consistent pain point

Decided this could be helpful - Platform testing capabilities and constraints comparison. At least it helps me with learning coverage as I venture further into these. 

Capability / Constraint	Guidewire	Salesforce	Dynamics 365 / Power Platform	SAP (S/4HANA)
Metadata-driven development	(✔️) Config layers, product model	✔️ Core concept	✔️ Solutions + Dataverse	(✔️) Mostly configuration, less metadata-portable
Proprietary programming language	✔️ Gosu	✔️ Apex	— (PowerFx only for Canvas, but not core platform)	✔️ ABAP
Strict platform resource limits	(✔️) Some internal limits	✔️ Governor limits	(✔️) API limits & throttling	(✔️) Performance constraints by module
Vendor-controlled releases with required regression	✔️ Product model upgrades & patches	✔️ Seasonal releases	✔️ Wave updates	✔️ Transport-based releases & upgrade cycles
Automated test impact analysis supported	—	(✔️) Through metadata diffs + DX	(✔️) Via solution diffs & pipelines	✔️ Transport-level impact analysis
Native test automation tooling	✔️ Guidewire Testing Framework	(✔️) Apex tests + UI Test Builder (limited)	(✔️) EasyRepro / Playwright guidance but not “native”	✔️ SAP S/4HANA Test Automation Tool
UI layer highly changeable / automation fragile	(✔️) Angular-based UI, moderate	✔️ Lightning DOM changes often	✔️ Model-driven apps update frequently	(✔️) Fiori stable but customizable
Complex cross-module business processes	✔️ Policy ↔ Billing ↔ Claims	(✔️) Depends on org complexity	(✔️) Depends on app footprint	✔️ Core ERP complexity across modules
Strong CI/CD support from vendor	(✔️) Limited compared to others	✔️ Salesforce DX	✔️ Azure DevOps + Build Tools	(✔️) SAP CI/CD + ChaRM
Easy ephemeral environment creation	—	✔️ Scratch orgs	✔️ Dev/Test environments via Admin Center	— (environments heavy, transports rely on fixed landscapes)
Heavy dependency on realistic test data	✔️ Policy & claims data	(✔️) For integration flows	(✔️) For model-driven logic	✔️ Mandatory for end-to-end flows
Contractual constraints on AI-generated code/config	(✔️) Vendor & client contracts commonly restrictive	(✔️) Org policies vary	(✔️) Varies by tenant/governance	(✔️) Strong compliance usually restricts
Complex upgrade regression risk	✔️ High	(✔️) Medium	(✔️) Medium	✔️ Very high
Platform-driven integration patterns (APIs, services)	✔️ SOAP/REST internal services	✔️ REST/Bulk/Messaging	✔️ Dataverse APIs + Azure	✔️ BAPIs/IDocs/OData
Stable API layer for automation	(✔️) Good internal APIs	✔️ Strong API surface	✔️ Dataverse APIs stable	✔️ Strong API layer but complex

Out of all the difference, this one is the most defining: ephemeral test environments: Salesforce, Dynamics 365 AND vendor-native automation tooling: SAP, Guidewire. 

Habit of AI in Quality Engineering

It was June 2024, and I was preparing to meet a journalist for an interview due to my recent appointment as Director for AI-Driven Application Testing at CGI Finland. The journalist drove the interview with two requests: 

• Demo and show! 
• What is our approach at CGI on this

New to the company, it took me a bit of reflecting on what is our approach and if we have *one* or *many* approaches. A year and a half later is a good time to reflect on the ideas I put together back then, and what my reality ended up looking like. 

For the demos I chose back then, I had three: 

GenAI code reviews and even if I had ended up with production experience, I still follow on a monthly basic the slow start of using genAI code reviews in customer projects. After all, the bottleneck with genAI moved from writing code to reading code, and some reports include numbers such as writing code 2-5x as fast, but code reviews being 90% slower. 

GenAI pair testing and using commonly available genAI tools as external imagination for purposes of exploratory testing. After all, from screenshot to ideas and observations, I already had more help from the tools than from majority of tester colleagues too focused on empirical proofreading of requirements. 

Digital twinning for a test expert was already on version 2 as people at CGI had used my CC-BY materials to experiment with genAI helper that would have my ideas of exploratory testing as context. While I might not need the answer of what would I advise, it was a fun demo of insights of having been building a base for being able to do this for two decades of open materials. 

Copiloting test automation code with polyglot approach of various languages and driving forward relevant efforts towards automating tests. After all, we tend to want deterministic examples we can track rather than regenerated things moving control away from whoever is operating the quality signaling efforts. What made this interesting is the foundation of hands-on experience from 2021 onwards and the roman numerals example where humans outperform the genAI. 

There were things at the company I had not become aware of back then, that all had later an impact on how I would think about building the habits

• CGI DigiShore is an AI solution for modernizing legacy applications. I found a lot of value in generating artifacts to understand yesterday's code for testing purposes, and building concepts towards DigiShore Coverage for Testing. 
• CGI AppFactory is a delivery concept where we optimize delivery teams of humans and agents. While the official messaging might not so say, I learned from discussions with my fellow developers  that regardless of titles, the delivery mode has a foundation in exploratory testing and understanding how we would continuously explore while documenting tests with modern automation. 
• CGI Navi is an artifact generator that can be run in hosted mode when all your data should not leave and ship to someone further away. With it I learned more on contractual trust relationships between organizations, and driving genAI use forward in practice even when that trust for some sets of data cannot be in place. 

I didn't only look at CGI Intellectual Property, but the wider community with our partners, whatever was cooking in the ways AI was making its way to commercial testing tools for test repositories, test automation and test data, and what became available in open source as well as in Finland. 

In hindsight, that 1.5 years was well used in modeling the AI black box for function and structure making comparisons and recognizing uniqueness of approaches easier. It allowed for recognizing the need of a choice to build a stronger habit of AI in quality engineering in practice, and thus now drive forward a significant agent-to-human -ratio for contemporary exploratory testing. 

Looking back at the slides and the reality that unfolded. I decided in June 2024 that my approach towards the change we need in testing would be four steps: 

We have established our understanding with AI we continue to collaborate on observing and integrating the latest in the field. We have recognized the automation transformation needs significant work and is both a foundation for AI in testing and any modernized approaches to software development with worthwhile quality signaling. We have started moving selected public materials to our open source for test sharing (https://github.com/QE-at-CGI-FI) and adjusted my previously used CC-BY to CC-BY-NC-SA, and open yet more restrictive license. This reflect what brought me to CGI: industrialization at scale of resultful exploratory testing I learned to teach over the last three decades. And finally, we have done what we do with our clients and their various stakeholders, often in multivendor teams. 

Building the habit of AI has required deeper understanding of data sensitivity classifications, isolating different sensitivity data, negotiating reasonable sandboxes for use of leading packaged genAI tooling both from a UI and programmatically from APIs, and having those sensitivity considerations leading to a hosted solution. It has driven my personal explorer's agent to human ratio to over 20, and while measuring time it saves isn't feasible, it is driving a real change in how I test with contemporary exploratory testing with a pipeline of task-based agents to capture some knowledge we have previously accepted as tacit. 

Finally, the journalist asked me for the approach. For the approach, I combined all that is dear to me:

• Resultful testing where the level of practice needs to be better than what we experience in scale
• Scaling habits by democratizing knowledge, and allowing time to learn in layers
• Open information while working towards business incentives that allow for sustainable work
• Learning by doing, learning by teaching
• Scaling by practices and tools
• Better quality signals with metrics

The community, both at CGI and at large has been on this journey with me. With a variety of clients more curious than ready for the change of habits, our habits take a continuous balance of awareness of where in the journey we are with different clients. 

If back then I was unsure if there needed to be one way or if this way fits, I now know I work in a community of curious professionals that see cultivation of many routes and integrating for greater benefit something of value. Some corporations are built as community. 

Dual-purpose Task Design

Dual-purpose thinking is big in the world right now. Dual-purpose technologies are innovations with application for both civilian and military uses, such as AI, drones and cybersecurity perspectives with any software we rely on. A few mentions of these in a conference I was listening in to, and my mind was building connections to using dual-purpose to explain exploratory testing. 

In the last weeks, I have found myself explaining to my fellow consultants that our CVs do work for us when we sleep or work on something else. I started explaining this to help people see why they should update their CVs, and what I see in the background processes. We get requests to let our customers know the scale of certain experiences, and we collect that information from the CV collections. Those CVs have a dual purpose. They serve as your personal entry to introduction with interesting gigs. But they also serve as a corporate entry to things that are bigger than the individual. Usually without us having to actively work on them. 

Today, as I was ensemble testing a message passed through an API and processed through stages, I realized that dual-use thinking is something core to how I think about exploratory testing too.  I design my testing tasks so that instead of them being separate, they are dual use. I optimize time by actively seeing the overlap. And it saves me a ton of trouble we were facing today. 

I described this in a post on LinkedIn: 

An ensemble testing session today showed the difference between manual testing, automated testing and contemporary exploratory testing in a fairly simple and concrete way.

Imagine you are testing with a message to API, with the three approaches.

Manual testing is when you run the message through postman. You get confused on what values you changed last because you did not name your tests for whatever you tried just now, there was no version control and information on what value X means is held in your head. The baseline versions of these you carefully document in Jira Xray, but not with the message. That you create manually from templates whenever you need it.

Automated testing is when you spend significant time in creating the message with whatever keywords you have, and then verifying the right contents with whatever keywords you have. Because of the mess of keywords and rules around how carefully these must be crafted to qualify, from a combo of values in a message to automated testing, there's quite a distance.

Because the first is fast and the second is slow, you carefully keep the two of them separate by having separate tasks. Maybe even separate people, but even when they are both on you, intertwining the two is not an option.

I push for the third option that I keep calling contemporary exploratory testing. Edit your messages in version controlled files in VSCode. Send them over with simple or complicated key keywords. The main point is you can leave your new combinations behind in version control as you discover them. Structure them towards whatever has all the right checks in place when the time is right, but build ways of structuring and naming that help document the concepts your inputs and checks have. See that you are building something while exploring that helps you refresh your results.

This all summed up as a picture with texts on the ideas I was working through. I seek ideas that are beyond what I see in day to day, hands-on testing with people capable of "manual testing" and "test automation". I seek Contemporary Exploratory Testing with Programmatic Tests.