Joep Schuurkes (Posts about exploratory testing)

Thinking in threes about software testing

Joep Schuurkes — Sat, 30 May 2026 22:00:00 GMT

In the episode "Thinking magically with Ramsey Dukes" of the "What magic is this?"-podcast, Ramsey Dukes (pen name of Lionel Snell) says some interesting things about thinking in twos as opposed to thinking in threes. (The quotes are from a section starting at 01:03:47 and ending at 01:08:02. Transcription mine.)

Thinking in twos and threes from the perspective of Magic

Thinking in twos, dualistic thinking, Ramsey Dukes calls a trick:

[...] the brain is constantly making distinctions. And, so I suggested that when we have these dualities, God - Devil is what we are sort of brought up with. [...] The good, the bad. Now when that gets polarized. You see other people are always the bad. They're the devil. And I say that if you look closely it's actually a trick. All these things are tricks. [...] There isn't a little gate in the middle with a guard there, saying: "Fill in the form. Are you on this side or are you on that side?" It's a whole spectrum. Everything's connected.

Instead of thinking in twos, Ramsey Dukes recommends thinking in threes:

So I said that if instead, we trained ourselves or encouraged ourselves to think in threes. God, Devil, Trickster.

Where the third element of a duality is not the middle ground:

It's clearly a bug, but what is the bug exactly?

Joep Schuurkes — Sat, 07 Mar 2026 23:00:00 GMT

A couple weeks ago I was updating some of our Playwright tests, when Playwright decided to mess with the execution order of the tests. The tests in question are a group of tests that cover a whole process from start to end. So we run them in serial mode, "to ensure they will always run together and in order". And yet, they were not running in order.

While that's obviously a bug, it wasn't clear what exactly the bug was. Was it the tests not running in the correct order? Or was that only a symptom of some other bug? And what was triggering this bug? Because we've had these tests running in serial mode for a while, but had never seen this behavior before.

The problem as we encountered it

The tests that triggered the issue were similar to these:

Some of the things I did after being off for a few weeks

Joep Schuurkes — Tue, 01 Jul 2025 22:00:00 GMT

I wish people would share more of what they do day-to-day. Especially testers and quality engineers. Show the work you do. On the other hand, I do realize that that's hard. You don't want to share sensitive information. You want to respect the privacy of your team. And then there's the whole practical side. How much of the context do you need to share to have an intelligible story?

Since my current project is open source and my team works in public, I have more options than most. So let's see if this kind of post works out.

Last week was my first week back at work after a few weeks of vacation. So an important part of my week was catching up. Going through emails and chat messages. Talk with my fellow team members. Participate in the usual meetings. Some stuff that didn't make it into this post. More importantly, there are three things I did, that give a good idea of what a significant part of my job looks like.

The nine skills of exploratory testing

Joep Schuurkes — Sat, 14 Dec 2024 23:00:00 GMT

Exploratory testing is a learned skill, as I claimed in my previous post "Being intentional about exploratory testing". In that post I mentioned the importance of two skills: noticing what there is to notice and deciding what to do next. Turns out it's not the first time I mentioned that pair of skills. In a post about how to teach Agile, I quoted John Mason's "Researching Your Own Practice, The Discipline of Noticing":

"All professional development could be described as changes in sensitivity to notice and accumulation of alternative actions to initiate." (p. 147)

That does raise the question if the skills of exploratory testing can't be made a little more specific. After giving it some thought, I came up with seven additional skills, making a total of nine. For some reasons they ended up as questions rather than nouns. I like how that makes this post less of a checklist and more of a tool for self-reflection. Each skill could be its own blog post, so I'm going to focus on one key element of each skill.

Being intentional about exploratory testing

Joep Schuurkes — Fri, 08 Nov 2024 23:00:00 GMT

This is the second post in a (to be) three-part series about my statement "The difference between a test case and a requirement is the moment of discovery."

In the previous post I distinguished test cases that are translated requirements from ones that aren't. This is something I learned from James Lyndsay. As he describes in "Why Exploration has a Place in any Strategy":

Some tests are designed to find risks. They're made on-the-fly and run once. Some are designed to tell us about retained value. They're made once, and run forever after. You need both: they tell you different things.

The tests with a focus on value are based on requirements, on things we know we want, they are prescribed (as in: written before). The tests with a focus on risks are exploratory, they are based on our decisions in the moment, we look for surprises and decide how we feel about those surprises.

One thing I've noticed through the years, is that a lot more exploratory testing is happening than we give credit for. It's hidden, a required but implicit part of the work. We do it, but we're not intentional about it.

Today I want to argue that it pays to be more intentional about exploratory testing. Before I get there, however, I want to explain what exploratory testing is, because there are still plenty of misconceptions going around.

The difference between a test case and a requirement is the moment of discovery

Joep Schuurkes — Sun, 26 May 2024 22:00:00 GMT

There are several straightforward ways to distinguish a test case from a requirement. A test case tells you how to check some kind of thing about the application, a requirement tells you that the application should do some kind of thing. A test case is written by a tester, a requirement by a business analyst. A test case takes the shape of an action and an evaluation of the result, a requirement takes the form of a sentence like "product ABC shall do XYZ."¹

A less straightforward, but more interesting way to distinguish a test case and a requirement, is this:

The difference between a test case and a requirement is the moment of discovery.²

In this post I want to explore the meaning of that statement. In the next post I'll explore how looking at requirements and test cases in this way, can help us to do better testing. So this post will be a bit more philosophical, the next one more practical.

My five favorite testing questions

Joep Schuurkes — Wed, 03 May 2023 06:57:25 GMT

Recently I realized there are a few testing questions I use a lot. They lie at the top of my testing toolbox, so to speak. Together they shape my testing style, making it easier for me to discover certain things, but probably also harder to find other kinds of things. So here are my five favorite testing questions, in no particular order.

What if there are zero, one, many, lots of this thing?

Last year I expressed my surprise on Mastodon how many times I've found bugs by asking the question: "What if there are 0 / 1 / several / lots of this thing?" And if you're working closely enough to the code, you should also ask about "null".

Quite a few people responded to my message. Turns out it's a very common pattern in TDD. And Brian Marick¹ remembered it standing out when he was looking into fixed bugs in the Linux kernel they used in the '80s. Personally I learned it from Elisabeth Hendrickson's "Test Heuristics Cheat Sheet", which found a new home last year at the Ministry of Testing.

Solving Black Box Puzzle 31 with data analysis

Joep Schuurkes — Sun, 28 Apr 2019 11:16:34 GMT

James Lyndsay has created a number of amazing Black Box Puzzles: tiny applications that challenge you to figure out what they do. (You can support him in creating more of these at his Patreon page.) Two of these Puzzles, 29 and 31, not only have a GUI to explore, but also an API.

And that gave me an idea. If you explore these Puzzles through their GUI, you start from the inputs. You try out different inputs in the hope of discovering a pattern in the outputs. And then that pattern feeds back into your exploration.
With an API, however - and because of the nature of Puzzle 31 - it becomes easy to get the outputs for all possible combinations of inputs. Which means you can start your exploration from the outputs instead of the inputs.

Before I tell you how and what I did, three important remarks.
First of all, I will be spoiling the solution to the Puzzle in this blog post. So this is the right moment to go and solve Puzzle 31 for yourself first. Or at least go play a bit with it, so you have an idea what the inputs and outputs are.
Secondly, I had already solved the Puzzle through the GUI a few months ago. So it was more of a "Can I find the solution this way as well?" than a "Can I find the solution?" thing.
Finally, the code and the spreadsheet I created (linked throughout, also available on GitHub here), are not very clean. I thought about tidying them up, but my two reasons for not doing so are (1) laziness; (2) the way they are now gives a more honest picture of what I did.

Three arguments against the verification-validation dichotomy

Joep Schuurkes — Tue, 24 Mar 2015 19:53:24 GMT

Last week while talking with two colleagues, one of them mentioned the verification/validation thing. And I noticed it made me feel uneasy. Because I know well enough what is meant by the distinction, but on a practical level I simply can't relate to it. When I think about what I do as a software tester and how verification versus validation applies to it, nothing happens. Blank mind. Crickets. Tumbleweed. So after giving it some thought, I present you with three arguments against the verification-validation dichotomy.

First of course, we have the obligatory interlude of defining these two terms. A place to start is the Wikipedia page on Software verification and validation. Unfortunately it contains conflicting definitions, so if anyone cares enough, please do fix. Luckily there's also the general Verification and validation page of Wikipedia, which gives us (among others) the tl;dr version of the distinction:

Verification: Are we building the product right?
Validation: Are we building the right product?

Finally there's the ISTQB glossary v2.4 that borrows from ISO 9000:

Verification: Confirmation by examination and through provision of objective evidence that specified requirements have been fulfilled.
Validation: Confirmation by examination and through provision of objective evidence that the requirements for a specific intended use or application have been fulfilled.

Now on to the three arguments.

Test cases, can't do 'm no more

Joep Schuurkes — Sat, 06 Jul 2013 18:19:32 GMT

Continuing the style of my previous blog post...

Some days ago I found myself no longer able to think in test cases while testing. Of course, it's not as if I was using test design techniques to generate test cases one day and woke up the next day to find myself unable to do it anymore. But still, about a week ago I figured I had explored enough to be able to write down the test cases I wanted to execute and found myself staring at a blank page (well ok, empty Excel sheet) feeling alienated from what I was planning to do.

So what do I mean when saying "thinking in test cases". Simply put, you take a piece of functionality, let a test design technique loose on it and there you go: a set of test cases to execute. Combine test design techniques over the different pieces of functionality as required and you're all covered test strategy-wise. Or that's the idea.