Joep Schuurkes (Posts about software testing)

It's clearly a bug, but what is the bug exactly?

Joep Schuurkes — Sat, 07 Mar 2026 23:00:00 GMT

A couple weeks ago I was updating some of our Playwright tests, when Playwright decided to mess with the execution order of the tests. The tests in question are a group of tests that cover a whole process from start to end. So we run them in serial mode, "to ensure they will always run together and in order". And yet, they were not running in order.

While that's obviously a bug, it wasn't clear what exactly the bug was. Was it the tests not running in the correct order? Or was that only a symptom of some other bug? And what was triggering this bug? Because we've had these tests running in serial mode for a while, but had never seen this behavior before.

The problem as we encountered it

The tests that triggered the issue were similar to these:

Reflections after writing a release advice

Joep Schuurkes — Sat, 07 Feb 2026 23:00:00 GMT

Recently I had to write a release advice. My first thought was: "I have close to two decades of experience in testing. I got this!" My second thought was: "I haven't written a release advice in over ten years, how do I do this again?"

Luckily the first thought reflected reality a lot better than the second one. And after reviews by my team members, my project lead, and our manager, I'm proud of the end result. Even though in some ways the document has ended up more like a release decision, than a release advice. (More on that later.)

Lead with the actual advice

Initially I was going to start the document with a management summary, then an introduction, an overview of the testing that was done, the relevant known issues and remaining risks, then the actual release advice and finally some ideas on how to improve our testing further. Quite soon I realized that didn't make sense.

Read more… (4 min remaining to read)

Testing types - beyond definitions

Joep Schuurkes — Mon, 08 Dec 2025 23:00:00 GMT

Testing types in software testing are a mess. It's not just "regression testing" that can mean more things than there are people in the room. It's true for basically all of them.

Just last week a teammate said "We should also do integration testing." So I asked "What do you mean by integration testing?". And the reply was: "Testing the frontend and backend together using Playwright." Which made me think "I see what you mean, but I would never call that integration testing." and made me reply: "Yeah, we should."

The typical solution to this problem is to start defining the different testing types. To me, that's just one approach of many to make sense of testing types. So far I've come up with an additional five.

We do not all have the same job

Joep Schuurkes — Sat, 15 Nov 2025 23:00:00 GMT

At the first conference I ever attended - or I think it was the first, in any case it was Let's Test in 2012 -, I asked someone "So what kind of applications do you test?" And the reply was: "One of the onboard computers of a Saab fighter plane." While at that time I was the test lead for an enterprise service bus.

We do not all have the same job.

In a recent episode of the Quality Remarks podcast hosted by Keith Klain, Paul Holland told the story of an organization where running the test automation took 13 people two weeks. And no, I'm sorry, they didn't go into any detail about what happened in those two weeks. In my current project our pipeline takes 0 people about 17 minutes to run.

We do not all have the same job.

Some of the things I did after being off for a few weeks

Joep Schuurkes — Tue, 01 Jul 2025 22:00:00 GMT

I wish people would share more of what they do day-to-day. Especially testers and quality engineers. Show the work you do. On the other hand, I do realize that that's hard. You don't want to share sensitive information. You want to respect the privacy of your team. And then there's the whole practical side. How much of the context do you need to share to have an intelligible story?

Since my current project is open source and my team works in public, I have more options than most. So let's see if this kind of post works out.

Last week was my first week back at work after a few weeks of vacation. So an important part of my week was catching up. Going through emails and chat messages. Talk with my fellow team members. Participate in the usual meetings. Some stuff that didn't make it into this post. More importantly, there are three things I did, that give a good idea of what a significant part of my job looks like.

The difference between a test case and a requirement is the moment of discovery

Joep Schuurkes — Sun, 26 May 2024 22:00:00 GMT

There are several straightforward ways to distinguish a test case from a requirement. A test case tells you how to check some kind of thing about the application, a requirement tells you that the application should do some kind of thing. A test case is written by a tester, a requirement by a business analyst. A test case takes the shape of an action and an evaluation of the result, a requirement takes the form of a sentence like "product ABC shall do XYZ."¹

A less straightforward, but more interesting way to distinguish a test case and a requirement, is this:

The difference between a test case and a requirement is the moment of discovery.²

In this post I want to explore the meaning of that statement. In the next post I'll explore how looking at requirements and test cases in this way, can help us to do better testing. So this post will be a bit more philosophical, the next one more practical.

Read more… (4 min remaining to read)

So you want to become a test engineer?

Joep Schuurkes — Sun, 25 Feb 2024 23:00:00 GMT

Becoming a test engineer these days is probably harder than it was for me back in 2006. Back then, there was no test automation, we worked in the slow rhythm of waterfall, and for years I was in a team with other testers or at least had a test manager to bounce ideas off. These days, there's a good chance none of these are true as you start as a test engineer.

While most of these changes are good ones (please don't take test automation or agile away), it does make me empathize with anyone who starts their career as a test engineer today. The pace is higher and the skill set is broader. More importantly, you need to navigate your career while no one is really sure where to position testers in their organization. That's not a straightforward environment to start a career in.

So here are four pieces of advice I'd give myself if I'd start my career in testing today:

testing can be many different things
you're a software engineer that specializes in testing
the end-game is leadership skills
shape your career in a way that suits you

Read more… (4 min remaining to read)

My five favorite testing questions

Joep Schuurkes — Wed, 03 May 2023 06:57:25 GMT

Recently I realized there are a few testing questions I use a lot. They lie at the top of my testing toolbox, so to speak. Together they shape my testing style, making it easier for me to discover certain things, but probably also harder to find other kinds of things. So here are my five favorite testing questions, in no particular order.

What if there are zero, one, many, lots of this thing?

Last year I expressed my surprise on Mastodon how many times I've found bugs by asking the question: "What if there are 0 / 1 / several / lots of this thing?" And if you're working closely enough to the code, you should also ask about "null".

Quite a few people responded to my message. Turns out it's a very common pattern in TDD. And Brian Marick¹ remembered it standing out when he was looking into fixed bugs in the Linux kernel they used in the '80s. Personally I learned it from Elisabeth Hendrickson's "Test Heuristics Cheat Sheet", which found a new home last year at the Ministry of Testing.

"Tester" is an overloaded variable

Joep Schuurkes — Thu, 04 Aug 2022 09:39:25 GMT

Yesterday I came across a post on LinkedIn explaining why there are no testers in Scrum¹. What struck me most about the post was the amount of work the word "tester" was doing. In one sentence it meant one thing (a role in a team), in the next sentence something else (a step in the process), and so on. Hence the title of this post: the word "tester" was being used as an overloaded variable. So let's do some unpacking.

Testers to people management

"Having a tester" means that there are people with the official title of "tester" or "QA engineer" or whatever within the company. For the purposes of people management², there's a distinction between this role and the other roles in the company. This allows for more specific expectations about the role, for different career paths and salary scales, etc.

Read more… (4 min remaining to read)

It's time to retire our test case management tools

Joep Schuurkes — Sun, 19 Jul 2020 14:38:17 GMT

Recently the topic of test case management tools popped up a few times in my surroundings. In almost all cases I'd recommend against using these kinds of tools and I found myself able to give a few reasons, but also found that my thoughts lacked the clarity I'd like them to have. Hence this blog post, to force myself to think more deeply and communicate more clearly.

Before I go into that, there are a few things this blog post is not about. I won't be really going into what effect test cases have on test execution, or rather if test cases are a good tool to use when doing actual testing. Personally I don't think they are and I wrote about my inability to use them in this post from July 2013. For some deeper thoughts on this, I recommend James Bach's and Aaron Hodder's article "Test cases are not testing: Towards a culture of test performance".

What I do want to cover in this post is managing test cases. Having a collection of test cases stored somewhere to re-use across releases and reporting their pass/fail numbers. Both are important use cases for a test case management tool and both are in my opinion a bad idea.