Joep Schuurkes (Posts about semantics)

Testing types - beyond definitions

Joep Schuurkes — Mon, 08 Dec 2025 23:00:00 GMT

Testing types in software testing are a mess. It's not just "regression testing" that can mean more things than there are people in the room. It's true for basically all of them.

Just last week a teammate said "We should also do integration testing." So I asked "What do you mean by integration testing?". And the reply was: "Testing the frontend and backend together using Playwright." Which made me think "I see what you mean, but I would never call that integration testing." and made me reply: "Yeah, we should."

The typical solution to this problem is to start defining the different testing types. To me, that's just one approach of many to make sense of testing types. So far I've come up with an additional five.

We do not all have the same job

Joep Schuurkes — Sat, 15 Nov 2025 23:00:00 GMT

At the first conference I ever attended - or I think it was the first, in any case it was Let's Test in 2012 -, I asked someone "So what kind of applications do you test?" And the reply was: "One of the onboard computers of a Saab fighter plane." While at that time I was the test lead for an enterprise service bus.

We do not all have the same job.

In a recent episode of the Quality Remarks podcast hosted by Keith Klain, Paul Holland told the story of an organization where running the test automation took 13 people two weeks. And no, I'm sorry, they didn't go into any detail about what happened in those two weeks. In my current project our pipeline takes 0 people about 17 minutes to run.

We do not all have the same job.

The difference between a test case and a requirement is the moment of discovery

Joep Schuurkes — Sun, 26 May 2024 22:00:00 GMT

There are several straightforward ways to distinguish a test case from a requirement. A test case tells you how to check some kind of thing about the application, a requirement tells you that the application should do some kind of thing. A test case is written by a tester, a requirement by a business analyst. A test case takes the shape of an action and an evaluation of the result, a requirement takes the form of a sentence like "product ABC shall do XYZ."¹

A less straightforward, but more interesting way to distinguish a test case and a requirement, is this:

The difference between a test case and a requirement is the moment of discovery.²

In this post I want to explore the meaning of that statement. In the next post I'll explore how looking at requirements and test cases in this way, can help us to do better testing. So this post will be a bit more philosophical, the next one more practical.

Quality: different purposes, different definitions

Joep Schuurkes — Wed, 24 Aug 2022 17:41:25 GMT

For years when asked to define quality, I've said "value to a person who matters"¹. Not too long ago I used that definition in the first post of my four-part series "Thinking about quality". However, in the fourth post of that series I also said that quality is something emergent. And I continued with:

We can have long discussions about what quality is, but that's a different question from how do you get quality?

Today I took this one step further, when I realized that depending on the context, I talk very differently about quality. And while I may not define 'quality' explicitly in every conversation, implicitly I'm still using different definitions. That alone, I think is interesting: instead of a single, general definition of quality that always applies, I have different definitions depending on their purpose².

"Tester" is an overloaded variable

Joep Schuurkes — Thu, 04 Aug 2022 09:39:25 GMT

Yesterday I came across a post on LinkedIn explaining why there are no testers in Scrum¹. What struck me most about the post was the amount of work the word "tester" was doing. In one sentence it meant one thing (a role in a team), in the next sentence something else (a step in the process), and so on. Hence the title of this post: the word "tester" was being used as an overloaded variable. So let's do some unpacking.

Testers to people management

"Having a tester" means that there are people with the official title of "tester" or "QA engineer" or whatever within the company. For the purposes of people management², there's a distinction between this role and the other roles in the company. This allows for more specific expectations about the role, for different career paths and salary scales, etc.

Getting [name] from "Name: [name]" in Python - an engineering problem

Joep Schuurkes — Mon, 26 Aug 2019 19:27:38 GMT

Today I was presented with an interesting engineering problem. (Important later: context was the code of an auto-test.) Given a string of the format "Name: [name]", what's the best way to get the [name] in Python?

There are several options:

lstrip()
split()
replace()
string slicing
regex

So let's look at each of them and then I'll explain which one I prefer and why. All examples are in Python 3.6, using the Python Interpreter.

Your CI/CD pipeline does not run regression tests

Joep Schuurkes — Mon, 05 Aug 2019 21:47:51 GMT

CI/CD pipelines

The purpose of a CI/CD pipeline is to allow you to deliver small changes in a fast and controlled way. Without any tests in your pipeline you would gain a lot of speed. You'd also lose a lot control, which is why people in general do run tests in their pipeline. The purpose of these tests is to check if that stage of the pipeline meets the minimum level of acceptable quality for that stage.

For example, commit stage tests will consist of mostly unit tests, a few integration tests, and even fewer end-to-end tests, because early in the pipeline speed is more important than comprehensiveness. When I commit my changes, I want the results fast enough so that I will wait for them - ready to fix any issue that might occur.

Regression testing

There are many definitions of regression testing, as you can read in Arborosa's blog post on the topic. I have always defined regression testing along the lines of "testing the parts that weren't impacted by a change to see if they really weren't impacted." (Which is really weird if you start thinking about it: something is regression testing depending on your knowledge of the system and the change.)

The tests in your pipeline are regression tests, …

Most of the tests that run in your pipeline are regression tests. Your commits are small and you have a lot of tests, so most of those will cover parts of the system that shouldn't have been impacted by your changes. So yes, regression tests.

Regression testing, it means less than you think

Joep Schuurkes — Sun, 03 Jan 2016 18:42:26 GMT

The past weeks I have made several attempts at a blog post about regression testing. About how we use it to refer to different things: tests running on a CI server, people executing test scripts, etc. And about how often the term really doesn't mean much at all, yet nobody questions you when you use it: "What are you doing?" "Regression testing." "Oh good, carry on." The point of the post would be to argue we should use the term 'regression testing' a lot less, because most of the time we can be more specific without having to be more verbose.

However, the more I thought about (what I would qualify as) proper regression testing, the more I felt that regression versus progression (or progressive) testing is a distinction without difference. One interesting observation in this regard is that "regression testing" returns 30 times more results on Google than "progression testing" and "progressive testing" combined. So what's going on here if we have a dichotomy with one member producing so much more discussion than the other? And there's more: regression testing is commonly contrasted with test types like functional testing and usability testing. But how then should I categorize a regression test focusing on functionality¹?

Why the testing/checking debate is so messy - a fruit salad analogy

Joep Schuurkes — Sun, 15 Nov 2015 11:17:12 GMT

Five days ago James Thomas posted the following in the Software Testing & Quality Assurance group on LinkedIn:

Are Testing and Checking different or not?
This article by Paul Gerrard explains why we shouldn't be trying to draw a distinction between checking and testing, but should be paying more attention to the skills of the testers we employ to do the job.

I posted a reply there, but I think I can do better than those initial thoughts, so here we go.

Let's imagine the following scene: Alice and Bob are preparing a fruit salad together.
Alice: "Ok, let's make a nice fruit salad. We need some apples and some fruit."
Bob: "Euh, aren't apples fruit?"
Alice: "Yes. Of course. But when I say 'fruit', I mean 'non-apple fruit'."

What's the word for the part of testing that's not checking?

Joep Schuurkes — Mon, 17 Aug 2015 18:19:25 GMT

The question I asked

Yesterday I asked on twitter:

Question: what's the proper word for the part of testing that's not checking? #cdt #testing #semantics
- Joep Schuurkes (@j19sch) August 16, 2015

The reason I asked, is that I noticed I needed that word in discussions about testing and checking. If checking is part of testing - and in the RST namespace it most definitely is, see 'Testing and checking refined' -, then what can I contrast checking with? Contrasting checking with testing (as in 'checking versus testing') isn't going to work: there's one thing that's checking and then there's this other thing, testing, that contains that one thing and some other stuff¹, but it's like a completely different thing. See the difference? Conceptually that just doesn't work - at least not in my mind.

The answers I got

So I figured I'd ask twitter in all its infinite testing wisdom and lo and behold, not only did people reply, a discussion ensued with the following people (listed in no particular order) participating in different configurations: @eddybruin, @mariakedemo, @SandroIbig, @TestPappy, @dwiersma, @ilarihenrik, @PhilipHoeben, @huibschoots and @deefex. Thank you all!