A couple weeks ago I was updating some of our Playwright tests, when Playwright decided to mess with the execution order of the tests. The tests in question are a group of tests that cover a whole process from start to end. So we run them in serial mode, “to ensure they will always run together and in order”. And yet, they were not running in order.

While that’s obviously a bug, it wasn’t clear what exactly the bug was. Was it the tests not running in the correct order? Or was that only a symptom of some other bug? And what was triggering this bug? Because we’ve had these tests running in serial mode for a while, but had never seen this behavior before.

The problem as we encountered it

The tests that triggered the issue were similar to these:

Read more…

Recently I had to write a release advice. My first thought was: “I have close to two decades of experience in testing. I got this!” My second thought was: “I haven’t written a release advice in over ten years, how do I do this again?”

Luckily the first thought reflected reality a lot better than the second one. And after reviews by my team members, my project lead, and our manager, I’m proud of the end result. Even though in some ways the document has ended up more like a release decision, than a release advice. (More on that later.)

Lead with the actual advice

Initially I was going to start the document with a management summary, then an introduction, an overview of the testing that was done, the relevant known issues and remaining risks, then the actual release advice and finally some ideas on how to improve our testing further. Quite soon I realized that didn’t make sense.

Read more…

In Chapter 18 of “Taking Testing Seriously” (I skipped ahead, sue me), Keith Klain says:

So much today is about “continuous everything” - continuous deployment, continuous integration - and speed. The pace at which people are doing things is really, really disruptive to thinking deeply about problems. Then the goal gets displaced from getting good products and services to customers quickly, and turns into, “How do we back something out quickly when we inevitably screw up?” It’s continuous everything - except continuous thinking.
- Taking Testing Seriously, James Bach, Michael Bolton et al., 2025, p. 424

He has a point. But I got there by first kind of disagreeing with what he said. Because “continuous everything” is not about speed. It’s about risk. Or rather, about managing certain risks: the risk of (code) integration and the risks of delivery/deployment. And how fast you go, should not just be determined by those two specific risks. (Or even worse, by the fact that you just think it’s really cool that you can release every commit to your users.) It should be determined by your overall risk appetite, which is a much broader perspective.

Read more…

Testing types in software testing are a mess. It’s not just “regression testing” that can mean more things than there are people in the room. It’s true for basically all of them.

Just last week a teammate said “We should also do integration testing.” So I asked “What do you mean by integration testing?”. And the reply was: “Testing the frontend and backend together using Playwright.” Which made me think “I see what you mean, but I would never call that integration testing.” and made me reply: “Yeah, we should.”

The typical solution to this problem is to start defining the different testing types. To me, that’s just one approach of many to make sense of testing types. So far I’ve come up with an additional five.

Read more…

At the first conference I ever attended - or I think it was the first, in any case it was Let’s Test in 2012 -, I asked someone “So what kind of applications do you test?” And the reply was: “One of the onboard computers of a Saab fighter plane.” While at that time I was the test lead for an enterprise service bus.

We do not all have the same job.

In a recent episode of the Quality Remarks podcast hosted by Keith Klain, Paul Holland told the story of an organization where running the test automation took 13 people two weeks. And no, I’m sorry, they didn’t go into any detail about what happened in those two weeks. In my current project our pipeline takes 0 people about 17 minutes to run.

We do not all have the same job.

Read more…

In a previous post I claimed that any set of principles (or methodology for that matter) for good software development should cover these five areas:

• collaboration
• software engineering
• work management
• product
• reflect & experiment

I mentioned these five again in my post “What is a scrum master?”, but that’s all I’ve said about them. That changes with this post. I will provide a brief description for each of the five areas. I’ll share some ideas how a list of the five areas of Agile might be useful. And I’ll reflect on the question: Are they the five areas of good software development or of Agile software development?

Read more…

How does a project get to be a year late?
…One day at a time.
- The Mythical Man-Month (anniversary edition), Frederick P. Brooks Jr., p. 153

Last week I found a minor bug in the application my team is building. For every election committee, we have a page showing all sessions of that committee. These sessions are shown as a row of cards. Playing around with the number of sessions, I discovered that the row keeps on growing, resulting in a horizontal scroll bar:

Read more…

I wish people would share more of what they do day-to-day. Especially testers and quality engineers. Show the work you do. On the other hand, I do realize that that’s hard. You don’t want to share sensitive information. You want to respect the privacy of your team. And then there’s the whole practical side. How much of the context do you need to share to have an intelligible story?

Since my current project is open source and my team works in public, I have more options than most. So let’s see if this kind of post works out.

Last week was my first week back at work after a few weeks of vacation. So an important part of my week was catching up. Going through emails and chat messages. Talk with my fellow team members. Participate in the usual meetings. Some stuff that didn’t make it into this post. More importantly, there are three things I did, that give a good idea of what a significant part of my job looks like.

Read more…

A question I don’t seem to be able to let go of is: where does a tester fit into a software development team? More specifically, how essential are testers to the team?

I think that there are basically three options:

1. Your testers do not add something essential.
2. It’s not clear if your testers add something essential.
3. Your testers do add something essential.

Your testers do not add something essential

I suspect this is becoming more and more common. Thanks to Agile, CI/CD, DevOps, improved tooling and changing expectations, you should not have testers that find the dumb bugs I was expected to find at the start of my career. If a programmer wrote a > where it should have been a >=, we should expect a unit test and not a tester to catch it.

Read more…