Joep Schuurkes (Posts about test automation)

Some of the things I did after being off for a few weeks

Joep Schuurkes — Tue, 01 Jul 2025 22:00:00 GMT

I wish people would share more of what they do day-to-day. Especially testers and quality engineers. Show the work you do. On the other hand, I do realize that that's hard. You don't want to share sensitive information. You want to respect the privacy of your team. And then there's the whole practical side. How much of the context do you need to share to have an intelligible story?

Since my current project is open source and my team works in public, I have more options than most. So let's see if this kind of post works out.

Last week was my first week back at work after a few weeks of vacation. So an important part of my week was catching up. Going through emails and chat messages. Talk with my fellow team members. Participate in the usual meetings. Some stuff that didn't make it into this post. More importantly, there are three things I did, that give a good idea of what a significant part of my job looks like.

What do you fix when you fix a test?

Joep Schuurkes — Sat, 24 Aug 2024 22:00:00 GMT

You ran the tests¹ - or a pipeline did it for you - and some of them failed. Time to fix the tests! But what is it exactly that needs fixing?

There are quite a few things that might make a test fail:

an issue with the build
an issue with the pipeline (if that's where the test runs)
an issue in the environment the code under test is running on
an issue in the environment the test code is running on
a bug in the code under test
a mistake in the test code
a mistake in what the test should test

Arguably, on the last three describe a test that fails. The test did its job detecting a problem. In the first four we didn't even get that far. The issues prevented the test from doing its job. So in those cases, it's not the test(s) as such that need fixing.

Tackling test automation in a new language

Joep Schuurkes — Sat, 10 Feb 2024 23:00:00 GMT

While there's value in learning all the ins-and-outs of one particular language, its ecosystem and its testing libraries, I think there's also a lot of value in having experience in several. Or at least, in two. If you only know one, you don't really know what's essential and what's incidental to the one set of tools you know. You don't know from experience in what ways things could be different.

Picking up a new language is not trivial though, especially if it's your second one. There will be a lot to learn. You will notice similarities between the new language and the one(s) you already know. Sometimes those similarities will help you, sometimes they will mislead you.

Also, it's more than picking up a new language. There are also the testing libraries you will use and the language's ecosystem (e.g. how to install those libraries¹ or how to set up a pre-commit hook with a linter). That's quite a package.

Metrics, models and conversations

Joep Schuurkes — Sun, 12 Mar 2023 09:11:25 GMT

A few weeks ago someone suggested we should start measuring how many automated test cases versus how many manual test cases we have. Luckily it was part of a longer list of suggested metrics and it was also presented in that way: a list of options, to be discussed later. And the reason I say "luckily" is because I knew I disagreed with using that metric, but didn't have a good explanation as to why. At that moment I could have given a minutes-long monologue about how that metric doesn't fit how I think we should be thinking about testing. However, I have zero expectations that such a monologue could have worked in that situation. It would be all over the place as I philosophize about testing and make all kinds of connections and analogies. A convincing argument that does not make.

So since then, I've been thinking: how would I explain my position on measuring manual versus automated test cases? Something to the point, something relatable, something that acknowledges the need behind asking for this metric. And then it hit me: use cooking as an analogy. Now I know plenty of people have made this analogy before. However, I don't remember reading everything I will cover in this post somewhere else. Also, there's value in different people saying similar things, but each in their own voice.

Your CI/CD pipeline does not run regression tests

Joep Schuurkes — Mon, 05 Aug 2019 21:47:51 GMT

CI/CD pipelines

The purpose of a CI/CD pipeline is to allow you to deliver small changes in a fast and controlled way. Without any tests in your pipeline you would gain a lot of speed. You'd also lose a lot control, which is why people in general do run tests in their pipeline. The purpose of these tests is to check if that stage of the pipeline meets the minimum level of acceptable quality for that stage.

For example, commit stage tests will consist of mostly unit tests, a few integration tests, and even fewer end-to-end tests, because early in the pipeline speed is more important than comprehensiveness. When I commit my changes, I want the results fast enough so that I will wait for them - ready to fix any issue that might occur.

Regression testing

There are many definitions of regression testing, as you can read in Arborosa's blog post on the topic. I have always defined regression testing along the lines of "testing the parts that weren't impacted by a change to see if they really weren't impacted." (Which is really weird if you start thinking about it: something is regression testing depending on your knowledge of the system and the change.)

The tests in your pipeline are regression tests, …

Most of the tests that run in your pipeline are regression tests. Your commits are small and you have a lot of tests, so most of those will cover parts of the system that shouldn't have been impacted by your changes. So yes, regression tests.

How this tester writes code

Joep Schuurkes — Sat, 08 Jun 2019 16:21:38 GMT

A long time ago (March 2015) I wrote a post titled "Test automation - five questions leading to five heuristics". Later that year Rich Rogers asked for a follow-up. To which I replied I should do a follow-up post (ahum) "soon". Then last Wednesday Noah Sussman said on twitter: 'I don't know that I've *ever* seen "this is how testers write code"'. To which I replied "challenge accepted", so now here we are, me writing a blog post about how I as a tester write code.

The format of this post turned out to be advice based on my experiences, so the usual disclaimers apply. And feel free to leave a comment if you have any feedback!

The basics

use an IDE

An IDE is not just an advanced text editor. It understands your code - to a degree, since it's not interpreting, compiling or executing the code. So not only allows an IDE you to manipulate your code as text, it also allows you to manipulate your code as code.

Solving Black Box Puzzle 31 with data analysis

Joep Schuurkes — Sun, 28 Apr 2019 11:16:34 GMT

James Lyndsay has created a number of amazing Black Box Puzzles: tiny applications that challenge you to figure out what they do. (You can support him in creating more of these at his Patreon page.) Two of these Puzzles, 29 and 31, not only have a GUI to explore, but also an API.

And that gave me an idea. If you explore these Puzzles through their GUI, you start from the inputs. You try out different inputs in the hope of discovering a pattern in the outputs. And then that pattern feeds back into your exploration.
With an API, however - and because of the nature of Puzzle 31 - it becomes easy to get the outputs for all possible combinations of inputs. Which means you can start your exploration from the outputs instead of the inputs.

Before I tell you how and what I did, three important remarks.
First of all, I will be spoiling the solution to the Puzzle in this blog post. So this is the right moment to go and solve Puzzle 31 for yourself first. Or at least go play a bit with it, so you have an idea what the inputs and outputs are.
Secondly, I had already solved the Puzzle through the GUI a few months ago. So it was more of a "Can I find the solution this way as well?" than a "Can I find the solution?" thing.
Finally, the code and the spreadsheet I created (linked throughout, also available on GitHub here), are not very clean. I thought about tidying them up, but my two reasons for not doing so are (1) laziness; (2) the way they are now gives a more honest picture of what I did.

Test automation - five questions leading to five heuristics

Joep Schuurkes — Tue, 24 Mar 2015 19:53:24 GMT

I wrote a follow-up to this post in June 2019 in which I revisit the heuristics from this post.

In 1984 Abelson and Sussman said in the Preface to 'Structure and Interpretation of Computer Programs':

Our design of this introductory computer-science subject reflects two major concerns. First, we want to establish the idea that a computer language is not just a way of getting a computer to perform operations but rather that it is a novel formal medium for expressing ideas about methodology. Thus, programs must be written for people to read, and only incidentally for machines to execute. Second, we believe that the essential material to be addressed by a subject at this level is not the syntax of particular programming-language constructs, nor clever algorithms for computing particular functions efficiently, nor even the mathematical analysis of algorithms and the foundations of computing, but rather the techniques used to control the intellectual complexity of large software systems. [emphasis mine]

This oft-quoted sentence I emphasized, is even more true if the purpose of our programs is test automation¹. So let's say you run your test automation program and the result is a list of passes and fails. The purpose of testing is to produce information. You could say that this list of results qualifies as information and I would disagree. I would say it is data, data in need of interpretation. When we attempt this interpretation, we should consider the following five questions.

Lessons learned in some test automation

Joep Schuurkes — Mon, 18 Feb 2013 20:46:02 GMT

In the past two weeks I built a test tool in VBA (Visual Basic for Applications). I did this, because two weeks ago my fellow tester showed me an important test we have to do at least once for each major release. The test consists of having the application generate three reports in Excel format. On two of these reports you apply a set of filters and take the sum of a few columns, rinse and repeat. Then you add several of those sums together and the results of those calculations should match the numbers in the third report. So basically, the point of the test is to check if the numbers in the three reports add up. And it's a lot of work, i.e. about two days.

After being shown how to perform the test, the first thought that popped into my head, was: "Boring!" The second thought was: "It's automatable!" And since there was little else to do - delivery of the new test environments was delayed - I changed it from automatable to automated.

So now, after two weeks, I have this tool in VBA. For each of the three reports it contains a sheet in which you define the sets of filters and sums. If you click a big button, the report is opened and the filters and sums are applied. There's also a fourth sheet to do the second set of calculations and to do the check if the numbers match. This last part is not done in VBA; it's all formulas in Excel.