My five favorite testing questions

Recently I realized there are a few testing questions I use a lot. They lie at the top of my testing toolbox, so to speak. Together they shape my testing style, making it easier for me to discover certain things, but probably also harder to find other kinds of things. So here are my five favorite testing questions, in no particular order.

What if there are zero, one, many, lots of this thing?

Last year I expressed my surprise on Mastodon how many times I’ve found bugs by asking the question: “What if there are 0 / 1 / several / lots of this thing?” And if you’re working closely enough to the code, you should also ask about “null”.

Quite a few people responded to my message. Turns out it’s a very common pattern in TDD. And Brian Marick¹ remembered it standing out when he was looking into fixed bugs in the Linux kernel they used in the ‘80s. Personally I learned it from Elisabeth Hendrickson’s “Test Heuristics Cheat Sheet”, which found a new home last year at the Ministry of Testing.

How would this work for the user?

Often there’s a large distance between discovery (discovering what users need) and implementation. Not just in time, but also in who’s involved. Information from discovery becomes an initiative, becomes an epic, which gets broken down into work items and often those again in tasks. Somewhere along the way - where exactly varies - developers get involved. However, despite their access to the results of discovery, the day-to-day work of developers is very much focused around a single work item with acceptance criteria or requirements. And that’s a problem. Working this way nudges developers towards limiting themselves to just delivering single work items according to spec, losing sight of the bigger picture.

To counteract this nudge, I like to ask: “How would this work for the user?” What problem is the user trying to solve? What are they trying to achieve? And can I imagine how the user gets their thing done, step-by-step, using the software? Often enough, there are gaps. Or some things are presented in a way or expected to be done in a way that would not make sense to a user.

The great thing about this question is that you can start asking it early, during refinement or even before that. A rephrasing of this question for that stage is: “Can I imagine myself demoing this feature?”

What about time?

For some reason, people seem to default to a functional programming-like perspective when thinking about software. There’s input, the input gets transformed one or multiple times, until we get output. This makes us overlook that stuff changes over time. And keeping track of time, also means having date and time as input. Hence the testing question: “What about time?”

Time data as input

Time data as input is tricky, as illustrated so well by Noah Sussman’s “Falsehoods programmers believe about time”.

It gets even trickier when time data is input without you fully realizing it. A few months ago I found a bug where a web application took midnight current day in the user’s time zone, converted it to UTC and then used the date part of that UTC timestamp to determine the date. So for people in time zones ahead of UTC (basically the whole Eastern hemisphere) the application had an incorrect date².

Stuff happens in time

As Ray Cummings wrote: “Time is what keeps everything from happening at once.”³ Unfortunately we work with computers, so we not only have to deal with things changing over time, but also with several things that do happen all at once, i.e. concurrency. Things can be fast or slow (like internet connections). Sometimes one thing is slower than usual and another faster, i.e. a race condition. (I once “fixed” a race condition by adding a single line of logging.)

Sometimes things also stay the same over time. Decisions have an effect far after they were made. The most extreme example I have come across was around 2010. Noticing that the data model was odd, I asked why that was the case. The answer was that when the mainframe was implemented in 1974, they had retained the data model of the punch card system that was being replaced by the mainframe. And there also was some really old data in there, since the product it supported was life insurance policies.

I learned this heuristic of thinking about time from Rapid Software Testing‘s “Heuristic Test Strategy Model”. It has a great mnemonic for product elements, SFDIPOT, where the “T” stands for time.

What can I discover by interacting with it?

Basically, exploratory testing. You can’t come up with everything you want to test beforehand.⁴ You also can’t come up with everything you want to test based solely on acceptance criteria, requirements, domain knowledge, etc. Some things you can only discover through interacting with the software. And a lot of things can be discovered a lot more easily through exploratory testing than in any other way. Instead of testing based on what you can imagine about the software in your head, you use the software itself as your “external imagination”⁵.

And do note that “What can I discover by interacting with it?” is not limited to interacting with the software in the same way as a user would. If there’s something to interact with (a UI, an API, a database, a configuration file, a line of code), you can do exploratory testing.

I learned how to do exploratory testing by participating in James Lyndsay‘s excellent workshop “Getting a Grip on Exploratory Testing”. I wrote about that experience in this post from 2012.

What do you mean by <some term>?

Usually people understand each other just fine. They use the same term and even though there will never be a perfect 100% overlap in meaning, there is sufficient overlap that for all practical purposes, they are using the same term in the same way. Sometimes, however, there is a shallow agreement. People think they agree on what a term means, but in at least one important aspect, they don’t. When you suspect that’s the case, it’s time to ask: “What do you mean by <some term>?”

One of the biggest bugs I ever found was when I asked: “What exactly do you mean by ‘buffering’?” One party thought it meant they needed to buffer the first 100 undeliverable messages and could discard the rest. The other party thought that during their downtime every undeliverable message would be buffered for later delivery. My simple question in preparing the end-to-end test got the whole project sent back to the architect.

I think I learned to do this as part of getting my degree in philosophy. Having a good idea of what a philosopher means with a specific term and how that fits into their world view, is a crucial part of understanding what a philosopher wrote.

Enough about my favorite testing questions. What are your favorite one? Are some of them the same as mine? Which other ones do you use? What things are easier to find thanks to your favorite questions? And what things might you overlook?