Ensuring quality for a peculiar piece of software

TestBash Brighton logo
2 October 2025
Joep Schuurkes
Staff Test Engineer
Dutch Electoral Council
@joeposaurus@chaos.social
https://smallsheds.garden

What's peculiar about it?

used once a year (or not at all, or multiple times)

350+ organisations

most users are volunteers

delivery weeks before usage

air-gapped server and clients

Why is it peculiar?

software for Dutch elections

voting on paper ballots

data entry per polling station

adding up all the votes

determining who gets a seat

a stack of voting ballots a sports hall with tables where people are counting ballots a room with people sitting behind computers

One way to build it

clear requirements

clear deadline

very few and stable integrations

lock ourselves away for 18 months

Another way to build it

modern software development practices still apply

always be delivering

always be collecting feedback

The four practices I will cover

use cases

thin vertical slices

zero bug policy

working in public

quality is many small things

done right

Not just those four, also

just-in-time refinement

domain knowledge

purpose

pipeline with quality gates

and many more...

My role in all of this

test & quality engineer

not testing every change

attending to quality in general

which includes some testing

use cases

My previous experiences

thick documents

confusing diagrams

too much detail

done before I start

an UML digram of a use case

There's a book!

the cover of the book Writing Effective Use Cases by Alistair Cockburn

Writing Effective Use Cases
- Alistair Cockburn

Why not user stories?

user stories are for work management

a Lego Idiana Jones in the grass

Tree structure

☁️ cloud

🪁 kite

🌊 sea

🐟 fish

🐚 shell

Effective use cases

subject - verb - object

"The application determines the data is valid."

different levels of formality

Use case example

scereeenshot of a use case with a main success scenario and a few extentions

How use cases have helped us

better understanding by team

better understanding by stakeholders

input for UX design

input for epic kick-off

double check before closing epic

The reality

requires investing energy

some people engage more than others

short-term and long-term value

thin vertical slices

Slices, thin and vertical

how do you split up the work?

Elephant Carpaccio-workshop

drawing of an elephant with yellow vertical lines drawn over it to indicate slices

Thin slices

small steps of progress

instead of...

feedback on what you're working on now
instead of last week

Vertical slices

frontend - backend - database

instead of...

rich feedback
instead of narrow feedback

The reality

thin-ish and vertical-ish slices

small epics

challenge of incremental feedback

challenge of greenfield project

zero bug policy

Zero bug policy

A flowchart showing that if you find a bug the question is: Can it stay like this forever? If not, fix it. If it can, forget about it.

The why

no backlog full of bugs

done is done,
so done needs to be good enough

The first recorded instance of an actual bug (a moth) disrupting the operation of the Harvard Mark II on September 9, 1947

The reality

discipline and vigilance

team culture

recently postponed a fix

working in public

It's all public!

the code

the repository github.com/kiesraad/abacus

the designs figma

the pipelines and releases

the documentation

the project board

the test environments abacus-test.nl

But why?

because we have to

because we think we should

the guarantees required for good elections: voting secrecy, independence, accessibility, transparency, integrity, verifiability, freedom to vote, right to vote, uniqueness

What it has brought us so far

fuzz testing

Putting seat calculations in Dutch election software to the (fuzz) test

What it has brought us so far

six external pull requests (so far)

A pull request making a minor fix to the comment of a unit test

What it has brought us so far

involvement of municipalities

A release of the application published on GitHub

The reality

it takes more than making the repository public

documentation

reaching out to people

responding to feedback

The four practices I covered

use cases

thin vertical slices

zero bug policy

working in public

closing thoughts

this basic Agile stuff(TM),
it works!

adapt practices based on
outcome(s) and principles

We Tried Baseball and It Didn't Work - Ron Jeffries

quality is something emergent
- Anne-Marie Charrett

you don't test quality in

you don't sprinkle on
a little quality at every step

quality is many small things

done right

thank you!

a pile of little red pencils
slides at smallsheds.garden/my-talks/