our testing types

Romanian Testing Conference logo
Joep Schuurkes - @j19sch
slides at j19sch.github.io
Mendix logo

testing type or test type?

Google Ngram test type vs testing type

testing our testing types

"experience in functional testing, whitebox testing, regression testing, mobile testing and UI testing"

"we should do integration testing"

is it really that difficult?

How Google tests software

"Instead of distinguishing between code, integration, and system testing, Google uses the language of small, medium, and large tests, emphasizing scope over form."

How Google Tests Software - Whittaker, Arbon, Carollo

testing our testing types

Wittgenstein: philosophy is an activity


Merriam-Webster's definition of definition


integration testing
Testing performed to expose defects in the interfaces and in the interactions between integrated components or systems.
regression testing
Testing of a previously tested component or system following modification to ensure that defects have not been introduced or have been uncovered in unchanged areas of the software, as a result of the changes made.
negative testing
Tests aimed at showing that a component or system does not work. Negative testing is related to the tester's attitude rather than a specific test approach or test design technique, e.g., testing with invalid input values or exceptions.
source: ISTQB Glossary

definitions all the way down

black-box testing
Testing, either functional or non-functional, without reference to the internal structure of the component or system.

definitions all the way down

black-box testing
Testing, either functional or non-functional, without reference to the internal structure of the component or system.
A minimal part of a system that can be tested in isolation.
A collection of interacting elements organized to accomplish a specific function or set of functions.

definitions all the way down

black-box testing
Testing, either functional or non-functional, without reference to the internal structure of the component or system.
(non-)functional testing
Testing conducted to evaluate the compliance of a component or system with (non-)functional requirements.
The capability of the software product to adhere to standards, conventions or regulations in laws and similar prescriptions.
functional requirement
A requirement that specifies a function that a component or system must be able to perform.
non-functional requirement
A requirement that describes how the component or system will do what it is intended to do.

definitions all the way down

black-box testing
Testing, either functional or non-functional, without reference to the internal structure of the component or system.
The process consisting of all lifecycle activities, both static and dynamic, concerned with planning, preparation and evaluation of software products and related work products to determine that they satisfy specified requirements, to demonstrate that they are fit for purpose and to detect defects.


relations between concepts

assume a starting point of knowledge


Select all squares not a pipe


white-box vs black-box testing

functional vs non-functional testing

dynamic vs static testing

scripted vs exploratory testing

negative vs positive testing

terms without opposite

agile testing

developer testing

regression testing


explore meaning through what something isn't

what does it mean if there is no opposite?

danger of false dichotomies


Wittgenstein in 1922


26 April 1889: born in Vienna

1906 - 1911: studies engineering in Berlin and Manchester

1911: arrives in Cambridge to study philosophy with Bertrand Russell

1914-1918: fights for the Austrian army on the Russian and Italian front

1919: finishes Tractatus Logico-Philosophicus

1919: becomes a school teacher

1921: Tractatus Logico-Philosophicus published

Tractatus Logico-Philosophicus

sense - the empirical
propositions corresponding to a state of affairs in the world

senseless - the logical
tautologies and contradictions

nonsense - the transcendental
art, ethics, values, aesthetics, philosophy, meaning of life and religion

philosophy is solved!

all the elements

Periodic table of elements

all the elements

quality characteristic testing
performance, security, functional, accessibility, usability

device type testing (?)
desktop, mobile, tablet, browser, mainframe testing

all the elements

unit testing

unit - component - system - integration - end-to-end
unit - api - ui

test automation pyramid

here be pyramids - Joris Meerts
Test automation pyramid, Cohn 2009

all the elements

elements in a category clarify each other's meaning

in what way are they the same?
in what way are they the different?


food combinations


manual unit testing

manual API testing

automated usability testing


scripted user acceptance testing

static integration testing

white-box end-to-end testing


dynamic functional whitebox regression mobile UI exploratory pair testing


combine testing types and see what happens

is the result

... ?


Wittgenstein in 1947


1929: returns to Cambridge, starts lecturing in 1930

1936-1937: writes most of part I of Philosophical Investigations

1938: becomes German citizen (Anschluss), applies for British citizenship

1939: elected Professor of Philosophy at Cambridge (resigns in 1947)

1941-1944: works as a hospital porter and as a laboratory assistant

1947-1949: writes Part II of Philosophical Investigations

28 April 1951: dies in Cambridge (aged 62)

1953: Philosophical Investigations published

the problem with the Tractatus

the problem

The statement that a point in the visual field has two different colours at the same time is a contradiction.
The logical product of two atomic propositions can neither be a tautology nor a contradiction.

the 'solution'

Not all atomic propositions are logically indepent of each other.

the result

The logical is no longer a clear boundary and the whole theory comes crashing down.

Philosophical Investigations

"[...] we are committed to a faulty view of language, one that thinks that, to every meaningful word there must correspond some object " - Ray Monk

five red apples (PI 1)
names of numbers, names of colours, names of objects

the beetle in the box (PI 293)
pain and private languages

a behavior theory of language

"[...] a radical break with the idea that language always functions in one way, always serves the same purpose: to convey thoughts - which may be about houses, pains, good and evil, or anything else you please." (PI 304)

"For a large class of cases of the employment of the word 'meaning' - though not for all - this word can be explained in this way: the meaning of a word is its use in language." (PI 43)

more Wittgenstein

How to read Wittgenstein
- Ray Monk

Wittgenstein, The duty of genius
- Ray Monk

Wittgenstein's Vienna
- Allan Janik and Stephen Toulmin

How to read Wittgenstein - Ray Monk

software development model

xkcd software development

software development model

V-model acceptance testing

V-model acceptance testing

acceptance test-driven development (ATDD)


software development model

agile testing

developer testing

automated testing

software development model

testing types do not exist on their own

part of an approach to software development

social aspects

people talking

social aspects

pair testing

mob testing

crowd testing

social aspects

end-to-end testing

black-box testing

technical testing, automated testing

security testing

social aspects

language does not exist outside of the social

testing types affect how people interact


"X" testing

anything can be "X"

so the meaning of "X" testing can be anything

most of the time, meaning is usage

today's heuristics

  1. definitions
  2. opposites
  3. all the elements
  4. combinations
  5. software development method
  6. social aspects


clarity of meaning matters in communication

if meaning is usage... clarifying trumps correcting

engage in dialogue
ask "can you show me?"

thank you!

Sli.do – C089

Joep Schuurkes - @j19sch