Why Testing Needs Better Vocabulary

The Fabled Test Pyramid

The Test Pyramid is good advice, but it is an insufficient guide to implementation (which is why I assumed that you’ve seen it before in some form or form, it may have been one of these).

The Army We Have

Before we try to change the world, let’s first attempt to understand it as it is. We do have quite a few words to describe various levels of a testing hierarchy.

“The Enemy We Have”

This quote from Donald Rumsfield has a corollary, though I couldn’t find attribution for it: “you have to go to war with the enemy you have, not the enemy you want.” In other words, you have to acknowledge and understand your problem before you can solve it.

Unit Tests

Unit tests are whitebox (or graybox) tests of some basic unit of source code, which will require developer-level access and understanding to create. These tests are entirely isolated from external dependencies. The defects they detect are technical gaps or inconsistencies that could potentially cause crashes, generic error dialogs, the loss and corruption of data, or something else.

System Integration Tests

System integration tests are graybox or blackbox tests that assess the interactions between two live systems. Developers don’t need to create these tests, but their usefulness may be limited unless the system and its components were built or modified by developers to be testable in this way. They can and should be isolated from all external dependencies, and doing this effectively may require developer support.

Functional Tests

Functional tests are blackbox tests of a complete live system in an environment that is reasonably production-like. These tests will have a high dependence on external systems, but these dependencies should be limited and managed.

More Definitions

Terms I used but didn’t define above:

  • Whitebox tests have full knowledge of the system’s design and implementation. They enable you to examine all of its parts and the way they fit together. A strict definition would require that this exclude any test that requires the system to actually be operating (whether traditional tests are “operating” the system is debatable).
  • Graybox tests have some knowledge of the system’s design, implementation, and the ability to inspect the system’s state while it’s running. In practical terms, you can think of your test as a graybox if it entails attaching a debugger to something.
  • Blackbox tests interact with only one system. The system may be configured in a way that facilitates testing, but it does not expose any special access to its internal state.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Women Who Code

Women Who Code

We are a 501(c)(3) non-profit organization dedicated to inspiring women to excel in technology careers. https://www.womenwhocode.com/