From 87b2d5091fed04dc5863b474d2e0662ca3f0936f Mon Sep 17 00:00:00 2001 From: Eric Paniagua Date: Sun, 6 Sep 2020 15:12:13 -0700 Subject: [PATCH] Add testing documentation. Signed-off-by: Eric Paniagua --- docs/codebase/glossary.md | 55 ++++++++++++++++++ docs/codebase/testing/automated-testing.md | 56 +++++++++++++++++++ .../codebase/{ => testing}/release-testing.md | 2 + 3 files changed, 113 insertions(+) create mode 100644 docs/codebase/testing/automated-testing.md rename docs/codebase/{ => testing}/release-testing.md (97%) diff --git a/docs/codebase/glossary.md b/docs/codebase/glossary.md index 03e314c5..d0f93e6a 100644 --- a/docs/codebase/glossary.md +++ b/docs/codebase/glossary.md @@ -7,6 +7,10 @@ It is, as yet, incredibly incomplete. Entries should be included in dictionary order. When describing an identifier of any kind from the codebase, it should be written using identical capitalization to its use in the code and surrounded by backticks: `identifierName` +## Firmware Terminology + +These terms commonly arise when discussing the firmware. + ### Cycle The `loop` method in one's sketch file is the heart of the firmware. It runs - @@ -40,3 +44,54 @@ A point where the core firmware calls [event handlers](#event-handler), allowing An Arduino library prepared to work with Kaleidoscope. They implement methods from the `kaleidoscope::Plugin` (usually a subset of them). See [event handlers](#event-handler) above for a list of methods. + +## Testing + +These terms arise when discussing the testing framworks and related tests. + +### Sim Harness + +An abstraction used in testing to inject events into or invoke actions on the +simulated firmware. This abstraction comprises half of the interface to the +[test simulator](#test-simulator). + +### Sim State + +An abstraction used in testing to encapsulate, snapshot, and examine firmware +state. This abstraction comprises half of the interface to the [test +simulator](#test-simulator). + +### Test + +An indivial assertion or expectation that must hold for a [test +case](#test-case) to pass. + +### Test Case + +An indivual `TEST*` macro invocation. Its body consists of one or [tests](#test) +and optionally other code, e.g. to invoke the [test harness](#test-harness). +Note that gtest uses the non-standard term _Test_ for what we call a [Test +Case](#test-case). + +### Test File + +An individual file containing one or more [test suites](#test-suite). + +### Test Fixture + +A class comprising setup, teardown, and other code and common state to make +writing [test cases](#test-case) easieer. A fresh object of the fixture class +associated with a [test suite](#test-suite) is constructed for each run of each +[teset case](#test-case) in the [test suite](#test-suite). + +### Test Simulator + +An abstraction wrapping a virtual firmware build that allows performing +actions against the virtual firmware and reading state out of the virtual +firmware. The interface to the test simular is comprised of the [sim +harness](#sim-harness) and the [sim state](#sim-state). + +### Test Suite + +A collection of related [test cases](#test-case), optionally with an associated +[test fixture](#test-fixture). diff --git a/docs/codebase/testing/automated-testing.md b/docs/codebase/testing/automated-testing.md new file mode 100644 index 00000000..1d5857a8 --- /dev/null +++ b/docs/codebase/testing/automated-testing.md @@ -0,0 +1,56 @@ +# Automated Testing + +## Testing with gtest/gmock + +Before feature requests or bug fixes can be merged into master, the folowing +steps should be taken: + +- Create a new test suite named after the issue, e.g. `Issue840`. +- Add a test case named `Reproduces` that reproduces the bug. It should fail if + the bug is present and pass if the bug is fixed. +- Merge the proposed fix into a temporary testing branch. +- The reproduction test should fail. +- Add a test called "HasNotRegresed" that detects a potential regression. + It should pass with the fix and fail before the fix. +- Comment out and keep the `Reproduces` test case as documentation. + +For an example, see keyboardio:Kaleidoscope/tests/issue_840. + +### Adding a New Test Case + +For general information on writing test with gtest/gmock see [gtest +docs](https://github.com/google/googletest/tree/master/googletest/docs) and +[gmock docs](https://github.com/google/googletest/tree/master/googlemock/docs). + +1. Create a new test file, if appropriate. +1. Choose a new test suite name and create a new test fixture, if appropriate. +1. Write your test case. + +The final include in any test file should be `#include +"testing/common/setup-googletest"` which should be followed by the macro +invocation `SETUP_GOOGLETEST()`. This will take care of including headers +commonly used in tests in addtion to gtest and gmock headers. + +Any test fixtures should inherit from `VirtualDeviceTest` which wraps the test +sim APIs and provides common setup and teardown functionality. The appropriate +header is already imported by `setup-googletest.h` + +### Test Infrastructure + +If you need to modify or extend test infrastructure to support your use case, +it can currently be found under `keyboardio:Kaleidoscope/testing`. + +### Style + +TODO(obra): Fill out this section to your liking. + +You can see samples of the desired test style in the [example tests](#examples). + +### Examples + +All existing tests are examples and may be found under +`keyboardio:Kaleidoscope/tests`. + +## Testing with Aglais/Papilio + +TODO(obra): Write (or delegate the writing of) this section. diff --git a/docs/codebase/release-testing.md b/docs/codebase/testing/release-testing.md similarity index 97% rename from docs/codebase/release-testing.md rename to docs/codebase/testing/release-testing.md index 439c3d55..3def39d1 100644 --- a/docs/codebase/release-testing.md +++ b/docs/codebase/testing/release-testing.md @@ -2,6 +2,8 @@ Before a new release of Kaleidoscope, the following test process should be run through on all supported operating systems. +Always run all of the [automated tests](automated-testing.md) to verify there are no regressions. + (As of August 2017, this whole thing really applies to Model01-Firmware, but we hope to generalize it to Kaleidoscope) # Tested operating systems