Add wrap-up items to testing/TODO.txt.

Signed-off-by: Eric Paniagua <epaniagua@google.com>
pull/898/head
Eric Paniagua 4 years ago
parent 87b2d5091f
commit ef2b587b56

@ -1,26 +1,15 @@
Build System
============
Current behavior
- `make` builds googletest, common, kaleidoscope, and issue_840
- This is intended :)
- It is also intended that hello-simulator be omitted, since it has a test which is intended to fail to smoke test the framework
FIXED - `make clean` builds googletest, common, kaleidoscope, and issue_840, then manually runs make clean for each
- Err... not intended
FIXED - Ideally `make clean` would recursively invoke `make clean` or equivalent
FIXED - `make issue_840` (or another target in $(TEST_DIRS)) builds googletest, common, kaleidoscope, and issue_840
- Not intended
- Should just build googletest, common, and the target director (issue_840 in this case)
Desired (currently missing) behavior
- Support for docker as done for `make docker-simulator-tests`
- I tried this, but wasn't able to get cmake (for googletest) to play well with Docker
Duplication of compiler flags across Makefiles under this directory
- Intended, but definitely shouldn't be in handwritten copies
Other duplication across Makefiles
- Intended, but already drifing; also, shouldn't be in manual copies
Duplication of delegate.mk
- Intended, to support invoking kaleidoscope-builder, but definitely shouldn't be manually copied
Undesired
- Duplication of compiler flags across Makefiles under this directory
- Intended, but definitely shouldn't be in handwritten copies
- Other duplication across Makefiles
- Intended, but already drifing; also, shouldn't be in manual copies
- Duplication of delegate.mk
- Intended, to support invoking kaleidoscope-builder, but definitely shouldn't be manually copied
Directory structure
- Tests (e.g. testing/hello-simulator, testing/kaleidoscope, testing/issue_840) should be moved into a directory
@ -30,3 +19,86 @@ Directory structure
- tests/hid/usb/bootkeyboard/foo
- Test infracture (now in testing/common/) should be moved into testing/ after the actual tests are moved out
- Test infrastructure should be able to be hierarchically organized
Future Work
===========
Issue 867
- Infra changes: none needed
- Testing 867
- Design a set of layers that allows all the test cases you would like, and configure them in a sketch.
- If you need multiple sets of layers, the current build mechanism requires splitting
the test code in two, each with its corresponding sketch.
- Navigate between layers using sim_.Press(row, col) and sim_.Release(row, col) as a user
would with the test sketch loaded.
- Verify that the right keys are pressed (based on the current layer) using
EXPECT_THAT(state->HidReports()->Keyboard(0), Contains(Key_A))
- See issue_840 for example usage.
Testing LED activation
- Infra changes
- Create an LedState class to hold a snapshot of LED activations.
- Add an led_state_ member to State together with whatever accessors are convenient.
- If the accessors are complicated or you just want to namespace that part of the global state,
create a state module for LEDs following the pattern implemented for HIDState.
- Of course, the HIDReportObserver stuff doesn't apply here.
- Modify State::Snapshot to copy the current LED state into led_state_
- This must be a copy, not a reference, as it may be desirable to compare state objects
generated at different times.
- Consider whether there are any sensible LED-specific "matchers" (in the gmock sense) that could
be added to make LED tests more readable.
- See https://github.com/google/googletest/blob/master/googlemock/docs/cook_book.md for instructions
and examples on writing new matchers
- Add USB simulation, eg to test Focus and FocusSerial
- It would be nice if there were a way to specify the test firmware configuration directly in the test
(ie *_test.cpp) files
Design of the Testing Simulator
===============================
Two halves of the simulator, each an abstraction to isolate tests from firmware implementation details
and thereby decrease the maintence cost of tests.
- SimHarness
- This is what lets you prod the firmware, e.g. with a change in keyswitch state
- Any new ways of feeding input to the simulated firmware should do so through this abstraction
- State
- This is a snapshot of the global firmware state. It is currently only intended to be accessible via
the snapshot returned by RunCycle.
- Should always be a copy of firmware state, never a reference or pointer to it.
- In general it is desirable to be able to compare aspects of state from two different "times" in
the simulation.
Both halves may end up with modular structure as complexity grows. This is fine as long as the basic
abstractions and division of responsibilities are maintained.
BEFORE MERGING
==============
- The bundle repo should merge branch `epan/build/justlib`. The test infra depends on the changes in
that branch.
- Consider whether you want to add the following before or after merging (and delay merging if appropriate):
- Fleshing out the coverage of HID reports by HID state.
- Adding LED output capture to State.
- Please impement a proper logging solution (rather than dumping almost everything to stdout/stderr and
needing to sort through it manually at fixed verbosity)
- Some desirable properties of a logging system
- Global verbosity levels (e.g. INFO, WARNING, ERROR, FATAL)
- Custom verbosity levels
- Local verbosity levels, eg per file, ideally configurable in a central and obvious location like the command line
- See parseCommandLine in virtual/cores/main.cpp for the hook that would need to be overridden
- Logging macros that compile away to nothing in production builds (ie where NDEBUG is defined)
- This allows logging directly from the firmware code at no space or time cost to the real firmware
- glog is a good example of some desirable capabilities in a logging system, if not a good system to adopt.
For portability for developers, please get the test build system to play well with Docker
- (right now, cmake isn't playing well with Docker)
Update testing documentation as appropriate to relfect changes to test infra or to directory structures.
- See docs/codebase/testing/automated-testing.md
- Add links into the codebase as appropriate
What I'd do with Another Solid Week
===================================
- Add coverage of remaining HID report types
- Add capture of LED state
- Pick several reported bugs and write up tests to reproduce them and then regression tests once they've been fixed
- My main goal here would be to exercise the framwork to identify pain points or other areas for improvement
- Write a test-specific plugin that supports snapshotting state each time one of its event handlers is called
- More formally document the test simulator's design and usage
- If there were time left and earlier steps were to bear out the need...
- Add the ability to inject synthetic events or events at a given delay into the future
- Improve the simulator's concept of time and trigger events against the firmware, interleaving them in the right
order with a time-based priority queue

Loading…
Cancel
Save