theonepath
2339aabeb1
Added new documentation regarding the process of conducting new unit tests for the project. Documentation covers what libraries are used in test-driven development, along with how they are used. Documentation covers how to create new unit tests according to a defined standard, as well as using pytest fixtures.
269 lines
11 KiB
Markdown
269 lines
11 KiB
Markdown
# Testing CLog
|
||
Clog makes use of a few development libraries for automated testing and unit
|
||
testing. Those libraries (currently) are primaily pytest, my[py], flake8,
|
||
and tox.
|
||
|
||
Here is a brief understanding of each library for how it's used:
|
||
|
||
* Pytest – automated unit testing of each function/method/class structure in
|
||
the code base. Pytest is also used with the `coverage` extension to indicate
|
||
how much of the code base is touched by the unit tests.
|
||
* Mypy – using static type-checking to ensure that type annotations are as
|
||
accurate as possible. Also identify bad practices that may appear when writing
|
||
statements/code blocks before refactorisation, and raise errors when mypy is
|
||
ran.
|
||
* Flake8 – to keep written code as close to the defined PEP8 standard, this
|
||
library is used to enforce style guidance. However, not all styling is
|
||
conventient and can be annoying to deal with when own standards are used. For
|
||
this reason, some flake8 rules are ignored in configuration files.
|
||
* tox – ensure that the code base is backwards compatible with earlier versions
|
||
(currently 3.6 or later) of Python. If any version is listed as failing, then
|
||
modules should be reconsidered to support older, compatible, syntax up to the
|
||
latest version of Python.
|
||
|
||
With these, Clog can ensure that code behaves the way it is expected, best
|
||
practices are adopted and forced were appropriate, and it's backwards
|
||
compatible with earlier versions and not just the latest release.
|
||
|
||
## Test-Driven Development
|
||
If you wish to test the library against the defined systems, then a few
|
||
prerequisites are required in order to setup the environment correctly. The
|
||
basic steps are as followed: clone project – setup tools with pip – run tools.
|
||
|
||
### Setting Up the Environment
|
||
It's usually always simpler and easier to clone the repository if you intend to
|
||
use a library, either for building from scratch or to develop on the library.
|
||
For this, you will require Git, and to run the following commands:
|
||
|
||
```
|
||
git clone https://git.closedless.xyz/ClosedLess/clog.git
|
||
cd clog
|
||
```
|
||
|
||
Afterwards, you can run pip to install and setup the required tools and
|
||
environment settings in editable mode.
|
||
|
||
#### For Linux
|
||
```
|
||
pip3 install -e .
|
||
pip3 install -r requirements-dev.txt
|
||
```
|
||
|
||
#### For Windows
|
||
```
|
||
pip install -e .
|
||
pip install -r requirements-dev.txt
|
||
```
|
||
|
||
This will install all the required packages necessarily for carrying out
|
||
test-driven development. Those are the following packages (with versions):
|
||
|
||
* `flake8==4.0.1`
|
||
* `tox==3.25.0`
|
||
* `pytest==7.1.2`
|
||
* `pytest-cov==3.0.0`
|
||
* `mypy==0.950`
|
||
|
||
### Running the Tools and Expectations
|
||
Assuming that `pip` did not report any errors when installing the required
|
||
packages, all the tools required should be available from the console window.
|
||
From either a terminal, Command-Prompt, or PowerShell, the following commands
|
||
can be ran with the defined inputs:
|
||
```
|
||
mypy clog # 'mypy .' will cause mypy to run on the entire workspace
|
||
flake8
|
||
pytest
|
||
tox
|
||
```
|
||
|
||
Each of the above lines are commands to run each tool individually. Mypy is the
|
||
only command which requires a file, or directory, to point to when it runs.
|
||
|
||
Every tool is expected to return no errors or warnings once it has finished
|
||
executing. If mypy is ran using `mypy .`, then some errors and/or warning may
|
||
be shown due to static type-checking of modules found in the `testing/` and
|
||
root directory spaces. These files are not important for ensuring type-checking
|
||
and may be ignored entirely (this is why `clog` is given as the directory).
|
||
|
||
Flake8 is a bit too strict regarding some guidance rules. For this reason,
|
||
certain rules are defined to be ignored by flake8 when analysing, and
|
||
additional rules are ignored for specific files. For example, the rules W503
|
||
and W504 define where a break should occur when dealing with multiline
|
||
operators. According to PEP8, a line-break should be placed before the operator
|
||
and not afterwards to help readability. With this, the W504 rules is ignored:
|
||
|
||
```py
|
||
# best practice
|
||
if (my_var == 1
|
||
and other_var is not None): # W503 & PEP8 standard
|
||
# do something...
|
||
|
||
# bad practice
|
||
if (my_var == 1 and
|
||
other_var is not None): # W504 & non-PEP8 standard
|
||
# do something...
|
||
```
|
||
|
||
Other rules may be ignored from the configuration, and testing modules are
|
||
completely ignored using the descriptor tag at the start of the module. To
|
||
find out which rules are ignored, consult the `setup.cfg` configuration file
|
||
in the root directory of the project.
|
||
|
||
If tox is producing errors, check that all Python interpreters are installed
|
||
to the local machine, and whose path to executable is located in the Path
|
||
environment variable. In the case that not all interpreters are available, it
|
||
would be reasonable to ignore the errors produced by tox and ensure that tox
|
||
passes for at least the installed interpreters. Note that if development to
|
||
the project is to be conducted, then **all** interpreters required by tox are
|
||
to be installed on the local machine to guarantee compatibility of code with
|
||
earlier versions of Python. (CAUTION: SUBMITTED PRs WHICH **DO NOT** PASS ALL
|
||
TOX TESTS **WILL BE REJECTED** UNTIL CORRECTED.)
|
||
|
||
Pytest is expected to pass 80 tests and skip 1 (v0.1.0). The tests are designed
|
||
to ensure robustness of each module and its contents within the code base. Some
|
||
test parameters will check that certain conditions fail purposefully to
|
||
indicate that erroneous arguments are handled accordingly. If any tests are
|
||
identified as failing on a local machine, an Issue may be raised, subject to
|
||
the code base not being modified. In the event that code has been modified,
|
||
this must be explicitly stated, and any new functionality added **should** have
|
||
their own unit tests defined appropriately. (CAUTION: SUBMITTED PRs WHICH
|
||
**DO NOT** PASS ALL PYTEST TESTS **WILL BE REJECTED** UNTIL CORRECTED.)
|
||
|
||
## Performing Pytest Unit Testing
|
||
If you wish to add new features to the project and submit a PR with a new
|
||
enhancement, then it's important to create unit tests for the additional
|
||
features that have been added. This section will define how to go about
|
||
defining new unit tests.
|
||
|
||
### Naming Testing Modules
|
||
If a new module in the code base has been added with new features, then a new
|
||
module should also be defined in the `testing/` directory. The name of this
|
||
module should start with the prefix `test_` followed by the suffix
|
||
`[new module name].py`. Therefore, if a new module created was `helloworld.py`,
|
||
then the corresponding test module should be named `test_helloworld.py`, and
|
||
all unit tests related to `helloworld.py` should be placed in this module.
|
||
|
||
If a new feature was added to an existing module in the code base, then the new
|
||
unit tests for this may be added to the respectful, existing test module.
|
||
|
||
### Defining Pytest Unit Tests
|
||
Unit tests are defined using a verbose function header, and potentially a class
|
||
body to simply group different tests together. For a function to become a
|
||
pytest unit test, the prefix `test` must be present on a function header. For
|
||
this reason, the adopted prefix is `test_` followed by a verbose summary of
|
||
the unit test, using the `snake_case` naming convention.
|
||
|
||
Here is a basic example of how to implement a valid unit test for Clog, which
|
||
is taken in part from existing modules in the code base.
|
||
|
||
**`helloworld.py`**
|
||
```py
|
||
class Foo:
|
||
"""A dummy Singleton class"""
|
||
__instance__ = None
|
||
|
||
def __new__(cls):
|
||
"""Construct a new instance of the class,
|
||
or return an existing instance."""
|
||
# check if an instance does not already exist
|
||
if cls.__instance__ is None:
|
||
# make a new Singleton instance
|
||
cls.__instance__ = super(Foo, cls).__new__(cls)
|
||
|
||
return cls.__instance__ # always return the Singleton instance
|
||
```
|
||
|
||
**`test_helloworld.py`**
|
||
```py
|
||
import pytest
|
||
|
||
from helloworld import Foo
|
||
|
||
|
||
bar = Foo() # instantiate a new global Singleton instance
|
||
|
||
class TestFoo:
|
||
"""class represents grouping of unit tests related to Foo."""
|
||
|
||
# using pytest `mark.parametrize`, multiple values can be passed
|
||
# to the function, and if one assertion fails, other values can
|
||
# still be tested by pytest.
|
||
@pytest.mark.parametrize(
|
||
"value, expected", [
|
||
(Foo(), Foo()),
|
||
(bar, bar),
|
||
(Foo(), bar),
|
||
(bar, Foo())
|
||
]
|
||
)
|
||
def test_new_foo_instance(self, value, expected):
|
||
"""Verfiy the identity of all `value`s against what is `expected`.
|
||
If any assertion fails, then the class is not a Singleton."""
|
||
assert value is expected # assert identity is the same
|
||
```
|
||
|
||
Note, docstrings and comments are not mandatory in testing modules. Given the
|
||
verboseness of the function header, there's a clear indication of what the
|
||
test is checking. Comments may be adopted if function bodies contain setup code
|
||
for a test – most tests will simply contain an `assert` statement.
|
||
|
||
### Creating Pytest Fixtures
|
||
You may want to construct your own fixture for testing to help with assisting.
|
||
Clog already defines a fixture called `capture_stdpipe`, which uses
|
||
`monkeypatch` to interrupt a call to write to a standard PIPE, and store the
|
||
write value to a temporary buffer, which can be accessed when required. All
|
||
fixtures should be defined in the `conftest.py` module, found in the root
|
||
directory of the project.
|
||
|
||
#### Using the `capture_stdpipe` fixture
|
||
If you are testing functionality regarding standard PIPE streams, this fixture
|
||
will be very handy for ensuring integrity of the message's body through a
|
||
write process to a PIPE, with respect to any formatting. Both `STDOUT` and
|
||
`STDERR` can be accessed via this fixture.
|
||
|
||
**`conftest.py`**
|
||
```py
|
||
import sys
|
||
|
||
import pytest
|
||
|
||
|
||
@pytest.fixture
|
||
def capture_stdpipe(monkeypatch):
|
||
stream_buf = {"stdout": "", "stderr": "", "writes": 0}
|
||
|
||
def mimic_stdout(chars):
|
||
stream_buf['stdout'] += chars
|
||
stream_buf['writes'] += 1
|
||
|
||
def mimic_stderr(chars):
|
||
stream_buf['stderr'] += chars
|
||
stream_buf['writes'] += 1
|
||
|
||
|
||
monkeypatch.setattr(sys.stdout, 'write', mimic_stdout)
|
||
monkeypatch.setattr(sys.stderr, 'write', mimic_stderr)
|
||
|
||
return stream_buf
|
||
```
|
||
|
||
The stream buffer can hold one single write to both standard PIPE streams
|
||
before data is then overwritten and lost. To access the data from the buffer in
|
||
a unit test function, the fixture can be passed as a function parameter.
|
||
|
||
**`helloworld.py`**
|
||
```py
|
||
def check_that_the_fixture_captures_output(capture_stdpipe):
|
||
# print will go to STDOUT by default
|
||
print("Hello World!")
|
||
# access the buffer and compare
|
||
assert capture_stdpipe['stdout'] == "Hello World!"
|
||
```
|
||
|
||
New fixtures defined for unit testing can be used in a similar fashion to how
|
||
`capture_stdpipe` is used. For more information regarding how to use fixtures
|
||
and further pytest features, consult the official documentations.
|
||
|
||
Author: Ethan Smith-Coss.
|
||
Copyright (c) 2022-23
|