From 2339aabeb1405fc03f89bae6a7acb4d3e6c42b3c Mon Sep 17 00:00:00 2001 From: theonepath Date: Mon, 16 May 2022 17:17:07 +0100 Subject: [PATCH] Added documentation TESTING.md Added new documentation regarding the process of conducting new unit tests for the project. Documentation covers what libraries are used in test-driven development, along with how they are used. Documentation covers how to create new unit tests according to a defined standard, as well as using pytest fixtures. --- testing/TESTING.md | 268 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 268 insertions(+) create mode 100644 testing/TESTING.md diff --git a/testing/TESTING.md b/testing/TESTING.md new file mode 100644 index 0000000..74d8faf --- /dev/null +++ b/testing/TESTING.md @@ -0,0 +1,268 @@ +# Testing CLog +Clog makes use of a few development libraries for automated testing and unit +testing. Those libraries (currently) are primaily pytest, my[py], flake8, +and tox. + +Here is a brief understanding of each library for how it's used: + +* Pytest – automated unit testing of each function/method/class structure in +the code base. Pytest is also used with the `coverage` extension to indicate +how much of the code base is touched by the unit tests. +* Mypy – using static type-checking to ensure that type annotations are as +accurate as possible. Also identify bad practices that may appear when writing +statements/code blocks before refactorisation, and raise errors when mypy is +ran. +* Flake8 – to keep written code as close to the defined PEP8 standard, this +library is used to enforce style guidance. However, not all styling is +conventient and can be annoying to deal with when own standards are used. For +this reason, some flake8 rules are ignored in configuration files. +* tox – ensure that the code base is backwards compatible with earlier versions +(currently 3.6 or later) of Python. If any version is listed as failing, then +modules should be reconsidered to support older, compatible, syntax up to the +latest version of Python. + +With these, Clog can ensure that code behaves the way it is expected, best +practices are adopted and forced were appropriate, and it's backwards +compatible with earlier versions and not just the latest release. + +## Test-Driven Development +If you wish to test the library against the defined systems, then a few +prerequisites are required in order to setup the environment correctly. The +basic steps are as followed: clone project – setup tools with pip – run tools. + +### Setting Up the Environment +It's usually always simpler and easier to clone the repository if you intend to +use a library, either for building from scratch or to develop on the library. +For this, you will require Git, and to run the following commands: + +``` +git clone https://git.closedless.xyz/ClosedLess/clog.git +cd clog +``` + +Afterwards, you can run pip to install and setup the required tools and +environment settings in editable mode. + +#### For Linux +``` +pip3 install -e . +pip3 install -r requirements-dev.txt +``` + +#### For Windows +``` +pip install -e . +pip install -r requirements-dev.txt +``` + +This will install all the required packages necessarily for carrying out +test-driven development. Those are the following packages (with versions): + +* `flake8==4.0.1` +* `tox==3.25.0` +* `pytest==7.1.2` +* `pytest-cov==3.0.0` +* `mypy==0.950` + +### Running the Tools and Expectations +Assuming that `pip` did not report any errors when installing the required +packages, all the tools required should be available from the console window. +From either a terminal, Command-Prompt, or PowerShell, the following commands +can be ran with the defined inputs: +``` +mypy clog # 'mypy .' will cause mypy to run on the entire workspace +flake8 +pytest +tox +``` + +Each of the above lines are commands to run each tool individually. Mypy is the +only command which requires a file, or directory, to point to when it runs. + +Every tool is expected to return no errors or warnings once it has finished +executing. If mypy is ran using `mypy .`, then some errors and/or warning may +be shown due to static type-checking of modules found in the `testing/` and +root directory spaces. These files are not important for ensuring type-checking +and may be ignored entirely (this is why `clog` is given as the directory). + +Flake8 is a bit too strict regarding some guidance rules. For this reason, +certain rules are defined to be ignored by flake8 when analysing, and +additional rules are ignored for specific files. For example, the rules W503 +and W504 define where a break should occur when dealing with multiline +operators. According to PEP8, a line-break should be placed before the operator +and not afterwards to help readability. With this, the W504 rules is ignored: + +```py +# best practice +if (my_var == 1 + and other_var is not None): # W503 & PEP8 standard + # do something... + +# bad practice +if (my_var == 1 and + other_var is not None): # W504 & non-PEP8 standard + # do something... +``` + +Other rules may be ignored from the configuration, and testing modules are +completely ignored using the descriptor tag at the start of the module. To +find out which rules are ignored, consult the `setup.cfg` configuration file +in the root directory of the project. + +If tox is producing errors, check that all Python interpreters are installed +to the local machine, and whose path to executable is located in the Path +environment variable. In the case that not all interpreters are available, it +would be reasonable to ignore the errors produced by tox and ensure that tox +passes for at least the installed interpreters. Note that if development to +the project is to be conducted, then **all** interpreters required by tox are +to be installed on the local machine to guarantee compatibility of code with +earlier versions of Python. (CAUTION: SUBMITTED PRs WHICH **DO NOT** PASS ALL +TOX TESTS **WILL BE REJECTED** UNTIL CORRECTED.) + +Pytest is expected to pass 80 tests and skip 1 (v0.1.0). The tests are designed +to ensure robustness of each module and its contents within the code base. Some +test parameters will check that certain conditions fail purposefully to +indicate that erroneous arguments are handled accordingly. If any tests are +identified as failing on a local machine, an Issue may be raised, subject to +the code base not being modified. In the event that code has been modified, +this must be explicitly stated, and any new functionality added **should** have +their own unit tests defined appropriately. (CAUTION: SUBMITTED PRs WHICH +**DO NOT** PASS ALL PYTEST TESTS **WILL BE REJECTED** UNTIL CORRECTED.) + +## Performing Pytest Unit Testing +If you wish to add new features to the project and submit a PR with a new +enhancement, then it's important to create unit tests for the additional +features that have been added. This section will define how to go about +defining new unit tests. + +### Naming Testing Modules +If a new module in the code base has been added with new features, then a new +module should also be defined in the `testing/` directory. The name of this +module should start with the prefix `test_` followed by the suffix +`[new module name].py`. Therefore, if a new module created was `helloworld.py`, +then the corresponding test module should be named `test_helloworld.py`, and +all unit tests related to `helloworld.py` should be placed in this module. + +If a new feature was added to an existing module in the code base, then the new +unit tests for this may be added to the respectful, existing test module. + +### Defining Pytest Unit Tests +Unit tests are defined using a verbose function header, and potentially a class +body to simply group different tests together. For a function to become a +pytest unit test, the prefix `test` must be present on a function header. For +this reason, the adopted prefix is `test_` followed by a verbose summary of +the unit test, using the `snake_case` naming convention. + +Here is a basic example of how to implement a valid unit test for Clog, which +is taken in part from existing modules in the code base. + +**`helloworld.py`** +```py +class Foo: + """A dummy Singleton class""" + __instance__ = None + + def __new__(cls): + """Construct a new instance of the class, + or return an existing instance.""" + # check if an instance does not already exist + if cls.__instance__ is None: + # make a new Singleton instance + cls.__instance__ = super(Foo, cls).__new__(cls) + + return cls.__instance__ # always return the Singleton instance +``` + +**`test_helloworld.py`** +```py +import pytest + +from helloworld import Foo + + +bar = Foo() # instantiate a new global Singleton instance + +class TestFoo: + """class represents grouping of unit tests related to Foo.""" + + # using pytest `mark.parametrize`, multiple values can be passed + # to the function, and if one assertion fails, other values can + # still be tested by pytest. + @pytest.mark.parametrize( + "value, expected", [ + (Foo(), Foo()), + (bar, bar), + (Foo(), bar), + (bar, Foo()) + ] + ) + def test_new_foo_instance(self, value, expected): + """Verfiy the identity of all `value`s against what is `expected`. + If any assertion fails, then the class is not a Singleton.""" + assert value is expected # assert identity is the same +``` + +Note, docstrings and comments are not mandatory in testing modules. Given the +verboseness of the function header, there's a clear indication of what the +test is checking. Comments may be adopted if function bodies contain setup code +for a test – most tests will simply contain an `assert` statement. + +### Creating Pytest Fixtures +You may want to construct your own fixture for testing to help with assisting. +Clog already defines a fixture called `capture_stdpipe`, which uses +`monkeypatch` to interrupt a call to write to a standard PIPE, and store the +write value to a temporary buffer, which can be accessed when required. All +fixtures should be defined in the `conftest.py` module, found in the root +directory of the project. + +#### Using the `capture_stdpipe` fixture +If you are testing functionality regarding standard PIPE streams, this fixture +will be very handy for ensuring integrity of the message's body through a +write process to a PIPE, with respect to any formatting. Both `STDOUT` and +`STDERR` can be accessed via this fixture. + +**`conftest.py`** +```py +import sys + +import pytest + + +@pytest.fixture +def capture_stdpipe(monkeypatch): + stream_buf = {"stdout": "", "stderr": "", "writes": 0} + + def mimic_stdout(chars): + stream_buf['stdout'] += chars + stream_buf['writes'] += 1 + + def mimic_stderr(chars): + stream_buf['stderr'] += chars + stream_buf['writes'] += 1 + + + monkeypatch.setattr(sys.stdout, 'write', mimic_stdout) + monkeypatch.setattr(sys.stderr, 'write', mimic_stderr) + + return stream_buf +``` + +The stream buffer can hold one single write to both standard PIPE streams +before data is then overwritten and lost. To access the data from the buffer in +a unit test function, the fixture can be passed as a function parameter. + +**`helloworld.py`** +```py +def check_that_the_fixture_captures_output(capture_stdpipe): + # print will go to STDOUT by default + print("Hello World!") + # access the buffer and compare + assert capture_stdpipe['stdout'] == "Hello World!" +``` + +New fixtures defined for unit testing can be used in a similar fashion to how +`capture_stdpipe` is used. For more information regarding how to use fixtures +and further pytest features, consult the official documentations. + +Author: Ethan Smith-Coss. +Copyright (c) 2022-23