Added new documentation regarding the process of conducting new unit
tests for the project. Documentation covers what libraries are used in
test-driven development, along with how they are used.

Documentation covers how to create new unit tests according to a defined
standard, as well as using pytest fixtures.

2022-05-16 17:17:07 +01:00

11 KiB

Raw Blame History

Testing CLog

Clog makes use of a few development libraries for automated testing and unit testing. Those libraries (currently) are primaily pytest, my[py], flake8, and tox.

Here is a brief understanding of each library for how it's used:

Pytest – automated unit testing of each function/method/class structure in the code base. Pytest is also used with the coverage extension to indicate how much of the code base is touched by the unit tests.
Mypy – using static type-checking to ensure that type annotations are as accurate as possible. Also identify bad practices that may appear when writing statements/code blocks before refactorisation, and raise errors when mypy is ran.
Flake8 – to keep written code as close to the defined PEP8 standard, this library is used to enforce style guidance. However, not all styling is conventient and can be annoying to deal with when own standards are used. For this reason, some flake8 rules are ignored in configuration files.
tox – ensure that the code base is backwards compatible with earlier versions (currently 3.6 or later) of Python. If any version is listed as failing, then modules should be reconsidered to support older, compatible, syntax up to the latest version of Python.

With these, Clog can ensure that code behaves the way it is expected, best practices are adopted and forced were appropriate, and it's backwards compatible with earlier versions and not just the latest release.

Test-Driven Development

If you wish to test the library against the defined systems, then a few prerequisites are required in order to setup the environment correctly. The basic steps are as followed: clone project – setup tools with pip – run tools.

Setting Up the Environment

It's usually always simpler and easier to clone the repository if you intend to use a library, either for building from scratch or to develop on the library. For this, you will require Git, and to run the following commands:

git clone https://git.closedless.xyz/ClosedLess/clog.git
cd clog

Afterwards, you can run pip to install and setup the required tools and environment settings in editable mode.

For Linux

pip3 install -e .
pip3 install -r requirements-dev.txt

For Windows

pip install -e .
pip install -r requirements-dev.txt

This will install all the required packages necessarily for carrying out test-driven development. Those are the following packages (with versions):

flake8==4.0.1
tox==3.25.0
pytest==7.1.2
pytest-cov==3.0.0
mypy==0.950

Running the Tools and Expectations

Assuming that pip did not report any errors when installing the required packages, all the tools required should be available from the console window. From either a terminal, Command-Prompt, or PowerShell, the following commands can be ran with the defined inputs:

mypy clog   # 'mypy .' will cause mypy to run on the entire workspace
flake8
pytest
tox

Each of the above lines are commands to run each tool individually. Mypy is the only command which requires a file, or directory, to point to when it runs.

Every tool is expected to return no errors or warnings once it has finished executing. If mypy is ran using mypy ., then some errors and/or warning may be shown due to static type-checking of modules found in the testing/ and root directory spaces. These files are not important for ensuring type-checking and may be ignored entirely (this is why clog is given as the directory).

Flake8 is a bit too strict regarding some guidance rules. For this reason, certain rules are defined to be ignored by flake8 when analysing, and additional rules are ignored for specific files. For example, the rules W503 and W504 define where a break should occur when dealing with multiline operators. According to PEP8, a line-break should be placed before the operator and not afterwards to help readability. With this, the W504 rules is ignored:

# best practice
if (my_var == 1
    and other_var is not None): # W503 & PEP8 standard
    # do something...

# bad practice
if (my_var == 1 and
    other_var is not None): # W504 & non-PEP8 standard
    # do something...

Other rules may be ignored from the configuration, and testing modules are completely ignored using the descriptor tag at the start of the module. To find out which rules are ignored, consult the setup.cfg configuration file in the root directory of the project.

If tox is producing errors, check that all Python interpreters are installed to the local machine, and whose path to executable is located in the Path environment variable. In the case that not all interpreters are available, it would be reasonable to ignore the errors produced by tox and ensure that tox passes for at least the installed interpreters. Note that if development to the project is to be conducted, then all interpreters required by tox are to be installed on the local machine to guarantee compatibility of code with earlier versions of Python. (CAUTION: SUBMITTED PRs WHICH DO NOT PASS ALL TOX TESTS WILL BE REJECTED UNTIL CORRECTED.)

Pytest is expected to pass 80 tests and skip 1 (v0.1.0). The tests are designed to ensure robustness of each module and its contents within the code base. Some test parameters will check that certain conditions fail purposefully to indicate that erroneous arguments are handled accordingly. If any tests are identified as failing on a local machine, an Issue may be raised, subject to the code base not being modified. In the event that code has been modified, this must be explicitly stated, and any new functionality added should have their own unit tests defined appropriately. (CAUTION: SUBMITTED PRs WHICH DO NOT PASS ALL PYTEST TESTS WILL BE REJECTED UNTIL CORRECTED.)

Performing Pytest Unit Testing

If you wish to add new features to the project and submit a PR with a new enhancement, then it's important to create unit tests for the additional features that have been added. This section will define how to go about defining new unit tests.

Naming Testing Modules

If a new module in the code base has been added with new features, then a new module should also be defined in the testing/ directory. The name of this module should start with the prefix test_ followed by the suffix [new module name].py. Therefore, if a new module created was helloworld.py, then the corresponding test module should be named test_helloworld.py, and all unit tests related to helloworld.py should be placed in this module.

If a new feature was added to an existing module in the code base, then the new unit tests for this may be added to the respectful, existing test module.

Defining Pytest Unit Tests

Unit tests are defined using a verbose function header, and potentially a class body to simply group different tests together. For a function to become a pytest unit test, the prefix test must be present on a function header. For this reason, the adopted prefix is test_ followed by a verbose summary of the unit test, using the snake_case naming convention.

Here is a basic example of how to implement a valid unit test for Clog, which is taken in part from existing modules in the code base.

helloworld.py

class Foo:
    """A dummy Singleton class"""
    __instance__ = None

    def __new__(cls):
        """Construct a new instance of the class,
           or return an existing instance."""
        # check if an instance does not already exist
        if cls.__instance__ is None:
            # make a new Singleton instance
            cls.__instance__ = super(Foo, cls).__new__(cls)
        
        return cls.__instance__ # always return the Singleton instance

test_helloworld.py

import pytest

from helloworld import Foo


bar = Foo() # instantiate a new global Singleton instance

class TestFoo:
    """class represents grouping of unit tests related to Foo."""
    
    # using pytest `mark.parametrize`, multiple values can be passed
    # to the function, and if one assertion fails, other values can
    # still be tested by pytest.
    @pytest.mark.parametrize(
        "value, expected", [
            (Foo(), Foo()),
            (bar, bar),
            (Foo(), bar),
            (bar, Foo())
        ]
    )
    def test_new_foo_instance(self, value, expected):
        """Verfiy the identity of all `value`s against what is `expected`.
           If any assertion fails, then the class is not a Singleton."""
        assert value is expected # assert identity is the same

Note, docstrings and comments are not mandatory in testing modules. Given the verboseness of the function header, there's a clear indication of what the test is checking. Comments may be adopted if function bodies contain setup code for a test – most tests will simply contain an assert statement.

Creating Pytest Fixtures

You may want to construct your own fixture for testing to help with assisting. Clog already defines a fixture called capture_stdpipe, which uses monkeypatch to interrupt a call to write to a standard PIPE, and store the write value to a temporary buffer, which can be accessed when required. All fixtures should be defined in the conftest.py module, found in the root directory of the project.

Using the `capture_stdpipe` fixture

If you are testing functionality regarding standard PIPE streams, this fixture will be very handy for ensuring integrity of the message's body through a write process to a PIPE, with respect to any formatting. Both STDOUT and STDERR can be accessed via this fixture.

conftest.py

import sys

import pytest


@pytest.fixture
def capture_stdpipe(monkeypatch):
    stream_buf = {"stdout": "", "stderr": "", "writes": 0}

    def mimic_stdout(chars):
        stream_buf['stdout'] += chars
        stream_buf['writes'] += 1
    
    def mimic_stderr(chars):
        stream_buf['stderr'] += chars
        stream_buf['writes'] += 1


    monkeypatch.setattr(sys.stdout, 'write', mimic_stdout)
    monkeypatch.setattr(sys.stderr, 'write', mimic_stderr)
    
    return stream_buf

The stream buffer can hold one single write to both standard PIPE streams before data is then overwritten and lost. To access the data from the buffer in a unit test function, the fixture can be passed as a function parameter.