Writing tests for plugin¶

When developing a plugin it is important to write tests. The main concern of running tests is that the test environment has to be separated from the production environment and care should be taken to avoid any unwanted change to the user’s database. You may have noticed that aiida_core has its own test framework for developments. While it is possible to use the same framework for the plugins, it is not ideal as any tests of plugins has to be run with the verdi devel tests command-line interface. Special profiles also have to be set mannually by the user and in automated test environments.

AiiDA ships with tools to simplify tests for plugins. The recommended way is to use the pytest framework, while the unittest package is also supported. Internally, test environments are created and managed by the aiida.manage.fixtures.fixture_manager() defined in aiida.manage.fixtures.

Using the pytest framework¶

In this section we will introduce using the pytest framework to write tests for plugins.

Preparing the fixtures¶

One important concept of the pytest framework is the fixture. A fixture is something that a test requires. It could be a predefined object that the test act on, resources for the tests, or just some code you want to run before the test starts. Please see pytest’s documentation for details, especially if you are new to writing testes.

To utilize the fixture_manager, we first need to define the actual fixtures:

# -*- coding: utf-8 -*-
###########################################################################
# Copyright (c), The AiiDA team. All rights reserved.                     #
# This file is part of the AiiDA code.                                    #
#                                                                         #
# The code is hosted on GitHub at https://github.com/aiidateam/aiida_core #
# For further information on the license, see the LICENSE.txt file        #
# For further information please visit http://www.aiida.net               #
###########################################################################
"""
For pytest
This file should be put into the root directory of the package to make
the fixtures available to all tests.
"""
from __future__ import division
from __future__ import print_function
from __future__ import absolute_import
import tempfile
import shutil
import pytest

from aiida.manage.fixtures import fixture_manager


@pytest.fixture(scope='session')
def aiida_profile():
    """setup a test profile for the duration of the tests"""
    with fixture_manager() as fixture_mgr:
        yield fixture_mgr


@pytest.fixture(scope='function')
def new_database(aiida_profile):
    """Get a the database for the test and clean it up after it finishes"""
    aiida_profile.reset_db()
    return


@pytest.fixture(scope='function')
def new_workdir():
    """get a new temporary folder to use as the computer's wrkdir"""
    dirpath = tempfile.mkdtemp()
    yield dirpath
    shutil.rmtree(dirpath)

The aiida_profile fixture initialize the fixture_manager yields it to the test function. By using the with clause, we ensure that the test profile to run tests are destroyed in the end. The scope of this fixture should be session, since there is no need to re-initialize the test profile mid-way. The next fixture new_database request the aiida_profile fixture and tells the received FixtureManager instance to reset the database. By requesting the new_database fixture, the test function will start with a fresh aiida environment. The next fixture, new_workdir, returns an temporary directory for file operations and delete it when the test is finished. You may also want to define other fixtures such as those setup and return Data nodes or prepare calculations.

To make these fixtures available to all tests, they can be put into the conftest.py in root level of the package or tests sub-packages. The code shown above can be downloaded here.

Import statements in tests¶

When running test, it is important that you DO NOT explicitly load the aiida database via load_dbenv(), which could result in corruption of your database with actual data. However, many AiiDA modules, such as those in aiida.orm cannot be loaded without calling load_dbenv() first. Modules in your plugin may also import such aiida modules at the top level. Hence, they can not be imported directly in test modules. To solve this issue, import should be delayed until the test profile has been loaded. You can always import these required modules inside the test function. A better way is to define a fixture as a loader for module imports. For example, instead of having:

import aiida.orm as orm

at the module level, you can define a fixture:

@pytest.fixture(scope='module')
def orm(aiida_profile):
    import aiida.orm as orm
    return orm

and simply request this fixture for your test function:

def test_load_dataclass(orm):
    """Test loading a data class defined by the plugin"""
    from aiida.plugins import DataFactory
    MyData = DataFactory('myplugin.maydata')

We set 'scope='module' to declare that this is module scope fixture and avoids repetitively doing the import for each test. It is also possible to group many imports in a single fixture:

@pytest.fixture(scope='module')
def imps(aiida_profile):
    """Return an class with all imports as its attributes"""
    class Imports(object):
        import aiida.orm as orm

    return Imports


def test_load_dataclass(imps):
    """Test loading a data class defined by the plugin"""
    MyData = imps.orm.DataFactory('my_plugin.maydata')

Requesting the aiida_profile fixture in the imps fixture guarantees that the test environment will be loaded before the any import statement are executed.

Running the tests¶

Finally, to run the tests, simply type:

pytest

in your terminal from the code directory. The discovery of the tests will be handled by pytest (file, class and function name should start with the word test)

Note

Your terminal will print something out during the creation of a test profile. Do not panic, as and the aiida profile and database are completely isolated and will not affect your .aiida folder and file repositories. Internally, at temporary folder is used as the .aiida folder and the test database are created using the pgtest package.

Using the unittest framework¶

The uniitest package is included in the python standard library. It is widely used despite some limitations (it is also used for testing aiida_core). We provide a aiida.manage.fixtures.PluginTestCase to be used for inheritance. By default, each test method in the test case class runs with a fresh aiida database. Due to the limitation of uniitest, sub-clasess of PluginTestCase has to be run with the special runner in aiida.manage.fixtures.TestRunner. To run the actually tests, you need to prepare a run script in python:

import uniitest
from aiida.manage.fixtures import TestRunner

test = unittest.deaultTestLoader.discover('.')
TestRunner().run(tests)

Save it as run_tests.py and tests can be discovered and run using:

python run_test.py

Migrating existing AiidaTestCase tests¶

The pytest framework can also be used to run unittest tests. Here, we will explain how to migrate existing tests for the plugins, written as sub-classes of AiidaTestCase to work with pytest. First, let’s see a typical test class using the unittest:

from aiida.plugins import DataFactory

# Assuming our new date type has entry point myplugin.complex
ComplexData = DataFactory("myplugin.complex")

class TestComplexData(AiidaTestCase):

    def setUp(self):
        """Clean up database for each test"""
        self.clean_db()

    def store_complex(self, comp_num):
        """Store a complex number, returns pk"""
        comdata = ComplexData()
        comdata.value = comp_num
        return comdata.pk

    def test_complex_store(self):
        """Test if the complex numbers can be stored"""

        comdata = ComplexData()
        comdata.value = 1 + 2j
        comdata.store()

    def test_complex_retrieve(self):
        """Test if the complex

        comp_num = 1 + 2j
        pk = self.store_complex(cnum)
        comdata = load_node(pk)
        self.assertEqual(comdata.value == comp_num)

We can modify this test class using some of the pytest features to allow it to be run with pytest directly, as shown below:

# Assuming our new date type has entry point myplugin.complex
import unittest
import pytest


@pytest.fixture(scope='module')
def module_import(aiida_profile, request):
    from aiida.plugins import DataFactory
    ComplexData = DataFactory("myplugin.complex")
    for name, value in locals():
        setattr(resquest.module, name, value)


@pytest.mark.usefixtures('module_import')
class TestComplexData(TestCase):
    """Test ComplexData. Compatible with pytest."""

    @pytest.fixture(autouse=True)
    def reset_db(aiida_profile):
        aiida_profile.reset_db()
        yield
        aiida_profile.reset_db()

    def store_complex(self, comp_num):
        comdata = ComplexData()
        comdata.value = comp_num
        return comdata.pk

    def test_complex_store(self):
        """Test if the complex numbers can be stored"""
        comdata = ComplexData()
        comdata.value = 1 + 2j
        comdata.store()

    def test_complex_retrieve(self):
        """
        Test if the complex number stored can be retrieved
        """
        comp_num = 1 + 2j
        pk = self.store_complex(cnum)
        comdata = load_node(pk)
        self.assertEqual(comdata.value == comp_num)

To allow pytest to run the tests, we first swap the AiidaTestCase with the generic TestCase. We define a module scope fixture module_import to import the required AiiDA modules and make them available in the module namespace. All previous module levels imports should be encapsulated inside this fixture. The request is a built-in fixture in pytest to allow introspect of the function from which the fixture is requested. Here, we simply add every things in the function scope back into the module of the class which requested the fixture.

Instead of the setUp and tearDown methods, we define a reset_db fixture to reset the database for every tests. The autouse=True flag tells all test methods inside the class to use it automatically.

When migrating your code to use the pytest, you may define a base class with these modifications and use it as the superclass for other test classes.