[pytest-dev] Using Pytest 2.5.2 to Automate Testing of a Non-Python Application

May 1, 2014

      Firstly, I'd like to thank Holger Krekel and the rest of the pytest team 
for all their hard work (including the very good documentation and videos).

I'm a novice pytest user and I wanted to run a scenario by more 
experienced pytest users, before potentially going down the wrong path. 
  I would appreciate feedback on the helper functions below (limited 
testing suggests that they do work) and suggestions for alternative 
approaches.

Thank you,

Peter Santoro

============================

GIVEN:

Pytest will be used to automate the running and evaluation of tests for 
a system that is not written in Python.  I have already prototyped this 
in Python without using pytest.  I'm currently converting my prototype 
code to conform with pytest APIs, rules, and conventions.  So far so good.

There will eventually be hundreds of tests, each of which requires the
copying of a test specific directory tree structure from a source 
location to a destination location (required by the system being 
tested).  I want to avoid copying any extra directory
trees, because:

1) it will increase the time it takes the copying fixture(s) to run

2) it will also cause the system actually running the tests
(outside of pytest, but run via a fixture) to do unnecessary work (i.e.
waste time by running too many tests)

I also want to limit the amount of work required to create each test
function, so  I plan to use helper functions that utilize the unique
docid to collect the specific test results from the system being
tested and use asserts to compare the result with the previously stored 
expected results.

The proposed test function naming convention is
test_doctype_docyear_schema_docid, where:

   doctype = document type
   docyear = document's year
   schema = document's xml schema version
   docid = globally unique document id

In addition to using pytest markers, this naming convention (via pytest 
-k command line option), will allow for easy selection of the tests to 
be run from the pytest command line.

The proposed source location directory tree structure (holds test input
files and expected results record):

                 docyear_schema
                     |
                   ------- doctype
                              |
                              ------------ docid

The destination directory tree structure is fixed per the software that 
will be actually running the tests:

                location based on doctype
                         |
                         ------------ docid

POSSIBLE SOLUTIONS TO DIRECTORY TREE COPYING FIXTURE(S):

1) Write a separate copying fixture function for each test;
however, this seems like too much work.

2) Use pytest markers to logically group tests (see marker related
helper functions below), but this becomes problematic if only a small
subset of the marked group(s) needs to be run.

3) Use a suitable test function naming convention that would allow
a single copying fixture function to handle the copying for all selected 
tests.  This of course requires that the copying fixture knows what 
tests actually requested itself (see fixture_test* functions below). 
This is my preferred approach.

# marker helper functions

def markers(node):
     for v in node.keywords.values():
         if isinstance(v, (MarkDecorator, MarkInfo)):
             yield v

def marker_names(node):
     return [m.name for m in markers(node)]

def request_markers(request):
     for n in request.node.items:
         yield from markers(n)

def request_marker_names(request):
     return [m.name for m in request_markers(request)]

# helper functions to get the applicable tests for a given fixture

def fixture_test_names(name, request):
     return [n.name for n in fixture_tests(name, request)]

def fixture_tests(name, request):
     for n in request.node.items:
         if name in n.fixturenames:
             yield n

# sample hook function using a helper function to get list of marker
names to make decisions on

def pytest_collection_modifyitems(session, config, items):  # hook
     for n, item in enumerate(items):
         print(n, item.nodeid, list(marker_names(item)))

         if 'func2' in item.nodeid:
             del items[n]

# sample fixture using helper functions to get tests
associated with itself

@pytest.fixture(scope='session')
def fcopy(request):
     """copy files for selected tests that use the fcopy fixture"""

     print(list(fixture_tests('fcopy', request)))
     print(fixture_test_names('fcopy', request))

# sample fixture using helper functions to get list of marker names

@pytest.fixture(scope='session')
def fcopy2(request):
"""copy files associated with certain marker(s)"""

     print(list(request_markers(request)))
     print(request_marker_names(request))

# sample test functions that use the fcopy fixture

@pytest.mark.mymarker1
@pytest.mark.mymarker2
def test_function1(fcopy):
     pass

@pytest.mark.mymarker3
@pytest.mark.mymarker4
def test_function2(fcopy):
     pass