[pytest-dev] introducing collection boundaries to support expanding fixtures to non-python and testsuites as python packages

[Ronny Pfannschmidt]

Hi everyone,

i would like to break open a number of limitations we have in the 
collection system.

1. parametrize is tied painfully into Python Items

I want to finish my experiment for function definitions and generalize them

This would mean that instead of multiple python collectors invoking 
parametrize internally to functions,
we would collect function definitions.

This would be represented in the collection tree instead of just being a 
implementation detail of the python generate_tests hook

the generate_tests hook would generalize so that it would apply to any 
"definition" not just a Python Function Definition.

(im torn between keeping item or intoducing a Parameterization Node, 
aining insights on that will have to wait until definition is part of 
the collection tree)


2. pytests unawareness of tests in packages that may/must install vs 
tests in local test folders create painful breakages with doctests/tests 
inside of packages that must install/reinstall for testruns


i'd like to enable a basic awareness of installed tests vs local tests 
vs editable tests (and fail pytest if installed tests differ from the 
local files)

this also includes better awareness of paths/meta paths and pep420


3. testsuites for reuse or purposes like system testing cannot natively 
be shipped as python packages,

collecting them creates painful effects wrt local vs remote filenames as 
well as collection paths vs import paths


it should be possible to have testsuites that ship as pytohn packages 
and to parametrize/configure them.


for that i would like to propose a system building on top of the 
packaging awareness of pytest, and extending it with mechanisms to 
either call defined testsuites and/or referring to external testuites 
and including configured versions of those in current testruns

exact mechanisms for configuration/nesting/"testsuite cycles" and more 
are yet to be determined


but examples i'd like to support are stuff like having tools and 
frameworks provide basic tests for stuff like "base behaviors of well 
behaved plugins of a framework", "sanity tests for features of of 
framework plugins"


i'd love to get inputs/considerations and concerns as i'll focus on 
those topics in order in the next few months.


-- Ronny