Mailman 3 October 2019 - pytest-dev

Fwd: Future of pytest-xdist/execnet?
by Isaul Vargas Oct. 25, 2019

Oct. 25, 2019

This looks promising on the surface. I think we should outline what a new concurrent test runner should look like. These are things in my wish list: * Tests are ran as a queue for each process. Currently x-dist batches the tests to N workers, and maybe grabs a test or two in that batch. But if any test takes a lot longer than expected, that batch is held up. * The ability to pin tests to a worker. Maybe you have a class of tests that need a single shared resource that would be difficult to pickle, so using a marker allows you to do this. There should be markers to allow grouping of tests to a single worker. Probably at the class and module level. * Add tooling to make it easy for fixtures to share data among processes. Maybe a fixture runs a long process to generate data, but the data is easily pickleable. There should be way to have a fixture initialize once, and share the result with other processes. * Allow some form of locking for function scope fixtures so that they can block when running concurrently. This could be useful for things that are complicated to share, but you want only one process to run at at time during setup or teardown. I'll be happy to test this new project and provide feedback. I am really excited that is happening. On Thu, Oct 24, 2019 at 8:46 AM Bruno Oliveira <nicoddemus(a)gmail.com> wrote: > Hi everyone, > > For some time now execnet has been in maintenance only mode, and even so > very few people are willing to maintain it; lately just myself and I’m not > a good choice given that I don’t know the codebase at all, plus I have tons > on my plate already. This poses a problem because often we have bug reports > or feature requests in pytest-xdist which would require fixing or > improving execnet, and are currently left without solution. > > Ronny and I have on occasion discussed the possibility of changing execnet‘s > backend , with the current contenders being multiprocessing for local > distribution and mitogen for remote distribution. > > I would like to write down some thoughts and gather opinions from the > mailing list members. > multiprocessing > > Aside from not needing to maintain execnet anymore, being able to use > multiprocessing would bring several benefits: > > - > > pytest -s would work just fine, because multiprocessing does not use > stdout/stderr for communication. This is a often requested feature but > which we can’t (with current expertise) implement on execnet. > - > > It would automatically support anything that can be picked to be > transmitted across the wire. Currently execnet does not support custom user > types and some builtins (frozen sets for example). > > Without execnet we would lose the ability of running the tests in > different interpreter versions, but I believe this has been supplanted by > tox in the ecosystem. > mitogen > > Using mitogen for remote execution would provide automatic bootstrap of > the Python environment, which is not currently supported by xdist‘s ssh > remote execution, greatly limiting its usefulness in real-time scenarios. > > This is Ronny’s idea and he can add more here if wanted. > pytest-xdist2? > > All the changes mentioned above would break backward compatibility *hard*, > because details of the execnet implementation are currently exposed to > users in the command-line: > > pytest -d --tx popen//python > > And in several hooks: > > def pytest_xdist_newgateway(gateway): > """ called on new raw gateway creation. """ > > (gateway is an execnet object) > > To incorporate the new backends, we would need to severely break both > command-line and existing hooks. Given the level of breakage this could > cause, just bumping the major version in this case doesn’t seem enough, it > would be a disservice to users given that probably nobody pins pytest-xdist > to <=2, and it would break the world with hard to understand error > messages once a first major release was released. > > Because of this, we have been discussing creating a new package, > pytest-xdist2 (other suggestions are welcome), without any backward > compatibility guarantees with pytest-xdist. > > pytest-xdist2 would, at first, only support local test distribution using > multiprocessing (-n flag), no new hooks, and no remote execution. I’ve > made a proof of concept here: > > https://github.com/pytest-dev/pytest-xdist/pull/479 > > So it definitely seems possible. One immediate benefit is that the proof > of concept above supports pytest -s already, and can transfer anything > that’s pickled over the wire. > > So I would like to know what people think of the above ideas, if they are > good/bad, and/or suggest alternatives that we are not seeing right now. > > Ronny, please feel free to add to this email and correct anything I got > wrong. > > Cheers, > Bruno > _______________________________________________ > pytest-dev mailing list > pytest-dev(a)python.org > https://mail.python.org/mailman/listinfo/pytest-dev >

2 1

pytest 5.2.2
by Bruno Oliveira Oct. 25, 2019

Oct. 25, 2019

pytest 5.2.2 has just been released to PyPI. This is a bug-fix release, being a drop-in replacement. To upgrade:: pip install --upgrade pytest The full changelog is available at https://docs.pytest.org/en/latest/changelog.html. Thanks to all who contributed to this release, among them: * Albert Tugushev * Andrzej Klajnert * Anthony Sottile * Bruno Oliveira * Daniel Hahler * Florian Bruhin * Nattaphoom Chaipreecha * Oliver Bestwalter * Philipp Loose * Ran Benita * Victor Maryama * Yoav Caspi Happy testing, The pytest Development Team

1 0

Building the Rationale for upcoming changes in the node construction apis
by RonnyPfannschmidt Oct. 24, 2019

Oct. 24, 2019

Hi everyone, in https://github.com/pytest-dev/pytest/pull/5975 i have startet to prepare a POC with a ambitious goal, i want to remove much of the mind boggling smartness from the constructors and i want to streamline note constructors in order to reduce the coupling in the different types of nodes. The tight coupling and tricky construction of nodes has repeatedly foiled my attempts to refactor both fixtures/fixture caching and setupstate as well as actually introducing FunctionDefinition in the public api (and shifting generate_tests to it) and retiring Instance. So from my pov its not sustainable to leave them be, as they prevent other required improvements. However as far as i can tell its next to impossible without breaking things. The Rough idea i have for iteratively solving it is to a) introduce Node.from_parent as the convention for node construction b) deprecate Node.__init__ being invoked outside of such a function c) destroy all the things and have a cupcake Whats needed in addition is a slightly more eloquent version of the description for the documentation. -- Ronny

2 1

Re: [pytest-dev] Future of pytest-xdist/execnet?
by Anthony Sottile Oct. 24, 2019

Oct. 24, 2019

Just my 2c, I think I might would be valuable to create two new plugins: - one which handles local concurrency via multiprocessing (pipes, shmem, etc.) - one which handles remote concurrency (sockets, client/server, etc.) from a usage standpoint I expect the 99% to be the former suggested plug-in (for current usage of xdist). At least for me all of my (admittedly little) usage is of vanilla xdist on the local machine. I also think by separating concerns the two plugins could focus more on implementing their core functionality (and therefore do a better job of it). only if there's very clear identical behavior should we consider making a shared library or including in core. I definitely think there's a lot of opportunities here to improve on the status quo (and provide some significant value for those currently using xdist). I've collected a few (I don't want to call them complaints) words from a few heavy users (pip, cryptography to name a few) where I think a new approach could provide significant improvements while they're both essentially distributed systems problems, I think the local vs. remote plugins can probably take slightly different approaches/trade-offs on reliability (especially with local multiprocessing likely being much more reliable) I'd be happy to help work/help on this, though I probably don't have the cycles to properly "own" such an effort Anthony

1 0

Future of pytest-xdist/execnet?
by Bruno Oliveira Oct. 24, 2019

Oct. 24, 2019

Hi everyone, For some time now execnet has been in maintenance only mode, and even so very few people are willing to maintain it; lately just myself and I’m not a good choice given that I don’t know the codebase at all, plus I have tons on my plate already. This poses a problem because often we have bug reports or feature requests in pytest-xdist which would require fixing or improving execnet, and are currently left without solution. Ronny and I have on occasion discussed the possibility of changing execnet‘s backend , with the current contenders being multiprocessing for local distribution and mitogen for remote distribution. I would like to write down some thoughts and gather opinions from the mailing list members. multiprocessing Aside from not needing to maintain execnet anymore, being able to use multiprocessing would bring several benefits: - pytest -s would work just fine, because multiprocessing does not use stdout/stderr for communication. This is a often requested feature but which we can’t (with current expertise) implement on execnet. - It would automatically support anything that can be picked to be transmitted across the wire. Currently execnet does not support custom user types and some builtins (frozen sets for example). Without execnet we would lose the ability of running the tests in different interpreter versions, but I believe this has been supplanted by tox in the ecosystem. mitogen Using mitogen for remote execution would provide automatic bootstrap of the Python environment, which is not currently supported by xdist‘s ssh remote execution, greatly limiting its usefulness in real-time scenarios. This is Ronny’s idea and he can add more here if wanted. pytest-xdist2? All the changes mentioned above would break backward compatibility *hard*, because details of the execnet implementation are currently exposed to users in the command-line: pytest -d --tx popen//python And in several hooks: def pytest_xdist_newgateway(gateway): """ called on new raw gateway creation. """ (gateway is an execnet object) To incorporate the new backends, we would need to severely break both command-line and existing hooks. Given the level of breakage this could cause, just bumping the major version in this case doesn’t seem enough, it would be a disservice to users given that probably nobody pins pytest-xdist to <=2, and it would break the world with hard to understand error messages once a first major release was released. Because of this, we have been discussing creating a new package, pytest-xdist2 (other suggestions are welcome), without any backward compatibility guarantees with pytest-xdist. pytest-xdist2 would, at first, only support local test distribution using multiprocessing (-n flag), no new hooks, and no remote execution. I’ve made a proof of concept here: https://github.com/pytest-dev/pytest-xdist/pull/479 So it definitely seems possible. One immediate benefit is that the proof of concept above supports pytest -s already, and can transfer anything that’s pickled over the wire. So I would like to know what people think of the above ideas, if they are good/bad, and/or suggest alternatives that we are not seeing right now. Ronny, please feel free to add to this email and correct anything I got wrong. Cheers, Bruno

5 4

Fwd: Re: Future of pytest-xdist/execnet?
by RonnyPfannschmidt Oct. 24, 2019

Oct. 24, 2019

-------- Weitergeleitete Nachricht -------- Betreff: Re: [pytest-dev] Future of pytest-xdist/execnet? Datum: Thu, 24 Oct 2019 16:25:43 -0400 Von: pytest-dev-owner(a)python.org An: ich(a)ronnypfannschmidt.de Your message has been rejected, probably because you are not subscribed to the mailing list and the list's policy is to prohibit non-members from posting to it. If you think that your messages are being rejected in error, contact the mailing list owner at pytest-dev-owner(a)python.org.

1 0

[proposal] global and per item data stashes
by Ronny Pfannschmidt Oct. 16, 2019

Oct. 16, 2019

Problem: for the storage of helper objects that either tackle certain global tracking details or that track information over the testprotocol lifecycle of test items, we use monkeypatchs and generally just munge around in either the pytest config object or item objects. I would like to propose a new API putting those details into consideration. config.get_stash(owner, type=pytest.NamespaceStash) # valid over the pytest session lifetime item.get_stash(owner, type=pytest.NamespaceStash) # valid across the run-test protocol cycle of item Each stash would be a context manager and construction would get access to both item and config. This would for example enable to express the junit xml logger object and configuration as `config.get_stash(__name__, JunitXMlTracker)` Just as it would allow other plugin to store node lifecycle related details within the node using a well known mechanism Incidentally this would also be a good fit for the fixture system and setup-state in general to store information. The basic assumption being that only the stashes of the items that are currently active are valid, and other stashes are torn down, then fixtures as well as the fixture request would nicely fit that storage mechanism, and would also generalize across the node tree from session down to individual items. (considering the layout, it might be sensible to even replace config.get_stash with something like session.get_stash) The idea is still rather fuzzy in my head and i would love to di either text based brainstorming on it, or a actual video call. -- Ronny Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander

3 5

pytest 4.6.6
by Bruno Oliveira Oct. 13, 2019

Oct. 13, 2019

pytest 4.6.6 has just been released to PyPI. Backport of bug fixes for the last version to support Python 2.7 and 3.4, being a drop-in replacement. To upgrade:: pip install --upgrade pytest The full changelog is available at https://docs.pytest.org/en/4.6-maintenance/changelog.html Thanks to all who contributed to this release, among them: * Anthony Sottile * Bruno Oliveira * Michael Goerz Happy testing, The pytest Development Team

1 0

transferring ownership of pytest-ordering
by Frank Tobia Oct. 9, 2019

Oct. 9, 2019

Hi there, I wrote pytest-ordering but I haven't been doing a great job of maintaining it. I'd like to transfer ownership to the pytest-dev group if that's amenable. I think all the requirements for transferring are fulfilled but I'd appreciate calling out if I missed something ( https://github.com/pytest-dev/pytest/blob/master/CONTRIBUTING.rst#submittin… ). Here's the repo: https://github.com/ftobia/pytest-ordering Thanks in advance. -Frank

2 3

pytest 5.2.1
by Bruno Oliveira Oct. 6, 2019

Oct. 6, 2019

pytest 5.2.1 has just been released to PyPI. This is a bug-fix release, being a drop-in replacement. To upgrade:: pip install --upgrade pytest The full changelog is available at https://docs.pytest.org/en/latest/changelog.html. Thanks to all who contributed to this release, among them: * Anthony Sottile * Bruno Oliveira * Florian Bruhin * Hynek Schlawack * Kevin J. Foley * tadashigaki Happy testing, The pytest Development Team

1 0