[pytest-dev] Future of pytest-xdist/execnet?

[RonnyPfannschmidt]

Hi Freddy,


unfortunately pytest-concurrent is fundamentally broken for managing 
fixtures and other details,
as things are my suggestion is to avoid it.


-- Ronny

Am 24.10.19 um 15:22 schrieb Rietdijk:
> Hi,
>
> I think having a simple `pytest-multiprocessing` or 
> `pytest-concurrent` would be extremely useful and cover a lot of 
> people's use cases and would strongly recommend to have that as a 
> separate plugin from something bigger that does feature hooks and/or 
> remote execution.. It seems though as such a package already exists, 
> https://github.com/reverbc/pytest-concurrent, which offers --concmode 
> and --concworkers instead.
>
> Frederik
>
> On Thu, Oct 24, 2019 at 2:46 PM Bruno Oliveira <nicoddemus at gmail.com 
> <mailto:nicoddemus at gmail.com>> wrote:
>
>     Hi everyone,
>
>     For some time now |execnet| has been in maintenance only mode, and
>     even so very few people are willing to maintain it; lately just
>     myself and I’m not a good choice given that I don’t know the
>     codebase at all, plus I have tons on my plate already. This poses
>     a problem because often we have bug reports or feature requests in
>     |pytest-xdist| which would require fixing or improving |execnet|,
>     and are currently left without solution.
>
>     Ronny and I have on occasion discussed the possibility of changing
>     |execnet|‘s backend , with the current contenders being
>     |multiprocessing| for local distribution and |mitogen| for remote
>     distribution.
>
>     I would like to write down some thoughts and gather opinions from
>     the mailing list members.
>
>
>         multiprocessing
>
>     Aside from not needing to maintain |execnet| anymore, being able
>     to use |multiprocessing| would bring several benefits:
>
>      *
>
>         |pytest -s| would work just fine, because |multiprocessing|
>         does not use |stdout/stderr| for communication. This is a
>         often requested feature but which we can’t (with current
>         expertise) implement on |execnet|.
>
>      *
>
>         It would automatically support anything that can be picked to
>         be transmitted across the wire. Currently execnet does not
>         support custom user types and some builtins (frozen sets for
>         example).
>
>     Without |execnet| we would lose the ability of running the tests
>     in different interpreter versions, but I believe this has been
>     supplanted by |tox| in the ecosystem.
>
>
>         mitogen
>
>     Using |mitogen| for remote execution would provide automatic
>     bootstrap of the Python environment, which is not currently
>     supported by |xdist|‘s ssh remote execution, greatly limiting its
>     usefulness in real-time scenarios.
>
>     This is Ronny’s idea and he can add more here if wanted.
>
>
>         pytest-xdist2?
>
>     All the changes mentioned above would break backward compatibility
>     *hard*, because details of the |execnet| implementation are
>     currently exposed to users in the command-line:
>
>     |pytest -d --tx popen//python |
>
>     And in several hooks:
>
>     |def pytest_xdist_newgateway(gateway): """ called on new raw
>     gateway creation. """ |
>
>     (|gateway| is an |execnet| object)
>
>     To incorporate the new backends, we would need to severely break
>     both command-line and existing hooks. Given the level of breakage
>     this could cause, just bumping the major version in this case
>     doesn’t seem enough, it would be a disservice to users given that
>     probably nobody pins pytest-xdist to |<=2|, and it would break the
>     world with hard to understand error messages once a first major
>     release was released.
>
>     Because of this, we have been discussing creating a new package,
>     |pytest-xdist2| (other suggestions are welcome), without any
>     backward compatibility guarantees with |pytest-xdist|.
>
>     |pytest-xdist2| would, at first, only support local test
>     distribution using |multiprocessing| (|-n| flag), no new hooks,
>     and no remote execution. I’ve made a proof of concept here:
>
>     https://github.com/pytest-dev/pytest-xdist/pull/479
>
>     So it definitely seems possible. One immediate benefit is that the
>     proof of concept above supports |pytest -s| already, and can
>     transfer anything that’s pickled over the wire.
>
>     So I would like to know what people think of the above ideas, if
>     they are good/bad, and/or suggest alternatives that we are not
>     seeing right now.
>
>     Ronny, please feel free to add to this email and correct anything
>     I got wrong.
>
>     Cheers,
>     Bruno
>
>     _______________________________________________
>     pytest-dev mailing list
>     pytest-dev at python.org <mailto:pytest-dev at python.org>
>     https://mail.python.org/mailman/listinfo/pytest-dev
>
>
> _______________________________________________
> pytest-dev mailing list
> pytest-dev at python.org
> https://mail.python.org/mailman/listinfo/pytest-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20191024/0dc14137/attachment.html>