<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hi Freddy,</p>
<p><br>
</p>
<p>unfortunately pytest-concurrent is fundamentally broken for
managing fixtures and other details,<br>
as things are my suggestion is to avoid it.</p>
<p><br>
</p>
<p>-- Ronny<br>
</p>
<div class="moz-cite-prefix">Am 24.10.19 um 15:22 schrieb
Rietdijk:<br>
</div>
<blockquote type="cite"
cite="mid:CAOQtOH2UmVAvGh0bW+ATtpWYFJ0v8=gZj8yF1eeYmXoEkjcPKA@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div>Hi,</div>
<div><br>
</div>
<div>I think having a simple `pytest-multiprocessing` or
`pytest-concurrent` would be extremely useful and cover a lot
of people's use cases and would strongly recommend to have
that as a separate plugin from something bigger that does
feature hooks and/or remote execution.. It seems though as
such a package already exists, <a
href="https://github.com/reverbc/pytest-concurrent"
target="_blank" moz-do-not-send="true">https://github.com/reverbc/pytest-concurrent</a>,
which offers --concmode and --concworkers instead.<br>
</div>
<div><br>
</div>
<div>Frederik</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Oct 24, 2019 at 2:46
PM Bruno Oliveira <<a href="mailto:nicoddemus@gmail.com"
moz-do-not-send="true">nicoddemus@gmail.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>
<p style="margin:1.2em 0px">Hi everyone,</p>
<p style="margin:1.2em 0px">For some time now <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">execnet</code>
has been in maintenance only mode, and even so very few
people are willing to maintain it; lately just myself
and I’m not a good choice given that I don’t know the
codebase at all, plus I have tons on my plate already.
This poses a problem because often we have bug reports
or feature requests in <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">pytest-xdist</code>
which would require fixing or improving <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">execnet</code>,
and are currently left without solution.</p>
<p style="margin:1.2em 0px">Ronny and I have on occasion
discussed the possibility of changing <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">execnet</code>‘s
backend , with the current contenders being <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">multiprocessing</code>
for local distribution and <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">mitogen</code>
for remote distribution.</p>
<p style="margin:1.2em 0px">I would like to write down
some thoughts and gather opinions from the mailing list
members.</p>
<h2 id="gmail-m_554320163752560948multiprocessing"
style="margin:1.3em 0px
1em;padding:0px;font-weight:bold;font-size:1.4em;border-bottom:1px
solid rgb(238,238,238)">multiprocessing</h2>
<p style="margin:1.2em 0px">Aside from not needing to
maintain <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">execnet</code>
anymore, being able to use <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">multiprocessing</code>
would bring several benefits:</p>
<ul style="margin:1.2em 0px;padding-left:2em">
<li style="margin:0.5em 0px">
<p style="margin:0.5em 0px"><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">pytest -s</code>
would work just fine, because <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">multiprocessing</code>
does not use <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">stdout/stderr</code>
for communication. This is a often requested feature
but which we can’t (with current expertise)
implement on <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">execnet</code>.</p>
</li>
<li style="margin:0.5em 0px">
<p style="margin:0.5em 0px">It would automatically
support anything that can be picked to be
transmitted across the wire. Currently execnet does
not support custom user types and some builtins
(frozen sets for example).</p>
</li>
</ul>
<p style="margin:1.2em 0px">Without <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">execnet</code>
we would lose the ability of running the tests in
different interpreter versions, but I believe this has
been supplanted by <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">tox</code>
in the ecosystem.</p>
<h2 id="gmail-m_554320163752560948mitogen"
style="margin:1.3em 0px
1em;padding:0px;font-weight:bold;font-size:1.4em;border-bottom:1px
solid rgb(238,238,238)">mitogen</h2>
<p style="margin:1.2em 0px">Using <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">mitogen</code>
for remote execution would provide automatic bootstrap
of the Python environment, which is not currently
supported by <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">xdist</code>‘s
ssh remote execution, greatly limiting its usefulness in
real-time scenarios.</p>
<p style="margin:1.2em 0px">This is Ronny’s idea and he
can add more here if wanted.</p>
<h2 id="gmail-m_554320163752560948pytest-xdist2-"
style="margin:1.3em 0px
1em;padding:0px;font-weight:bold;font-size:1.4em;border-bottom:1px
solid rgb(238,238,238)">pytest-xdist2?</h2>
<p style="margin:1.2em 0px">All the changes mentioned
above would break backward compatibility <strong>hard</strong>,
because details of the <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">execnet</code>
implementation are currently exposed to users in the
command-line:</p>
<pre style="font-family:Consolas,Inconsolata,Courier,monospace;font-size:1em;line-height:1.2em;margin:1.2em 0px"><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;background-color:rgb(248,248,248);white-space:pre-wrap;overflow:auto;border-radius:3px;border:1px solid rgb(204,204,204);padding:0.5em 0.7em;display:block">pytest -d --tx popen//python
</code></pre>
<p style="margin:1.2em 0px">And in several hooks:</p>
<pre style="font-family:Consolas,Inconsolata,Courier,monospace;font-size:1em;line-height:1.2em;margin:1.2em 0px"><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;white-space:pre-wrap;overflow:auto;border-radius:3px;border:1px solid rgb(204,204,204);display:block;padding:0.5em;background:rgb(35,36,31) none repeat scroll 0% 0%;color:rgb(248,248,242)"><span style="color:rgb(249,38,114)"><span style="color:rgb(102,217,239)">def</span> <span style="color:rgb(166,226,46)">pytest_xdist_newgateway</span><span style="color:rgb(248,248,242)">(gateway)</span>:</span>
<span style="color:rgb(230,219,116)">""" called on new raw gateway creation. """</span>
</code></pre>
<p style="margin:1.2em 0px">(<code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">gateway</code>
is an <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">execnet</code>
object)</p>
<p style="margin:1.2em 0px">To incorporate the new
backends, we would need to severely break both
command-line and existing hooks. Given the level of
breakage this could cause, just bumping the major
version in this case doesn’t seem enough, it would be a
disservice to users given that probably nobody pins
pytest-xdist to <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline"><=2</code>,
and it would break the world with hard to understand
error messages once a first major release was released.</p>
<p style="margin:1.2em 0px">Because of this, we have been
discussing creating a new package, <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">pytest-xdist2</code>
(other suggestions are welcome), without any backward
compatibility guarantees with <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">pytest-xdist</code>.</p>
<p style="margin:1.2em 0px"><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">pytest-xdist2</code>
would, at first, only support local test distribution
using <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">multiprocessing</code>
(<code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">-n</code>
flag), no new hooks, and no remote execution. I’ve made
a proof of concept here:</p>
<p style="margin:1.2em 0px"><a
href="https://github.com/pytest-dev/pytest-xdist/pull/479"
target="_blank" moz-do-not-send="true">https://github.com/pytest-dev/pytest-xdist/pull/479</a>
</p>
<p style="margin:1.2em 0px">So it definitely seems
possible. One immediate benefit is that the proof of
concept above supports <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">pytest -s</code>
already, and can transfer anything that’s pickled over
the wire.</p>
<p style="margin:1.2em 0px">So I would like to know what
people think of the above ideas, if they are good/bad,
and/or suggest alternatives that we are not seeing right
now.</p>
<p style="margin:1.2em 0px">Ronny, please feel free to add
to this email and correct anything I got wrong.</p>
<p style="margin:1.2em 0px">Cheers,<br>
Bruno</p>
</div>
</div>
_______________________________________________<br>
pytest-dev mailing list<br>
<a href="mailto:pytest-dev@python.org" target="_blank"
moz-do-not-send="true">pytest-dev@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/pytest-dev"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://mail.python.org/mailman/listinfo/pytest-dev</a><br>
</blockquote>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
pytest-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:pytest-dev@python.org">pytest-dev@python.org</a>
<a class="moz-txt-link-freetext" href="https://mail.python.org/mailman/listinfo/pytest-dev">https://mail.python.org/mailman/listinfo/pytest-dev</a>
</pre>
</blockquote>
</body>
</html>