[py-dev] pytest-xdist limitations in interactions between different versions of python

[meme dough]

Hi Holger,

I have a few questions for py.test with xdist and I'm not sure if
these are known and I missed the something.

When I kick off py.test with dist=each mode I've noticed 3 limitations
in interactions between different versions of python.

First if start py.test with python2.x then can specify any 2.x txes,
so the following is fine:

py.test --tb=short -r sfxX --dist=each
--tx=popen//python=/usr/local/python246/bin/python
--tx=popen//python=/usr/local/python255/bin/python
--tx=popen//python=/usr/local/python265/bin/python

However problems if txes for 3.0:

py.test -v --tb=short -r sfxX --dist=each
--tx=popen//python=/usr/local/python246/bin/python
--tx=popen//python=/usr/local/python255/bin/python
--tx=popen//python=/usr/local/python265/bin/python
--tx=popen//python=/usr/local/python301/bin/python
========================= test session starts =========================
platform linux2 -- Python 2.6.4 -- pytest-1.3.2 -- /usr/bin/python
test path 1: /home/memedough/work/projects/pytest-cov
[gw0] popen//python=/usr/local/python246/bin/python -- platform
linux2, Python 2.4.6-final-0 cwd:
/home/memedough/work/projects/pytest-cov-
/usr/local/python246/bin/python
[gw1] popen//python=/usr/local/python255/bin/python -- platform
linux2, Python 2.5.5-final-0 cwd:
/home/memedough/work/projects/pytest-cov-
/usr/local/python255/bin/python
[gw2] popen//python=/usr/local/python265/bin/python -- platform
linux2, Python 2.6.5-final-0 cwd:
/home/memedough/work/projects/pytest-cov-
/usr/local/python265/bin/python
[gw3] popen//python=/usr/local/python301/bin/python -- platform
linux2, Python 3.0.1-final-0 cwd:
/home/memedough/work/projects/pytest-cov-
/usr/local/python301/bin/python
[master] starting full item collection ...
[master] collected 7 items
[gw0] txnode ready to receive tests
[gw3] node down, error:
[gw2] txnode ready to receive tests
[gw1] txnode ready to receive tests

Then I have to ctrl-c.

Also problems if txes for 3.1:

py.test -v --tb=short -r sfxX --dist=each
--tx=popen//python=/usr/local/python246/bin/python
--tx=popen//python=/usr/local/python255/bin/python
--tx=popen//python=/usr/local/python265/bin/python
--tx=popen//python=/usr/local/python312/bin/python
========================= test session starts =========================
platform linux2 -- Python 2.6.4 -- pytest-1.3.2 -- /usr/bin/python
test path 1: /home/memedough/work/projects/pytest-cov
[gw0] popen//python=/usr/local/python246/bin/python -- platform
linux2, Python 2.4.6-final-0 cwd:
/home/memedough/work/projects/pytest-cov-
/usr/local/python246/bin/python
[gw1] popen//python=/usr/local/python255/bin/python -- platform
linux2, Python 2.5.5-final-0 cwd:
/home/memedough/work/projects/pytest-cov-
/usr/local/python255/bin/python
[gw2] popen//python=/usr/local/python265/bin/python -- platform
linux2, Python 2.6.5-final-0 cwd:
/home/memedough/work/projects/pytest-cov-
/usr/local/python265/bin/python
[gw3] popen//python=/usr/local/python312/bin/python -- platform
linux2, Python 3.1.2-final-0 cwd:
/home/memedough/work/projects/pytest-cov-
/usr/local/python312/bin/python
[master] starting full item collection ...
[master] collected 7 items
[gw3] node down, error:

==========================  in 1.79 seconds ===========================

I didn't have to ctrl-c then, it varies I think.


Second, if start py.test with python 3.0 then can specify 3.0 / 3.1,
so the following is fine:

py.test --tb=short -r sfxX --dist=each
--tx=popen//python=/usr/local/python301/bin/python
--tx=popen//python=/usr/local/python312/bin/python

However problems if txes for 2.x:

py.test -v --tb=short -r sfxX --dist=each
--tx=popen//python=/usr/local/python301/bin/python
--tx=popen//python=/usr/local/python265/bin/python
=========================== test session starts ===========================
platform linux2 -- Python 3.0.1 -- pytest-1.3.2 --
/usr/local/python301/bin/python
test path 1: /home/memedough/work/projects/pytest-cov
[gw0] popen//python=/usr/local/python301/bin/python -- platform
linux2, Python 3.0.1-final-0 cwd:
/home/memedough/work/projects/pytest-cov-
/usr/local/python301/bin/python
[gw1] popen//python=/usr/local/python265/bin/python -- platform
linux2, Python 2.6.5-final-0 cwd:
/home/memedough/work/projects/pytest-cov-
/usr/local/python265/bin/python
[master] starting full item collection ...
[master] collected 7 items
[gw1] node down, error: Traceback (most recent call last):
  File "/usr/local/python301/lib/python3.0/site-packages/execnet-1.0.7-py3.0.egg/execnet/gateway_base.py",
line 725, in executetask
    do_exec(co, loc)
  File "<string>", line 1, in do_exec
  File "", line 9, in <module>
  File "/usr/local/python301/lib/python3.0/site-packages/pytest_xdist-1.4-py3.0.egg/xdist/mypickle.py",
line 142, in receive
    return self._unpickle(pickled_obj)
  File "/usr/local/python301/lib/python3.0/site-packages/pytest_xdist-1.4-py3.0.egg/xdist/mypickle.py",
line 147, in _unpickle
    return self._ipickle.loads(pickled_obj)
  File "/usr/local/python301/lib/python3.0/site-packages/pytest_xdist-1.4-py3.0.egg/xdist/mypickle.py",
line 103, in loads
    res = unpickler.load()
  File "/usr/local/python265/lib/python2.6/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/local/python265/lib/python2.6/pickle.py", line 1090, in load_global
    klass = self.find_class(module, name)
  File "/usr/local/python265/lib/python2.6/pickle.py", line 1124, in find_class
    __import__(module)
ImportError: No module named copyreg


============================  in 0.29 seconds =============================


Third, if start py.test with python 3.1 then can specify 3.1, so the
following is fine:

py.test --tb=short -r sfxX --dist=each
--tx=popen//python=/usr/local/python312/bin/python
--tx=popen//python=/usr/local/python312/bin/python

However problems if txes for 3.0 or 2.x:

py.test -v --tb=short -r sfxX --dist=each
--tx=popen//python=/usr/local/python312/bin/python
--tx=popen//python=/usr/local/python301/bin/python
========================== test session starts ==========================
platform linux2 -- Python 3.1.2 -- pytest-1.3.2 --
/usr/local/python312/bin/python
test path 1: /home/memedough/work/projects/pytest-cov
[gw0] popen//python=/usr/local/python312/bin/python -- platform
linux2, Python 3.1.2-final-0 cwd:
/home/memedough/work/projects/pytest-cov-
/usr/local/python312/bin/python
[gw1] popen//python=/usr/local/python301/bin/python -- platform
linux2, Python 3.0.1-final-0 cwd:
/home/memedough/work/projects/pytest-cov-
/usr/local/python301/bin/python
[master] starting full item collection ...
/usr/local/python312/lib/python3.1/site-packages/py-1.3.2-py3.1.egg/py/_test/pluginmanager.py:100:
UserWarning: Module pytest_cov_init was already imported from
/usr/local/python312/lib/python3.1/site-packages/pytest_cov-0.16a1-py3.1.egg/pytest_cov_init.py,
but /home/memedough/work/projects/pytest-cov is being added to
sys.path
  from pkg_resources import iter_entry_points
[master] collected 7 items
[gw1] node down, error: Traceback (most recent call last):
  File "/usr/local/python312/lib/python3.1/site-packages/execnet-1.0.7-py3.1.egg/execnet/gateway_base.py",
line 725, in executetask
    do_exec(co, loc)
  File "<string>", line 1, in do_exec
  File "", line 9, in <module>
  File "/usr/local/python312/lib/python3.1/site-packages/pytest_xdist-1.4-py3.1.egg/xdist/mypickle.py",
line 142, in receive
    return self._unpickle(pickled_obj)
  File "/usr/local/python312/lib/python3.1/site-packages/pytest_xdist-1.4-py3.1.egg/xdist/mypickle.py",
line 147, in _unpickle
    return self._ipickle.loads(pickled_obj)
  File "/usr/local/python312/lib/python3.1/site-packages/pytest_xdist-1.4-py3.1.egg/xdist/mypickle.py",
line 103, in loads
    res = unpickler.load()
  File "/usr/local/python301/lib/python3.0/pickle.py", line 813, in load
    dispatch[key[0]](self)
  File "/usr/local/python301/lib/python3.0/pickle.py", line 1058, in load_global
    klass = self.find_class(module, name)
  File "/usr/local/python301/lib/python3.0/pickle.py", line 1092, in find_class
    __import__(module, level=0)
ImportError: No module named copy_reg


===========================  in 0.34 seconds ============================

The 2.x problem is:

py.test -v --tb=short -r sfxX --dist=each
--tx=popen//python=/usr/local/python312/bin/python
--tx=popen//python=/usr/local/python265/bin/python
========================== test session starts ==========================
platform linux2 -- Python 3.1.2 -- pytest-1.3.2 --
/usr/local/python312/bin/python
test path 1: /home/memedough/work/projects/pytest-cov
[gw0] popen//python=/usr/local/python312/bin/python -- platform
linux2, Python 3.1.2-final-0 cwd:
/home/memedough/work/projects/pytest-cov-
/usr/local/python312/bin/python
[gw1] popen//python=/usr/local/python265/bin/python -- platform
linux2, Python 2.6.5-final-0 cwd:
/home/memedough/work/projects/pytest-cov-
/usr/local/python265/bin/python
[master] starting full item collection ...
/usr/local/python312/lib/python3.1/site-packages/py-1.3.2-py3.1.egg/py/_test/pluginmanager.py:100:
UserWarning: Module pytest_cov_init was already imported from
/usr/local/python312/lib/python3.1/site-packages/pytest_cov-0.16a1-py3.1.egg/pytest_cov_init.py,
but /home/memedough/work/projects/pytest-cov is being added to
sys.path
  from pkg_resources import iter_entry_points
[master] collected 7 items
[gw0] txnode ready to receive tests
[gw1] node down, error: pickled_obj = "(S'slaveready'\np162\n(t(dp164\ntp166\n."

    def unpickle_callback(pickled_obj):
        if pickled_obj is uniqueendmarker:
            return callback(endmarker)
        try:
>           obj = self._unpickle(pickled_obj)

callback   = <bound method TXNode.callback of <TXNode 'gw1' down=false>>
endmarker  = -1
pickled_obj = "(S'slaveready'\np162\n(t(dp164\ntp166\n."
self       = <xdist.mypickle.PickleChannel object at 0x2d1da90>
uniqueendmarker = <object object at 0x2cae610>

/usr/local/python312/lib/python3.1/site-packages/pytest_xdist-1.4-py3.1.egg/xdist/mypickle.py:173:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <xdist.mypickle.PickleChannel object at 0x2d1da90>
pickled_obj = "(S'slaveready'\np162\n(t(dp164\ntp166\n."

    def _unpickle(self, pickled_obj):
        if isinstance(pickled_obj, self._channel.__class__):
            return pickled_obj
>       return self._ipickle.loads(pickled_obj)

pickled_obj = "(S'slaveready'\np162\n(t(dp164\ntp166\n."
self       = <xdist.mypickle.PickleChannel object at 0x2d1da90>

/usr/local/python312/lib/python3.1/site-packages/pytest_xdist-1.4-py3.1.egg/xdist/mypickle.py:147:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <xdist.mypickle.ImmutablePickler object at 0x2d1da50>
string = "(S'slaveready'\np162\n(t(dp164\ntp166\n."

    def loads(self, string):
>       f = py.io.BytesIO(string)
E       TypeError: 'str' does not support the buffer interface

self       = <xdist.mypickle.ImmutablePickler object at 0x2d1da50>
string     = "(S'slaveready'\np162\n(t(dp164\ntp166\n."

/usr/local/python312/lib/python3.1/site-packages/pytest_xdist-1.4-py3.1.egg/xdist/mypickle.py:100:
TypeError

I have to ctrl-c then.

So:

If 2.x then can have tx for 2.x but problems if tx for 3.x.

If 3.0 then can have tx for 3.0 / 3.1 but problems if tx for 2.x.

If 3.1 then can have tx for 3.1 but problems if tx for 2.x / 3.0.

Do I have something wrong in my env or is it normal?

:)