bizarre behavior with wheels and namespace packages
Not sure if this belongs on distutils-sig or import-sig, but I'm experiencing slightly odd behavior with wheels and namespace packages. Fetch a namespace package source distribution, e.g. twitter.common.python, and build two copies of it, one with bdist_egg and one with bdist_wheel. This leaves me with: twitter.common.python-0.3.1-py2.6.egg twitter.common.python-0.3.1-py26-none-any.whl Explode them into the following directories (respectively): egg_dist whl_dist Then attempt: PYTHONPATH=egg_dist python -c 'import twitter.common.python' (works) PYTHONPATH=whl_dist python -c 'import twitter.common.python' (drumroll) Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: No module named twitter.common.python This is slightly unexpected. Absent from the .whl distribution are the __init__.pys with the proper declare_namespace declarations. Even more bizarrely: PYTHONPATH=whl_dist python -c 'import pkg_resources' Traceback (most recent call last): File "<string>", line 1, in <module> File "/Users/wickman/Python/CPython-2.6.9/lib/python2.6/site-packages/pkg_resources.py", line 2736, in <module> add_activation_listener(lambda dist: dist.activate()) File "/Users/wickman/Python/CPython-2.6.9/lib/python2.6/site-packages/pkg_resources.py", line 698, in subscribe callback(dist) File "/Users/wickman/Python/CPython-2.6.9/lib/python2.6/site-packages/pkg_resources.py", line 2736, in <lambda> add_activation_listener(lambda dist: dist.activate()) File "/Users/wickman/Python/CPython-2.6.9/lib/python2.6/site-packages/pkg_resources.py", line 2274, in activate list(map(declare_namespace, self._get_metadata('namespace_packages.txt'))) File "/Users/wickman/Python/CPython-2.6.9/lib/python2.6/site-packages/pkg_resources.py", line 1870, in declare_namespace path = sys.modules[parent].__path__ KeyError: 'twitter' So even pkg_resources is surprised as well. If I do the following: PYTHONPATH=whl_dist python -c 'import site; site.addsitedir("whl_dist"); import twitter.common.python' then it works, since the twitter.common.python-0.3.1-py2.6-nspkg.pth gets processed properly and module.__path__s populated accordingly. I would expect pkg_resources to handle this fine but apparently it doesn't. Any ideas? Bug in pkg_resources, wheel or PEBKAC? Thanks! ~brian
More data:
This works fine with python 3
PYTHONPATH=whl_dist python3 -c 'import twitter.common.python'
but pkg_resource still barfs
PYTHONPATH=whl_dist python3 -c 'import pkg_resources'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "<frozen importlib._bootstrap>", line 1565, in _find_and_load
File "<frozen importlib._bootstrap>", line 1532, in
_find_and_load_unlocked
File
"/Users/wickman/Python/CPython-3.3.3/lib/python3.3/site-packages/setuptools-2.0.2-py3.3.egg/pkg_resources.py",
line 2725, in <module>
File
"/Users/wickman/Python/CPython-3.3.3/lib/python3.3/site-packages/setuptools-2.0.2-py3.3.egg/pkg_resources.py",
line 681, in subscribe
File
"/Users/wickman/Python/CPython-3.3.3/lib/python3.3/site-packages/setuptools-2.0.2-py3.3.egg/pkg_resources.py",
line 2725, in <lambda>
File
"/Users/wickman/Python/CPython-3.3.3/lib/python3.3/site-packages/setuptools-2.0.2-py3.3.egg/pkg_resources.py",
line 2259, in activate
File
"/Users/wickman/Python/CPython-3.3.3/lib/python3.3/site-packages/setuptools-2.0.2-py3.3.egg/pkg_resources.py",
line 1843, in declare_namespace
KeyError: 'twitter'
I did find a related wheel issue:
https://bitbucket.org/pypa/wheel/issue/46/delete-namespace-__init__py-based-...
But this seems to describe the opposite behavior, i.e. that the installer
should be deleting __init__.pys instead of creating them.
In the meantime, my two options are building an installer that populates
__init__.pys, or instead just calling site.addsitedir each time I call
dist.activate(). I'm guessing the latter is preferable as it does not
unnecessarily introduce a pkg_resources dependency.
~brian
On Fri, Mar 7, 2014 at 8:15 PM, Brian Wickman
Not sure if this belongs on distutils-sig or import-sig, but I'm experiencing slightly odd behavior with wheels and namespace packages.
Fetch a namespace package source distribution, e.g. twitter.common.python, and build two copies of it, one with bdist_egg and one with bdist_wheel.
This leaves me with: twitter.common.python-0.3.1-py2.6.egg twitter.common.python-0.3.1-py26-none-any.whl
Explode them into the following directories (respectively): egg_dist whl_dist
Then attempt:
PYTHONPATH=egg_dist python -c 'import twitter.common.python'
(works)
PYTHONPATH=whl_dist python -c 'import twitter.common.python'
(drumroll)
Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: No module named twitter.common.python
This is slightly unexpected. Absent from the .whl distribution are the __init__.pys with the proper declare_namespace declarations. Even more bizarrely:
PYTHONPATH=whl_dist python -c 'import pkg_resources' Traceback (most recent call last): File "<string>", line 1, in <module> File "/Users/wickman/Python/CPython-2.6.9/lib/python2.6/site-packages/pkg_resources.py", line 2736, in <module> add_activation_listener(lambda dist: dist.activate()) File "/Users/wickman/Python/CPython-2.6.9/lib/python2.6/site-packages/pkg_resources.py", line 698, in subscribe callback(dist) File "/Users/wickman/Python/CPython-2.6.9/lib/python2.6/site-packages/pkg_resources.py", line 2736, in <lambda> add_activation_listener(lambda dist: dist.activate()) File "/Users/wickman/Python/CPython-2.6.9/lib/python2.6/site-packages/pkg_resources.py", line 2274, in activate list(map(declare_namespace, self._get_metadata('namespace_packages.txt'))) File "/Users/wickman/Python/CPython-2.6.9/lib/python2.6/site-packages/pkg_resources.py", line 1870, in declare_namespace path = sys.modules[parent].__path__ KeyError: 'twitter'
So even pkg_resources is surprised as well. If I do the following:
PYTHONPATH=whl_dist python -c 'import site; site.addsitedir("whl_dist"); import twitter.common.python'
then it works, since the twitter.common.python-0.3.1-py2.6-nspkg.pth gets processed properly and module.__path__s populated accordingly. I would expect pkg_resources to handle this fine but apparently it doesn't.
Any ideas? Bug in pkg_resources, wheel or PEBKAC?
Thanks! ~brian
participants (1)
-
Brian Wickman