Mailman 3 March 2014 - Distutils-SIG

nspkg.pth files break $PYTHONPATH overrides
by Barry Warsaw 27 Mar '14

27 Mar '14

Apologies for cross-posting, but this intersects setuptools and the import system, and I wanted to be sure it reached the right audience. A colleague asked me why a seemingly innocent and common use case for developing local versions of system installed packages wasn't working, and I was quite perplexed. As I dug into the problem, more questions than answers came up. I finally (think! I) figured out what is happening, but not so much as to why, or what can/should be done about it. This person had a local checkout of a package's source, where the package was also installed into the system Python. He wanted to be able to set $PYTHONPATH so that the local package wins when he tries to import it. E.g.: % PYTHONPATH=`pwd`/src python3 but this didn't work because despite the setting of PYTHONPATH, the system version of the package was always found first. The package in question is lazr.uri, although other packages with similar layouts will also suffer the same problem, which prevents an easy local development of a newer version of the package, aside from being a complete head-scratcher. The lazr.uri package is intended to be a submodule of the lazr namespace package. As such, the lazr/__init__.py has the old style way of declaring a namespace package: try: import pkg_resources pkg_resources.declare_namespace(__name__) except ImportError: import pkgutil __path__ = pkgutil.extend_path(__path__, __name__) and its setup.py declares a namespace package: setup( name='lazr.uri', version=__version__, namespace_packages=['lazr'], ... One of the things that the Debian "helper" program does when it builds a package for the archive is call `$python setup.py install_egg_info`. It's this command that breaks $PYTHONPATH overriding. install_egg_info looks at the lazr.uri.egg-info/namespace_packages.txt file, in which it finds the string 'lazr', and it proceeds to write a lazr-uri-1.0.3-py3.4-nspkg.pth file. This causes other strange and unexpected things to happen: % python3 Python 3.4.0 (default, Mar 22 2014, 22:51:25) [GCC 4.8.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.modules['lazr'] <module 'lazr'> >>> sys.modules['lazr'].__path__ ['/usr/lib/python3/dist-packages/lazr'] It's completely weird that sys.modules would contain a key for 'lazr' when that package was never explicitly imported. Even stranger, because a fake module object is stuffed into sys.modules via the .pth file, tracing imports with -v gives you no clue as to what's happening. And while sys.modules['lazr'] has an __path__, it has no other attributes. I really don't understand what the purpose of the nspkg.pth file is, especially for Python 3 namespace packages. Here's what the nspkg.pth file contains: import sys,types,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('lazr',)); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('lazr',types.ModuleType('lazr')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p) The __path__ value is important here because even though you've never explicitly imported 'lazr', when you *do* explicitly import 'lazr.uri', the existing lazr module object's __path__ takes over, and thus the system lazr.uri package is found even though both lazr/ and lazr/uri/ should have been found earlier on sys.path (yes, sys.path looks exactly as expected). So the presence of the nspkg.pth file breaks $PYTHONPATH overriding. That seems bad. ;) If you delete the nspkg.path file, then things work as expected, but even this is a little misleading! I think the Debian helper is running install_egg_info as a way to determine what namespace packages are defined, so that it can actually *remove* the parent's __init__.py file and use PEP 420 style namespace packages. In fact, in the Debian python3-lazr.uri binary package, you find no system lazr/__init__.py file. This is why removing the nspkg.pth file works. So I thought, why not conditionally define setup(..., namespace_packages) only for Python 2? This doesn't work because the Debian helper will see that no namespace packages are defined, and thus it will leave the original lazr/__init__.py file in place. This then breaks $PYTHONPATH overriding too because of __path__ extension of the pre-PEP 420 code only *appends* the local development path. IOW, the system import path is the first element of a 2-element list on lazr.__path__. While the local import path is the second element, in this case too the local import fails. It seems like what you want for Python 3 (and we're talking >= 3.2 here) is for there to be neither a nspkg.pth file, nor the lazr/__init__.py file, and let PEP 420 do it's thing. In fact if you set things up this way, $PYTHONPATH overriding works exactly as expected. Because I don't know why install_egg_info is installing the nspkg.pth file, I don't know which component needs to be changed: * Change setuptools install_egg_info command to not install an nspkg.pth file even for namespace_package declare packages, at least under Python 3. This behavior seems pretty nasty all by itself because it magically and untraceably installs stripped down module objects in sys.modules when Python first scans the import path. * Change the Debian helper to remove the nspkg.pth file, or not call install_egg_info *and* continue to remove <nspkg>/__init__.py in Python 3 so as to take advantage of PEP 420. It's nice to know that PEP 420 actually represents something sane. :) For added bonus, we have this additional oddity: % PYTHONPATH=`pwd`/src python3 Python 3.4.0 (default, Mar 22 2014, 22:51:25) [GCC 4.8.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.modules['lazr'] <module 'lazr'> >>> sys.modules['lazr'].__path__ ['/usr/lib/python3/dist-packages/lazr'] >>> import lazr.uri >>> lazr.uri.__file__ '/usr/lib/python3/dist-packages/lazr/uri/__init__.py' >>> sys.modules['lazr'] <module 'lazr' from '/home/barry/projects/ubuntu/lazruri/trusty/src/lazr/__init__.py'> >>> sys.modules['lazr'].__path__ ['/home/barry/projects/ubuntu/lazruri/trusty/src/lazr', '/usr/lib/python3/dist-packages/lazr'] Notice how importing lazr.uri *replaces* sys.modules['lazr'] with the local development one, even though it still imports lazr.uri from the system path. I'm not exactly sure how this happens, but I've traced that to _LoaderBasics.exec_module()'s call of _call_with_frames_removed(), which exec's lazr.uri's code object into that module's __dict__. Nothing in lazr/uri/__init__.py should be doing that, afaict from both visual inspection of the code and disassembling the compiled code object. Hopefully I've explained the situation correctly and lucidly. Below I'll describe how to set up a reproducible environment on a Debian machine. Thoughts and comments are welcome! Cheers, -Barry % sudo apt-get install python3-lazr.uri % cd tmp % bzr branch lp:lazr.uri trunk % cd trunk % PYTHONPATH=`pwd`/src python3 (Then try things at the Python prompt from above.)

5 8

tourist here, with a dumb RTFM question
by Michael Bayer 27 Mar '14

27 Mar '14

Hi Distutils ! I don’t follow this list and haven’t looked at it in a long time. However, I’m learning via twitter that a brand new setuptools release that’s just gone out has just removed the “Feature” mechanism. Now as you’re all rolling your eyes and preparing to bang out frustrated replies how this was well announced and deprecated and warned about and wow really didn’t you know the term paper was due today, OK first off let me say I am sorry! I am not a distutils/setuptools maintainer. I am just someone that uses it, and as I include setuptools in my setup.py, I am also getting thousands of other people who download my product to use it as well. And I don’t read the setuptools changelog! Or the setuptools blog. Or this list. I assume you guys have it under control (and you certainly do!). There seem to be other people like me (people who write very widely used software) who also seem surprised! And that is surprising too (as I am usually the only person to be surprised by these things that should not be surprising). So I hope you can hold back your frustration with my clueless RTFMness long enough to answer these questions: 1. where was this announced? I’m wondering why there weren’t loud blaring blog posts and tweets all over the place stating that on March 6, dozens of major packages are going to all break. 2. what is the plan for unmaintained packages and old versions of existing packages on pypi that “import Feature” and can no longer be installed? Is it just the case that a large amount of pypi just won’t install anymore? 3. What, if any, is the recommended approach going forward to a Python package that wishes to specify a command-line flag during install. Here’s multiple choice: a. you can use new setuptools API <some new way to add —flags> b. you can roll it yourself in setup.py using <hack X> c. your setup.py should never accept any kind of flags as that interferes with <up-and-coming use case Q> d. other If choice c., then here is another question. My library includes optional C extensions. On certain platforms, these C extensions don’t build (like on Pypy, or on Windows if you don’t have compilation tools installed). In this regard it gracefully degrades. But also, I want to be able to have a command line option to determine this as well. Because! Maybe I’m installing within a test suite where I need to test the entire library without the C extensions built. Maybe some user has found a bug in the C extensions, and that user needs to temporarily install the tool without the extensions built. Other cases for flags are, maybe your library can build with or without SSL support, something like that. Keep in mind, I actually *won’t* be getting bug reports about this because my setup.py already gracefully degrades to distutils! But still, I’d like to have my —without-cextensions flag somehow. Thanks for listening and again I apologize for not following the setuptools changelog on a regular basis! - mike

8 22

setuptools tracker
by "Martin v. Löwis" 24 Mar '14

24 Mar '14

We are still hosting a roundup installation for setuptools, at http://bugs.python.org/setuptools/. Is this still needed? If not: what should we do with it? Regards, Martin

3 6

waf can be used to compile Python extension modules
by Daniel Holth 24 Mar '14

24 Mar '14

One reason people use distutils is because it can compile Python extension modules. Here's an example from waf that compiles Python extensions without using distutils. The problem of invoking the compiler has been solved by many build systems and is not just a unique and mysterious distutils feature. https://code.google.com/p/waf/source/browse/#git%2Fdemos%2Fpython

9 26

PEP464 - Removal of the PyPI Mirror Authenticity API
by Donald Stufft 21 Mar '14

21 Mar '14

Hello! I’d like to propose PEP464, the removal of the PyPI Mirror Authenticity API which was originally described in PEP381. The text of the PEP is below, or it can be viewed online at https://python.org/dev/peps/pep-0464/ PEP: 464 Title: Removal of the PyPI Mirror Authenticity API Version: $Revision$ Last-Modified: $Date$ Author: Donald Stufft <donald(a)stufft.io> BDFL-Delegate: Richard Jones <richard(a)python.org> Discussions-To: distutils-sig(a)python.org Status: Draft Type: Process Content-Type: text/x-rst Created: 02-Mar-2014 Post-History: 03-Mar-2014 Replaces: 381 Abstract ======== This PEP proposes the deprecation and removal of the PyPI Mirror Authenticity API, this includes the /serverkey URL and all of the URLs under /serversig. Rationale ========= The PyPI mirroring infrastructure (defined in PEP 381) provides a means to mirror the content of PyPI used by the automatic installers, and as a component of that, it provides a method for verifying the authenticity of the mirrored content. This PEP proposal the removal of this API due to: * No known implementations that utilize this API are known, this includes `pip <http://www.pip-installer.org/en/latest/>`_ and `setuptools <http://pythonhosted.org//setuptools/>`_. * Because this API uses DSA it is vulnerable to leaking the private key if there is *any* bias in the random nonce. * This API solves one small corner of the trust problem, however the problem itself is much larger and it would be better to have a fully fledged system, such as `The Update Framework <https://python.org/dev/peps/pep-0458/>`_, instead. Due to the issues it has and the lack of use it is the opinion of this PEP that it does not provide any practical benefit to justify the additional complexity. Plan for Deprecation & Removal ============================== Immediately upon the acceptance of this PEP the Mirror Authenticity API will be considered deprecated and mirroring agents and installation tools should stop accessing it. Instead of actually removing it from the current code base (PyPI 1.0) the current work to replace PyPI 1.0 with a new code base (PyPI 2.0) will simply not implement this API. This would cause the API to be "removed" when the switch from 1.0 to 2.0 occurs. If PyPI 2.0 has not been deployed in place of PyPI 1.0 by Sept 01 2014 then this PEP will be implemented in the PyPI 1.0 code base instead (by removing the associated code). No changes will be required in the installers, however PEP 381 compliant mirroring clients, such as `bandersnatch <https://pypi.python.org/pypi/bandersnatch/>`_ and `pep381client <https://pypi.python.org/pypi/pep381client/>`_ will need to be updated to no longer attempt to mirror the /serversig URLs. Copyright ========= This document has been placed in the public domain. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

3 8

wheel questions
by Thomas Heller 21 Mar '14

21 Mar '14

1. question: Can wheels support a postinstall script? Background: I'm in the process of setting up a build script for that part of our company's software that is ported to Python 3.4. I am using either wheels built by 'pip wheel' and cache them locally, or, if this doesn't work (numpy and pywin32 for example), convert bdist_wininst installers to wheels with 'wheel convert xxx.exe'. Works fine for numpy and some other stuff, but not for pywin32. pywin32 contains a postinstall script that installs some dlls into the right locations and also does other stuff. Can a wheel run an included postinstall (and preremove) script? 2. question: I am building wheels for py2exe. If the wheel is built with python 3.3, the resulting wheel filename is py2exe-0.9.0.3-py33-none-any.whl. If the wheel is built with python 3.4, the filename is py2exe-0.9.0.3-py34-none-any.whl. Both are (kind-of) pure python distributions. 'pip install' can install the former both for Python 3.3 and Python 3.4, the latter only for Python 3.4. The only reason can be the filename. Does the -py34- part mean that *at least* Python 3.4 is required, and does -py33- mean that *at least* Python 3.3 is required? What would -cp33- or -cp34- mean? Sorry about these questions, but I'm lost what these filenames really mean (and how they are created). Thanks, Thomas

2 3

[ANN]: distlib 0.1.8 released on PyPI
by Vinay Sajip 19 Mar '14

19 Mar '14

I've released version 0.1.8 of distlib on PyPI [1]. For newcomers, distlib is a library of packaging functionality which is intended to be usable as the basis for third-party packaging tools. The main changes in this release are as follows: * Fixed issue #45: Improved thread-safety in SimpleScrapingLocator. * Fixed issue #42: Handling of pre-release legacy version numbers now mirrors setuptools logic. * Added exists, verify, update, is_compatible and is_mountable methods to the Wheel class (the update method fixed issue #41). * Added a search method to the PackageIndex class. * Fixed a bug in the Metadata.add_requirements method. * Allowed versions with a single numeric component and a local version component (tracking changes to PEP 440).* Corrected spelling of environment variable used for the stub launcher on OS X. * Avoided using pydist.json in 1.0 wheels (bdist_wheel writes a non- conforming pydist.json). * Improved computation of ABI tags on Python versions where SOABI is not available, and improved computation of compatibility tags on OS X to allow for multiple architectures and older OS X versions. A more detailed change log is available at [2]. Please try it out, and if you find any problems or have any suggestions for improvements, please give some feedback using the issue tracker! [3] Regards, Vinay Sajip [1] https://pypi.python.org/pypi/distlib/0.1.8 [2] http://pythonhosted.org/distlib/overview.html#change-log-for-distlib [3] https://bitbucket.org/pypa/distlib/issues/new

1 0

Redirects for old distribute documentation?
by PJ Eby 18 Mar '14

18 Mar '14

I was browsing through some docs on a database API for Python today when I noticed that they had a link to http://pythonhosted.org/distribute/easy_install.html -- which is defunct, along with most everything else under pythonhosted.org/distribute/. Perhaps it would be a good idea for the missing pages to redirect to some active documentation instead?

1 0

bizarre behavior with wheels and namespace packages
by Brian Wickman 08 Mar '14

08 Mar '14

Not sure if this belongs on distutils-sig or import-sig, but I'm experiencing slightly odd behavior with wheels and namespace packages. Fetch a namespace package source distribution, e.g. twitter.common.python, and build two copies of it, one with bdist_egg and one with bdist_wheel. This leaves me with: twitter.common.python-0.3.1-py2.6.egg twitter.common.python-0.3.1-py26-none-any.whl Explode them into the following directories (respectively): egg_dist whl_dist Then attempt: PYTHONPATH=egg_dist python -c 'import twitter.common.python' (works) PYTHONPATH=whl_dist python -c 'import twitter.common.python' (drumroll) Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: No module named twitter.common.python This is slightly unexpected. Absent from the .whl distribution are the __init__.pys with the proper declare_namespace declarations. Even more bizarrely: PYTHONPATH=whl_dist python -c 'import pkg_resources' Traceback (most recent call last): File "<string>", line 1, in <module> File "/Users/wickman/Python/CPython-2.6.9/lib/python2.6/site-packages/pkg_resources.py", line 2736, in <module> add_activation_listener(lambda dist: dist.activate()) File "/Users/wickman/Python/CPython-2.6.9/lib/python2.6/site-packages/pkg_resources.py", line 698, in subscribe callback(dist) File "/Users/wickman/Python/CPython-2.6.9/lib/python2.6/site-packages/pkg_resources.py", line 2736, in <lambda> add_activation_listener(lambda dist: dist.activate()) File "/Users/wickman/Python/CPython-2.6.9/lib/python2.6/site-packages/pkg_resources.py", line 2274, in activate list(map(declare_namespace, self._get_metadata('namespace_packages.txt'))) File "/Users/wickman/Python/CPython-2.6.9/lib/python2.6/site-packages/pkg_resources.py", line 1870, in declare_namespace path = sys.modules[parent].__path__ KeyError: 'twitter' So even pkg_resources is surprised as well. If I do the following: PYTHONPATH=whl_dist python -c 'import site; site.addsitedir("whl_dist"); import twitter.common.python' then it works, since the twitter.common.python-0.3.1-py2.6-nspkg.pth gets processed properly and module.__path__s populated accordingly. I would expect pkg_resources to handle this fine but apparently it doesn't. Any ideas? Bug in pkg_resources, wheel or PEBKAC? Thanks! ~brian

1 1

recommended setuptools version
by Benedek Zoltan 08 Mar '14

08 Mar '14

Hi, I'm a bit confused with the setuptools versioning. When I install it by ez_setup.py it installs the 3.0.2 version, but on pypi the latest version is 2.2. Is there a difference between the two? Which is the recommended one? Thanks Zoltan

4 3