Mailman 3 March 2013 - Distutils-SIG

Re: [Distutils] Setuptools-Distribute merge announcement
by Vinay Sajip March 14, 2013

March 14, 2013

Carl Meyer <carl <at> oddbird.net> writes: > > On 03/13/2013 06:54 PM, PJ Eby wrote: > > Jason Coombs (head of the Distribute project) and I are working on > > merging the bulk of the improvements distribute made into the > > setuptools code base. He has volunteered to take over maintenance of > > setuptools, and I welcome his assistance. > > This merge is very good news for Python packaging. Kudos and thanks to > you and Jason. > > This also seems like an appropriate opportunity to say: thank you for > building the first working packaging system for Python (the one that all > popular packaging tools are still based on), and for all the work you've > put into it over the years. > Emphatic +1! Regards, Vinay Sajip

1 0

Distribute: will not build on Python 3 with --install-XXX arguments
by Paul Moore March 14, 2013

March 14, 2013

What I'm doing is checking out distribute, then trying to install it into a temporary directory. However, when I do this, I seem to be getting the files in python 2 form, *not* having been converted using 2to3. The resulting build is therefore broken. Can anyone suggest what might be going on here? I've been looking at this for hours now, and I'm no nearer to working out what is going on :-( I'm running on Windows 7, 64-bit, using Python 3.3 in Powershell. The same happens in cmd.exe and Python 3.2. There are NO packages installed in my Python installation, it's a straight install from the python.org installer. PS 13:26 C:\Work\Scratch >hg clone http://bitbucket.org/tarek/distribute dtest real URL is https://bitbucket.org/tarek/distribute requesting all changes adding changesets adding manifests adding file changes added 1160 changesets with 2491 changes to 443 files updating to branch default 98 files updated, 0 files merged, 0 files removed, 0 files unresolved PS 13:26 C:\Work\Scratch >cd dtest PS 13:26 C:\Work\Scratch\dtest >mkdir a Directory: C:\Work\Scratch\dtest Mode LastWriteTime Length Name ---- ------------- ------ ---- d---- 05/03/2013 13:26 a PS 13:26 C:\Work\Scratch\dtest >py setup.py install --install-purelib=aa\purelib --install-platlib=aa\platlib --install-scripts=aa\scripts --install-headers=aa\headers --install-data=aa\data [... lots of setup.py output clipped ...] byte-compiling aa\purelib\setuptools\tests\test_sdist.py to test_sdist.cpython-33.pyc File "aa\purelib\setuptools\tests\test_sdist.py", line 152 except UnicodeDecodeError, e: ^ SyntaxError: invalid syntax byte-compiling aa\purelib\setuptools\tests\test_test.py to test_test.cpython-33.pyc byte-compiling aa\purelib\setuptools\tests\test_upload_docs.py to test_upload_docs.cpython-33.pyc byte-compiling aa\purelib\setuptools\tests\__init__.py to __init__.cpython-33.pyc byte-compiling aa\purelib\setuptools\__init__.py to __init__.cpython-33.pyc byte-compiling aa\purelib\site.py to site.cpython-33.pyc byte-compiling aa\purelib\_markerlib\markers.py to markers.cpython-33.pyc byte-compiling aa\purelib\_markerlib\__init__.py to __init__.cpython-33.pyc running install_egg_info Writing aa\purelib\distribute-0.6.36-py3.3.egg-info After install bootstrap. Creating aa\purelib\setuptools-0.6c11-py3.3.egg-info Creating aa\purelib\setuptools.pth PS 13:27 C:\Work\Scratch\dtest > It's that SyntaxError that confirms that the generated files are invalid :-( Paul

3 6

distlib - installer support vs runtime support
by Paul Moore March 11, 2013

March 11, 2013

As part of a discussion on a pip issue, it was noted that if pip depends on distlib for installer capabilities (e.g., locators, wheel installation, version and requirement parsing and matching, etc) then the user needs distlib installed as well as pip. That shouldn't be an issue for an end user, any more than having pip itself installed would be. However, there are some capabilites of distlib (notably the "exports" functionality that replaces setuptools' entry points) that are intended to be used at runtime, by the end user. (I suspect, but haven't checked, that the exe wrapper functionality depends on exports as well - it certainly does in setuptools). That means that the end user *is* affected by the fact that pip depends on distlib. For example, if the user needs a later version of distlib, it's quite likely that he won't be able to "pip install" it, as pip typically has trouble upgrading its own dependencies (for obvious reasons...) Would it be worth considering splitting distlib into two separate parts - one that is intended solely for writers of installers and similar tools, and another for "runtime support" functions that end users would use? It may not be a practical thing to achieve, but it would be worth at least understanding the trade-offs involved. Paul. PS The same issue exists in setuptools, but as far as I am aware, setuptools was deliberately designed to provide both the installation tools and runtime support in the one package. (And IIUC, even with setuptools, pkg_resources is the only component that includes runtime support, so in theory it is possible to split the two parts up).

6 9

PEP 427 (wheel) clarifications
by Daniel Holth March 10, 2013

March 10, 2013

A few more clarifications to PEP 427 "wheel". + The ``b'#!pythonw'`` convention is allowed. ``b'#!pythonw'`` indicates + a GUI script instead of a console script. + ... { "hash": "sha256=ADD-r2urObZHcxBW3Cr-vDCu5RJwT4CaRTHiFmbcIYY" } +(The hash value is the same format used in RECORD.) + +What's the deal with "purelib" vs. "platlib"? + Wheel preserves the historic "purelib" vs. "platlib" distinction + even though both map to the same install location in any system the + author could find. + + For example, a wheel with "Root-Is-Purelib: false" with all its files + in ``{name}-{version}.data/purelib`` is equivalent to a wheel with + "Root-Is-Purelib: true" with those same files in the root, and it + is legal to have files in both the "purelib" and "platlib" categories. + + In practice a wheel should have only one of "purelib" or "platlib" + depending on whether it is pure Python or not and those files should + be at the root.

2 1

Add optional password_command .pypirc value
by Matt Behrens March 10, 2013

March 10, 2013

After doing some research tonight on storing/accessing passwords in the OSX Keychain (http://asktherelic.com/2013/03/07/storing-command-line-passwords-in-keychai…), I was curious why the .pypirc doesn't support something like this when asking for the password during 'upload', to not have your pypi password in plaintext on your system. As far as I can see from the source, the password is read straight from the .pypirc config: https://bitbucket.org/tarek/distribute/src/188dcdb7f0873f1b382e8bde65377c5f… and fails if the password value doesn't exist: https://bitbucket.org/tarek/distribute/issue/291/allow-password-to-be-omitt… I'm curious about implementing: 1. a password_command to support integration with external password tools (1password, keychain, keyring python lib) The implementation from the program I am trying to emulate, pianobar, is here: https://github.com/PromyLOPh/pianobar/blob/master/src/main.c#L135 just a /bin/sh for nix/osx. Could run cmd.exe for windows cross-platform compatibility. 2. better notification to the user about trying to upload with an empty password or using get_pass if empty password The only other reference to something like this is from several years ago here: http://bugs.python.org/issue4394 Does this seem like it's worth making a patch for? -- Matt Behrens

3 2

Add optional password_command .pypirc value
by Matt Behrens March 9, 2013

March 9, 2013

After doing some research last night on storing/accessing passwords in the OSX Keychain (http://asktherelic.com/2013/03/07/storing-command-line-passwords-in-keychai…), I was curious why the .pypirc doesn't support something like this when asking for the password during 'upload', to not have your pypi password in plaintext on your system. As far as I can see from the source, the password is read straight from the .pypirc config: https://bitbucket.org/tarek/distribute/src/188dcdb7f0873f1b382e8bde65377c5f… and fails if the password value doesn't exist: https://bitbucket.org/tarek/distribute/issue/291/allow-password-to-be-omitt… I'm curious about implementing: 1. a password_command to support integration with external password tools (1password, keychain, keyring python lib) The implementation from the program I am trying to emulate, pianobar, is here: https://github.com/PromyLOPh/pianobar/blob/master/src/main.c#L135 just a /bin/sh for nix/osx. Could run cmd.exe for windows cross-platform compatibility. 2. better notification to the user about trying to upload with an empty password or using get_pass if empty password The only other reference to something like this is from several years ago here: http://bugs.python.org/issue4394 Does this seem like it's worth making a patch? -- Matt Behrens

6 8

distribute-0.6.35 fails on python 3.3?
by Chris Withers March 6, 2013

March 6, 2013

Hi All, Can anyone else reproduce this? I have a Python 3.3 virtualenv and I wanted to upgrade one of the packages as follows: $ bin/pip install -U manuel Requirement already up-to-date: manuel in ./lib/python3.3/site-packages Downloading/unpacking distribute from http://pypi.python.org/packages/source/d/distribute/distribute-0.6.35.tar.g… (from manuel) Downloading distribute-0.6.35.tar.gz (644kB): 644kB downloaded Running setup.py egg_info for package distribute Traceback (most recent call last): File "<string>", line 3, in <module> File "./setuptools/__init__.py", line 2, in <module> from setuptools.extension import Extension, Library File "./setuptools/extension.py", line 5, in <module> from setuptools.dist import _get_unpatched File "./setuptools/dist.py", line 103 except ValueError, e: ^ SyntaxError: invalid syntax Complete output from command python setup.py egg_info: Traceback (most recent call last): File "<string>", line 3, in <module> File "./setuptools/__init__.py", line 2, in <module> from setuptools.extension import Extension, Library File "./setuptools/extension.py", line 5, in <module> from setuptools.dist import _get_unpatched File "./setuptools/dist.py", line 103 except ValueError, e: ^ SyntaxError: invalid syntax Can other people reproduce this? When was that Python 3 incompatible except clause introduced? cheers. Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

9 12

PEP 426 (Metadata 2.0) - Requires-Dist and setuptools/distribute
by Paul Moore March 5, 2013

March 5, 2013

In thinking about how virtualenv would describe the packages it wants to preload in PEP 426 metadata form, it occurred to me that there are scenarios with setuptools and distribute where it's not obvious how to state the requirement you want. Specifically, if you want to install setuptools if it is present, but if not fall back to distribute (for example, if you have a local package repository and no access to PyPI, but setuptools may or may not be present). I appreciate this is a fairly obscure case. It comes up with virtualenv because virtualenv uses locally-available distributions by default, only going to PyPI if it has to. So in that case (depending on user options) I could genuinely want to pick whichever of setuptools or distribute is present, and I don't care which, as it saves a network lookup. I'm actually using the distlib locator API, not the PEP 426 fields themselves, but (a) distlib locators use the same syntax, as far as I'm aware, and (b) I think the Requires-Dist syntax makes a good language for specifying distribution requirements in any context, so I'd hate to end up with 2 slightly-different forms. If the answer is that the spec doesn't support that, then fine. I'll have to manually code for it. But I'd hate to write code I didn't need to :-) Paul.

4 6

Library instability on PyPI and impact on OpenStack
by Mark McLoughlin March 5, 2013

March 5, 2013

Hey, (I sent this to the wrong list and was directed here by Nick. I wasn't aware of the very promising sounding PEP426 and haven't read it yet, so apologies on that. Just wanted to resend my mail here ASAP to prevent the discussion happening on the wrong list. Thanks!) Generally speaking, when a project has a large list of dependencies on libraries outside of its control, it can take one of two approaches to those dependencies: 1) specify the minimum required version of each library and assume new releases of all your dependencies will be backwards compatible with previous versions. You feel safe that if an incompatible version of the library is released, it will be a completely new stream and you can adopt the new stream at your leisure. I'm much more familiar with C libraries than any other language. Somehow, C library maintainers seem to understand the need for this approach and so you've got mechanisms like libtool managed sonames and parallel installable libraries like gtk, gtk2, gtk3. 2) specify exactly what version of each library to use, because you assume all of your dependencies are constantly changing their APIs and breaking your app This is what you see in the Ruby (bundler) and Java (maven) worlds. For distribution packagers, it presents horrific problems - either you accept that you're going to be packaging (potentially many) different versions of the same library and that any time a security issue comes up you need to patch each version. Personally, I tend to pour scorn on Ruby and Java folks for the chaotic nature that pushes app developers down this "I need to control the whole stack of dependencies because they're so unstable" path. OTOH, I hear the Perl community are pretty good about taking the first approach. I always felt that the Python community tended more towards the former approach, but there always exceptions to the rule - to unfairly pick one one project, sqlalchemy seems to have an API that often changes incompatibly. However, OpenStack is starting to get burned more often and some are advocating taking the second approach to managing our dependencies: http://lists.openstack.org/pipermail/openstack-dev/2013-February/thread.htm… http://lists.openstack.org/pipermail/openstack-dev/2013-February/thread.htm… http://lists.openstack.org/pipermail/openstack-dev/2012-November/002075.html It's probably not worthwhile for everyone to try and read the nuances of those threads. The tl;dr is we're hurting and hurting bad. Is this a problem the OpenStack and Python communities want to solve together? Or does the Python community fundamentally seem themselves as taking the same approach as the Ruby and Java communities? Maybe it sounds like I'm trolling, but this is an honest question. What I'd really like to hear back is "please, please OpenStack keep using the first approach and we can work through the issues you're seeing and together make Python better". We can totally do that IMHO. If people want an example of the kind of stuff things we need to work on: http://lists.openstack.org/pipermail/openstack-dev/2013-February/006048.html Thanks, Mark.

9 33

Metadata files, dist-info/egg-info and migration paths
by Paul Moore March 4, 2013

March 4, 2013

This is peripherally related to PEP 426, to the extent that PEP 426 specifies that the distribution metadata goes in the dist-info directory defined by PEP 376. The dist-info directory conceptually replaces the old de-facto standard egg-info directory. But neither PEP 376 nor PEP 426 mention anything about what should happen to the *other* files that currently reside in egg-info. These are basically setuptools extensions, for things like namespace packages, entry points, zipped egg support, etc. As far as I am aware, recent versions of distribute look for the setuptools metadata files in the dist-info directory if it's present. So for projects using distribute, moving all of the metadata files to dist-info makes sense. But there's no release of setuptools that supports this, so what should happen there? The issue is with built distributions. Core python distutils still writes an egg-info directory (and that won't change till 3.4, and only distribute supports Python 3, so no issue there). Setuptools and distribute both write to egg-info, so there's no compatibility issue there either. But the wheel format uses dist-info internally, and installation is defined as only producing a dist-info directory (by unpacking the one in the wheel). There are two questions here that bear discussion. First of all, when creating a wheel, should builders put custom metadata files from the existing egg-info data into the dist-info directory. I would suggest that yes, they should, as otherwise that data is lost completely - in particular, setuptools entry points (and hence executable wrappers) fail without the entry_points.txt file. There is some support in distlib for replacement functionality in some of these areas (exports, the EXPORTS file and script wrappers) but this is at an early stage and there's no migration path defined yet that I'm aware of. The more difficult question is what should happen when a wheel is installed. At the moment, tools write out the dist-info directory and that's it. That works fine for projects using distribute, or ones that don't use setuptools-style metadata. But projects using setuptools under Python 2 won't be able to see the metadata. Should we require that in order to use wheels, distribute should be used (or a suitably patched setuptools, should that become available)? Or should wheel installers write a legacy egg-info directory for use by setuptools (I'd suggest that this should only happen on Python 2, and even then probably only if a specific "legacy" flag was set). I have no real knowledge of what to do here - my suggestions above are relatively uninformed, and in particular I have little knowledge of what is common among people still using Python 2. What do the experts think? Paul.

3 3