Mailman 3 February 2013 - Distutils-SIG

Author and Maintainer fields in PEP 426 (was Re: PEP 426 is now the draft spec for distribution metadata 2.0)
by Nick Coghlan 23 Feb '13

23 Feb '13

On Sun, Feb 24, 2013 at 1:51 PM, Chris Jerdonek <chris.jerdonek(a)gmail.com> wrote: > On Sun, Feb 17, 2013 at 2:10 AM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: >> >> Maintainer (optional) >> --------------------- >> >> A string containing the maintainer's name at a minimum; additional >> contact information may be provided. >> >> Note that this field is intended for use when a project is being >> maintained by someone other than the original author: it should be >> omitted if it is identical to ``Author``. > > I'm wondering whether Metadata 2.0 can help in rectifying the fact > that the contents of the Author field are blown away by the contents > of the Maintainer field when used with current tools (e.g. distutils, > Distribute/setuptools, PyPI) as described in issues 16403 and 16108, > etc ([1], [2]). If backwards compatibility is the issue, maybe > Metadata 2.0 can help by providing the way forward. > > [1] http://bugs.python.org/issue16403 > [2] http://bugs.python.org/issue16108 The core problem is distutils currently only emits metadata v1.1, which has no separate field for maintainer data. Once it emits v2.0 in Python 3.4, the upload should indeed provide both pieces of information, allowing PyPI (and other indexes) to display them more appropriately (e.g. as "Contact" and "Originally Created By"). I also agree that with the way the author field is used by distutils, "Contact" would be a more appropriate label in the index UI than "Author". Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0

Re: [Distutils] PEP 426 is now the draft spec for distribution metadata 2.0
by Chris Jerdonek 23 Feb '13

23 Feb '13

On Sun, Feb 17, 2013 at 2:10 AM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > > Maintainer (optional) > --------------------- > > A string containing the maintainer's name at a minimum; additional > contact information may be provided. > > Note that this field is intended for use when a project is being > maintained by someone other than the original author: it should be > omitted if it is identical to ``Author``. I'm wondering whether Metadata 2.0 can help in rectifying the fact that the contents of the Author field are blown away by the contents of the Maintainer field when used with current tools (e.g. distutils, Distribute/setuptools, PyPI) as described in issues 16403 and 16108, etc ([1], [2]). If backwards compatibility is the issue, maybe Metadata 2.0 can help by providing the way forward. [1] http://bugs.python.org/issue16403 [2] http://bugs.python.org/issue16108 --Chris

2 2

pkg_resources: inconsistency when using namespace packages
by Manlio Perillo 23 Feb '13

23 Feb '13

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi. I have find an inconstistency when using the pkg_resources module with namespace packages. I'm using setuptools 0.6c11-py2.5.egg Here is an interactive session: >>> import pkg_resources >>> dist = pkg_resources.Requirement.parse('wsgix') >>> dist = pkg_resources.get_provider(dist) >>> dist.get_resource_filename(dist, 'y') '/home/manlio/projects/hg/wsgix/wsgix/y' >>> dist_2 = pkg_resources.get_provider('wsgix') >>> dist_2.get_resource_filename(dist_2, 'y') '/home/manlio/projects/hg/wsgix/wsgix.forms.jqueryui/wsgix/y' The get_provider function returns two different distributions, when passing a string or Requirement object. My problem is with the resource API, since get_provider is called with the object passed (and I pass a string with the package name); instead the get_distribution function always convert the object passed to a Requirement object Thanks Manlio -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlEn258ACgkQscQJ24LbaURXCwCcDeuCWFUANQwsH0Q9CBATghha ttMAn1GmzdWmefLyP/7rz0/yBKjfcOyU =85ks -----END PGP SIGNATURE-----

2 2

The Wheel specification and Unicode filenames
by Vinay Sajip 21 Feb '13

21 Feb '13

The Wheel specification talks about supporting Unicode in the filename of wheel files, but is mute on the subject of the names of the entries in the archive. It would be good to have clarity on this point. The Python docs for 2.x and 3.x tell us: There is no official file name encoding for ZIP files. If you have unicode file names, you must convert them to byte strings in your desired encoding before passing them to write(). WinZip interprets all file names as encoded in CP437, also known as DOS Latin. The "your desired encoding" is, I think, too loose for wheel files, as we want interoperability between implementations. We should mandate CP437 encoding if we want the files to be examinable on Windows in e.g. WinZip or 7-Zip. On Linux, file-roller seems to be unable to display Unicode, whether you use CP437 for the filenames or whether you use utf-8. Regards, Vinay Sajip

3 4

[buildout] http://downloads.buildout.org/1/bootstrap.py resulting in buildout 2.0.1
by Chris Withers 21 Feb '13

21 Feb '13

Hi All, Am I the only one who has buildouts ending up using 2.0.1 when using: http://downloads.buildout.org/1/bootstrap.py ...which I thought would keep zc.recipe.egg and zc.buildout pinned pre 2.0? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

4 7

Delay acceptance of PEP 426 until after PyCon US?
by Nick Coghlan 20 Feb '13

20 Feb '13

Previously, there was some urgency to getting PEP 426 approved, as it was one of the conditions on getting wheel support into pip. However, Daniel revised the wheel spec to allow metadata v1.1, thus removing that dependency (since wheel dependencies work almost as well with the de facto setuptools metadata as they will with PEP 426). As one of the main reactions I'm getting to PEP 426 at the moment is a lot of confusion about how it fits into the larger picture of "fixing Python packaging", I'm considering postponing acceptance of the standard until after PyCon US. This will give me time to document more of the longer term possibilities for addressing deficiencies in the current packaging ecosystem, as well as being clear on *which parts* of the problem PEP 426 is trying to solve (as that is one of the other complaints I have received: PEP 426 doesn't solve every problem we currently have. It isn't meant to, but that may not be clear because proposals for other parts of the problem aren't even being discussed). Packaging will no doubt once again be a topic at the language summit and of course we have the mini-summit on Friday night and the panel on Saturday, all of which should help me address peoples concerns without their eyes glazing over at the walls of text I tend to publish when trying to explain complex topics over email. Thoughts? Regards, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

3 5

Wheel support added to distlib
by Vinay Sajip 20 Feb '13

20 Feb '13

Following the acceptance of PEP 427 (The Wheel Binary Package Format 1.0), I've added support for Wheel to distlib [1]. There are still some issues cropping up in my Windows and OS X tests - the test code uses pip to install some test distributions to build wheels from, and the issues appear to be related to the versions of pip on those machines. I expect to have those resolved soon. The documentation has been updated [2], but I'll just highlight some aspects of the API. Building wheels is simple: from distlib.wheel import Wheel wheel = Wheel() # Set the distribution's identity wheel.name = 'name_of_distribution' wheel.version = '0.1' # Set the tags you need, if the defaults don't fit your needs. # The filename will be computed automatically. wheel.pyver = ['py32'] wheel.abi = ['none'] wheel.arch = ['linux_x86_64'] # Indicate where the files to go in the wheel are to be found paths = { 'prefix': '/path/to/installation/prefix', 'purelib': '/path/to/purelib', # only one of purelib 'platlib': '/path/to/platlib', # or platlib should be set 'scripts': '/path/to/scripts', 'headers': '/path/to/headers', 'data': '/path/to/data', } wheel.dirname = '/where/you/want/the/wheel/to/go' # Now build wheel.build(paths) If the 'data', 'headers' and 'scripts' keys are absent, or point to paths which don't exist, nothing will be added to the wheel for these categories. The 'prefix' key and one of 'purelib' or 'platlib' *must* be provided, and the paths referenced should exist. Installing from wheels is similarly easy: from distlib.wheel import Wheel wheel = Wheel('/path/to/my_dist-0.1-py32-none-any.whl') # Indicate where the files in the wheel are to be installed to. # All the keys should point to writable paths. paths = { 'prefix': '/path/to/installation/prefix', 'purelib': '/path/to/purelib', 'platlib': '/path/to/platlib', 'scripts': '/path/to/scripts', 'headers': '/path/to/headers', 'data': '/path/to/data', } # Now install. The method accepts a ``dry_run`` keyword # argument which goes through the installation procedure # but doesn't actually install anything. wheel.install(paths) Although some work has been done to add wheel support to pip, you don't need this to build wheels for existing PyPI distributions if you use distlib. The following script, wheeler.py, shows how you can use an unpatched, vanilla pip to build wheels: https://gist.github.com/vsajip/4988471 Obviously this work has just been done so there will be some rough edges, but I would welcome feedback from anyone who wants to try it out. If you want to report issues, you can do it on the BitBucket tracker [3]. Coverage stats are available at [4]. Regards, Vinay Sajip [1] https://bitbucket.org/vinay.sajip/distlib/ [2] http://distlib.readthedocs.org/en/latest/tutorial.html#using-the-wheel-api [3] https://bitbucket.org/vinay.sajip/distlib/issues/new [4] http://www.red-dove.com/distlib/coverage/

5 17

Building wheels for dependencies with distlib, wheeler.py and vanilla pip.
by Vinay Sajip 20 Feb '13

20 Feb '13

I've updated the wheeler.py script [1] to build wheels for dependencies by default, though you can turn that off by specifying --no-deps. It's smart about not repeating work if dependencies are common across multiple distributions you specify: $ python wheeler.py pygments flask Finding the dependencies of pygments ... Finding the dependencies of flask ... Pipping Werkzeug==0.8.3 ... Pipping Jinja2==2.6 ... Pipping Pygments==1.6 ... Pipping Flask==0.9 ... The following wheels were built: Werkzeug-0.8.3-py27-none-any.whl Jinja2-2.6-py27-none-any.whl Pygments-1.6-py27-none-any.whl Flask-0.9-py27-none-any.whl It seems to work for Pyramid, too (Marcus asked about it), though there are some 2.x/3.x issues relating to Unicode, zip file entries and csv files that I'm investigating (Pyramid has some test fixture files with Unicode filenames). You should pull the latest changes from distlib before using wheeler.py, as I fixed some errors in Unicode handling. Regards, Vinay Sajip [1] https://gist.github.com/vsajip/4988471

1 0

PEP 376, Uninstallation and interoperability
by Vinay Sajip 19 Feb '13

19 Feb '13

I've been thinking about uninstallation of distributions, and it seems possible that PEP 376 is underspecified if we want uninstallation to work interoperably between different tools that implement the PEP. It seems desirable that uninstallation should undo the process of installation as far as possible. This of course means removing files that were installed, but it also means removing directories which were created during installation. While not saying so explicitly, PEP 376 strongly implies that only files are named in RECORD, and not directories. When removing directories during uninstallation, an initial candidate set of directories to cull would be the ancestor directories of all files named in RECORD, up to a certain point. That "certain point" would exclude any of the directories shared by distributions, and their ancestors. Those shared directories are the ones defined in the sysconfig installation schemes under the keys 'purelib', 'platlib', 'include', 'scripts' and 'data'. There's also a notional 'base' (equivalent to prefix as in sys.prefix), but for all schemes as currently defined, this is the same as 'data'. So the lib/pythonX.Y/site-packages, bin, include etc. directories and their ancestor directories would be immune from deletion, as we would want. Although sysconfig gives *default* values for these locations, it seems likely that tools will offer users options to override the defaults using e.g. command line parameters or configuration files. If such options are made use of and the overriding values used are not recorded, then there is no obvious way of determining where to stop with directory deletion during uninstallation. Another question arises - what is to be done when directories, which are meant to be deleted during uninstallation, contain files which were not put there during installation of the distribution being removed? It would be useful for packaging tools to inform users about the extraneous, unexpected files and perhaps get confirmation before proceeding with directory deletion. However, since shared directories are likely to contain files put there by other installation operations, you wouldn't want to include these when checking for unexpected files. Unless I've missed something obvious, I think the lack of information about shared directories needs to be rectified. One simple way of achieving this is to add to the files in .dist-info another text file, perhaps called SHARED, which lists the values actually used for 'purelib', 'platlib' etc. at installation time, in a key=value format (one per line in the file). The file should contain entries for at least all of the keys used in sysconfig, but it can also contain other user-defined keys (e.g. some tools might have the concept of a specific directories for man pages or HTML documentation, or sysconfig might grow more keys in the future). What do you think about this? I'd like your comments. Regards, Vinay Sajip

7 17

Wheel (pep 427) reference implementation
by Daniel Holth 18 Feb '13

18 Feb '13

A friendly reminder that wheel is implemented, useful and working as long as you are using distribute 0.6.28 or better. Qwcode's pip fork supports it. When you install a wheel you get .dist-info and can easily have no .egg-info in your virtualenv. Would anyone care to lend a hand with the Sphinx API docs which I don't know how to set up? Code is at https://bitbucket.org/dholth/wheel . Most of what will be the API is in wheel/install.py Thanks! Daniel Holth

1 0