Mailman 3 March 2013 - Distutils-SIG

Importable wheels using distlib/distil
by Vinay Sajip 29 Mar '13

29 Mar '13

> I'm not top-posting, but trying to keep GMane happy ... Since wheels are .zip files, they can sometimes be used to provide functionality without needing to be installed. Whereas .zip files contain no convention for indicating compatibility with a particular Python, wheels do contain this compatibility information. Thus, it is possible to check if a wheel can be directly imported from, and the wheel support in distlib allows you to take advantage of this using the mount() and unmount() methods. When you mount a wheel, its absolute path name is added to sys.path, allowing the Python code in it to be imported. (A DistlibException is raised if the wheel isn't compatible with the Python which calls the mount() method.) You don't need mount() just to add the wheel's name to sys.path, or to import pure-Python wheels. The mount() method goes further than just enabling Python imports - any C extensions in the wheel are also made available for import. For this to be possible, the wheel has to be built with additional metadata about extensions - a JSON file called EXTENSIONS which serialises an extension mapping dictionary. This maps extension module names to the names in the wheel of the shared libraries which implement those modules. Running unmount() on the wheel removes its absolute pathname from sys.path and makes its C extensions, if any, also unavailable for import. Wheels built with distil contain the EXTENSIONS metadata, so can be mounted complete with C extensions: $ distil download -d /tmp simplejson Downloading simplejson-3.1.2.tar.gz to /tmp/simplejson-3.1.2 63KB @ 73 KB/s 100 % Done: 00:00:00 Unpacking ... done. $ distil package --fo=wh -d /tmp /tmp/simplejson-3.1.2/ The following packages were built: /tmp/simplejson-3.1.2-cp27-none-linux_x86_64.whl $ python Python 2.7.2+ (default, Jul 20 2012, 22:15:08) [GCC 4.6.1] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from distlib.wheel import Wheel >>> w = Wheel('/tmp/simplejson-3.1.2-cp27-none-linux_x86_64.whl') >>> w.mount() >>> import simplejson._speedups >>> dir(simplejson._speedups) ['__doc__', '__file__', '__loader__', '__name__', '__package__', 'encode_basestring_ascii', 'make_encoder', 'make_scanner', 'scanstring'] >>> simplejson._speedups.__file__ '/home/vinay/.distlib/dylib-cache/simplejson/_speedups.so' >>> This, IMO, makes the wheel format more useful than it already is :-) Regards, Vinay Sajip

7 16

Merge catalog-sig and distutils-sig
by Donald Stufft 29 Mar '13

29 Mar '13

Is there much point in keeping catalog-sig and distutils-sig separate? It seems to me that most of the same people are on both lists, and the topics almost always have consequences to both sides of the coin. So much so that it's often hard to pick *which* of the two (or both) lists you post too. Further confused by the fact that distutils is hopefully someday going to go away :) Not sure if there's some official process for requesting it or not, but I think we should merge the two lists and just make packaging-sig to umbrella the entire packaging topics. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

13 22

Builders vs Installers
by Paul Moore 28 Mar '13

28 Mar '13

There's a longer-term issue that occurred to me when thinking about pip's role as a "builder" or an "installer" (to use Nick's terminology). As I understand Nick's vision for the future, installers (like pip) will locate built wheels and download and install them, and builders (like distutils and bento) will be responsible for building wheels. But there's an intermediate role which shouldn't get forgotten in the transition - the role that pip currently handles with the "pip wheel" command. This is where I specify a list of distributions, and pip locates sdists, downloads them, checks dependencies, and ultimately builds all of the wheels. I'm not sure whether the current idea of builders includes this "locate, download and resolve dependencies" function (distutils and bento certainly don't have that capability). I imagine that pip will retain some form of the current "pip wheel" capability that covers this requirement, but maybe as the overall picture of the new design gets clarified, this role should be captured. Paul

9 74

python setup.py bdist_rpm fails on RHEL5 x86_64
by Sean Carolan 27 Mar '13

27 Mar '13

Hello everyone: I'm attempting to use the bdist_rpm flag to build a plain vanilla, Python 2.7 RPM for RHEL 5 x86_64, but the build command fails. Since you all are the distutils experts I figured you might have seen this before. I also submitted a bug to bugs.python.org: http://bugs.python.org/issue17553? Anyone have some pointers on how to make this build work? thanks Sean

1 1

PEP439 and backward compat / easy_install / distlib
by holger krekel 26 Mar '13

26 Mar '13

Hi Richard, all, two first notes on PEP439. backward compat with present-day release files: the PEP should state it as a goal or at least discuss it in some depth. In that context, the choice of providing a bootstrap for pip rather than easy_install needs reasoning. One problem with pip, compared to easy_install, is that it doesn't support eggs which is a problem particularly on Windows machines where often no fitting C compiler is available. If the remedy here is to support wheels and recommend it's use, it is still a backward compatibility problem: many users will not be able to use the builtin-supported installer to install todays existing egg release files. setuptools and distlib: Even if Python3.4+ had a mature distlib providing minimal setuptools functionality, how would it work for the typical "python setup.py install" which is invoked by pip? Often those setup.py scripts depend on a setuptools package. I am highlighting these two backward-compat aspects because otherwise we might run into this problem: http://xkcd.com/927/ and i understood that most people involved in improving the packaging ecology want to avoid that :) best, holger

10 32

A new, experimental packaging tool: distil
by Vinay Sajip 26 Mar '13

26 Mar '13

I've created a new tool called distil which I'm using to experiment with packaging functionality. Overview -------- It's based on distlib and has IMO some interesting features. With it, one can: * Install projects from PyPI and wheels (see PEP 427). Distil does not invoke setup.py, so projects that do significant computation in setup.py may not be installable by distil. However, a large number of projects on PyPI *can* be installed, and dependencies are detected, downloaded and installed. For those distributions that absolutely *have* to run setup.py, distil can create wheels using pip as a helper, and then install from those wheels. * Optionally upgrade installed distributions, whether installed by distil or installed by pip. * Uninstall distributions installed by distil or pip. * Build source distributions in .tar.gz, .tar.bz2, and .zip formats. * Build binary distributions in wheel format. These can be pure-Python, or have C libraries and extensions. Support for Cython and Fortran (using f2py) is possible, though currently distil cannot install Cython or Numpy directly because of how they use setup.py. * Run tests on built distributions. * Register projects on PyPI. * Upload distributions to PyPI. * Upload documentation to http://pythonhosted.org/. * Display dependencies of a distribution - either as a list of what would be downloaded (and a suggested download order), or in Graphviz format suitable for conversion to an image. Getting started is simple (documentation is at [2]): * Very simple deployment - just copy distil.py[1] to a location on your path, optionally naming it to distil on POSIX platforms. There's no need to install distlib - it's all included. * Uses either a system Python or one in a virtual environment, but by default installs to the user site rather than system Python library locations. * Offers tab-completion and abbreviation of commands and parameters on Bash-compatible shells. Logically, packaging activities can be divided into a number of categories or roles: * Archiver - builds source distributions from a source tree * Builder - builds binary distributions from source * Installer - installs source or binary distributions This version of distil incorporates (for convenience) all of the above roles. There is a school of thought which says that that these roles should be fulfilled by separate programs, and that's fine for production quality tools - it's just more convenient for now to have everything in one package for an experimental tool like distil. Actual Improvements ------------------- Despite the fact that distil is in an alpha stage of development and has received no real-world exposure like the existing go-to packaging tools, it does offer some improvements over them: * Dependency resolution can be performed without downloading any distributions. Unlike e.g. pip, you are told which additional dependencies will be downloaded and installed, before any download occurs. * Better information is stored for uninstallation. This allows better feedback to be given to users during uninstallation. * Dependency checking is done during uninstallation. Say you've installed a distribution A, which pulled in dependencies B and C. If you request an uninstallation of B (or C), distil will complain that you can't do this because A needs it. When you uninstall A, you are offered the option to uninstall B and C as well (assuming you didn't install something else that depends on B or C, after installing A). * By default, installation is to the user site and not to the system Python, so you shouldn't need to invoke sudo to install distributions for personal use which are not for specific projects/virtual environments. * There's no need to "install" distil - the exact same script will run with any system Python or any venv (subject to Python version constraints of 2.6, 2.7, 3.2 or greater). Bootstrapping pip ----------------- I've used distil to bootstrap pip, then used that pip to install other stuff. I created a fresh PEP 405 venv with nothing in it, used distil to install a wheel[3] for my distribute fork which runs on Python 2.x and 3.x, then used distil to install pip from PyPI. Finally, to test pip, I installed SQLAlchemy (using pip) from PyPI. See [4] for the transcript. I would welcome any feedback you could give regarding distil/distlib. There is of course a lot more testing to do, but I consider these initial findings to be promising, and worth sharing. If you find any problems, you can raise issues at [5]. Regards, Vinay Sajip [1] https://bitbucket.org/vinay.sajip/docs-distil/downloads/distil.py [2] https://pythonhosted.org/distil/ [3] https://bitbucket.org/vinay.sajip/distribute3/downloads/ [4] https://gist.github.com/vsajip/5243936 [5] https://bitbucket.org/vinay.sajip/distlib/issues/new

4 12

PEP 439 updated
by Richard Jones 26 Mar '13

26 Mar '13

Hi all, I've updated PEP 439 to note the outcome of the recent discussion regarding setuptools dependencies and a couple of other minor things. The changes are viewable here: http://hg.python.org/peps/diff/0d57c70eff91/pep-0439.txt Richard

2 1

Re: [Distutils] A new, experimental packaging tool: distil
by Vinay Sajip 26 Mar '13

26 Mar '13

Paul Moore <p.f.moore <at> gmail.com> writes: > Interesting! Is the source available anywhere? (Not distil.py, but the Not in a public VCS repo - I'm not soliciting contributions, as I'm still experimenting with a few features. But it'd be nice if the packaging-savvy readers here played with it just as users, and gave some feedback from that perspective. > source of the big chunk of embedded zipfile that appears to contain > the bulk of the functionality...) That zipfile contains distlib, some CLI support code and a packager.py, which contains the distil core functionality. Regards, Vinay Sajip

1 0

zc.buildout 2.1.0 released
by Jim Fulton 24 Mar '13

24 Mar '13

The new release has 2 major features: - Conditional sections allow you have buildout options that depend on the environment buildout runs in (OS, Python version, etc.) https://pypi.python.org/pypi/zc.buildout/2.1.0#conditional-sections - Meta-recipes allow you to scale the scope of buildouts sanely. https://pypi.python.org/pypi/zc.buildout/2.1.0#meta-recipe-support As well as a number of bug fixes. https://pypi.python.org/pypi/zc.buildout/2.1.0#id3 Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton

2 1

The pypa account on BitBucket
by Nick Coghlan 24 Mar '13

24 Mar '13

Hey pip/virtualenv folks, does one of you control the pypa placeholder account on BitBucket? (it seems possible, given it was created shortly after the Github account). I've been pondering the communicating-with-the-broader-community issue (especially in relation to http://simeonfranklin.com/blog/2013/mar/17/my-pycon-2013-poster/) and I'm thinking that the PSF account is the wrong home on BitBucket for the meta-packaging documentation repo. The PSF has traditionally been hands off relative to the actual development activities, and I don't want to change that. Instead, I'd prefer to have a separate team account, and also talk to Vinay about moving pylauncher and distlib under that account. I can create a different account if need be, but if one of you controls pypa, then it would be good to use that and parallel the pip/virtualenv team account on GitHub. If you don't already control it, then I'll write to BitBucket support to see if the account is actually being used for anything, and if not, if there's a way to request control over it. Failing that, I'll settle for a similar-but-different name, but "pypa" is definitely my preferred option. Regards, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

11 19