> I'm not top-posting, but trying to keep GMane happy ...
Since wheels are .zip files, they can sometimes be used to provide
functionality without needing to be installed. Whereas .zip files contain no
convention for indicating compatibility with a particular Python, wheels do
contain this compatibility information. Thus, it is possible to check if a
wheel can be directly imported from, and the wheel support in distlib allows
you to take advantage of this using the mount() and unmount() methods. When you
mount a wheel, its absolute path name is added to sys.path, allowing the Python
code in it to be imported. (A DistlibException is raised if the wheel isn't
compatible with the Python which calls the mount() method.)
You don't need mount() just to add the wheel's name to sys.path, or to import
pure-Python wheels. The mount() method goes further than just enabling Python
imports - any C extensions in the wheel are also made available for import. For
this to be possible, the wheel has to be built with additional metadata about
extensions - a JSON file called EXTENSIONS which serialises an extension
mapping dictionary. This maps extension module names to the names in the wheel
of the shared libraries which implement those modules.
Running unmount() on the wheel removes its absolute pathname from sys.path and
makes its C extensions, if any, also unavailable for import.
Wheels built with distil contain the EXTENSIONS metadata, so can be mounted
complete with C extensions:
$ distil download -d /tmp simplejson
Downloading simplejson-3.1.2.tar.gz to /tmp/simplejson-3.1.2
63KB @ 73 KB/s 100 % Done: 00:00:00
Unpacking ... done.
$ distil package --fo=wh -d /tmp /tmp/simplejson-3.1.2/
The following packages were built:
/tmp/simplejson-3.1.2-cp27-none-linux_x86_64.whl
$ python
Python 2.7.2+ (default, Jul 20 2012, 22:15:08)
[GCC 4.6.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from distlib.wheel import Wheel
>>> w = Wheel('/tmp/simplejson-3.1.2-cp27-none-linux_x86_64.whl')
>>> w.mount()
>>> import simplejson._speedups
>>> dir(simplejson._speedups)
['__doc__', '__file__', '__loader__', '__name__', '__package__',
'encode_basestring_ascii', 'make_encoder', 'make_scanner', 'scanstring']
>>> simplejson._speedups.__file__
'/home/vinay/.distlib/dylib-cache/simplejson/_speedups.so'
>>>
This, IMO, makes the wheel format more useful than it already is :-)
Regards,
Vinay Sajip
Is there much point in keeping catalog-sig and distutils-sig separate?
It seems to me that most of the same people are on both lists, and the topics almost always have consequences to both sides of the coin. So much so that it's often hard to pick *which* of the two (or both) lists you post too. Further confused by the fact that distutils is hopefully someday going to go away :)
Not sure if there's some official process for requesting it or not, but I think we should merge the two lists and just make packaging-sig to umbrella the entire packaging topics.
-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
There's a longer-term issue that occurred to me when thinking about
pip's role as a "builder" or an "installer" (to use Nick's
terminology).
As I understand Nick's vision for the future, installers (like pip)
will locate built wheels and download and install them, and builders
(like distutils and bento) will be responsible for building wheels.
But there's an intermediate role which shouldn't get forgotten in the
transition - the role that pip currently handles with the "pip wheel"
command. This is where I specify a list of distributions, and pip
locates sdists, downloads them, checks dependencies, and ultimately
builds all of the wheels. I'm not sure whether the current idea of
builders includes this "locate, download and resolve dependencies"
function (distutils and bento certainly don't have that capability).
I imagine that pip will retain some form of the current "pip wheel"
capability that covers this requirement, but maybe as the overall
picture of the new design gets clarified, this role should be
captured.
Paul
Hello everyone:
I'm attempting to use the bdist_rpm flag to build a plain vanilla, Python
2.7 RPM for RHEL 5 x86_64, but the build command fails. Since you all are
the distutils experts I figured you might have seen this before. I also
submitted a bug to bugs.python.org:
http://bugs.python.org/issue17553?
Anyone have some pointers on how to make this build work?
thanks
Sean
Hi Richard, all,
two first notes on PEP439.
backward compat with present-day release files: the PEP should state
it as a goal or at least discuss it in some depth. In that context, the
choice of providing a bootstrap for pip rather than easy_install needs
reasoning. One problem with pip, compared to easy_install, is
that it doesn't support eggs which is a problem particularly on
Windows machines where often no fitting C compiler is available. If the
remedy here is to support wheels and recommend it's use, it is still a
backward compatibility problem: many users will not be able to use the
builtin-supported installer to install todays existing egg release files.
setuptools and distlib: Even if Python3.4+ had a mature distlib
providing minimal setuptools functionality, how would it work for the
typical "python setup.py install" which is invoked by pip? Often those
setup.py scripts depend on a setuptools package.
I am highlighting these two backward-compat aspects because otherwise
we might run into this problem: http://xkcd.com/927/ and i understood
that most people involved in improving the packaging ecology want
to avoid that :)
best,
holger
I've created a new tool called distil which I'm using to experiment with
packaging functionality.
Overview
--------
It's based on distlib and has IMO some interesting features. With it, one can:
* Install projects from PyPI and wheels (see PEP 427). Distil does not invoke
setup.py, so projects that do significant computation in setup.py may not be
installable by distil. However, a large number of projects on PyPI *can* be
installed, and dependencies are detected, downloaded and installed. For those
distributions that absolutely *have* to run setup.py, distil can create
wheels using pip as a helper, and then install from those wheels.
* Optionally upgrade installed distributions, whether installed by distil or
installed by pip.
* Uninstall distributions installed by distil or pip.
* Build source distributions in .tar.gz, .tar.bz2, and .zip formats.
* Build binary distributions in wheel format. These can be pure-Python, or have
C libraries and extensions. Support for Cython and Fortran (using f2py) is
possible, though currently distil cannot install Cython or Numpy directly
because of how they use setup.py.
* Run tests on built distributions.
* Register projects on PyPI.
* Upload distributions to PyPI.
* Upload documentation to http://pythonhosted.org/.
* Display dependencies of a distribution - either as a list of what would be
downloaded (and a suggested download order), or in Graphviz format suitable
for conversion to an image.
Getting started is simple (documentation is at [2]):
* Very simple deployment - just copy distil.py[1] to a location on your path,
optionally naming it to distil on POSIX platforms. There's no need to install
distlib - it's all included.
* Uses either a system Python or one in a virtual environment, but by default
installs to the user site rather than system Python library locations.
* Offers tab-completion and abbreviation of commands and parameters on
Bash-compatible shells.
Logically, packaging activities can be divided into a number of categories or
roles:
* Archiver - builds source distributions from a source tree
* Builder - builds binary distributions from source
* Installer - installs source or binary distributions
This version of distil incorporates (for convenience) all of the above roles.
There is a school of thought which says that that these roles should be
fulfilled by separate programs, and that's fine for production quality tools -
it's just more convenient for now to have everything in one package for an
experimental tool like distil.
Actual Improvements
-------------------
Despite the fact that distil is in an alpha stage of development and has
received no real-world exposure like the existing go-to packaging tools, it
does offer some improvements over them:
* Dependency resolution can be performed without downloading any distributions.
Unlike e.g. pip, you are told which additional dependencies will be
downloaded and installed, before any download occurs.
* Better information is stored for uninstallation. This allows better feedback
to be given to users during uninstallation.
* Dependency checking is done during uninstallation. Say you've installed a
distribution A, which pulled in dependencies B and C. If you request an
uninstallation of B (or C), distil will complain that you can't do this
because A needs it. When you uninstall A, you are offered the option to
uninstall B and C as well (assuming you didn't install something else that
depends on B or C, after installing A).
* By default, installation is to the user site and not to the system Python, so
you shouldn't need to invoke sudo to install distributions for personal use
which are not for specific projects/virtual environments.
* There's no need to "install" distil - the exact same script will run with any
system Python or any venv (subject to Python version constraints of 2.6, 2.7,
3.2 or greater).
Bootstrapping pip
-----------------
I've used distil to bootstrap pip, then used that pip to install other stuff.
I created a fresh PEP 405 venv with nothing in it, used distil to install a
wheel[3] for my distribute fork which runs on Python 2.x and 3.x, then used
distil to install pip from PyPI. Finally, to test pip, I installed SQLAlchemy
(using pip) from PyPI. See [4] for the transcript.
I would welcome any feedback you could give regarding distil/distlib. There is
of course a lot more testing to do, but I consider these initial findings to be
promising, and worth sharing. If you find any problems, you can raise issues
at [5].
Regards,
Vinay Sajip
[1] https://bitbucket.org/vinay.sajip/docs-distil/downloads/distil.py
[2] https://pythonhosted.org/distil/
[3] https://bitbucket.org/vinay.sajip/distribute3/downloads/
[4] https://gist.github.com/vsajip/5243936
[5] https://bitbucket.org/vinay.sajip/distlib/issues/new
Hi all,
I've updated PEP 439 to note the outcome of the recent discussion
regarding setuptools dependencies and a couple of other minor things.
The changes are viewable here:
http://hg.python.org/peps/diff/0d57c70eff91/pep-0439.txt
Richard
Paul Moore <p.f.moore <at> gmail.com> writes:
> Interesting! Is the source available anywhere? (Not distil.py, but the
Not in a public VCS repo - I'm not soliciting contributions, as I'm still
experimenting with a few features. But it'd be nice if the packaging-savvy
readers here played with it just as users, and gave some feedback from that
perspective.
> source of the big chunk of embedded zipfile that appears to contain
> the bulk of the functionality...)
That zipfile contains distlib, some CLI support code and a packager.py, which
contains the distil core functionality.
Regards,
Vinay Sajip
Hey pip/virtualenv folks, does one of you control the pypa placeholder
account on BitBucket? (it seems possible, given it was created shortly
after the Github account).
I've been pondering the communicating-with-the-broader-community issue
(especially in relation to
http://simeonfranklin.com/blog/2013/mar/17/my-pycon-2013-poster/) and
I'm thinking that the PSF account is the wrong home on BitBucket for
the meta-packaging documentation repo. The PSF has traditionally been
hands off relative to the actual development activities, and I don't
want to change that.
Instead, I'd prefer to have a separate team account, and also talk to
Vinay about moving pylauncher and distlib under that account.
I can create a different account if need be, but if one of you
controls pypa, then it would be good to use that and parallel the
pip/virtualenv team account on GitHub. If you don't already control
it, then I'll write to BitBucket support to see if the account is
actually being used for anything, and if not, if there's a way to
request control over it. Failing that, I'll settle for a
similar-but-different name, but "pypa" is definitely my preferred
option.
Regards,
Nick.
--
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
