In article , Paul Moore wrote:

I'd like to reopen the discussions on how the new packaging module will handle/support binary distributions in Python 3.3. The previous thread (see http://mail.python.org/pipermail/python-dev/2011-October/113956.html) included a lot of good information and discussion, but ultimately didn't reach any firm conclusions.

First question - is this a Windows only problem, or do Unix/MacOS users want binary support? My feeling is that it's not an issue for them, at least not enough that anyone has done anything about it in the past, so I'll focus on Windows here.

I haven't been following this discussion that closely but I'm rather surprised that the need for binary distributions for Python packages on non-Windows platforms would be in question. Just as on Windows, it's not a given that all Unix or Mac OS X end-user systems will have the necessary development tools installed (C compiler, etc) to build C extension modules. Today, the most platform-independent way of distributing these are with binary eggs: the individual binary eggs are, of course, not platform-independent but the distribution and installation mechanism is or should be. Sure, there are other ways, like pushing the problem back to the OS distributor (e.g. Debian, Red Hat, et al) or, as in the case of Mac OS X where there isn't a system package manager in the same sense, to a third-party package distributor (like MacPorts, Homebrew, or Fink). Or you can produce platform-specific installers for each platform which also seems heavy-weight. Has anyone analyzed the current packages on PyPI to see how many provide binary distributions and in what format? -- Ned Deily, nad@acm.org

On 30 October 2011 18:04, Ned Deily wrote:

Has anyone analyzed the current packages on PyPI to see how many provide binary distributions and in what format?

A very quick and dirty check: dmg: 5 rpm: 12 msi: 23 dumb: 132 wininst: 364 egg: 2570 That's number of packages with binary distributions in that format. It's hard to be sure about egg distributions, as many of these could be pure-python (there's no way I know, from the PyPI metadata, to check this). This is 2913 packages with some form of binary distribution out of 16615 that have a single release on PyPI. I skipped 398 with multiple releases as I wasn't sure how to capture the data for those... I suspect they include some important cases, though (I know lxml is in there, for example). So: 17% of packages have any binary release. Of those, 88% have eggs, 12% have wininst and the rest are under 5%. Put another way, 2% of all packages have wininst installers. And 15% have eggs. That's not a lot. Paul.

I like binary distribution even under Linux. I access some Linux machines using same Linux distribution and some of them doesn't have "python-dev" package or even "build-essensials". (because they are netbooting so have restricted rootfs size) So I want build binary package by myself and distribute it to virtualenv on such machines. In this case, absolute path of virtualenv is not fixed. So "bdist_dumb --relative" or egg is good for me. On Sun, Oct 30, 2011 at 11:09 PM, Paul Moore wrote:

I'd like to reopen the discussions on how the new packaging module will handle/support binary distributions in Python 3.3. The previous thread (see http://mail.python.org/pipermail/python-dev/2011-October/113956.html) included a lot of good information and discussion, but ultimately didn't reach any firm conclusions.

First question - is this a Windows only problem, or do Unix/MacOS users want binary support? My feeling is that it's not an issue for them, at least not enough that anyone has done anything about it in the past, so I'll focus on Windows here.

Second question - is there a problem at all? For the majority of Windows users, I suspect not. The existing bdist_wininst and bdist_msi formats have worked fine for a long time, offer Windows integration and a GUI installer, and in the case of MSI offer options for integrating with corporate distribution policies that some users consider significant, if not essential. (Binary eggs are a third, and somewhat odd, case - a number of projects have started distributing binary eggs, but I don't know what benefits they have over bdist_wininst in particular, as easy_install will read bdist_wininst installers. Perhaps a setuptools/distribute user could comment. For now I'll assume that binary eggs will slowly go away as packaging gets more widely adopted).

So that leaves a minority who (1) prefer integration with packaging, (2) need to work with virtual environments or custom local builds, (3) need binary extensions in some or all of their environments and (4) don't want to have to build all the binaries they need from scratch.

Given the scale of the issue, it seems likely that putting significant effort into addressing it is unwise. In particular, it seems unlikely that developers are going to move en masse to a new distribution format just to cater for this minority. On the other hand, for people who care, the fact that packaging (currently) offers no direct support for consuming binary distributions is a fairly obvious hole. And having to build from source just to install into a virtual environment could be a showstopper.

The bdist_wininst format is relatively amenable to manipulation - it's little more than a zip file, after all. So writing 3rd party code to install the contents via packaging shouldn't be hard (I've done some proof of concept work, and it isn't :-)) Vinay's proposal to use the resource mechanism and some custom hooks would work, but I'd like to see a small amount of extra direct support added to packaging to make things cleaner. Also, if packaging supported plugins to recognise new distribution formats, this would make it possible to integrate the extra code seamlessly.

The MSI format is a little more tricky, mainly because it is a more complex format and (as far as I can tell from a brief check) files are stored in the opaque CAB format, so the only way of getting data out is to do a temporary install somewhere. But I see no reason why that isn't achievable.

So, my proposal is as follows:

1. I will write a 3rd party module to take bsist_wininst and bdist_msi modules and install them using packaging 2. Where packaging changes are useful to make installing binaries easier, I'll request them (by supplying patches) 3. I'll look at creating a format-handling plugin mechanism for packaging. If it's viable, I'll post patches 4. If it seems useful, my module could be integrated into the core packaging module

I don't intend to do anything about a GUI, or modify the existing formats at all. These don't interest me, particularly, so I'll leave them to someone who has a clear picture of what they want in those areas, and the time to develop it.

For 3.3 at least, I'd expect developers to continue distributing bdist_wininst or bdist_msi format files. We'll see what happens with binary eggs.

Unix/MacOS users who care will need to propose something themselves.

Does anyone have any comments?

Paul. _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com

-- INADA Naoki 

Paul Moore writes:

Hang on. I'm talking here about repackaging the binary files in the MSI file for use in a pysetup install invocation. As pysetup has no GUI, and doesn't integrate with Add/Remove, there's no issue here. If you want a GUI and Add/Remove integration, just run the MSI. Or am I missing something? We seem to be at cross purposes here, I suspect I'm missing your point.

As you say later in your post, we're probably just coming at this from two different perspectives. I think you mentioned the possible need to install to a temporary location just to extract files from the CAB; then you would presumably need to uninstall again to remove the Add/Remove Programs entry created when you installed to the temporary location (or else I misunderstood your meaning here).

...

It would be important to retain the flexibility offered by setup.cfg hooks, as I don't believe any out-of-the-box approach will work for the range of use cases on Windows (think Powershell scripts, Visio templates and other Microsoft Office integration components).

Why? Again, if this is purely as a means to consume bdist_xxx files, then the only flexibility needed is enough to cater for any variations in data stored in the bdist_xxx format. The wininst format is easy here - it has directories PLATLIB, PURELIB, DATA, SCRIPTS and HEADERS (corresponding to the installation --install-xxx parameters) and that's all. As long as the module is flexible enough to deal with that, it can read anything bdist_wininst can produce.

My point is really that a one-size-fits-all DATA location is unlikely to cater to all use cases. The flexibility offered by setup.cfg together with hooks gets around the limitation of a single location for data.

Ah, I think I see what you are getting at. If someone uses the new features and flexibility of packaging to create a fancy custom install scheme, how do they bundle up a binary distribution from that? My (current) answer is that I don't know. The packaging module as it stands only offers the legacy bdist_xxx formats, so the answer is "run pysetup run bdist_wininst on it". If that breaks (as it is likely to - wininst format isn't very flexible) then tough, you're out of luck.

Yes, that's what I was getting at. Regards, Vinay Sajip

Am 31.10.2011 09:07, schrieb Vinay Sajip:

Paul Moore writes:

...

Hang on. I'm talking here about repackaging the binary files in the MSI file for use in a pysetup install invocation. As pysetup has no GUI, and doesn't integrate with Add/Remove, there's no issue here. If you want a GUI and Add/Remove integration, just run the MSI. Or am I missing something? We seem to be at cross purposes here, I suspect I'm missing your point.

As you say later in your post, we're probably just coming at this from two different perspectives. I think you mentioned the possible need to install to a temporary location just to extract files from the CAB; then you would presumably need to uninstall again to remove the Add/Remove Programs entry created when you installed to the temporary location (or else I misunderstood your meaning here).

This presumption is false (as is the claim that you need to install the MSI to get at the files). It's quite possible to extract the files from the MSI without performing the installation. There are actually two ways to do that: a) perform an "administrative" installation, which unpacks the files to disk but doesn't actually perform any installation procedure, or b) use the MSI API to extract first the CAB file, and then the files in the CAB file. This would be a bit work to do if you want to find out the full path names of the individual files, but it could work in theory.

...

Why? Again, if this is purely as a means to consume bdist_xxx files, then the only flexibility needed is enough to cater for any variations in data stored in the bdist_xxx format. The wininst format is easy here - it has directories PLATLIB, PURELIB, DATA, SCRIPTS and HEADERS (corresponding to the installation --install-xxx parameters) and that's all. As long as the module is flexible enough to deal with that, it can read anything bdist_wininst can produce.

My point is really that a one-size-fits-all DATA location is unlikely to cater to all use cases. The flexibility offered by setup.cfg together with hooks gets around the limitation of a single location for data.

I'm sure bdist_wininst can be augmented to support arbitrary "base prefixes" (assuming that is the flexibility you talk about). It would just need a list of what directory names are prefixes, The MSI format is designed to provide exactly that flexibility of arbitrarily mapping source folders to destination folders during installation. bdist_msi would just need to be taught to interpret setup.cfg files.

...

Ah, I think I see what you are getting at. If someone uses the new features and flexibility of packaging to create a fancy custom install scheme, how do they bundle up a binary distribution from that? My (current) answer is that I don't know. The packaging module as it stands only offers the legacy bdist_xxx formats, so the answer is "run pysetup run bdist_wininst on it". If that breaks (as it is likely to - wininst format isn't very flexible) then tough, you're out of luck.

Yes, that's what I was getting at.

Hmm. You are just describing a bug, not an inherent limitation. Regards, Martin

Paul Moore writes:

Agreed - the "one size fits all" data location is a limitation. I'm not sure that in practical terms it is a big issue, though - it's been like that since the wininst format was designed, and nobody has ever complained. There are certainly cases where packages have needed to implement more or less clumsy workarounds (for example, not including documentation in binary distributions) but it's obviously never been enough of an issue to prompt people to fix it. The egg format has the same limitation, as far as I'm aware, so clearly even the "eggs solve everything" crowd don't feel it's a real issue

Yes, but with setup.py you had the option of running any Python code to move things around using a post-install script, so people could get around those limitations, albeit in a completely ad hoc way. So there was nothing to fix, but no standard way of achieving what you wanted in out-of-the-ordinary scenarios.

I'd love to see a binary format that was as flexible and powerful as building from source, which allowed OS integration where the user wanted it while still supporting venvs and non-system installations, and which was widely adopted by distribution authors. Oh, and can I have a pony? Sadly, I don't have the time or understanding of the various requirements to deliver something like that.

Well, from the point of view of venvs and PEP 404, it's certainly topical and worth trying to get some traction behind this particular pony. If bdist_pony is easy enough to use and doesn't close any existing doors, then there's no obvious reason why distribution authors wouldn't use it for future releases of their distributions. Regards, Vinay Sajip

On Mon, 31 Oct 2011 05:59:09 -0400 "Eric V. Smith" wrote:

It might be true that such systems don't need binary packages on PyPI, but the original question is about binary package support for the packaging module on non-Windows systems. I think the answer is clearly "yes": I have such systems without compilers. If I build packages on a staging server, I would want to put them on an internal PyPI-like server, for consumption by packaging. So packaging would need to consume these binary packages.

And it's not only compilers, it's also external libraries (which are generally not installed by default). For example, to compile pyOpenSSL, you first need to fetch the OpenSSL development headers. Regards Antoine.

Hi,

I'd like to reopen the discussions on how the new packaging module will handle/support binary distributions in Python 3.3. The previous thread (see http://mail.python.org/pipermail/python-dev/2011-October/113956.html) included a lot of good information and discussion, but ultimately didn't reach any firm conclusions.

I’m sorry there was no reply from the core group of packaging contributors. I read the messages as they flew by and wanted to reply on a lot of points, but didn’t get the time to do it. I hope the list subscribers won’t mind if I go through the threads in the coming days and make many replies. Cheers

On 31 October 2011 18:36, Ned Deily wrote:

In article ,  Paul Moore wrote:

...

On 30 October 2011 18:04, Ned Deily wrote:

...

Has anyone analyzed the current packages on PyPI to see how many provide binary distributions and in what format?

A very quick and dirty check:

dmg: 5 rpm: 12 msi: 23 dumb: 132 wininst: 364 egg: 2570

That's number of packages with binary distributions in that format. It's hard to be sure about egg distributions, as many of these could be pure-python (there's no way I know, from the PyPI metadata, to check this).

Thanks.  If you have access to the egg file name, you should be able to tell.  AFAIK, eggs with extension modules include the Distutils platform name in the file name preceded by a '-', so '-linux', '-win32', '-macosx' for the main ones.  Pure python eggs do not contain a platform name.  http://pypi.python.org/pypi/pyinterval/ is a random example of the former.

136 architecture-specific 2502 architecture independent About 5%. The numbers don't quite add up, so there's some funnies in there (possibly bad data that I'm not handling well) but it gives an idea. Counts by architecture: win32 70 linux-i686 43 win-amd64 33 linux-x86_64 26 macosx-10.3-fat 12 macosx-10.5-i386 11 macosx-10.6-universal 9 macosx-10.6-fat 8 macosx-10.3-i386 7 macosx-10.6-i386 6 macosx-10.7-intel 4 macosx-10.6-intel 3 macosx-10.6-x86_64 2 macosx-10.3-ppc 2 macosx-10.4-i386 2 macosx-10.4-ppc 2 py2.3-linux-i686 1 py2.4-linux-i686 1 gnu-0.3-i686-AT386 1 linux-ppc 1 cygwin-1.5.25-i686 1 py2.3 1 py2.4 1 py2.5 1 macosx-10.7-x86_64 1 macosx-10.4-universal 1 py2.5-linux-i686 1 Most of the 1-counts are bad data in some form. I'm not sure what this proves, to be honest, but what I take from it is: - Nearly all binary distributions are for Windows - Architecture-neutral eggs are common (but not relevant here as packaging can install from source with these) - Ignoring architecture-neutral eggs, most popular formats are wininst, egg, dumb(!!!) and msi - Even the most popular binary format (wininst) only accounts for 2% of all packages. Having said all of this, there are two major caveats I'd include: - Not everything is on PyPI. - This analysis ignores relative importance. It's hard to claim that numpy is no more significant than, say, "Products.CMFDynamicViewFTI" (whatever that might be - I picked it at random, so apologies to the author :-)) Paul.

Urgh. I guess that was already answered. Guess this'll teach me not to reply to a thread before waiting for ALL the messages to download over a low-bandwidth connection... (am on the road at the moment and catching up on stuff in spare cycles - sorry for the noise) On Fri, Nov 4, 2011 at 10:24 PM, PJ Eby wrote:

On Sun, Oct 30, 2011 at 6:52 PM, Paul Moore wrote:

...

On 30 October 2011 18:04, Ned Deily wrote:

...

Has anyone analyzed the current packages on PyPI to see how many provide binary distributions and in what format?

A very quick and dirty check:

dmg: 5 rpm: 12 msi: 23 dumb: 132 wininst: 364 egg: 2570

That's number of packages with binary distributions in that format. It's hard to be sure about egg distributions, as many of these could be pure-python (there's no way I know, from the PyPI metadata, to check this).

FYI, the egg filename will contain a distutils platform identifier (e.g. 'win32', 'macosx', 'linux', etc.) after the 'py2.x' tag if the egg is platform-specific. Otherwise, it's pure Python.

Martin v. Löwis writes:

Again, that's a bdist_msi implementation issue. It could generate custom actions that run the "proper" setup.cfg hooks (I presume - I have no idea what a setup.cfg hook actually is).

I know that custom hooks are quite powerful, but my comment was about having the functionality in Python. Here's an example of a working hooks.py: import os import sys if os.name == 'nt': def get_personal_path(): from ctypes import (wintypes, windll, create_unicode_buffer, WinError, c_int, HRESULT) from ctypes.wintypes import HWND, HANDLE, DWORD, LPWSTR, MAX_PATH CSIDL_PERSONAL = 5 # We use an older API to remain XP-compatible. SHGetFolderPath = windll.shell32.SHGetFolderPathW SHGetFolderPath.argtypes = [HWND, c_int, HANDLE, DWORD, LPWSTR] SHGetFolderPath.restype = DWORD path = create_unicode_buffer(MAX_PATH) hr = SHGetFolderPath(0, CSIDL_PERSONAL, 0, 0, path) if hr != 0: raise WinError() return path.value path = get_personal_path() del get_personal_path # Assume ~\Documents\WindowsPowerShell\Modules is in $PSModulePath, # which should be true in a default installation of PowerShell 2.0. psroot = os.path.join(path, 'WindowsPowerShell') psmodules = os.path.join(psroot, 'Modules') psscripts = os.path.join(psroot, 'Scripts') def setup(config): files = config['files'] if os.name != 'nt': files_to_add = 'virtualenvwrapper.sh = {scripts}' else: files_to_add = ('winfiles/ *.ps* = ' '{psmodules}/virtualenvwrapper\n' 'winfiles/ vew_profile.ps1 = {psscripts}') if 'resources' not in files: files['resources'] = files_to_add else: files['resources'] += '\n%s' % files_to_add def pre_install_data(cmd): if os.name == 'nt': cmd.categories['psmodules'] = psmodules cmd.categories['psscripts'] = psscripts cmd.categories['psroot'] = psroot which works with the following setup.cfg: [global] setup_hooks = hooks.setup [install_data] pre-hook.win32 = hooks.pre_install_data categories = cat1 = /path/one # comment cat2 = /path/two #[install_dist] #post-hook.win32 = hooks.post_install_dist [metadata] name = nemo version = 0.1 summary = New Environments Made, Obviously description = A tool to manage virtual environments download_url = UNKNOWN home_page = https://bitbucket.org/vinay.sajip/nemo author = Vinay Sajip author_email = vinay_sajip@yahoo.co.uk license = BSD classifier = Development Status :: 3 - Alpha Programming Language :: Python :: 3 Operating System :: OS Independent Intended Audience :: System Administrators Intended Audience :: Developers License :: OSI Approved :: BSD License requires_python = >= 3.3 [files] packages = nemo virtualenvwrapper scripts = nemo = nemo.main extra_files = hooks.py winfiles/* # Additional esources are added in hooks based on platform resources = nemo/scripts/** = {purelib} I'm curious to know how this level of flexibility can be achieved with the MSI format: I know one can code the equivalent logic in C (for example) in a custom action, but don't know how you can keep the logic in Python. Regards, Vinay Sajip

It seems to me that there are two separate things going on in this sample. It's not 100% clear that they are separate, at first glance, as packaging currently doesn't make a strong distinction between things going on at "build" time, and things going on at "install" time. This is essentially because the idea of binary installs is not fundamental to the design. (Thanks for sharing this example, btw, I hadn't really spotted this issue until I saw the code here).

Suppose you have two people involved - the "packager" who uses the source code to create a binary distribution (MSI, wininst, zip, doesn't matter - conceptually, it's a set of "final" files that need no further processing and can just be put in the correct locations on the target PC) and the "end user" who takes that binary distribution and installs it on his PC.

Some of the hook code is designed to run at "build" time (the stuff that adds the right resource files). This can be run on the packager's machine quite happily, as long as the packager is using the same OS as the end user. However, other parts of the hook code (the stuff that defines the custom categories) must run on the end user's PC, as it detects specific aspects of the target PC configuration.

In this case at least, the code *all* runs at installation time: the distributed package contains all files for all platforms, and at installation time the choice is made as to which files to actually install from the installation directory to the target directories. While this might not be ideal for all packagers, the only downside of having all files for all platforms available in a single distribution is disk space - an increasingly cheap commodity. OTOH there is some advantage in having a single package which would be usable on all platforms (supported by the package being distributed), albeit perhaps for a particular version of Python.

I think Martin is only really interested in the second type of hook here. I know that I am, insofar as they are the only type I would expect to need to support if I were building a new binary distribution format. But without the two types being more clearly separated, it's not obvious that it's possible to "just support one type" in quite that sense...

In terms of the flexibility required, the code to determine the "personal path" is being run at installation time, to determine the target folder for the PowerShell scripts. It's this kind of flexibility (by which I mean Python coded logic) that I don't see how to easily provide in the MSI format, short of recoding in e.g. C, in a custom action DLL or EXE. (This latter approach is what I've used in the PEP 397 launcher MSI.) Regards, Vinay Sajip

I'd provide a fixed custom action which gets hold of the installer session, and then runs a Python script. IIUC, it should be possible to map categories to entries in the Directory table, so that the Python script would actually configure the installer process before the installer actually starts installing the files. The DLL could be part of packaging, similar to how the bdist_wininst executable is part of distutils.

Presumably the code in the DLL would need to be independent of Python, and find the correct Python version to run? Perhaps a variable in the .MSI could serve to indicate the version dependency. It's certainly feasible, but needs specifying in more detail ... Regards, Vinay Sajip