Mailman 3 March 2008 - Distutils-SIG

How we can get rid of eggs for 2.6 and beyond
by Phillip J. Eby 02 Apr '08

02 Apr '08

So, after having some time to absorb the Python-Dev threads about setuptools, bootstrap, and all the rest, I think I see an opportunity to let people route around the "damage" of eggs, while still making it possible for the people who want to use easy_install or to put dependencies in their setup.py to get what they want, too. (And without them needing to install eggs, either.) At the same time, we can address the issues that remain around uninstalling packages, system vs. user packages, PYTHONPATH and site.py woes, and really pretty much every other complaint I've heard in the last few days about setuptools stomping on other people's stuff. (Even Paul's Windows issues, hopefully.) Now, you might be asking, "Okay, so why are you telling me about this? Why not just go fix setuptools?" Well, I *can't*. Not without some help from Python-Dev and the Distutils-SIG, to create an updated standard for installed package metadata, by updating PEP 262 ("A Database of Installed Python Packages") to include input from the system packaging folks, support for namespace packages, and support for setuptools-compatible dependency information. What's that got to do with anything? Well, without it, setuptools can't support uninstall or conflict management without using eggs to compartmentalize the installed files. And because it has to use eggs to do *that*, it has to munge .pth files and install its own site.py when installing to PYTHONPATH. All of this ugliness follows directly from the absence of a PEP 262-style installation database. Sure, setuptools could create its own version of this, and I almost did that four years ago. (If you look at the "open issues" part of PEP 262, you'll see my comments from back then.) I decided not to for two reasons: first, the distutils didn't support it yet, so it didn't help for conflict detection and avoidance in the real world at that point. Second, there were no uninstall tools for it, so I'd have had to write one myself. (Zed's "easy_f'ing_uninstall" to the contrary, it ain't easy, and I have an aversion to deleting stuff on people's systems without knowing what will break. There's a big difference between them typing 'rm -rf' themselves, and me doing it.) However, if tools exist and are distributed for such a "database", and *everybody* agrees to use it as an officially-blessed standard, then it should be possible for setuptools to co-exist with that framework, and we're all happy campers. In particular, the "installing eggs sucks" camp should be happy, because it'll be possible for me (or anyone else) to write a version of easy_install that doesn't install eggs any more, and setuptools-based packages can go back to having "setup.py install" install things the old way by default. So, to accomplish this, we (for some value of "we") need to: 1. Hash out consensus around what changes or enhancements are needed to PEP 262, to resolve the previously-listed open issues, those that have come up since (namespace packages, dependency specifications, canonical name/version forms), and anything else that comes up. 2. Update or replace the implementation as appropriate, and modify the distutils to support it in Python 2.6 and beyond. And "support it" means, "ensure that 'install' and *all* bdist commands update the database". The bdist_rpm, bdist_wininst, and bdist_msi commands, even bdist_dumb. (This should probably also include the add/remove programs stuff in the Windows case.) 3. Create a document for system packagers referencing the PEP and introducing them to what/why/how of the standard, in case they weren't one of the original participants in creating this. It will probably take some non-trivial work to do all this for Python 2.6, but it's probably possible, if we start now. I don't think it's critical to have an uninstall tool distributed with 2.6, as long as there's a reasonable way to bootstrap its installation later. Questions, comments... volunteers? :)

24 74

setup.py develop un-egg-spected result
by Oliver Schoenborn 31 Mar '08

31 Mar '08

I have a problem with "setup.py develop" installation. I have the following folder tree: pubsub \- setup.py \- extern \- module0.py \- core \- __init__.py \- module1.py \- module2.py \- ... I want my setup.py to install modules from pubsub/extern (such as module0.py) into site-packages, and to install the 'core' package in site-packages as pubsubcore: setup( ... py_modules = ['module0'], packages = ['pubsubcore'], package_dir = {'': 'extern', 'pubsubcore':'core'}, scripts = [], ... ) For sdist, bdist, bdist_egg etc, it works: the output shows module0 is in the "root" of the distribution, together with a package named "pubsubcore" which contains what is in 'pubsub/core'. HOWEVER, and here is the problem: when I do "setup.py develop" (this is on WinXP with Python 2.4.4 and the latest stable setuptools), only the contents of 'extern' get 'installed': C:\Documents and Settings\schoenb\My Documents\pubsub>setup.py develop running develop running egg_info writing external\PyPubSub.egg-info\PKG-INFO writing top-level names to external\PyPubSub.egg-info\top_level.txt writing dependency_links to external\PyPubSub.egg-info\dependency_links.txt writing manifest file 'external\PyPubSub.egg-info\SOURCES.txt' running build_ext Creating c:\python24\lib\site-packages\PyPubSub.egg-link (link to external) Adding PyPubSub 3.0a6 to easy-install.pth file Installed c:\documents and settings\schoenb\my documents\pubsub\extern Processing dependencies for PyPubSub==3.0a6 The above allows me, as the developer, to do "import module0" but not "import pubsubcore". Indeed, looking at \python24\lib\site-packages\easy-install.pth, there is only one new line added, pointing to c:\documents and settings\schoenb\my documents\pubsub\extern. There should be a second line added for the package, ie c:\documents and settings\schoenb\my documents\pubsub so that I (developer) can do "import pubsubcore", just as any user would be able to if I had done a full install. When I add the above line manually to the .pth file the "developer" installation works as expected. What am I doing wrong? Thanks, Oliver

4 7

Multiple Eggs in one setup file
by Dennis Kempin 27 Mar '08

27 Mar '08

Hi folks, i really like the setuptools project, it makes my life much easier, but recently i stumbled over the following problem: My Project constist of 2 main egg distributions and a bunch of "plugin" eggs. The first Problem ist: These projects are in the same source tree. like this: src/project/master src/project/plugins (namespace package) src/project/plugins/pa src/project/plugins/pb src/project/plugins/pc Well not a problem for setuptools since you can specify which packages to include into a distribution. But the problem is: when i call setup() multiple times in one setup.py file everything gets completly messed up. All egg files contain the whole project source. My second solution was to create multiple setup.py files (setup-master.py, setup-pa.py, and so on) and call them via system(). But this resulted in some conflicts in the build directory. So well.. is it possible (and especially how is it possible) to create multiple egg files within one python instance (one setup.py file that is executed)? And another question: When i have all the plugin egg files packaged. I want to create a new egg file (project-dist) to distribute the master and all of its plugins (which are simply copied to a /etc/ folder). Thus a single egg file that installs the master and copies some egg files to a specified folder. thanks in advance and best regards, Dennis

1 0

Multiple Egg files from one single setup.py
by Dennis Kempin 27 Mar '08

27 Mar '08

Hi folks, i really like the setuptools project, it makes my life much easier, but recently i stumbled over the following problem: My Project constist of 2 main egg distributions and a bunch of "plugin" eggs. The first Problem ist: These projects are in the same source tree. like this: src/project/master src/project/plugins (namespace package) src/project/plugins/pa src/project/plugins/pb src/project/plugins/pc Well not a problem for setuptools since you can specify which packages to include into a distribution. But the problem is: when i call setup() multiple times in one setup.py file everything gets completly messed up. All egg files contain the whole project source. My second solution was to create multiple setup.py files (setup-master.py, setup-pa.py, and so on) and call them via system(). But this resulted in some conflicts in the build directory. So well.. is it possible (and especially how is it possible) to create multiple egg files within one python instance (one setup.py file that is executed)? And another question: When i have all the plugin egg files packaged. I want to create a new egg file (project-dist) to distribute the master and all of its plugins (which are simply copied to a /etc/ folder). Thus a single egg file that installs the master and copies some egg files to a specified folder. thanks in advance and best regards, Dennis

2 1

CRLF in zipped packages?
by skip＠pobox.com 26 Mar '08

26 Mar '08

I pulled processing 0.50 from PyPI today in the form of a zip archive. I unzipped it on a Solaris machine using "unzip -a" but it thought all the files were binary (and zipinfo confirms this), so it didn't convert the CRLF line terminators to LF. Is this a side effect of the way the zip archive was constructed? I don't see anything conclusive which tells me the zip archive was built with distutils, but you folks are the packaging experts so I thought I'd ask here first. Is there something I can do simpler than a shell for loop and tr -d '\015' command to get rid of the annoying ^M characters? Thanks, Skip Montanaro

1 0

metadata/versions
by Chris Withers 26 Mar '08

26 Mar '08

Hi All, Apologies if these questions have been answered elsewhere, if they have, please point me at the answers. I've been trying to follow the discussion but the shear volume has overwhelmed me... 1. Is there a canonical way to tell what version a python package thinks it is? 2. Does the python package index contain dependency information? If not, why not? ;-) cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk

3 5

Re: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and beyond
by Phillip J. Eby 25 Mar '08

25 Mar '08

At 03:11 PM 3/25/2008 -0400, Alexander Michael wrote: >I agree that if every python package provided a >MSI/DEB/RPM/MPKG/etc., then there wouldn't be a need to install python >packages without a system-level installer. Even if they did, there would still be a need. System-level packages aren't generally retargetable to different locations, so there's no way to manage development sandboxes with them, or private application environments. Oh, and let's not forget shared hosting. No, system packages aren't even close to being the answer, for package authors *or* users.

1 0

Re: [Distutils] Question on using distutils.core.run_setup
by Terry Jones 22 Mar '08

22 Mar '08

>>>>> "Phillip" == Phillip J Eby <pje(a)telecommunity.com> writes: Phillip> At 07:00 AM 3/19/2008 +0100, Terry Jones wrote: >> I guess I'm missing something here? Phillip> Whoops, my bad. 'get_finalized_command' is a distutils *command Phillip> object* method, not a distribution method. I usually work with Phillip> code in command objects much more than with distribution objects. [snip] OK. Thanks a lot again, Terry

1 0

Re: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and beyond
by Phillip J. Eby 22 Mar '08

22 Mar '08

At 02:14 PM 3/22/2008 +0000, Paul Moore wrote: >For the system Python, I need: >- a single way to list what's installed (including version) >- a single way to uninstall items as needed >- a way (or more than one) to install 3rd party software *which ties >into the above* Right, and the PEP effort is devoted to having one way to store the information required, to tie these things together. If there is a standard way to store that info on your system, then it doesn't matter how many tools there are, you still have your "one way" to list what's installed or to uninstall things, because you just pick the one lister and/or uninstaller whose UI you prefer.

1 0

Adventures with x64, VS7 and VS8 on Windows
by Mark Hammond 21 Mar '08

21 Mar '08

Hi all, I hope the cross-post is appropriate. I've started playing with getting the pywin32 extensions building under the AMD64 architecture. I started building with Visual Studio 8 (it was what I had handy) and I struck a few issues relating to the compiler version that I thought worth sharing. * In trying to build x64 from a 32bit VS7 (ie, cross-compiling via the PCBuild directory), the python.exe project fails with: pythoncore fatal error LNK1112: module machine type 'X86' conflicts with target machine type 'AMD64' is this a known issue, or am I doing something wrong? * The PCBuild8 project files appear to work without modification (I only tried native compilation here though, not a cross-compile) - however, unlike the PCBuild directory, they place all binaries in a 'PCBuild8/x64' directory. While this means that its possible to build for multiple architectures from the same source tree, it makes life harder for tools like 'distutils' - eg, distutils already knows about the 'PCBuild' directory, but it knows nothing about either PCBuild8 or PCBuild8/x64. A number of other build processes also know to look inside a PCBuild directory (eg, Mozilla), so instead of formalizing PCBuild8, I think we should merge PCBuild8 into PCBuild. This could mean PCBuild/vs7 and PCBuild/vs8 directories with the "project" files, but binaries would still be generated in the 'PCBuild' (or PCBuild/x64) directory. This would mean the same tree isn't capable of hosting 2 builds from different VS compilers, but I think that is reasonable (if it's a problem, just have multiple source directories). I understand that PCBuild8 is not "official", but in the assumption that future versions of Python will use a compiler later than VS7, it makes sense to me to clean this up now - what are others opinions on this? * Re the x64 directory used by the PCBuild8 process. IMO, it makes sense to generate x64 binaries to their own directory - my expectation is that cross-compiling between platforms is a reasonable use-case, and we should support multiple achitectures for the same compiler version. This would mean formalizing the x64 directory in both 'PCBuild' and distutils, and leaving other external build processes to update as they support x64 builds. Does this make sense? Would this fatally break other scripts used for packaging (eg, the MSI framework)? * Wide characters in VS8: PC/pyconfig.h defines PY_UNICODE_TYPE as 'unsigned short', which corresponds with both 'WCHAR' and 'wchar' in previous compiler versions. VS8 defines this as wchar_t, which I'm struggling to find a formal definition for beyond being 2 bytes. My problem is that code which assumes a 'Py_UNICODE *' could be used in place of a 'WCHAR *' now fails. I believe the intent on Windows has always been "PyUNICODE == 'native unicode'" - should PC/pyconfig.h reflect this (ie, should pyconfig.h grow a version specific definition of PyUNICODE as wchar_t)? * Finally, as something from left-field which may well take 12 months or more to pull off - but would there be any interest to moving the Windows build process to a cygwin environment based on the existing autoconf scripts? I know a couple of projects are doing this successfully, including Mozilla, so it has precendent. It does impose a greater burden on people trying to build on Windows, but I'd suggest that in recent times, many people who are likely to want to build Python on Windows are already likely to have a cygwin environment. Simpler mingw builds and nuking MSVC specific build stuff are among the advantages this would bring. It is not worth adding this as "yet another windows build option" - so IMO it is only worth progressing with if it became the "blessed" build process for windows - if there is support for this, I'll work on it as the opportunity presents itself... I'm (obviously) only suggesting we do this on the trunk and am happy to make all agreed changes - but I welcome all suggestions or critisisms of this endeavour... Cheers, Mark

14 66