Mailman 3 April 2008 - Distutils-SIG

Shebang lines on Jython
by Philip Jenvey 16 Apr '08

16 Apr '08

Speaking of shebang lines, there's problems with setuptools shebang lines on Jython. Unfortunately it's going to require another patch or two to setuptools. The problem being that Jython's executable is a .sh or .bat file (that'll invoke java), and interpreters in shebang lines can't be interpreter files, i.e. another script. Running a .py script that shebangs the Jython .sh executable directly will result in something along the lines of a "test.py: line 2 import command not found" sh error. For posix, we'll just need to special case Jython on posix to make a shebang along the lines of: #!/usr/bin/env /opt/local/bin/jython Windows isn't as easy. setuptools' launcher.c similarly can't use a .bat file as the interpreter. A workaround would be to use cmd.exe / c in the same way we'll use /usr/bin/env on posix. launcher.c currently can't handle this, though, the problem being it can't correctly quote the cmd.exe /c line correctly. With: #!c:\windows\system32\cmd.exe /c c:\jython2.2\jython.bat launcher.c ends up calling: "c:\windows\system32\cmd.exe" "/c" "c:\jython2.2\jython.bat" "c: \scripts\myscript-script.py" when we need: "c:\windows\system32\cmd.exe" /c ""c:\jython2.2\jython.bat" "c: \scripts\myscript-script.py"" which is another set of quotes around the entire /c arg, and yes no quotes around /c (I have no idea why "/c" doesn't work). I guess we would have to patch launcher.c to deal with this special situation or make a custom version for Jython. That's pretty awful -- might anyone have a simpler way of handling this? -- Philip Jenvey

2 2

ez_setup.py fails with ImportError: No module named core
by Deron Meranda 16 Apr '08

16 Apr '08

For some reason I can not bootstrap setuptools 0.6c8 into a python. This is Python 2.5 running under AIX 5.2, and the site-packages directory is completely empty. I've tried to use just ez_setup.py with no options, as well as downloading the egg file locally into a temporary directory and using: python ez_setup.py -v -H "" setuptools-0.6c8-py2.5.egg I get this strange traceback every time: Downloading http://pypi.python.org/packages/2.5/s/setuptools/setuptools-0.6c8-py2.5.egg Traceback (most recent call last): File "ez_setup.py", line 267, in <module> main(sys.argv[1:]) File "ez_setup.py", line 200, in main from setuptools.command.easy_install import main File "build/bdist.linux-i686/egg/setuptools/__init__.py", line 2, in <module> File "build/bdist.linux-i686/egg/setuptools/extension.py", line 1, in <module> ImportError: No module named core Furthermore, the local egg file setuptools-0.6c8-py2.5.egg is removed from my working directory! My working directory is completely outside of the python path or site-packages dir. I also tried using the -d option to explicitly point to the site-packages dir, but the results are the same. I can not figure out what exactly is wrong. Any ideas? -- Deron Meranda

1 1

Re: [Distutils] ez_setup.py fails with ImportError: No module named core
by Phillip J. Eby 16 Apr '08

16 Apr '08

At 03:42 PM 4/16/2008 -0400, Deron Meranda wrote: >For some reason I can not bootstrap setuptools 0.6c8 into a python. > >This is Python 2.5 running under AIX 5.2, and the site-packages >directory is completely empty. I've tried to use just ez_setup.py >with no options, as well as downloading the egg file locally into >a temporary directory and using: > python ez_setup.py -v -H "" setuptools-0.6c8-py2.5.egg > >I get this strange traceback every time: > >Downloading >http://pypi.python.org/packages/2.5/s/setuptools/setuptools-0.6c8-py2.5.egg >Traceback (most recent call last): > File "ez_setup.py", line 267, in <module> > main(sys.argv[1:]) > File "ez_setup.py", line 200, in main > from setuptools.command.easy_install import main > File "build/bdist.linux-i686/egg/setuptools/__init__.py", line 2, > in <module> > File "build/bdist.linux-i686/egg/setuptools/extension.py", line > 1, in <module> >ImportError: No module named core This is an indication that your distutils installation is broken and/or missing. Make sure that if your OS has a 'python-dev' or 'python-devel' or 'python-distutils' package, that it's installed.

1 0

Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
by Phillip J. Eby 15 Apr '08

15 Apr '08

At 10:53 AM 3/17/2008 -0500, Guido van Rossum wrote: >I don't think this should play games with scripts being overridden or >whatever. If a bootstrap script is to be installed it should have a >separate name. I'm not sure what the advantage is of a bootstrap >script over "python -m bootstrap_module ..." though. And -m also makes explicit: 1. that it's a Python-specific tool 2. which Python version it will apply to >The PEP suggests that other package managers also benefit. How do they >benefit if the bootstrap script installs setuptools? Because those other package managers depend, in fact, on setuptools, or at least pkg_resources... which was why the original proposal was to just include pkg_resources in the first place. :) >I'd also like to avoid the specific name "easy_install" for any of >this. That's a "brand name" (and a misleading one if you ask me, but >that's politics again :-). Ok, so if someone will propose a name and API for the thing, I'll implement it. (Assuming the proposed API is sane and reasonably implementable, of course.)

9 20

shebang line modified by setuptools
by Cliff Wells 14 Apr '08

14 Apr '08

Sorry for breaking up the thread. I wasn't subscribed to the list (now I am) and apparently I stopped being CC'd at some point, so I'll have to sum up several things and address them here. 1) I agree that "system" scripts should use the "system" python (whatever that is defined to mean - for now it means /usr/bin/python) and that path should be hardcoded into the shebang line. 2) I disagree with Ben's position that: """ It's my position that the Python instance one uses for development should diverge as little as possible from the default system instance. Otherwise one is actively pursuing a recipe for dependency failures when one eventually deploys the result. """ The reason I disagree is that assumes the system is single user and/or single purpose. In my case (and many others) I'm concerned about a traditional shared hosting environment (not VPS). This means that there may be several versions of Python (and many, many ~/python2.x/site-packages directories). Diverging from the default system instance *in every single case* is the only way to make this workable. Not only do different accounts require a particular version of Python, but they require particular versions of installed packages (one might rely on Pylons 0.9.5 another on Pylons 0.9.7). 3) Since there is so much opposition to using /usr/bin/env in the shebang line, an acceptable alternative would be to hardcode the version-specific python instead (e.g. /usr/bin/python2.4). This seems to solve my particular issue while respecting Phillip's concerns about accidentally running an incorrect interpreter, and actually does a better job since Phillip's method (using /usr/bin/python) falsely assumes that the system Python will never change (untrue on Gentoo, Foresight, SourceMage and any other distro that supports "rolling upgrades"). 4) As far as the discussion about having three separate install areas, I think this is not necessary since it's already possible to have unlimited install areas (one per user) using features already built into setuptools. /usr/bin/python/ and /usr/local/bin/pythonX.X/ should suffice. If setuptools used versioning info along with a hardcoded path in the shebang line, this would become even more solid. I should also mention that some of this discussion is being made moot by virtualenv. I'll be using that for future stuff, but I have legacy installs to worry about as well. Regards, Cliff

1 0

Re: [Distutils] how to easily consume just the parts of eggs that are good for you
by Phillip J. Eby 14 Apr '08

14 Apr '08

At 10:49 PM 4/8/2008 -0400, Stanley A. Klein wrote: >On Tue, April 8, 2008 9:37 pm, Ben Finney ><bignose+hates-spam(a)benfinney.id.au> wrote: > > Date: Wed, 09 Apr 2008 11:37:07 +1000 > > From: Ben Finney <bignose+hates-spam(a)benfinney.id.au> > > Subject: Re: [Distutils] how to easily consume just the parts of eggs > > that are good for you > > To: Distutils-Sig(a)Python.Org > > > > > > zooko <zooko(a)zooko.com> writes: > >> eyes and said "So they are planning to reinvent apt!". > > > > That's pretty much my reaction, too. > >I have the same reaction. I'm curious. Have any of you actually read PEP 262 in any detail? I have seen precious little discussion so far that doesn't appear to be based on significant misunderstandings of either the purpose of reviving the PEP, or the mechanics of its proposed implementation. >I have tried in the past to use easy_install, but have run into problems >because there is no communication between easy_install and the rpm >database, resulting in failure of easy_install to recognize that >dependencies have already been installed using rpms. This problem doesn't exist with Python 2.5, unless you're using a platform that willfully strips out the installation information that Python 2.5 provides for these packages. >A database focused only on Python packages is highly inappropriate for >Linux systems, violates the Linux standards, and creates problems because >eggs are not coordinated with the operating system package manager. The revamp of PEP 262 is aimed at removing .egg files and directories from the process, by allowing system packagers to tell Python what files belong to them and should not be messed with. And conversely, allowing systems and installation targets *without* package managers to safely manage their Python installations. > The >way to achieve a database for Python would be to provide tools for >conversion of eggs to rpms and debs, Such tools already exist, although the conversion takes place from source distributions rather than egg distributions. >to have eggs support conformance to >the LSB and FHS, Applying LSB and FHS to the innards of Python packages makes as much sense as applying them to the contents of Java .jar files -- i.e., none. If it's unchanging data that's part of a program or library, then it's a program or library, just like static data declared in a C program or library. Whether the file extension is .py, .so, or even .png is irrelevant.

14 44

[Enthought-dev] Installation woes - again..
by Hans Meine 14 Apr '08

14 Apr '08

Hi, Dave P. has recently cross-posted here a mail from the enthought-dev thread I started, in which I describe serious problems installing the ETS with easy_install: https://mail.enthought.com/pipermail/enthought-dev/2008-April/013913.html I have tried setuptools 0.6_rc8 as distributed with Gentoo, 0.6_rc7 as installed by virtualenv, 0.6_rc8 as installed with ez_setup.py, and finally setuptools from SVN. I am getting really strange errors which others confirmed not to make much sense; please see the above description or the follow-up posts. (A detailed log is here: http://kogs-www.informatik.uni-hamburg.de/~meine/clean-ets-install.log ) I have worked around this specific problem for me, but I herewith offer to provide additional information in case anyone wants to get this fixed. Have a nice day, Hans

1 0

Re: [Distutils] Distutils-SIG Digest, Vol 60, Issue 29
by Stanley A. Klein 14 Apr '08

14 Apr '08

On Mon, April 14, 2008 1:37 am, distutils-sig-request(a)python.org wrote: > Message: 4 > Date: Mon, 14 Apr 2008 14:16:13 +1200 > From: Greg Ewing <greg.ewing(a)canterbury.ac.nz> > Subject: Re: [Distutils] how to easily consume just the parts of eggs > that are good for you > To: distutils-sig(a)python.org > > David Cournapeau wrote: >> There are two ways of looking at it, I think. One is to think that linux >> FHS (and unix in general) is totally broken. > > I don't think it's *totally* broken. I do think it goes > overboard with splitting things up and scattering them > around. I understand that there are reasons for some of > that, but I don't see why e.g. includes, library files > and other resources used by a package can't be kept > together. > Linux and the FHS aren't broken at all. They are just designed around a different concept than the single-user, proprietary software concept that Windows and Mac OS X are based on. They are based on multi-user systems with reusable software and multi-use software tools. They provide a natural environment for community developed and maintained Free/Open-Source Software (FOSS). You can think of Linux splitting things up and scattering them around. I think of Windows putting things that don't belong together in the same place just because they happen to be supplied by the same provider. Every Windows application is monolithic, because a proprietary provider who depends on something other than the OS buys it, pays the royalties for its use, hides it from the user by compiling it into the application binary, and includes it in the software package supplied. This doesn't usually happen in Unix or Linux. Providers depend on each other. That's the reason for dependencies. Trying to use FOSS on Windows creates issues that have to be addressed. BTW, if eggs are analogous to jars (as Eby states), eggs are absolutely not a complete packaging system and were never intended to be so. If you look at a java-related rpm package, you are likely to see a number of jar files along with a lot of other files. Stan Klein

1 0

how to easily consume just the parts of eggs that are good for you
by zooko 13 Apr '08

13 Apr '08

Folks: Here is a simple proposal: make the standard Python "import" mechanism notice eggs on the PYTHONPATH and insert them (into the *same* location) on the sys.path. This eliminates the #1 problem with eggs -- that they don't easily work when installing them into places other than your site-packages and that if you allow any of them to be installed on your system then they take precedence over your non-egg packages even you explicitly put those other packages earlier in your PYTHONPATH. (That latter behavior is very disagreeable to more than a few prorgammers.) This also preserves most of the value of eggs for many use cases. This is backward-compatible with most current use cases that rely on eggs. This is very likely forward-compatible with new schemes that are currently being cooked up and will be deployed in the future. Regards, Zooko

9 11

[issue7] Test 1
by Martin v. Löwis 13 Apr '08

13 Apr '08

New submission from Martin v. Löwis <martin(a)v.loewis.de>: Testing the installation ---------- messages: 7 nosy: loewis priority: bug status: unread title: Test 1 _______________________________________________ Setuptools tracker <setuptools(a)bugs.python.org> <http://bugs.python.org/setuptools/issue7> _______________________________________________

1 0

Re: [Distutils] shebang line modified by setuptools
by Stephen Waterbury 13 Apr '08

13 Apr '08

Gael Varoquaux wrote: > On Sat, Apr 12, 2008 at 09:41:08PM -0400, Stephen Waterbury wrote: >>> These are all broken and you should report bugs on them. I have >>> reported many for Ubuntu. A system application should only ever >>> depend on the system Python (or interpreter), never on the whims of >>> your $PATH. > >> I agree with Barry here -- I would even go further and say that >> the Python used by system apps (i.e., apps that the OS depends on) >> should be separate and should not be on the default PATH (so the user >> can't use it accidentally), and no non-system apps should use the >> system Python unless specifically installed by the user or sysadmin >> into the system Python's area. > > I agree with you, and not with Barry that all these are broken. If you > look at the apps that use "#!/usr/bin/env", they are all pretty > non-system apps, like for instance "f2py". I think it is good that these > apps use the default python: I don't want to have f2py to be running for > another python than my default one. I am actually quite surprised to see > that ipython is installed with a hardcoded path to /usr/bin/python. > > Anyway, I am not going to bug report anything, as my system works just > fine, thank you. Yes, your luck holds -- as does mine, for now ... but since Barry works for Canonical (besides being Python's new Release Manager), he has a civic responsibility to report Ubuntu Python bugs even if his own system works ... and since that can be quite time-consuming, I can't blame him for trying to recruit some help ... ;) Steve

2 1

Re: [Distutils] how to easily consume just the parts of eggs that are good for you
by Stanley A. Klein 11 Apr '08

11 Apr '08

On Fri, April 11, 2008 6:00 am, Dave Peterson <dpeterson(a)enthought.com> wrote: > > Message: 5 > Date: Thu, 10 Apr 2008 18:16:03 -0500 > From: Dave Peterson <dpeterson(a)enthought.com> > Subject: Re: [Distutils] how to easily consume just the parts of eggs > that are good for you > To: "Phillip J. Eby" <pje(a)telecommunity.com> > Cc: distutils-sig <distutils-sig(a)python.org> > > Phillip J. Eby wrote: >> At 03:48 PM 4/10/2008 -0500, Dave Peterson wrote: >>> Stanley A. Klein wrote: >>>> On Wed, 2008-04-09 at 18:17 -0500, Dave Peterson wrote: >>>>> I think I can sum up any further points by simply asking: "Should it >>>>> be safe to assume I can distribute my application via eggs / >>>>> easy_install just because it is written in Python?" >>>> >>>> I think that based on this discussion the bottom line answer to this >>>> question is "No". >>> >>> I agree that it seems like that's where things are headed in the >>> discussion. But the discussion doesn't always coincide with the >>> reality, right? I guess I'm trolling more for a statement from the >>> setuptools maintainer here. >>> >>> Particularly since I'm looking for an answer to my question about >>> should Enthought continue to invest time into a setuptools patch that >>> lets developers include docs, config files, etc. in eggs for >>> installation in a FHS-approved location at install time? >> >> I think it's more than reasonable to define a standard for including >> such data. I'm somewhat more skeptical about doing that installation >> automatically within easy_install. Likewise, I'm skeptical about >> doing other sorts of non-package, non-script installation. I'd like >> to see proposals that show due care to cross-platformness, >> uninstallability, etc. >> >> In other words, when it comes to a "patch" -- the documentation is >> going to count for a lot more than the code, and I'd rather see a >> concrete proposal well in advance of the patch. >> >> Sooner would be better than later, too, because it's likely that the >> plan for "non-egg installs" is going to be affected by the plan as well. > > I believe I understand, and agree, with your concerns. Let me be clear > on the status of where we are in our work: we've internally talked > through a number of design possibilities, and are now trying out (via > hacking on setuptools) how the one we thought was "best" worked out. In > particular, we're concerned about the difficulty of use in terms of even > just the use-cases we have for ETS projects. Also, since we do a bit > of cross-platform deployment, we're also investigating those effects on > the design as well. That being said, I don't think we're ready to put > forward a proposal that would withstand too much public scrutiny quite > yet - at least if the resulting discussion implied a significant time or > effort commitment on our part to carry the conversation forward. If it > sounded like we'd already figured it all out, I apologize for getting > people's hopes up! I just wanted to make sure further pursuit in this > direction on our part wasn't completely wasted. > > The above not withstanding, if anyone is interested in talking about it > / helping us, we certainly wouldn't ignore you. I just can't promise > immediate responses due to pretty pressing customer commitments on our > part. > > > Regarding the mention of 'uninstallability' above, I assume it would be > sufficient if the installed files were simply included in the generated > list of files provided by the "--record" option since there is currently > no uninstall command at all? Or is there something else you'd like to > see as an intermediate measure? I'd love to add an uninstall option to > easy_install as well (it's something we get hit about alot by our user > community) but there's only so many hours in a day. ;-) > Here are a few principles I think I've gleaned from this discussion: 1. Eggs are not intended to serve as substitutes for rpms, debs, and other platform package distribution systems. They are primarily intended as a convenience for developers during development. They can be used in distribution packaging in a manner analogous to .jar files in java. 2. Every platform wants to put the files somewhere different. Accordingly, Setuptools and Distutils need to support mapping the files to the preferred locations for the various platforms. 3. Setuptools and Distutils support some platform packaging systems, such as rpm. Packagers should distribute using those systems. If eggs are included in a package, they should be treated as any other file for that packaging system and not as a form of distribution. For example, an rpm should reasonably be able to contain egg files along with other distribution files. (Note that java-related rpms often contain .jar files.) Consideration should be given to adding features to existing support for platform packaging systems. Also, if egg files are to be included in a distribution package, it needs to be a two-step process (first make the eggs, then make the package). 4. It would probably help to have a style guide or automated converter for naming and placement of files in a source distribution to facilitate file location mappings and platform naming issues. For example, Windows accepts file names with spaces in them but Linux generally doesn't (and bdist_rpm produces errors if it sees them). For a multiplatform distribution either a style guide or a lint-type tool needs to address issues like this. Stan Klein

1 0

Re: [Distutils] [PATCH][setuptools] Add test_runner support to setup
by Phillip J. Eby 11 Apr '08

11 Apr '08

At 06:07 PM 4/11/2008 +0200, Tarek Ziadé wrote: >On Fri, Apr 11, 2008 at 6:01 PM, Phillip J. Eby <pje(a)telecommunity.com> wrote: > > > > > >was the patch satisfactory or is it still lacking? > > >Or do I need to do anything more for it to be applied? > > > > It's sitting in my folder of patches to eventually be applied. :) > >What about this one ? > >http://mail.python.org/pipermail/distutils-sig/2008-April/009187.html Yes. >Do you have a public tracker where we can submit patches ? No -- but I heard it's being worked on. My first preference would be a python.org hosted Roundup instance, especially if I can use my existing login. If that can't be arranged, I'll probably look into throwing up a Trac or Roundup myself -- eventually.

1 0

Batch installation of python extentions
by Elazar Leibovich 10 Apr '08

10 Apr '08

Hi, I need to install a big bunch of python extentions in a couple of offline windows XPcomputer. Is there any standard way to do that on the win32 platform? The best options I thought of is installing ez_install manually on every computer, and then install all the egg-files in a certain directory. Is there any better way to do that? Is there a standard way to distribute python extentions. Example extentions: scipy, pyserial, wxpython

1 0

Re: [Distutils] setuptools and pkg registration
by Phillip J. Eby 10 Apr '08

10 Apr '08

At 11:37 PM 4/10/2008 +0200, Jeroen Ruigrok van der Werven wrote: >Phillip, > >I am not sure (haven't verified) if my email ever got through to the list, >but with FreeBSD's Perl installation an additional module gets installed >that registers every CPAN installed module as a pkg in FreeBSD's pkg system. > >Given the recent bruhaha about the setuptools and eggs (hey, from a >programmer and sysadmin point of view I like 'm) I wonder if this would be >easier for the discussion if you could dump a prototype on the table. > >If it would help I can see if I can cook up something. At this point, it has become clear to me that the package database isn't really necessary for other systems. easy_install could simply not uninstall things that it doesn't find listed in its *own* metadata. Really, the only technical hurdle remaining to solving egg-free installation is setting up a namespace package's __path__ correctly, in a situation where: 1. The user has system packages contributing to the namespace in a "site" directory, 2. There is no __init__.py in the namespace under that site directory, 3. There are non-egg packages on PYTHONPATH that contribute to the same directory, and 4. Two versions of the same module within that namespace have been installed, one in site-packages and the other on PYTHONPATH Actually, there are various other criteria involved, such as whether pkg_resources is imported or not, and whether the setuptools site.py hack is employed in the PYTHONPATH directory. For example, if pkg_resources were imported (which normally happens when setuptools-installed scripts are run), it would be possible to have it do the sorting, and thus ensure that there's never a problem like that. What it would not fix, is the circumstance where you just pop into the Python interpreter with your PYTHONPATH set, if you have the above situation. And the above situation is going to occur anytime you're using PYTHONPATH to get a later version of something that uses a namespace package. So, the other alternative is to keep the 'site.py' hack in effect, but either change the contents of that site.py or add extra information to a .pth file so that the namespaces get sorted at that point. There would still be a hacked site module at that point, but at least upgrading the site.py would let us get rid of the need to load any .pth files from the directory, put eggs ahead of other things, etc. For that matter, the site.py wouldn't be needed unless you were installing things with namespace packages in it. (Unfortunately, looking at site-packages to see if the hack is needed won't help, since you might install a conflicting system package *later*.)

1 0

Re: [Distutils] [Enthought-dev] Installation woes - again..
by Dave Peterson 10 Apr '08

10 Apr '08

NOTE: cc'ing distutils-sig for help debugging potential problem with setuptools on gentoo. Hans Meine wrote: > Am Donnerstag, 10. April 2008 09:19:31 schrieb Hans Meine: > >> The same errors occur without virtualenv, so you're obviously right that >> this is setuptools-related. I could try again and really deinstall >> Gentoo's setuptools (instead of just locally installing it the official >> way), but I really doubt now that this would make a difference. >> > > So, I have even tried this - deinstalling setuptools altogether, and > installing everything from scratch. I used "script" to create a log file of > the process with all input and resulting output + errors: > http://kogs-www.informatik.uni-hamburg.de/~meine/clean-ets-install.log > Thanks for providing that full log! From looking at that, I'm really starting to suspect a bug in setuptools 0.6c8 as I can't make any sense of the following sequence of events. While it's possible we've done something wrong in our setup.py scripts, I have my doubts as this release appears to install fine for users on Windows, OSX, and RH. (For distutils-sig readers, Hans is using Gentoo.) 1) During the first install attempt of "ets", the log shows two lines reporting success with installing enthought.tvtk-2.0.2-py2.4-linux-i686.egg. One right after downloading and building enthought.tvtk from the source tarball, and one in the middle of the enthought.kiva build. 2) Regarding that enthought.kiva build, the log indicates that a dependency caused a search to be made for "enthought.kiva>=2.0.3.dev,<=2.0.3" and a match was found (enthought.kiva-2.0.3.tar.gz) The log then shows this building, apparently a successful build (no error messages, just warnings) up until the normally reported "Installed /path/to/egg" line where it mentions success with installing enthought.tvtk rather enthought.kiva. After that, it reports being unable to find the enthought.kiva distribution. 3) You then re-run the "easy_install -f ... ets" command which has no problem building enthought.kiva this time using the exact same source tarball. I have no idea what might be causing this sequence of events. Has anyone seen anything similar with setuptools? -- Dave

1 0

Re: [Distutils] Proposal: egg -> bdist_wininst converter
by Phillip J. Eby 10 Apr '08

10 Apr '08

At 09:10 AM 4/10/2008 +0100, Paul Moore wrote: >For me, on Windows, the biggest negative aspect of eggs is when people >provide binary installations (ones with compiled C extensions) in eggs >only. I prefer bdist_wininst/bdist_msi installers, and with compiled >code, I can't always build from source (complex library dependencies, >etc). > >So, given that an egg already has everything that is needed at >runtime, how easy would it be to create a tool which converted an egg >into a bdist_wininst installer? With that, I could happily forget >about eggs. http://mail.python.org/pipermail/python-dev/2008-March/077959.html Basically what you need to do is rewrite the zipfile index such that everything is prefixed with PURELIB/ or PLATLIB/, except for the EGG-INFO directory which must be renamed to match the original egg's filename, plus '-info' -- i.e., foo-1.2-py2.3-win32.egg should have its EGG-INFO renamed to PURELIB/foo-1.2-py2.3-win32.egg-info or PLATLIB/foo-1.2-py2.3-win32.egg-info. Once the egg is reformatted in this way, it's a perfectly valid zipfile that the existing bdist_wininst code could be reused to turn into an .exe.

2 1

Proposal: egg -> bdist_wininst converter
by Paul Moore 10 Apr '08

10 Apr '08

For me, on Windows, the biggest negative aspect of eggs is when people provide binary installations (ones with compiled C extensions) in eggs only. I prefer bdist_wininst/bdist_msi installers, and with compiled code, I can't always build from source (complex library dependencies, etc). So, given that an egg already has everything that is needed at runtime, how easy would it be to create a tool which converted an egg into a bdist_wininst installer? With that, I could happily forget about eggs. Paul. PS I am assuming that "eggs are only a packaging format" - and that all of the setuptools goodies like namespace support, entry points etc, that some projects benefit from, would still work whether the package was an egg or a conventionally-installed bdist_wininst...

1 0

Re: [Distutils] A Modest Proposal for "A Database of Installed Packages"
by Phillip J. Eby 09 Apr '08

09 Apr '08

At 05:05 PM 4/7/2008 -0400, Alexander Michael wrote: >a. I believe that having side-car files that sit alongside >packages because they have the same base name makes the database more >transparent to the uninitiated. I'm not aware that this was ever a stated design goal, nor why it should have any priority. OTOH, files named by distribution would be at least as, if not even *more* transparent than package names, so I don't see any particular benefit to this. >Just browsing a directory of python >packages will allow you to see what's going on. Moving like-names >files around manually maintains the integrity and availability of the >data. Moving anything manually, other than the *entire* directory, will be unlikely to retain any form of integrity, so it's best not to give the false impression that it would. >I think that having magic entries in an essentially "hidden" >directory somewhere will cause all sorts of trouble that could be >avoiding at the cost of a small bit of duplication. > b. I assume, perhaps incorrectly, that most distributions contain >only a single package. Very incorrectly, unless you mean a single top-level package. Odds are fairly good that if there's a package, there's probably at least a subpackage, too, like perhaps a tests subpackage. >That said, I do agree that if you are primarily interested in a >database of *distributions* (as opposed to *packages*) then something >like is proposed in PEP 262 makes more sense (but it would have to be >per directory and not site-wide due to the dynamic nature of the >python path). That's exactly what I want. The only reason I didn't just implement easy_install using a per-directory form of PEP 262 is that I wanted something done rather more immediately. That was years ago, so I can afford to be more patient now. :) >This is a trade-off between putting the metadata up >front in an obvious and easy to understand way so that it will >hopefully have a better chance of being noticed and maintained, versus >tucking it away hidden someplace so that even though it is broken, it >doesn't bother anyone until they care enough to fix it. *It is this >trade-off that I am exploring with this strawman "counter" proposal to >PEP 262.* Someone would have to be crazy to maintain this information by hand. So I'd actually consider it an advantage if the file format made this fact plain, by using something that's difficult for a human being to maintain, like say a pickle. ;-) OTOH, it's possible that some system packagers will not wish to use Python to generate the files, so using something a bit less complex would be a good idea. The format proposed by PEP 262 isn't really that bad of a trade-off in those terms. >2. The strawman proposal did not explicitly address how optional >add-on tools (like setuptools) might manage namespace packages. I think there's some mistunderstanding here about the proposal's goals. If the proposal doesn't work for setuptools, it doesn't work, period. The entire point is to allow setuptools to do its work without annoying the people who don't want to use it. >I agree with Floris that the best way to avoid magic is to actually >have the sub-packages in a namespace share the same parent directory >on disk. I agree with this also. The issue is that an __init__.py must exist for this to happen, but most system packaging tools (e.g. RPM) require that a given file be owned by at most one system package (i.e., distribution), whereas the contents of a namespace package are assembled from multiple distributions. That's the problem that needs solving, not runtime support for the namespace itself. >4. Concerns were raised about the performance penalty for using the >side-car style files without version numbers possibly not all of which >were located at the top-most level of the directory listed in the >python path. > >Any add-on tool that actually used the data would very likely need to >build a cache of the data using a more efficient representation, This is a misunderstanding of the point I raised. Floris merely asked why there were version numbers in .egg-info files, and I answered him. That doesn't actually have much, if anything, to do with the package database proposal. It's merely how installed distributions' versions can be recognized quickly at runtime, not anything to do with how potential installation conflicts are handled at installation time. easy_install uses eggs for installation simply because it need never worry about file ownership conflicts. There is a direct mapping from a distribution to its files: the contents of a zipfile or subdirectory. This also allows for (relatively) straightforward uninstallation. The goal of the proposal, then, is to have a way for easy_install to have another way to map from a distribution to its owned files (and vice versa), so that eggs are not necessary for normal, single-version installations.

2 1

Re: [Distutils] [Python-Dev] how to easily consume just the parts of eggs that are good for you
by Phillip J. Eby 09 Apr '08

09 Apr '08

At 11:48 PM 4/9/2008 +0100, Paul Moore wrote: >On 09/04/2008, Phillip J. Eby <pje(a)telecommunity.com> wrote: > > It would be, if .eggs were a packaging format, rather than a binary > > distribution/runtime format. > > > > Remember "eggs are to Python as jars are to Java" -- a Java .jar > > doesn't contain documentation either, unless it's needed at > > runtime. Same for configuration files. > >And yet, Java doesn't have an equivalent of easy_install for jar >files, to my knowledge. Actually, OSGi and Eclipse plugins and "feature sites" come quite close, and setuptools rips off many of its features from them. OSGi is basically a standard for additional .jar metadata to encompass dependencies and other info.

1 0

Distutils setting permissions during build
by Reuben Thomas 09 Apr '08

09 Apr '08

Hi, This is the only obvious up-to-date address or mailing list I could find, so feel free to direct me where I should go. I'm a Linux user, and normally I have a 0027 umask. This means that when I build a Python package using distutils, the files in the build directory are not world-readable (Python copies them there without preserving permissions, according to the comment in build_py.py: # XXX copy_file by default preserves mode, which appears to be the # wrong thing to do: if a file is read-only in the working # directory, we want it to be installed read/write so that the next # installation of the same module distribution can overwrite it # without problems. (This might be a Unix-specific issue.) Thus # we turn off 'preserve_mode' when copying to the build directory, # since the build directory is supposed to be exactly what the # installation will look like (ie. we preserve mode when # installing). This is all very well, but turning off "preserve_mode" is not enough to ensure this in the presence of a restrictive umask like mine; the permissions need rather to be explicitly set (as done for example by the GNU install tool, though that assumes that permissions are set at install time rather than build time). The obvious workaround is for me to remember to change my umask when I'm building python packages. That's OK, but is this problem recognised as such and might it be fixed in future? -- http://rrt.sc3d.org/ | God is the name of our ignorance (Grayling)

1 0

Re: [Distutils] [Python-Dev] how to easily consume just the parts of eggs that are good for you
by Phillip J. Eby 09 Apr '08

09 Apr '08

At 10:30 AM 4/9/2008 -0700, zooko wrote: >PEP 262 sounds like a non-starter to me because > >1. It appears to be backwards-incompatible with setuptools/ >easy_install/eggs, thus losing all the recently gained cooperation >that I mentioned in the previous paragraph, and No. It provides a forward-compatibility path for the distutils, so that easy_install doesn't have to install things in .egg format in the future. There's no compatibility breakage at all. >2. It defines a new database file, No, it defines *files*. One file per installed distribution, containing (among other things) an installation manifest. >where I would prefer either: > a. Doing away with database files entirely and relying on the >filesystem alone to hold that information, or ...which is what PEP 262 *does*. Unfortunately, PEP 262's title is bad for marketing, as you've effectively pointed out. It would be better titled something "package installation manifests" or "package contents files", or something of that sort. > b. Continuing to use the current ".pth" database file format, >possibly improved by having native support for .pth files in the >Python import machinery. These mechanisms are orthogonal to this issue. >3. Because of #2, it triggers programmers to exclaim "They are >planning to reinvent apt!", thus making it unlikely that the new >proposal will recapture the cooperation that setuptools has already >(slowly) gained. Yeah, we need a new name. Everybody is going off of "database of installed packages" and thinking "apt", because they aren't paying any closer attention. However, given that we are discussing this on Python-Dev and distutils-sig, I do think it's reasonable to expect (if perhaps not reasonable to receive) that people discussing the PEP have *read* the freaking PEP first, prior to trashing it or offering alternatives. And it's not like I'm personally offended or anything -- I didn't even write the PEP in question. But what's the point of having PEPs if people read nothing but their titles? We could just delete everything but PEP 0. :) >Perhaps PEP 262 and my proposal are not actually alternatives, but >are complementary. As I've already pointed out, your proposal does not address multiple installed versions of a package, and I see no sane way to modify it to do so. >What I want is for the already implemented, tested, and deployed >code- re-use features of setuptools/easy_install to be more widely >accepted. This is best and most easily achieved by fixing the two >most frequent objections to setuptools/easy_install: 1. That you >can't conveniently install into an arbitrary directory, and 2. that >it subverts the meaning of your PYTHONPATH. As I've already stated, the only way for these problems to be fixed is for easy_install to not install files in .egg form -- which also solves the general objection to using .eggs in the first place. And the only way to do that, is to have a way to keep track of what files are installed. Rather than have easy_install come up with its own way of doing that, I would prefer to share a standard with the distutils. Hence, the PEP discussion. For earlier versions of Python, it will still be possible to install and uninstall with setuptools using this approach. You just won't be able to uninstall pure distutils-based packages, unless you installed them using easy_install. Meanwhile, it has occurred to me that the easiest way of handling compatibility is not to require that other packaging tools mark their files for non-removability, but simply not allow easy_install to remove or overwrite anything that *isn't* claimed by a manifest. In that way, easy_install would be immediately usable in the new mode, without any updates to Python or to system packaging tools.

2 2

Re: [Distutils] how to easily consume just the parts of eggs that are good for you
by Stanley A. Klein 09 Apr '08

09 Apr '08

On Wed, April 9, 2008 4:27 pm, distutils-sig-request(a)python.org wrote: > Message: 5 > Date: Wed, 9 Apr 2008 21:21:09 +0100 > From: Floris Bruynooghe <floris.bruynooghe(a)gmail.com> > Subject: Re: [Distutils] how to easily consume just the parts of eggs > that are good for you > To: distutils-sig(a)python.org > > On Wed, Apr 09, 2008 at 02:26:31PM -0400, Stanley A. Klein wrote: >> I don't know what Windows add/remove >> programs function does, but all it might do is to run the executable to >> install packages and record the installation (as was previously done by >> third party programs) to facilitate clean removal. Unless you can >> perform >> more of the other functions I listed above, I doubt I would call >> add/remove a package manager. > > Ugh, you have yet to discover the horrors/wonders of MSI (I wish I was > as naive as you here!). A properly installed windows program will > install using an MSI database, registering each file, registry setting > etc. Often a setup.exe will still interface with the MSI database in > the background (they should, there's a C API for it too). MSI will > even do stuff like reference counting how many programs need a certain > file (in case you have something installed in a shared directory), > figure out what to do on conflict etc. They have many anc complicated > rules, options and features. > > As far as I am concerned MSI (and thus Add/Remove Programs) can be > treated as a package manager in windows. > I have 3 desktops and a laptop. They are all at least dual boot. One of the systems on each machine is Windows. All the others are Linux, including Fedora 7, Fedora Core 5, Ubuntu 7, or CENTOS 5 in some combination on each machine. My greatest use of Windows is at tax time, because the good tax programs aren't released for Linux. I also have a mid-1990's version of QuickBooks that I still use. Aside from those applications, my use of Windows is sporadic at best, maybe a few times every few months. I do everything else in Linux. My exposure to Windows is minimal, so my exposure to MSI is even less. I wouldn't call it naivete. I just don't do Windows. :-) Stan Klein

1 0

A Modest Proposal for "A Database of Installed Packages"
by Alexander Michael 09 Apr '08

09 Apr '08

I'll continue my fool hearty effort [1] to build a concrete proposal for "a database of installed packages" by offering up a sketch of a possible straw-man "solution". I realize that this is likely oversimplified to a fault, but I hope it will help us move forward. Apologies if the equivalent of this has been proposed and rejected before. My proposal is basically to make PKG-INFO functional and usable by: * Fixing the technical issues with requirements (i.e. dependencies) and naming as specified by PEP 314/345. * Modifying distutils to install PKG-INFO alongside each module file or package directory as a side-car file of the same name but with a special extension (.pyi or whatever). These files would be the place to include the optional list of installed files as well as the optional md5sums, if desired by the installer. Files in the package will be listed using relative paths, while far flung files (bin, shared, etc) will get full paths so that there is full allowance for relocating simple (nothing in bin or shared) modules and packages. Although optional, "python setup.py install" will include the installed file list by default. That's it. The intent is to provide just enough information to allow the development of tools to use it, for those that are interested, while being minimally invasive to developers that are not interested in such tools. To determine the current state of your python environment, walk sys.path looking for modules and packages, collecting PKG-INFO when available. No standard centralized database. Some of us will choose to opt-in to a particular installation management tool that might maintain a cache (centralized or per-directory) for efficiency, but that would be considered a performance optimization for that particular tool. We can also bootstrap older python installations by creating an online database (that can of course be downloaded by security conscious individuals for offline querying) that maps (module-file/package-directory name, md5sum) pairs to their respective PKG-INFO contents (no list of installed files) which can be queried by an automated sys.path walker to fill-in missing side-car files. Thus, I can opt-in to this scheme for python 2.5 by installing a distutils patch and meta-data side-car bootstrapper that does its best to identify what's on my sys.path. It would be quite tractable to maintain this for the python standard library and perhaps the official installations of a few major OS versions. Such a database could even be used for the community to provide metadata for packages that the developer didn't (again, furthering an opt-in mentality). Of course, even though it worked for CDDB, it would likely be too much to expect this level of coverage through user submitted entries. [1] http://mail.python.org/pipermail/distutils-sig/2008-March/009108.html

3 7

Re: [Distutils] how to easily consume just the parts of eggs that are good for you
by Stanley A. Klein 08 Apr '08

08 Apr '08

On Tue, April 8, 2008 9:37 pm, Ben Finney <bignose+hates-spam(a)benfinney.id.au> wrote: > Date: Wed, 09 Apr 2008 11:37:07 +1000 > From: Ben Finney <bignose+hates-spam(a)benfinney.id.au> > Subject: Re: [Distutils] how to easily consume just the parts of eggs > that are good for you > To: Distutils-Sig(a)Python.Org > > > zooko <zooko(a)zooko.com> writes: > >> I am skeptical that prorgammers are going to be willing to use a new >> database format. They already have a database -- their filesystem -- >> and they already have the tools to control it -- mv, rm, and >> PYTHONPATH. Many of them already hate the existence the >> "easy_instlal.pth" database file, and I don't see why a new database >> file would be any different. > > Moreover, many of us already have a database of *all* packages on the > system, not just Python-language ones: the package database of our > operating system. Adding another, parallel, database which needs > separate maintenance, and only applies to Python packages, is not a > step forward in such a situation. > >> They both agreed that it made perfect sense. I told one of them >> about the alternate proposal to define a new database file to >> contain a list of installed packages, and he sighed and rolled his >> eyes and said "So they are planning to reinvent apt!". > > That's pretty much my reaction, too. I have the same reaction. I don't install eggs (unless they are installed through my operating system package manager) or use easy_install. My systems have rpms (for Fedora and CENTOS) and debs (for Ubuntu). There are also a Linux Standards Base and a Unix Filesystem Hierarchy Standard (cited by the LSB) that rpms and debs generally enforce, but eggs often do not. I have tried in the past to use easy_install, but have run into problems because there is no communication between easy_install and the rpm database, resulting in failure of easy_install to recognize that dependencies have already been installed using rpms. Also, there are tools provided with rpm and apt to perform functions such as querying installed packages for their contents. I use "rpm -qil" frequently to find what files a package has installed on my system and where they are installed. A database focused only on Python packages is highly inappropriate for Linux systems, violates the Linux standards, and creates problems because eggs are not coordinated with the operating system package manager. The way to achieve a database for Python would be to provide tools for conversion of eggs to rpms and debs, to have eggs support conformance to the LSB and FHS, and to use rpm or apt for package management. Stan Klein

1 0

FW: [issue2581] Vista UAC/elevation support for bdist_wininst
by Mark Hammond 07 Apr '08

07 Apr '08

Hi all, I've just added this patch which adds basic UAC support to bdist_wininst in a fairly unobtrusive manner. I've added Thomas to the nosy list, but I'd welcome any feedback or concerns from everyone. Thanks, Mark -----Original Message----- From: Mark Hammond [mailto:report@bugs.python.org] Sent: Tuesday, 8 April 2008 12:43 PM To: mhammond(a)users.sourceforge.net Subject: [issue2581] Vista UAC/elevation support for bdist_wininst New submission from Mark Hammond <mhammond(a)users.sourceforge.net>: The attached patch adds basic UAC support to bdist_wininst created installers. A new option '--user-access-control' has been added to bdist_wininst, which is written to the INI file read by the installer. The installer reads this value: if it is 'force', elevation is always requested, if it is 'auto', elevation is requested when Python is installed in HKLM. 'none' (the default) means nothing UAC related happens at all. The elevation happens by having the installer re-execute itself using ShellExecute. I've also ensured the code builds for all versions of VS we support. As a result, it was necessary to change the old bdist_wininst project files to point to the later zlib version Python itself uses. All these changes are in the patch. ---------- assignee: mhammond components: Distutils files: bdist_wininst_uac.patch keywords: patch, patch messages: 65139 nosy: mhammond, theller severity: normal status: open title: Vista UAC/elevation support for bdist_wininst type: behavior versions: Python 2.6 Added file: http://bugs.python.org/file9978/bdist_wininst_uac.patch __________________________________ Tracker <report(a)bugs.python.org> <http://bugs.python.org/issue2581> __________________________________

1 0

Py2exe, on PyGame gives problems ...
by Stef Mientki 07 Apr '08

07 Apr '08

hello, I'm using the orginal Enthought suite 2.4.3 and added Pygame to it. Now everything works fine, until .. ... I try to build a windows executable with Py2Exe. All kinds of numarray modules are not detected by Py2Exe. Adding the missing numarray (which one, Enthought contains more of them), later on manual to the dirstibution didn't help. Does anyone recognize this problem, and maybe even have a solution ? thanks, Stef Mientki

1 0

Remove "current" Windows executables from Lib/distutils/command in svn?
by Mark Hammond 07 Apr '08

07 Apr '08

As I was preparing to check-in my cross-compilation patch, which includes a new x64 executable in the Lib/distutils/command directory, it occurs to me that we don't actually need it in subversion - and nor do we need wininst-9.0.exe. I believe these executables are in svn for historical reasons. The build process for these executables are now integrated into the core Python build process (although they are disabled by default). There is a good use-case for keeping executables built with older MSVC versions, but I don't see one for executables built with the current version. I'd like to propose we delete Lib/Distutils/command/wininst-9.0.exe, and enable the building of that project by default in the standard build process (and I'll setup the x64 build of the executable similarly). They will then need to be re-added when a future version moves away from Visual Studio 2008, but that seems like a reasonable way to approach things. Are there any objections to this, or should we stick with the status-quo and I'll add the new x64 executable? Or do things that way *just* for x64? Thanks, Mark.

5 6

[setuptools] Windows AMD64 installation
by Luis Bruno 07 Apr '08

07 Apr '08

Hi all, I installed setuptools-0.6c8 in a Python 2.5 AMD64 machine by dropping its .egg in site-packages, a 'temp.pth' file alongside pointing to it, and doing "python -m easy_install setuptools". Was there an easier way?

1 0

[setuptools] how to report bugs and patchs ?
by Tarek Ziadé 06 Apr '08

06 Apr '08

Hello, Sorry I must be naive, but I don't understand how setuptools community works at this time. In the README.txt of the package, it is written "Questions, comments, and bug reports should be directed to the distutils-sig mailing list..." But what happens when some bug reports or patches are just ignored in the mailing list ? (even after months) I would expect patch or bug reports to be added in a bug tracker, then rejected or accepted, even if it takes months for them to be treated. At least, registered somewhere. I have read that a bug tracker was going to be set for setuptools, but the current status is rather fuzzy, so I was just wondering if this is stilled planned... Regards, Tarek -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/

2 1

ReST formatting errors when registering package can invalidate HTML conversion
by Noah Gift 06 Apr '08

06 Apr '08

I thought I would post a generally help tip that Ian Bicking helped me with. I couldn't figure out why my ReST formatted long_description field would not get converted to HTML when I registered my egg with the cheeseshop: python setup.py register It turns out I had a minor error in my ReST formatting, I didn't underline a headline with enough length. So, the moral of the story is if you wonder why you cannot get HTML formatted ReST to display when you register your egg, it is possible you have a formatting error. It was tricky for me to discover, because I did not get any error message when I uploaded to the cheeseshop, and I just missed the minor formatting error when I was viewing the converted HTML document I used to preview what I created. Noah Gift / http://noahgift.com

2 2

--allow-host and local files
by Tarek Ziadé 05 Apr '08

05 Apr '08

Hello, while playing with the allow-host options in setuptools, I have noticed that it is restricted to URLs because url_ok() uses urlparse over the regular expressions that are provided This means that it is not possible to allow local folders to be visited since a "file://*" expression for example, will lead to an empty string: >>> import urlparse >>> urlparse.urlparse('file:///tmp/my/local/file.tgz')[1] '' This will make some links blocked and impossible to add as authorized resource: ... Link to file:///tmp/tmpE-LbUpbuildouttests/setuptools/ ***BLOCKED*** by --allow-hosts ... I would like to propose something: I think this would be easy to change by calling URL_SCHEME() over the url in url_ok(), before urlparse is called. If it is not an url we could then consider that the url is "safe" and return immediatly. if you think it is a good idea, i can provide a patch with a test, Regards, Tarek -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/

1 1

setuptools bug report: _get_unpatched() is way too clever (and also returns the wrong module)
by zooko 04 Apr '08

04 Apr '08

Bug report #1: _get_unpatched() returns a module from the system site-packages instead of the module from the PYTHONPATH which ought to have precedence, leading to setuptools-0.6c8 invoking setuptools-0.6a9 in the following stack trace: PYTHONPATH="/home/buildslave/slave-tahoe/dapper/build/support/lib/ python2.4/site-packages" python ./setup.py develop --prefix="/home/ buildslave/slave-tahoe/dapper/build/support" ... Running Nevow-0.9.18/setup.py -q bdist_egg --dist-dir /tmp/ easy_install-rk83KQ/Nevow-0.9.18/egg-dist-tmp-nJ8GSy Traceback (most recent call last): ... File "/home/buildslave/slave-tahoe/dapper/build/ setuptools-0.6c8.egg/setuptools/command/develop.py", line 102, in install_for_development File "/usr/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/ easy_install.py", line 518, in process_distribution Full details: https://dev.allmydata.com/buildbot-tahoe/builders/dapper/builds/1427/ steps/compile/logs/stdio Bug report #2: _get_unpatched() is way too clever for me to spend time trying to understand and debug right now: def _get_unpatched(cls): """Protect against re-patching the distutils if reloaded Also ensures that no other distutils extension monkeypatched the distutils first. """ What the heck? I don't want to think about that, so instead we're delisting Ubuntu Dapper as a supported platform, since it is the only one of our supported platforms on which we have this problem. Generalized complaint: There are a lot of things that are too clever in setuptools/ easy_install/eggs. Cleverness makes it work in more situations, which is good, but it also means that it is harder to know when it will fail or to understand why it failed. A simpler system that satisfied 90% of the use cases while failing in more obvious ways would be appreciated. (Provided, of course, that those 90% of use cases include the ones that I really care about.) See my earlier post which proposes such a simplification: http://mail.python.org/pipermail/python-dev/2008-March/078243.html Bug report #3: setuptools/easy_install/eggs needs a bug tracker. (Also, of course, a buildbot and a rich suite of automated unit tests.) Some of this message may have been inappropriate for distutils-sig, but what else am I going to do with this issue before I delist Dapper and forget all about it? I would be willing to host a trac instance to serve as a wiki and issue tracker for setuptools, and a buildbot. (But I don't want to have anything to do with subversion -- I much prefer darcs.) Regards, Zooko

4 9

Re: [Distutils] Easy_install fails, package somehow depends on itself and won't resolve
by Phillip J. Eby 04 Apr '08

04 Apr '08

At 02:52 PM 4/3/2008 -0400, Deron Meranda wrote: >The only thing I can think is that the system I'm installing on has a >slightly different >OS version. But it should be (and is) binary compatible. Setuptools currently uses exact matching on OS strings, so that's almost certainly the problem. If you want to contribute code to do platform matching, check out the pkg_resources docs under "Platform Utilities" and then look at the code for the existing Mac OS version compatibility checking. Be sure to include tests in your patch, and the tests (as well as the version comparison code itself) must be able to run on all platforms. Feel free to ask on the list if you run into any problems/questions.

1 0

[zc.buildout] upgrade with changed explicit recipe version
by Hanno Schlichting 04 Apr '08

04 Apr '08

Hi. I ran into a problem with zc.buildout while upgrading a customers buildout and wondered if the problem I encountered is a general one. The buildout in question had one recipe pinned down to an exact version as in: [buildout] parts = plone [plone] recipe = plone.recipe.plone==3.0.2 To upgrade this environment I changed the explicit version to 3.0.6: [buildout] parts = plone [plone] recipe = plone.recipe.plone==3.0.6 When rerunning the buildout script I got an error during uninstall of the plone part. It complained that the plone.recipe.plone recipe version 3.0.2 was not available. From my understanding of this, buildout parses the version requirements first and at the point where it wants to uninstall the plone part, only the newer 3.0.6 version of the recipe is available. The workaround was to change .installed.cfg and remove the version restriction, so any version of the recipe would be accepted to uninstall the part, but I wonder if there is a general problem here or if we are misusing the version restrictions? I have the feeling that buildout should either add the old recipe to the environment to satisfy the uninstall requirement or accept newer versions of a recipe to uninstall older versions. Thoughts? Hanno

2 2

[zc.buildout] buildout unable to parse 'simple' view
by Andreas Jung 03 Apr '08

03 Apr '08

We're running a local egg repository based on haufe.eggserver. Buildout won't find an egg called 'haufe.recipe.svnsetup' that is stored on the server. The server is configured properly within the find-links section. And I can see a successful HTTP request to the page in the logs. The underlying template generates the following HTML: <html> <body> .... <a href="http://eggs.haufe-ep.de:21111/eggs/download?package=haufe.recipe.subversion…">haufe.recipe.subversion-0.1.0dev_r942-py2.4.egg</a> <br /> <a href="http://eggs.haufe-ep.de:21111/eggs/download?package=haufe.recipe.subversion…">haufe.recipe.subversion-0.1.1dev_r997-py2.4.egg</a> <br /> <a href="http://eggs.haufe-ep.de:21111/eggs/download?package=haufe.recipe.svnbuild&a…">haufe.recipe.svnbuild-0.1.1dev_r1057-py2.4.egg</a> <br /> ... I wonder why buildout won't parse this HTML properly and find the related distribution?! (Error: Couldn't distribution for 'haufe.recipe.svnsetup'). Andreas

2 1

Easy_install fails, package somehow depends on itself and won't resolve
by Deron Meranda 03 Apr '08

03 Apr '08

I'm having an issue with easy_install from setuptools 0.6c8 with egg files containing C extensions. I have created an egg from a package which includes C code. When I try to install it on another system though, the egg itself will install, but then easy_install aborts with an error while trying to resolve it's dependencies (which it shouldn't have any dependencies). The created egg file is named pysqlite-2.4.1-py2.5-hp-ux-B.11.00-9000-800.egg easy_installing it though yields an error: ----- $ easy_install -v -H "" pysqlite-2.4.1-py2.5-hp-ux-B.11.00-9000-800.egg Processing pysqlite-2.4.1-py2.5-hp-ux-B.11.00-9000-800.egg Copying pysqlite-2.4.1-py2.5-hp-ux-B.11.00-9000-800.egg to /...<PATH>.../lib/python2.5/site-packages Removing pysqlite 2.3.2 from easy-install.pth file Adding pysqlite 2.4.1 to easy-install.pth file Saving /...<PATH>.../lib/python2.5/site-packages/easy-install.pth Installed /...<PATH>.../lib/python2.5/site-packages/pysqlite-2.4.1-py2.5-hp-ux-B.11.00-9000-800.egg Processing dependencies for pysqlite==2.4.1 Searching for pysqlite==2.4.1 Link to http://pypi.python.org/simple/pysqlite/ ***BLOCKED*** by --allow-hosts Couldn't retrieve index page for 'pysqlite' Scanning index of all packages (this may take a while) Link to http://pypi.python.org/simple/ ***BLOCKED*** by --allow-hosts No local packages or download links found for pysqlite==2.4.1 error: Could not find suitable distribution for Requirement.parse('pysqlite==2.4.1') ----- However I can still use the package just fine, and it seems the same require condition works: >>> import pkg_resources >>> pkg_resources.require('pysqlite==2.4.1') [pysqlite 2.4.1 (/...<PATH>.../lib/python2.5/site-packages/pysqlite-2.4.1-py2.5-hp-ux-B.11.00-9000-800.egg)] >>> import pysqlite2 >>> pysqlite2.__path__ ['/...<PATH>.../lib/python2.5/site-packages/pysqlite-2.4.1-py2.5-hp-ux-B.11.00-9000-800.egg/pysqlite2'] Everything else works just fine too. The C library gets extracted into ~/.python-eggs and using the package works flawlessly. The only thing I can think is that the system I'm installing on has a slightly different OS version. But it should be (and is) binary compatible. System use to build egg: $ uname -s -r -m HP-UX B.11.00 9000/800 System trying to install on: $ uname -s -r -m HP-UX B.11.11 9000/800 I should also note that the python executable (and its libraries) are identical on each system. Python was actually built on the first system, and copied to the second. Yes I know sqlite is built into python 2.5 so pysqlite isn't needed, but this should work anyway, right? Can anybody help me understand what is going on, and how can I get the egg file to install without an error? Thanks -- Deron Meranda

1 0

FW: [issue2513] 64bit cross compilation on windows
by Mark Hammond 02 Apr '08

02 Apr '08

FYI, I've uploaded a patch that provides for cross-compilation on Windows between 32 and 64 bit platforms - all comments invited! Thanks, Mark -----Original Message----- From: Mark Hammond [mailto:report@bugs.python.org] Sent: Sunday, 30 March 2008 6:01 PM To: mhammond(a)users.sourceforge.net Subject: [issue2513] 64bit cross compilation on windows New submission from Mark Hammond <mhammond(a)users.sourceforge.net>: I've taken the liberty of adding Trent, Christian and Martin to the nosy list as I know they are actively, if reluctantly interested in this. This patch allows the distutils to cross-compile on Windows. It has been tested on x86 and amd64 platforms, with both platforms successfully able to natively and cross-compile extension modules and create binary distributions. To cross-compile, specify '--plat-name' to the build command (valid values are 'win32', 'win-amd64' and 'win-ia64'). This option name was chosen to be consistent with the bdist_dumb command. I've included the docs I added below (which are also in the patch), but note that as with native compilation using distutils, it's not necessary to set any environment variables or do anything else special with your environment to make this work. The patch also adds a x64 target for the 'bdist_wininst' target, which it creates as distutils/command/wininst-9.0-amd64.exe. This executable is necessary even for bdist_wininst to work natively on x64, but is still included here for simplicity. To assist with testing, I've also added a distutils setup.py script to the PC/example_nt directory. This is capable of creating bdist_wininst executables for both native and cross platforms; 'setup.py build --platname=win-amd64 bdist_wininst' will create an amd64 installer on an x86 machine. The patch has not been tested with a Visual Studio environment without cross-compile tools installed - it will obviously fail, but its not clear how ugly this failure will be. Below is the text I added to docs/distutils/builtdist.rst: Cross-compiling on Windows ===================== Starting with Python 2.6, distutils is capable of cross-compiling between Windows platforms. In practice, this means that with the correct tools installed, you can use a 32bit version of Windows to create 64bit extensions and vice-versa. To build for an alternate platform, specify the :option:`--plat-name` option to the build command. Valid values are currently 'win32', 'win-amd64' and 'win-ia64'. For example, on a 32bit version of Windows, you could execute:: python setup.py build --plat-name=win-amd64 to build a 64bit version of your extension. The Windows Installers also support this option, so the command:: python setup.py build --plat-name=win-amd64 bdist_wininst would create a 64bit installation executable on your 32bit version of Windows. Note that by default, Visual Studio 2008 does not install 64bit compilers or tools. You may need to reexecute the Visual Studio setup process and select these tools. ---------- assignee: mhammond components: Distutils files: windows-cross-compile.patch keywords: 64bit, patch messages: 64743 nosy: Trent.Nelson, ctheune, loewis, mhammond severity: normal status: open title: 64bit cross compilation on windows type: feature request versions: Python 2.6 Added file: http://bugs.python.org/file9900/windows-cross-compile.patch __________________________________ Tracker <report(a)bugs.python.org> <http://bugs.python.org/issue2513> __________________________________

3 7

How we can get rid of eggs for 2.6 and beyond
by Phillip J. Eby 02 Apr '08

02 Apr '08

So, after having some time to absorb the Python-Dev threads about setuptools, bootstrap, and all the rest, I think I see an opportunity to let people route around the "damage" of eggs, while still making it possible for the people who want to use easy_install or to put dependencies in their setup.py to get what they want, too. (And without them needing to install eggs, either.) At the same time, we can address the issues that remain around uninstalling packages, system vs. user packages, PYTHONPATH and site.py woes, and really pretty much every other complaint I've heard in the last few days about setuptools stomping on other people's stuff. (Even Paul's Windows issues, hopefully.) Now, you might be asking, "Okay, so why are you telling me about this? Why not just go fix setuptools?" Well, I *can't*. Not without some help from Python-Dev and the Distutils-SIG, to create an updated standard for installed package metadata, by updating PEP 262 ("A Database of Installed Python Packages") to include input from the system packaging folks, support for namespace packages, and support for setuptools-compatible dependency information. What's that got to do with anything? Well, without it, setuptools can't support uninstall or conflict management without using eggs to compartmentalize the installed files. And because it has to use eggs to do *that*, it has to munge .pth files and install its own site.py when installing to PYTHONPATH. All of this ugliness follows directly from the absence of a PEP 262-style installation database. Sure, setuptools could create its own version of this, and I almost did that four years ago. (If you look at the "open issues" part of PEP 262, you'll see my comments from back then.) I decided not to for two reasons: first, the distutils didn't support it yet, so it didn't help for conflict detection and avoidance in the real world at that point. Second, there were no uninstall tools for it, so I'd have had to write one myself. (Zed's "easy_f'ing_uninstall" to the contrary, it ain't easy, and I have an aversion to deleting stuff on people's systems without knowing what will break. There's a big difference between them typing 'rm -rf' themselves, and me doing it.) However, if tools exist and are distributed for such a "database", and *everybody* agrees to use it as an officially-blessed standard, then it should be possible for setuptools to co-exist with that framework, and we're all happy campers. In particular, the "installing eggs sucks" camp should be happy, because it'll be possible for me (or anyone else) to write a version of easy_install that doesn't install eggs any more, and setuptools-based packages can go back to having "setup.py install" install things the old way by default. So, to accomplish this, we (for some value of "we") need to: 1. Hash out consensus around what changes or enhancements are needed to PEP 262, to resolve the previously-listed open issues, those that have come up since (namespace packages, dependency specifications, canonical name/version forms), and anything else that comes up. 2. Update or replace the implementation as appropriate, and modify the distutils to support it in Python 2.6 and beyond. And "support it" means, "ensure that 'install' and *all* bdist commands update the database". The bdist_rpm, bdist_wininst, and bdist_msi commands, even bdist_dumb. (This should probably also include the add/remove programs stuff in the Windows case.) 3. Create a document for system packagers referencing the PEP and introducing them to what/why/how of the standard, in case they weren't one of the original participants in creating this. It will probably take some non-trivial work to do all this for Python 2.6, but it's probably possible, if we start now. I don't think it's critical to have an uninstall tool distributed with 2.6, as long as there's a reasonable way to bootstrap its installation later. Questions, comments... volunteers? :)

24 74

Making setuptools play nicer with virtualenv
by Gael Varoquaux 02 Apr '08

02 Apr '08

Hi, I have just out-clevered myself using setuptools and virtualenv: * install foo using "python setup.py develop" (foo being ipython). * download some module bar you want to work on in an isolated environment * create this isolated environment using "virtualenv bar" * in the isolated environment "python setup.py develop" the bar module. * still in the isolated environment, try to import bar in a script installed by foo (aka ipython) --> fails because foo uses the system python, and virtualenv wants you to use its own python One very easy solution to make this work is to have the setuptools generated scripts use, under unices, "#!/usr/bin/env python" rather than "#!/usr/bin/python". This seems to me like a good solution, in general, to follow the user's expectations. Is this a change that would be possible? Cheers, Gaël

3 2

Making setuptools play nicer with virtualenv
by Gael Varoquaux 02 Apr '08

02 Apr '08

Hi, I have just out-clevered myself using setuptools and virtualenv: * install foo using "python setup.py develop" (foo being ipython). * download some module bar you want to work on in an isolated environment * create this isolated environment using "virtualenv bar" * in the isolated environment "python setup.py develop" the bar module. * still in the isolated environment, try to import bar in a script installed by foo (aka ipython) --> fails because foo uses the system python, and virtualenv wants you to use its own python One very easy solution to make this work is to have the setuptools generated scripts use, under unices, "#!/usr/bin/env python" rather than "#!/usr/bin/python". This seems to me like a good solution, in general, to follow the user's expectations. Is this a change that would be possible? Cheers, Gaël

1 0

setup.py develop un-egg-spected result
by Oliver Schoenborn 31 Mar '08

31 Mar '08

I have a problem with "setup.py develop" installation. I have the following folder tree: pubsub \- setup.py \- extern \- module0.py \- core \- __init__.py \- module1.py \- module2.py \- ... I want my setup.py to install modules from pubsub/extern (such as module0.py) into site-packages, and to install the 'core' package in site-packages as pubsubcore: setup( ... py_modules = ['module0'], packages = ['pubsubcore'], package_dir = {'': 'extern', 'pubsubcore':'core'}, scripts = [], ... ) For sdist, bdist, bdist_egg etc, it works: the output shows module0 is in the "root" of the distribution, together with a package named "pubsubcore" which contains what is in 'pubsub/core'. HOWEVER, and here is the problem: when I do "setup.py develop" (this is on WinXP with Python 2.4.4 and the latest stable setuptools), only the contents of 'extern' get 'installed': C:\Documents and Settings\schoenb\My Documents\pubsub>setup.py develop running develop running egg_info writing external\PyPubSub.egg-info\PKG-INFO writing top-level names to external\PyPubSub.egg-info\top_level.txt writing dependency_links to external\PyPubSub.egg-info\dependency_links.txt writing manifest file 'external\PyPubSub.egg-info\SOURCES.txt' running build_ext Creating c:\python24\lib\site-packages\PyPubSub.egg-link (link to external) Adding PyPubSub 3.0a6 to easy-install.pth file Installed c:\documents and settings\schoenb\my documents\pubsub\extern Processing dependencies for PyPubSub==3.0a6 The above allows me, as the developer, to do "import module0" but not "import pubsubcore". Indeed, looking at \python24\lib\site-packages\easy-install.pth, there is only one new line added, pointing to c:\documents and settings\schoenb\my documents\pubsub\extern. There should be a second line added for the package, ie c:\documents and settings\schoenb\my documents\pubsub so that I (developer) can do "import pubsubcore", just as any user would be able to if I had done a full install. When I add the above line manually to the .pth file the "developer" installation works as expected. What am I doing wrong? Thanks, Oliver

4 7