Mailman 3 March 2008 - Distutils-SIG

Re: [Distutils] Question on using distutils.core.run_setup
by Terry Jones 19 Mar '08

19 Mar '08

>>>>> "Phillip" == Phillip J Eby <pje(a)telecommunity.com> writes: Phillip> At 06:27 AM 3/18/2008 +0100, Terry Jones wrote: [snip] >> If so, how can I find where the thing(s) I installed now resides? [snip] Phillip> It'd be on the 'install_lib' command instance, not the Phillip> distribution. Try Phillip> dist.get_finalized_command('install_lib').install_dir instead. For the record / archives, I think this should be dist.get_command_obj('install_lib').install_dir Thanks again, Terry

2 1

Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
by Phillip J. Eby 18 Mar '08

18 Mar '08

At 09:01 PM 3/18/2008 +0000, Paul Moore wrote: >On 18/03/2008, Marius Gedminas <marius(a)pov.lt> wrote: > > +1 for something like this in the stdlib. > > > > os.path.join(os.path.dirname(__file__), 'foo') just has too many > > problems. > >Hmm, maybe I should put some of my money where my mouth is. I'd be >willing to work up a patch to add some key items into the pkgutils >module. However, I don't like the thought of reinventing the wheel - >Phillip, would you be OK with me stealing code where I can from >pkg_resources? Sure... though I doubt you'll have much luck, unless you drop support for resource_filename, and stick with resource_string()/resource_stream(). Supporting resource_filename() for zipfiles inherently requires that you extract the file somewhere, which is where all the jillions of lines of code for import providers and importer registrations and egg caches and all that other stuff come in. By contrast, returning a resource string or stream is easy, since you can just delegate to a __loader__ object's get_data, or fall back to filesystem access if the importer is None. In Python 2.5, the pkgutil module grew a bunch of helpful APIs that originally came from pkg_resources. (If you look at pkg_resources in the setuptools 0.7 trunk, it actually uses these new pkgutil APIs, instead of the bundled versions that live in the 0.6 branch.) You should be able to use pkgutil.get_loader(modulename) to get either a loader or None, and then look for a 'get_data' method from there.

2 1

Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
by Phillip J. Eby 18 Mar '08

18 Mar '08

At 12:31 AM 3/18/2008 -0500, Guido van Rossum wrote: >I am hoping that someone will create a simpler bootstrap module that >is able to download a file of pure Python code and install it, perhaps >by running its setup.py, assuming that it only depends on distutils >(or other things previously installed). I will welcome such a module >into the stdlib. I'm not sure a PEP is even needed, though interested >parties are certainly welcome to write a PEP specifying the behavior >first. With 2.6 and 3.0 slated for release in September, there should >be enough time to get this done before then. Unfortunately, as I've already tried to explain, "download a file ... and install it" is not a sufficiently well-specified requirement to implement a robust tool. Even if it is not to support arbitrary existing distutils sources, there still needs to be a way to document precisely what the tool does and does not support installing, so that users can produce correct files for it to consume, register them properly with PyPI, etc. And as I said before (perhaps not very well) the distutils documentation has already proven to be inadequate as such a specification, both for users to create these files -- and even more important -- for programs to consume them. (For example, distutils source distribution tarball filenames are not unambiguously machine-parseable.) That's why I think some specific "format" (i.e. conventions) have to be defined for this to work, even if it's merely a set of documented restrictions on a distutils-based layout, file naming conventions, versioning, etc. In other words, you can't have your cake and eat it, too. If there's to be a bootstrap tool, you must bless *some* set of packaging conventions, including file naming, version parsing, and so on. Can we use setuptools' version parsing scheme to identify the "latest stable version", for example? What about setuptools' filename component canonicalization and escaping rules? Frankly, I don't care what the conventions are, only that they be unambiguously defined and reasonably implementable for producers and consumers alike. I just want there to be *some* sort of robust, documented, standard installation bootstrap vector in the stdlib, so that setuptools, zc.buildout, and virtualenv don't have to maintain their own (or depend on setuptools maintaining its own). But not only have you rejected the *only* existing robust and well-documented conventions for automated processing of Python libraries, you say you "have no time for this part of the thread" when I ask what conventions you want to bless *instead*. So I'm at a bit of a loss for what we're supposed to do now.

2 1

Re: [Distutils] Question on using distutils.core.run_setup
by Phillip J. Eby 18 Mar '08

18 Mar '08

At 05:55 PM 3/18/2008 +0100, Tarek Ziadé wrote: >Cool ! Maybe this could be pushed into distutils ? It looks more >like a fix to me Well, it's not really even a substitute for run_setup(), let alone a fix. It just lets you run the script, not get access to its distribution object. I'm not sure that run_setup() is fixable, though, even in principle. But it's been too many years since I struggled with it, so I don't remember what the problem was. In fact, I know for sure that I've gone through more than one cycle of forgetting what the problem was after not touching it for a few years. :)

1 0

Re: [Distutils] Question on using distutils.core.run_setup
by Terry Jones 18 Mar '08

18 Mar '08

Hi Phillip & Tarek Thanks for the replies, and the code! Terry

1 0

Question on using distutils.core.run_setup
by Terry Jones 18 Mar '08

18 Mar '08

[Apologies if this is in the mailing list archives. I looked but didn't see anything, based on a search for run_setup] I'm trying to programmatically install something built using distutils. I found distutils.core.run_setup and can use it via >>> dist = run_setup('setup.py', ['-q', 'install']) Is that the recommended way to do an install from inside Python (as opposed to doing it on the command line)? If so, how can I find where the thing(s) I installed now resides? I saw dist.packages but that just has top-level package names. I could __import__ these (and then use module.__file__), but that's not a good solution as it may run code I don't want run. On my machine, I can see the packages have been installed under the system's python2.5/site-packages directory. But how can I determine that programmatically? I don't see anything useful on the distutils.dist.Distribution instance I'm getting back from run_setup. Thanks! Terry

3 3

Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
by Phillip J. Eby 17 Mar '08

17 Mar '08

At 01:59 PM 3/17/2008 -0500, Guido van Rossum wrote: >I have certainly personally encountered plenty of situations where I >wasn't able to complete an egg-based install because some dependency >was broken (e.g. not available for the Python version I was using). That's odd -- setuptools-based installs should be able to find and install packages from source. I have noticed a recent phenomenon where new developers upload *only* an egg to PyPI, without the source, but that's usually short-lived until someone points it out to them. Do you happen to know what packages you had this problem with? >I'm okay if setuptools, once it's been installed, runs some setup code >that creates the .egg-info directory and whatever else. This means I'm >also okay with the bootstrap module finding and invoking that setup >code. But I'm *not* okay with building any kind of egg management into >the bootstrap module. The bootstrap module must be be neutral w.r.t. >the package management style. Ok, well then we'll have to invent a new kind of binary package, whose name isn't 'egg'. Supporting distutils source packages is almost certainly a non-starter, if you want to avoid bringing the rest of setuptools into play. The only way to correctly determine what a source package contains is to run its setup script... and running unboxed setup scripts isn't safe because there are people who hardcode paths (or more precisely, use bad ways of computing them) in their setup scripts. I'm not saying the tool needs to guard against *malicious* scripts, just badly-written ones. (Setuptools does this with its "sandboxing" module, when running source packages' setup scripts.) So, if source is out, then some binary format is needed, which means defining the conventions for said format... i.e. "eggs lite" or "egg substitutes". :) > > So, it might be simpler all around to just clear up the > > "controversy". To the best of my recollection, only MAL and MvL have > > ever objected on Python-Dev to the idea of supporting eggs. > >You can add my name to the list. I've heard plenty of people speak >highly of eggs, but I've *also* heard from plenty of people (besides >MAL and MvL) who have serious difficulties with the concept of eggs. I did say "on Python-Dev", and you implied that it was not controversial with you, except for the maintenance-related concerns. I'm not fighting about this, but I would rather you were straight-up with your objections rather than deferring it to a controversy that "might go away in a few years". That way, I could at least attempt to do something about the concerns. OTOH, if your objections were non-specific and likely to stay that way, then I could have at least not wasted your time with any of this.

1 0

Re: [Distutils] PEP 365 (Adding the pkg_resources module)
by Phillip J. Eby 17 Mar '08

17 Mar '08

At 12:17 PM 3/17/2008 -0500, Guido van Rossum wrote: >There will be no egg support in the standard library. Are there any qualifications on that statement, or is this in the same category as "from __future__ import braces"?

2 1

Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
by Phillip J. Eby 17 Mar '08

17 Mar '08

At 08:48 AM 3/17/2008 -0500, Guido van Rossum wrote: >On Sun, Mar 16, 2008 at 7:06 PM, Phillip J. Eby <pje(a)telecommunity.com> wrote: > > So, if the consensus is that it would be better to have a module that > > only does bootstrap installs of pure-Python eggs from PyPI, I'm > > totally fine with that. > >Let's just do this; it will avoid a protracted discussion of the >merits of eggs, pkg_resources, and setuptools. Well, it might be replaced by a protracted discussion of how the module should work and what its API should be, but perhaps that would be a better one to have. :) So, the original proposal (from the previous thread about this) was that the module be named easy_install, and that it simply downloads setuptools and delegates to the "real" easy_install. That way, people can simply use "python -m easy_install ...", without worrying about whether setuptools has been installed yet. IIRC, other package management tools such as zc.buildout and workingenv can then be installed using easy_install. Any objections? Should I revise the PEP?

3 4

Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
by Phillip J. Eby 17 Mar '08

17 Mar '08

Quick summary of the below: I'm definitely fine with doing a simpler, pure-bootstrap module, if there's some consensus on what should go in it. I just wish we could've had this discussion last year, when OSAF was still able to fund the work... ;-) At 06:13 PM 3/16/2008 -0500, Guido van Rossum wrote: >Phillip asked me to give an opinion on his pkg_resources PEP. While >the PEP is short and sweet, the pkg_resources module itself is huge >(1800 non-blank lines; 16 classes plus 5 exceptions; it exports 67 >names in total according to __all__). And pkg_resources.txt is another >1700 lines of documentation. I find that hard to swallow. Is there >anyone besides Phillip who can claim he understands this module? Bob Ippolito actually wrote the very first version of pkg_resources. Others, such as Philip Jenvey of the Jython project, have provided patches. From previous discussions on the distutils-sig, I know that Jim Fulton has in-depth knowledge of both pkg_resources and easy_install. Of course, that's not the same as any of these guys volunteering to be maintainers. :) >If its inclusion is really meant just as a bootstrap to simplify >installing other package management solutions, as the PEP claims, I >would prefer to see something with a much smaller footprint. Actually, the PEP says: "pkg_resources is a module used to find and manage Python package/version dependencies and access bundled files and resources, including those inside of zipped .egg files. Currently, pkg_resources is only available through installing the entire setuptools distribution, but it does not depend on any other part of setuptools; in effect, it comprises the entire runtime support library for Python Eggs, and is independently useful." This kind of glosses over the part where this is also for runtime support of projects that use eggs. Which, these days, is, well, almost any large Python project, from Chandler to Enthought to Zope. > Surely >there is no need for example to have support for C extensions inside >zip files *as part of the bootstrap module*? It's a runtime; the PEP actually merely proposes that a further addition to be made to support bootstrapping, *also*. Otherwise, the PEP would be even shorter. :) The reason I proposed it this way was for simplicity -- and politics. Currently, people using setuptools in their setup.py have to include a similar bootstrap module to download setuptools if it's not available, and pkg_resources already has version checking logic and everything needed to find dependencies and download them. (Plus, I figured it'd be easier to just use what was already there and stable, rather than creating something different.) That was the simplicity part. The politics part was that: 1. I thought it would be less controversial to include the "runtime for eggs" than to include something that's just a bootstrapper for setuptools. However, MvL surprised me by actually being in *favor* of including a setuptools bootstrapper. 2. I thought that it would have broader acceptance if it was oriented towards bootstrapping *any* package, not just setuptools. So, if the consensus is that it would be better to have a module that only does bootstrap installs of pure-Python eggs from PyPI, I'm totally fine with that. >Unless I find someone besides Phillip who is interested in having this >included and is willing to help maintain it, I don't really think it >would be wise to accept this into the standard library. > >Phillip, in the PEP you mention that there are several other package >management tools that also like to use pkg_resources. Maybe you can >get some folks from those tools to speak up and explain what >pkg_resources means to them, and maybe even volunteer to co-own it >once it's in the standard library? The distutils-sig is the de facto place for discussions regarding those tools, so I've cc'd this there. Hopefully, one or more volunteers will step up if they want this.

3 2