Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 10:53 AM 3/17/2008 -0500, Guido van Rossum wrote:
I don't think this should play games with scripts being overridden or whatever. If a bootstrap script is to be installed it should have a separate name. I'm not sure what the advantage is of a bootstrap script over "python -m bootstrap_module ..." though.
And -m also makes explicit: 1. that it's a Python-specific tool 2. which Python version it will apply to
The PEP suggests that other package managers also benefit. How do they benefit if the bootstrap script installs setuptools?
Because those other package managers depend, in fact, on setuptools, or at least pkg_resources... which was why the original proposal was to just include pkg_resources in the first place. :)
I'd also like to avoid the specific name "easy_install" for any of this. That's a "brand name" (and a misleading one if you ask me, but that's politics again :-).
Ok, so if someone will propose a name and API for the thing, I'll implement it. (Assuming the proposed API is sane and reasonably implementable, of course.)
On Mon, Mar 17, 2008 at 11:12 AM, Phillip J. Eby <pje@telecommunity.com> wrote:
At 10:53 AM 3/17/2008 -0500, Guido van Rossum wrote:
I don't think this should play games with scripts being overridden or whatever. If a bootstrap script is to be installed it should have a separate name. I'm not sure what the advantage is of a bootstrap script over "python -m bootstrap_module ..." though.
And -m also makes explicit:
1. that it's a Python-specific tool 2. which Python version it will apply to
Right!
The PEP suggests that other package managers also benefit. How do they benefit if the bootstrap script installs setuptools?
Because those other package managers depend, in fact, on setuptools, or at least pkg_resources... which was why the original proposal was to just include pkg_resources in the first place. :)
On how much of pkg_resources do they depend? Or is that an unanswerable question?
I'd also like to avoid the specific name "easy_install" for any of this. That's a "brand name" (and a misleading one if you ask me, but that's politics again :-).
Ok, so if someone will propose a name and API for the thing, I'll implement it. (Assuming the proposed API is sane and reasonably implementable, of course.)
Perhaps it can be called package_bootstrap? I don't know enough about the required functionality to propose an API, but here goes anyway. Would be reasonable for it to have a command line that allows you to specify a package name, optionally a version string, and (very) optionally a server to contact (specified as an URL?). It should default to the highest non-experimental version that's applicable to the current Python version (assuming there's an easy way to determine this; I don't want it to try and download different versions until one works). It should not handle any kind of dependency management or package management administration. It should be able to download a Python-only module or package and install it into site-packages (or perhaps elsewhere if so directed via another optional command line flag). It should support zip, tar and tar.gz/tgz files (and perhaps tar.bz2). It should simply unpack the zip/tar file using the zip or tar modules, and extract the package/module into site-packages in such a way that it can be imported directly without messing with sys.path. It should not mess with .pth files, setup.py scripts, or eggs. If the contents of the tar/zip file has a toplevel directory with version numbers in its name (e.g. Django-0.96.1) it should skip that and just use the subdirectory whose name is the "pure" package name (e.g. django). Does this make sense? I'm happy to take push-back, this is just something I cooked up off the top of my head based on my experience with manually installing packages. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
It should be able to download a Python-only module or package and install it into site-packages (or perhaps elsewhere if so directed via another optional command line flag). It should support zip, tar and tar.gz/tgz files (and perhaps tar.bz2). It should simply unpack the zip/tar file using the zip or tar modules, and extract the package/module into site-packages in such a way that it can be imported directly without messing with sys.path. It should not mess with .pth files, setup.py scripts, or eggs.
Do you mean "existing eggs" or does that include the (potential .egg) package that is being installed? If I understood correctly, this bootstrap module currently supports installing eggs (although I'm not sure how they are supposed to work without the current way of keeping a .pth file). Is it *wanted* that eggs are being supported by core Python? Stefan
On Mon, Mar 17, 2008 at 11:55 AM, Stefan Behnel <stefan_ml@behnel.de> wrote:
Guido van Rossum wrote:
It should be able to download a Python-only module or package and install it into site-packages (or perhaps elsewhere if so directed via another optional command line flag). It should support zip, tar and tar.gz/tgz files (and perhaps tar.bz2). It should simply unpack the zip/tar file using the zip or tar modules, and extract the package/module into site-packages in such a way that it can be imported directly without messing with sys.path. It should not mess with .pth files, setup.py scripts, or eggs.
Do you mean "existing eggs" or does that include the (potential .egg) package that is being installed?
The latter.
If I understood correctly, this bootstrap module currently supports installing eggs (although I'm not sure how they are supposed to work without the current way of keeping a .pth file).
My proposal is that the boostrap module only be able to install non-egg archives.
Is it *wanted* that eggs are being supported by core Python?
No. There will be no egg support in the standard library. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
On 17/03/2008, Guido van Rossum <guido@python.org> wrote:
On Mon, Mar 17, 2008 at 11:55 AM, Stefan Behnel <stefan_ml@behnel.de> wrote:
Is it *wanted* that eggs are being supported by core Python?
No. There will be no egg support in the standard library.
This bothers me somewhat. At a certain level, all that egg files are is zip files with a different extension - and the zipimport module and PEP 302 certainly do support them. There is a lot more conceptual baggage associated with the egg "brand name", but unless that extra is clearly specified, a statement like "there will be no egg support" doesn't mean much. My view on PEP 365 is that it offers a standard way of doing from pkg_resources import resource_string foo_config = resource_string(__name__, 'foo.conf') which is basically a version of foo_config = open(os.path.join(os.path.dirname(__file__),'foo.conf').read() which also supports PEP 302 importers such as zipimport. This has nothing to do with eggs, and everything to do with solidifying the support for cleanly handling PEP 302 importers. For me, that would be far more useful that a package download&installer (as I'm on Windows, I tend to use bdist_wininst installers, and a bootstrap module which gave no uninstall capability would suck for me). The sticking point here, is deciding what parts of pkg_resources are OK, and which constitute "egg support". It may not be possible to come to a clear understanding of this. But I remain -1 on any module that just does download and install, with no uninstall capability. I don't like *eggs* for precisely this reason! Paul.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Guido van Rossum wrote:
On Mon, Mar 17, 2008 at 11:12 AM, Phillip J. Eby <pje@telecommunity.com> wrote:
At 10:53 AM 3/17/2008 -0500, Guido van Rossum wrote:
I don't think this should play games with scripts being overridden or whatever. If a bootstrap script is to be installed it should have a separate name. I'm not sure what the advantage is of a bootstrap script over "python -m bootstrap_module ..." though.
And -m also makes explicit:
1. that it's a Python-specific tool 2. which Python version it will apply to
Right!
The PEP suggests that other package managers also benefit. How do they benefit if the bootstrap script installs setuptools?
Because those other package managers depend, in fact, on setuptools, or at least pkg_resources... which was why the original proposal was to just include pkg_resources in the first place. :)
On how much of pkg_resources do they depend? Or is that an unanswerable question?
I'd also like to avoid the specific name "easy_install" for any of this. That's a "brand name" (and a misleading one if you ask me, but that's politics again :-).
Ok, so if someone will propose a name and API for the thing, I'll implement it. (Assuming the proposed API is sane and reasonably implementable, of course.)
Perhaps it can be called package_bootstrap? I don't know enough about the required functionality to propose an API, but here goes anyway.
Would be reasonable for it to have a command line that allows you to specify a package name, optionally a version string, and (very) optionally a server to contact (specified as an URL?). It should default to the highest non-experimental version that's applicable to the current Python version (assuming there's an easy way to determine this; I don't want it to try and download different versions until one works). It should not handle any kind of dependency management or package management administration.
It should be able to download a Python-only module or package and install it into site-packages (or perhaps elsewhere if so directed via another optional command line flag). It should support zip, tar and tar.gz/tgz files (and perhaps tar.bz2). It should simply unpack the zip/tar file using the zip or tar modules, and extract the package/module into site-packages in such a way that it can be imported directly without messing with sys.path. It should not mess with .pth files, setup.py scripts, or eggs. If the contents of the tar/zip file has a toplevel directory with version numbers in its name (e.g. Django-0.96.1) it should skip that and just use the subdirectory whose name is the "pure" package name (e.g. django).
Does this make sense? I'm happy to take push-back, this is just something I cooked up off the top of my head based on my experience with manually installing packages.
I would prefer to see it just: - Find a source distribution (the variants you specified) on the given server which matches the supplied version string, using the same semantics as the current 'pkg_resources' constraints. - Unpack the source distribution to a temp directory. - cd into that directory and use sys.executable to invoke 'setup.py install'. Any extra command-line arguments beyond those used to find the source distribution would be passed on to the 'install' command, which would allow alternate locations, etc. That makes the installation as much like a manual one as possible, and means that existing pacakges will be installable whether or not they know about setuptools, etc. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFH3sDr+gerLs4ltQ4RAhjWAKCbFP87mJN4UnVt0pzDs81JovVpoACdGI7A tohpRJnXah0/QnyQeYiqoYY= =9Cif -----END PGP SIGNATURE-----
Folks: (By the way, it was a pleasure to meet many of you in Real Life for the first time at Pycon.) Here is what I want: 1. The standard Python build tools by default produce machine- parseable package metadata, which can include package dependency information with reasonably well-defined semantics. Oh good! I already have this, since distutils in Python >= 2.5 produces .egg-info metadata in an easy-to-parse format. 2. This machine-parseable metadata is widely supported and understood by the Python community. In retrospect, it's too bad that it isn't named ".pkg-info" instead of ".egg-info", in order to avoid the fraught politics around the concept of "eggs". A concrete example of such a misunderstanding is the sad fact that many Linux distributions were in the habit of deleting this information from their Python packages, perhaps because they were under the impression that it was obviated by their packaging tools. The major distributions have all stopped doing this now. Unifying the created-by-default PKG-INFO files and the created-by- default .egg-info directories would be nice, too. 3. The standard Python library includes a tool to find and parse this metadata, so that I can write programmatic tests of my dependencies, like this: http://allmydata.org/trac/tahoe/browser/_auto_deps.py?rev=2062 This is one of the improvements that I was anticipating from pkg_resources going into the standard library. 4. The standard Python library includes a tool to find and read resources (other than Python modules) that came bundled in a Python package. Consider, for example, this snippets of code in Nevow: http://divmod.org/trac/browser/trunk/Nevow/setup.py?rev=13786#L10 http://divmod.org/trac/browser/trunk/Nevow/setup.py?rev=13786 http://divmod.org/trac/browser/trunk/Nevow/setup_egg.py?rev=2406 When Nevow uses pkg_resources to import its files such as "default.css", then it is able to find at runtime, even if is being imported from a py2exe or py2app zip, or on other platforms where its homegrown setup script and homegrown "find my file" function fail. So using pkg_resources (and setuptools to install it) makes "test_nevow" pass on all of the allmydata.org buildslaves: http://allmydata.org/buildbot/waterfall?show_events=false Regards, Zooko
On Mon, Mar 17, 2008 at 2:19 PM, zooko <zooko@zooko.com> wrote:
4. The standard Python library includes a tool to find and read resources (other than Python modules) that came bundled in a Python package.
Consider, for example, this snippets of code in Nevow:
http://divmod.org/trac/browser/trunk/Nevow/setup.py?rev=13786#L10 http://divmod.org/trac/browser/trunk/Nevow/setup.py?rev=13786 http://divmod.org/trac/browser/trunk/Nevow/setup_egg.py?rev=2406
When Nevow uses pkg_resources to import its files such as "default.css", then it is able to find at runtime, even if is being imported from a py2exe or py2app zip, or on other platforms where its homegrown setup script and homegrown "find my file" function fail. So using pkg_resources (and setuptools to install it) makes "test_nevow" pass on all of the allmydata.org buildslaves:
I think we're pretty close to this already. PEP 302 defines a getdata() method. Hopefully most PEP 302 implementations support it. The only thing missing IMO is a little function that does what getdata() does when there is no __loader__ object (i.e. when the default "import-from-filesystem" import method is used). -- --Guido van Rossum (home page: http://www.python.org/~guido/)
On 19/03/2008, Guido van Rossum <guido@python.org> wrote:
On Mon, Mar 17, 2008 at 2:19 PM, zooko <zooko@zooko.com> wrote:
4. The standard Python library includes a tool to find and read resources (other than Python modules) that came bundled in a Python package.
I think we're pretty close to this already. PEP 302 defines a getdata() method. Hopefully most PEP 302 implementations support it. The only thing missing IMO is a little function that does what getdata() does when there is no __loader__ object (i.e. when the default "import-from-filesystem" import method is used).
I'm currently working on an addition to pkgutil to provide this type of function. I'm considering going a little further (adding functions to get a file-like object, test for existence, and list available resources, modelled on the pkg_resources functions - but these extra ones are not supported by the loader protocol, so I'm undecided as to whether it's worth it, I'll see how complex the code gets). Once I have a patch, I'll post it to the tracker. What's the best approach? Code a patch for 3.0 and backport, or code for 2.6 and let the merging process do its stuff? Paul.
On Wed, Mar 19, 2008 at 11:34 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 19/03/2008, Guido van Rossum <guido@python.org> wrote:
On Mon, Mar 17, 2008 at 2:19 PM, zooko <zooko@zooko.com> wrote:
4. The standard Python library includes a tool to find and read resources (other than Python modules) that came bundled in a Python package.
I think we're pretty close to this already. PEP 302 defines a getdata() method. Hopefully most PEP 302 implementations support it. The only thing missing IMO is a little function that does what getdata() does when there is no __loader__ object (i.e. when the default "import-from-filesystem" import method is used).
I'm currently working on an addition to pkgutil to provide this type of function. I'm considering going a little further (adding functions to get a file-like object, test for existence, and list available resources, modelled on the pkg_resources functions - but these extra ones are not supported by the loader protocol, so I'm undecided as to whether it's worth it, I'll see how complex the code gets).
I'd only do what __loader__ offers. People can always wrap a StringIO around it.
Once I have a patch, I'll post it to the tracker. What's the best approach? Code a patch for 3.0 and backport, or code for 2.6 and let the merging process do its stuff?
Code for 2.6, let the merge do its thing. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
On 19/03/2008, Guido van Rossum <guido@python.org> wrote:
I'd only do what __loader__ offers. People can always wrap a StringIO around it.
Once I have a patch, I'll post it to the tracker. What's the best approach? Code a patch for 3.0 and backport, or code for 2.6 and let the merging process do its stuff?
Code for 2.6, let the merge do its thing.
http://bugs.python.org/issue2439 Let me know if you want any changes. If it's OK, I'll think about whether grander things would be of use. Paul.
On 19/03/2008, Guido van Rossum <guido@python.org> wrote:
I'm currently working on an addition to pkgutil to provide this type of function. I'm considering going a little further (adding functions to get a file-like object, test for existence, and list available resources, modelled on the pkg_resources functions - but these extra ones are not supported by the loader protocol, so I'm undecided as to whether it's worth it, I'll see how complex the code gets).
I'd only do what __loader__ offers. People can always wrap a StringIO around it.
Once I have a patch, I'll post it to the tracker. What's the best approach? Code a patch for 3.0 and backport, or code for 2.6 and let the merging process do its stuff?
Code for 2.6, let the merge do its thing.
I've had a patch in http://bugs.python.org/issue2439 for a few weeks now. It just adds a get_data function, on the basis that that's all the loader API offers. I think it's complete - Phillip Eby helped thrash out a couple of problems. Is there anything I can do to get it applied, or should I just leave it now for someone to decide if they care enough? (As it's a library change, I don't know to what extent the "no API changes after the next alpha" rule will apply). Paul.
Paul Moore wrote:
Is there anything I can do to get it applied, or should I just leave it now for someone to decide if they care enough? (As it's a library change, I don't know to what extent the "no API changes after the next alpha" rule will apply).
I'm looking into it now - assuming it applies cleanly and the tests run happily afterwards (and I don't see any reason it won't after reading the patch), it should be checked in later tonight. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org
On 17/03/2008, Phillip J. Eby <pje@telecommunity.com> wrote:
The PEP suggests that other package managers also benefit. How do they benefit if the bootstrap script installs setuptools?
Because those other package managers depend, in fact, on setuptools, or at least pkg_resources... which was why the original proposal was to just include pkg_resources in the first place. :)
I'm puzzled. We seem to be talking about adding a module to the stdlib whose basic function is to download and install setuptools? How is this better than just including setuptools in the stdlib? Personally, I have no problem per se with including setuptools in the stdlib. Maybe that means I miss the subtle benefit of this approach... I'm -1 on having a module which just installs setuptools. I'm +0 on including pkg_resources (as described in PEP 365) in the stdlib. I'm +lots on someone giving a clear explanation of the meaning and interrelationship of the various terms involved in this discussion (setuptools, easy_install, pkg_resources, eggs, "package managers" as distinct from setuptools, etc etc) so that the discussion gets some much-needed clarity :-( I'm -1 on adding anything until PEP 365 is updated to match what is being proposed, and then that amended PEP is submitted for discussion. Paul.
On Mon, Mar 17, 2008 at 11:35 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 17/03/2008, Phillip J. Eby <pje@telecommunity.com> wrote:
The PEP suggests that other package managers also benefit. How do they benefit if the bootstrap script installs setuptools?
Because those other package managers depend, in fact, on setuptools, or at least pkg_resources... which was why the original proposal was to just include pkg_resources in the first place. :)
I'm puzzled. We seem to be talking about adding a module to the stdlib whose basic function is to download and install setuptools? How is this better than just including setuptools in the stdlib?
I'm not in favor of this either.
Personally, I have no problem per se with including setuptools in the stdlib. Maybe that means I miss the subtle benefit of this approach...
I'm -1 on having a module which just installs setuptools. I'm +0 on including pkg_resources (as described in PEP 365) in the stdlib.
I'm +lots on someone giving a clear explanation of the meaning and interrelationship of the various terms involved in this discussion (setuptools, easy_install, pkg_resources, eggs, "package managers" as distinct from setuptools, etc etc) so that the discussion gets some much-needed clarity :-(
Right. But finding someone who can explain all this is apparently hard. All the owners of package managers seem busy...
I'm -1 on adding anything until PEP 365 is updated to match what is being proposed, and then that amended PEP is submitted for discussion.
Well, if we can reach consensus here first on what to put into PEP 365 first I'd be fine with updating the PEP as an afterthought, especially of the consensus also comes with working code (hopefully less than 2500 lines :-). -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
On Mon, Mar 17, 2008 at 11:35 AM, Paul Moore <p.f.moore@gmail.com> wrote:
I'm +lots on someone giving a clear explanation of the meaning and interrelationship of the various terms involved in this discussion (setuptools, easy_install, pkg_resources, eggs, "package managers" as distinct from setuptools, etc etc) so that the discussion gets some much-needed clarity :-(
Right. But finding someone who can explain all this is apparently hard. All the owners of package managers seem busy...
In preparing for my PyCon 2008 tutorial on eggs and buildout, I spent three full-time weeks carefully going over sources for distutils, setuptools and buildout to discover those aspects not documented. I can explain how they work, although I'm not sure this is the correct forum. I'd like to first offer my slides from my tutorial, 150 of them with detailed handout notes on many of them. http://wiki.python.org/moin/buildout/pycon2008_tutorial I'm happy to answer questions after that. I'm in the Hanada B room for OLPC at PyCon and on IRC #pycon. -Jeff
At 02:44 PM 3/17/2008 -0500, Jeff Rush wrote:
Guido van Rossum wrote:
On Mon, Mar 17, 2008 at 11:35 AM, Paul Moore <p.f.moore@gmail.com> wrote:
I'm +lots on someone giving a clear explanation of the meaning and interrelationship of the various terms involved in this discussion (setuptools, easy_install, pkg_resources, eggs, "package managers" as distinct from setuptools, etc etc) so that the discussion gets some much-needed clarity :-(
Right. But finding someone who can explain all this is apparently hard. All the owners of package managers seem busy...
In preparing for my PyCon 2008 tutorial on eggs and buildout, I spent three full-time weeks carefully going over sources for distutils, setuptools and buildout to discover those aspects not documented. I can explain how they work, although I'm not sure this is the correct forum. I'd like to first offer my slides from my tutorial, 150 of them with detailed handout notes on many of them.
Wow. I am skimming over the 44-page one on setuptools, and that is definitely the most comprehensive doc anyone has produced on it, aside from the official docs. Thank you!
I'm puzzled. We seem to be talking about adding a module to the stdlib whose basic function is to download and install setuptools? How is this better than just including setuptools in the stdlib?
I can do a review of such a module in an hour. To review setuptools (which I would have to do if it were to be included), I would likely need a week or so, full time.
Personally, I have no problem per se with including setuptools in the stdlib. Maybe that means I miss the subtle benefit of this approach...
Did you review setuptools and can vouch that it is written cleanly, follows the coding style, has correct documentation, and so on? Regards, Martin
On 17/03/2008, "Martin v. Löwis" <martin@v.loewis.de> wrote:
Personally, I have no problem per se with including setuptools in the stdlib. Maybe that means I miss the subtle benefit of this approach...
Did you review setuptools and can vouch that it is written cleanly, follows the coding style, has correct documentation, and so on?
No, I concede that when I say "I have no problem" with including setuptools, I'm assuming that someone does that review - and there's no reason to assume that anyone will do that (that's why I say "I have no problem with" rather than "I want"). But I don't see any practical difference with including setuptools and including a module that installs setuptools. Would you be happy with the standard library including a module whose sole function was to install a package that you weren't happy to include directly in the standard library? I wouldn't, personally. Paul
On Mon, Mar 17, 2008 at 12:12 PM, Paul Moore <p.f.moore@gmail.com> wrote:
But I don't see any practical difference with including setuptools and including a module that installs setuptools. Would you be happy with the standard library including a module whose sole function was to install a package that you weren't happy to include directly in the standard library? I wouldn't, personally.
Actually it's quite different -- the user or administrator is in control and they can choose not to install setuptools. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
But I don't see any practical difference with including setuptools and including a module that installs setuptools. Would you be happy with the standard library including a module whose sole function was to install a package that you weren't happy to include directly in the standard library? I wouldn't, personally.
If the actual code is not in Python, I'm not responsible for it. I don't have to deal with bug reports, and I don't have to help users with that feature. As users want to get the functionality anyway, and out-of-the-box, I consider it a useful compromise. Regards, Martin
participants (9)
-
"Martin v. Löwis" -
Guido van Rossum -
Jeff Rush -
Nick Coghlan -
Paul Moore -
Phillip J. Eby -
Stefan Behnel -
Tres Seaver -
zooko