Mailman 3 March 2008 - Distutils-SIG

Re: [Distutils] [Python-Dev] platform management
by Guido van Rossum 20 Mar '08

20 Mar '08

Great idea! Sounds like a PEP (informational, probably) would be good idea. On Tue, Mar 18, 2008 at 4:59 PM, Bill Janssen <janssen(a)parc.com> wrote: > I don't think this is bike-shedding. > > The debate about "AMD64" vs. "amd64" vs. "x86_64" reminded me that > I've been bit more and more frequently by bits of platform-specific > knowledge scattered around the standard library. The latest is the > code in distutils.unixccompiler that tries to figure out what flags to > send to the linker in order to add a dynamic library path lookup to a > shared library. > > There are lots of different ways of figuring out which platform is > being used, and they're all over the place. The code in > distutils.unixccompiler uses "sys.platform[:6]", and looks for > "darwin"; the code in urllib.py uses "os.name", and expects it to be > "mac", but later looks for "sys.platform == 'darwin'; posixfile > believes that platforms come with version numbers ("linux2", "aix4"); > pydoc and tarfile have tests for "sys.platform == 'mac'". tempfile > looks at os.sys.platform *and* os.name. > > Could well be that all of these are correct (I believe that "mac", for > instance, refers to the generations of the Mac OS before OS X). But > it means that when someone tries to update "Python" to a new major > version release for, say, OS X, it's really easy to miss things. And > the fact that the platform version is sometimes included and sometimes > not is also problematic; darwin9 is different from darwin8 in some > important aspects. > > It would be nice to > > (a) come up with a list of standard platform symbols, > (b) put those symbols somewhere in the documentation, > (c) have one way of discovering them, via sys or os or platform or > whichever module, > (d) add a standard way of discovering the platform version, and > (e) stamp out all the other ways that have crept in over the years. > > Bill > _______________________________________________ > Python-Dev mailing list > Python-Dev(a)python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/)

3 2

Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
by Phillip J. Eby 20 Mar '08

20 Mar '08

At 09:33 AM 3/20/2008 +0000, Paul Moore wrote: >1. No integration with the system packager (Windows, in my case). If I >do easy_install nose, then nose does not show up in add/remove >programs. That significantly affects the way I manage my PC. The long-term fix here is probably to have a platform-specific installer capable of either turning eggs into .msi or .exe installers, or of doing the add/remove programs integration directly. (Someone, of course, will have to step up to create such a tool.) >5. Auto-discovery doesn't always work. I'm sorry, I really can't >recall the example at the moment, but sometimes easy_install says it >can't find a package I *know* is available. Sometimes it does that to me, too. But then I look at the project's page in PyPI, and they don't have a link to a download page. Usually, they've got a link to a page on their site with instructions about downloading, but that doesn't directly link to any tarballs or anything. So I grab the link of the real download page and paste it into a -f option to easy_install, so it knows where to find the link from.

1 0

Consistent platform name for 64bit windows (was: distutils.util.get_platform() for Windows)
by mhammond＠keypoint.com.au 20 Mar '08

20 Mar '08

I'm reviving a very old thread based on discussions with Martin at pycon. > Sent: Monday, 23 July 2007 5:12 PM > Subject: Re: [Distutils] distutils.util.get_platform() for Windows Rather than forcing everyone to read the context, allow me to summarize: On 64bit Windows versions, we need a "string" that identifies the platform, and this string should ideally be used consistently. This original thread related to the files created by distutils (eg, pywin32-210.win???64??-py2.6.exe) but it seems obvious that we should be consistent wherever Python wants to display the platform (eg, in the startup banner, in platform.py, etc). In the old thread, there was a semi-consensus that 'x86_64' be used by distutils (and indeed, Lib/distutils/util.py in get_platform() has been changed, by me, to use this string), but the Python 'banner', for example, reports AMD64. Platform.py doesn't report much at all in this area, at least when pywin32 isn't installed, but it arguably should. Both Martin and I prefer AMD64 as the string, for various reasons. Firstly, it is less ugly than 'x86_64', and doesn't include an '_'/'-' which might tend to confuse parsing by humans or computers. Martin also made the point that AMD invented the architecture and AMD64 is their preferred name, so we should respect that. So, at the risk of painting a bike-shed, I'd like to propose that we adopt 'AMD64' in distutils (needs a change), platform.py (needs a change to use sys.getwindowsversion() in preference to pywin32, if possible, anyway), and the Python banner (which already uses AMD64). Any objections? Any strong feelings that using 'AMD' will confuse people with Intel processors? Strong feelings about the parsability of the name (PJE? <wink>)? Strong feelings about the color <wink>? Thanks, Mark

6 5

Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
by Phillip J. Eby 20 Mar '08

20 Mar '08

At 03:43 PM 3/18/2008 -0500, Guido van Rossum wrote: >Only very few people would care about writing a setup >script that works with this bootstrap module; basically only package >manager implementers. That's true today, sure, but as soon as it is widely available, others are sure to want to use it too. I just want a bright-line distinction between what is and isn't bootstrappable, rather than a murky region of "maybe, if you're not doing anything too complicated". >There seems to be a misunderstanding about what I am proposing we do >instead. The boostrap installer should only be powerful enough to >allow it to be used to install a real package manager like setuptools. Which is why PEP 365 proposed only downloading an archive to a cache directory, and optionally running something from it. It explicitly disavows "installation" of anything, since the downloaded archive wouldn't have been added to sys.path except for the duration of the bootstrap process, and no scripts were to be installed. (Indeed, apart from the methods it would have used to locate the archive on PyPI, and to determine what to run from inside it, there was nothing particularly egg-specific about the proposed bootstrapping process.) So, to fully egg-neutralize the bootstrapping approach, we need only know how to locate an appropriate archive, and how to determine what to run from it. For the latter, we could use the already-in-2.6 convention of running __main__ from a zipfile or directory. (Too bad distutils source distributions have an extra directory name embedded in them, so one can't just execute them directly. Otherwise, we could've just let people drop in a __main__.py next to setup.py. OTOH, maybe it would be enough to use setuptools' algorithm for finding setup.py to locate __main__.py, and I'm fairly sure *that* can be briefly expressed in the PEP.) The other open question is a naming convention and version detection, so that the bootstrap tool can identify which of the files listed on PyPI is suitable for its use. (Both with regard to the version selection, and file type.) However, if PyPI were to grow support for designating the appropriate files and/or versions in some other way, we wouldn't need a naming convention as such. Without one or the other, the bootstrap tool would have to grow a version parsing scheme of some type, and play guessing games with file extensions. (Which is one reason I limited PEP 365's scope to downloading eggs actually *uploaded* to PyPI, rather than arbitrary packages *linked* from PyPI.) So, if I had to propose something right now, I would be inclined to propose: * using setuptools' version parsing semantics for interpretation of alpha/beta/dev/etc. releases * having a bdist_bootstrap format that's essentially a bdist_dumb .zip file with the internal path prefixes stripped off, making it an importable .zip with a different file extension. (Or maybe just .pyboot.zip?) The filename convention would use setuptools' canonicalization and escaping of names and version numbers, to allow unambiguous machine parsing of the filename. A __main__ module would have to be present for the archive to be run, as opposed to just being downloaded to a temporary directory. * calling the bootstrap module 'bootstrap', as in 'python -m bootstrap projectname optionalversion'. The module would expose an API to allow it to be used programmatically as well as the command line, so that bootstrapped packages can use the bootstrap process to locate dependencies if they so desire. (Today's package management tools, at least, are all based on setuptools, so if it's not present they'll need to download that before beginning their own bootstrapping process.) Apart from keeping the PEP self-contained and short, is there anything in this that you think you would object to? (You may reserve the right, of course, to later not like something in the details of setuptools' version/filename rules, after I've put them into the PEP, or really, anything else. I'm just asking if there's anything that's obviously offensive at this point, before I spend time on a new PEP.)

5 6

Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
by Phillip J. Eby 19 Mar '08

19 Mar '08

At 12:59 PM 3/17/2008 -0500, Guido van Rossum wrote: >On Mon, Mar 17, 2008 at 12:45 PM, Phillip J. Eby ><pje(a)telecommunity.com> wrote: > > At 12:17 PM 3/17/2008 -0500, Guido van Rossum wrote: > > >There will be no egg support in the standard library. > > > > Are there any qualifications on that statement, or is this in the > > same category as "from __future__ import braces"? > >IIUC eggs are a method of package management that includes support for >dependencies, multiple versions, and C extensions in zip files, as >well as conventions for naming these and for encoding metadata (e.g. >how to find out the version or the dependencies). This whole set of >conventions is IMO too much to include into the stdlib ATM -- if only >because it has proved controversial in the past. Maybe a few years >from now it will be no longer controversial and then my objections >will disappear. So, does this mean that the bootstrap tool must not use eggs? That seems a little bit odd, in that setuptools will at least need its .egg-info directory to get installed, and all of the people who'll be using this initially will be using it precisely in order to have support for eggs... So, it might be simpler all around to just clear up the "controversy". To the best of my recollection, only MAL and MvL have ever objected on Python-Dev to the idea of supporting eggs. Note: I'm specifically segregating "egg support" from the topic of including setuptools or easy_install in the stdlib directly. There are many legitimate reservations and open questions about the latter, including availability of volunteer support, choice of defaults, whether to replace distutils with setuptools, etc. etc. I recognize and respect the validity of those issues, which is precisely why I withdrew setuptools from inclusion in Python 2.5. However, regarding support for eggs, my understanding is that there were only two objections to eggs -- even at the time of the 2.5 setuptools discussions. And even though MvL objects to the idea of eggs in *principle*, I didn't read his recent posts as objecting to having the bootstrap tool download and install eggs in *practice*. (Although I hope he will clarify that stance one way or the other.) That leaves MAL, whose objections to PEP 365 centered on the API (he said he was "+1 on the concepts being added to the stdlib, -1 on adding the module in its current state"). Among other concerns, he wanted pkg_resources to be split into pkgutil and a new "egglib" module. I don't have a problem with this in principle, if there were a pkg_resources module that reconstituted the merged API. (But there are some practical problems with that approach, such as trying to split namespace package support between two theoretically-unrelated modules.) I would guess, however, that MAL's issues with the pkg_resources API would not apply to a bootstrap module whose sole purpose was to download eggs and put them on sys.path. Or, perhaps he would object *more*, I don't know. We could certainly ask him, though. :) So, was there anyone else you were counting towards "controversy"? The only other person I recall objecting to setuptools in any way on Python-Dev was effbot, and IIUC his objections were practical/administrative re: supporting easy_install and setuptools, not to the idea of .egg support in general. In summary, I think the controversy on Python-Dev regarding .egg support has actually been over for some time now.

6 11

Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
by Phillip J. Eby 19 Mar '08

19 Mar '08

At 06:41 PM 3/19/2008 +0000, Paul Moore wrote: >I may just go for a simple registry of implementation classes. That's all simplegeneric *is*, in this context. A way of having "simple registries", since more than one was needed there. > Sorry, but if you >couldn't get sufficient agreement to simplegeneric to get it in a >"proper" location, It's a private implementation detail of pkgutil, not a stdlib feature, which is why no such agreement was requested or required.

1 0

Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
by Phillip J. Eby 19 Mar '08

19 Mar '08

At 05:04 PM 3/19/2008 +0000, Paul Moore wrote: >On 19/03/2008, Phillip J. Eby <pje(a)telecommunity.com> wrote: > > It's so that you can get loaders for modules that aren't imported yet > > -- and your code would need to handle this case too. > >Well, technically, it need not, as I don't *need* to implement the >exact functionality that pkg_resources does. My (personal) goal is to >standardise the functionality, not to cater for every possible use >case. > > > (You could handle it by actually going ahead and importing the module, as > > pkg_resources does, but there are other tools using pkgutil to e.g. > > inspect as-yet-unimported modules.) > >I'm not sure what you mean by that. There are no tools using >pkgutil.resource_stream, as it doesn't exist yet. So whether it >imports the module or not cannot be relevant (although it should be >documented). You asked why pkgutil.get_loader() was more complex than you'd expect. That's the question I answered, in both of the snippets you quoted above. That is, when I said, "your code would need to handle this case", I meant "your code for get_loader()", not "your code for resource_foo() methods". IOW, I was trying to encourage you to use get_loader() as-is rather than modifying or replacing it with something "simpler" that would break other uses of get_loader() in the stdlib. > > You don't really need to, because even though they're technically > > "undocumented", the intent was for them to be published APIs (apart > > from simplegeneric, which is an implementation detail). The docs > > just never got copied to the official docs. > >If it's just a matter of copying documentation from the PkgResources >page, that's no issue. However, only get_importer is documented on the >PkgResources page. I meant, the documentation was never copied from the *docstrings* of these functions in pkgutil; as you noted, not all of them are direct ports from pkg_resources. I refactored the framework there to include support for pyrun and pydoc's use cases as well. It's true that some additional documentation is needed, too, for anyone who wants to implement a PEP 302 loader or importer that wants to support enhanced features like listing modules or getting resources. >On reviewing the pkg_resources APIs, I only see the following that >look worth porting to pkgutil: > > * resource_exists(package, resource_name) > * resource_stream(package, resource_name) > * resource_string(package, resource_name) > * resource_isdir(package, resource_name) > * resource_listdir(package, resource_name) > >... > >Hmm, these go way beyond the simple (and optional) loader.get_data >interface from PEP 302. And yet the pkg_resources machinery is far >more complex than I'd want to see in the stdlib. Maybe I'll just >reinvent the wheel and see how far I get :-) So that future PEP 302 importers can define support for these features, please use pkgutil.simplegeneric to implement these, following the pattern set out by other pkgutil functions. E.g.: def resource_listdir(package, resource_name): return loader_listdir(get_loader(package), package, resource_name) Where 'loader_listdir' is a simplegeneric with default handlers registered for the built-in/stdlib loader types (zipimporter, None, and ImpLoader). If you do this, then new importer/loader types can use loader_listdir.register() to support the functionality, without monkeypatching or other hacks.

2 1

Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
by Phillip J. Eby 19 Mar '08

19 Mar '08

At 09:43 AM 3/19/2008 +0000, Paul Moore wrote: >On 19/03/2008, Phillip J. Eby <pje(a)telecommunity.com> wrote: > > >I don't particularly want to use undocumented functions - even within > > >the module that defines them. > > > > Er, you could always document them, you know. I just didn't get > > around to it before the whole "setuptools flap of 2005" wiped out my > > motivation to do any further work on Python 2.5. For the most part, > > they do have docstrings. I just never did the LaTeX work on > > them. And now, you can use reST instead of LaTeX. :) > >I could, but the problem is I don't really follow the code. My >motivation is to add useful functions, not document stuff that's >already there. You mentioned using get_loader to implement >resource_string. OK, but I'd have done something a lot simpler than >the code of get_loader, and I don't understand why the code in there >is necessary. It's so that you can get loaders for modules that aren't imported yet -- and your code would need to handle this case too. (You could handle it by actually going ahead and importing the module, as pkg_resources does, but there are other tools using pkgutil to e.g. inspect as-yet-unimported modules.) >Never mind, I'll implement what I'm planning on using my own code, and >ignore trying to understand the (corner cases of the) undocumented >functions. You don't really need to, because even though they're technically "undocumented", the intent was for them to be published APIs (apart from simplegeneric, which is an implementation detail). The docs just never got copied to the official docs. These APIs are also actively used by at least pydoc and pyrun in Python 2.5, so they're unlikely to go away or break.

2 1

Re: [Distutils] Question on using distutils.core.run_setup
by Terry Jones 19 Mar '08

19 Mar '08

>>>>> "Phillip" == Phillip J Eby <pje(a)telecommunity.com> writes: Phillip> At 04:03 AM 3/19/2008 +0100, Terry Jones wrote: >> >>>>> "Phillip" == Phillip J Eby <pje(a)telecommunity.com> writes: Phillip> At 06:27 AM 3/18/2008 +0100, Terry Jones wrote: >> >> If so, how can I find where the thing(s) I installed now resides? >> Phillip> It'd be on the 'install_lib' command instance, not the Phillip> distribution. Try Phillip> dist.get_finalized_command('install_lib').install_dir instead. >> >> For the record / archives, I think this should be >> dist.get_command_obj('install_lib').install_dir Phillip> No, it shouldn't. Check again. :) I get this (using the simplistic direct invocation of run_setup): $ python Python 2.5 (r25:51918, Sep 19 2006, 08:49:13) [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from distutils.core import run_setup >>> d = run_setup('setup.py', ['-q', 'install']) >>> d.get_finalized_command('install_lib').install_dir Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: Distribution instance has no attribute 'get_finalized_command' >>> d.get_command_obj('install_lib').install_dir '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/' I guess I'm missing something here? Terry

2 1

Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
by Phillip J. Eby 19 Mar '08

19 Mar '08

At 10:44 PM 3/18/2008 +0000, Paul Moore wrote: >Someone should document these new functions :-( Was it you that added >them? Yes, and it was specifically to start pulling some of that stuff out of pkg_resources so it'd be more widely available. > Assuming it was, what's the "simplegeneric" function doing in >there? It's a substitute for the adapter machinery that the corresponding code used when it lived in pkg_resources. Guido had just shot down adapting in favor of generic functions, so I used what I thought would be a more palatable name. :) >Can it be moved somewhere more sensible? Er, sure, if you can get that past Python-Dev. It's not a particularly full-featured generic function implementation, just a crude subset of the simplegeneric that's on PyPI. If I were doing it today, I'd have called it 'trivialgeneric' instead. ;) >I don't particularly want to use undocumented functions - even within >the module that defines them. Er, you could always document them, you know. I just didn't get around to it before the whole "setuptools flap of 2005" wiped out my motivation to do any further work on Python 2.5. For the most part, they do have docstrings. I just never did the LaTeX work on them. And now, you can use reST instead of LaTeX. :)

2 1