At 07:15 PM 4/18/2006 +0200, M.-A. Lemburg wrote:

Why should a 3rd party extension be hot-fixing the standard Python distribution ?

Because setuptools installs things in zip files, and older versions of pydoc don't work for packages zip files.

If you want to provide a hot fix for Python 2.3 and 2.4, you should make this a separate install, so that users are aware that their Python distribution is about to get modified in ways that have nothing to do with setuptools.

Their Python distribution is not "modified" -- new modules are merely placed on sys.path ahead of the stdlib. (Which, I might add, is a perfectly normal process in Python -- nothing stops users from installing their own version of pydoc or any other module via PYTHONPATH.) Note also that uninstalling setuptools by removing the .egg file or directory will effectively remove the hot fix, since the modules live in the .egg, not in the stdlib.

If an application wants to use setuptools for e.g. plugin management, then standard distutils features will get replaced by setuptools implementations which are not compatible to the standard distutils commands, effectively making it impossible to access the original versions.

Please do a little research before you spread FUD. The 'pkg_resources' module is used for runtime plugin management, and it does not monkeypatch anything.

Monkey patching is only a last resort in case nothing else works. In this case, it's not needed, since distutils provides the interfaces needed to extend its command classes via the setup() call.

The monkeypatching is there so that the easy_install command can build eggs for packages that use the distutils. It's also there so that other distutils extensions that monkeypatch distutils (and there are a few of them out there) will have a better chance of working with setuptools. I originally took a minimally-invasive approach to setuptools-distutils interaction, but it was simply not possible to provide a high-degree of backward and "sideward" compatibility without it. In fact, I seem to recall finding some behaviors in some versions of distutils that can't be modified without monkeypatching, although the details are escaping me at this moment.

...

As for discussion, Guido originally brought up the question here a few months ago, and it's been listed in PEP 356 for a while. I've also posted things related to the inclusion both here and in distutils-sig.

I know, but the discussions haven't really helped much in getting the setuptools design compatible with standard distutils.

That's because the job was already done. :) The setuptools design bends over backwards to be compatible with Python 2.3 and 2.4 versions of distutils, not to mention py2exe, Pyrex, and other distutils extensions, along with the quirky uses of distutils that exist in dozens of distributed Python packages. However, I think you and I may perhaps have different definitions of "compatibility". Mine is that things "just work" and users don't have to do anything special. For that definition, setuptools is extremely compatible with the standard distutils. In many situations it's more compatible than the distutils themselves, in that more things "just work". ;)

At 09:02 PM 4/18/2006 +0200, M.-A. Lemburg wrote:

Phillip J. Eby wrote:

...

At 07:15 PM 4/18/2006 +0200, M.-A. Lemburg wrote:

...

Why should a 3rd party extension be hot-fixing the standard Python distribution ?

Because setuptools installs things in zip files, and older versions of pydoc don't work for packages zip files.

That doesn't answer my question.

That is the answer to the question you asked: "why hot-fix?" Because setuptools uses zip files, and older pydocs crash when trying to display the documentation for a package (not module; modules were fine) that is in a zip file.

Third-party extension *should not do this* !

If you install setuptools, you presumably would like for things to work, and the hot fix eliminates a bug that interferes with them working. I'm not sure, however, what you believe any of that has to do with python-checkins or python-dev. The version of setuptools that will do this is not yet released, and the hotfix aspect will be documented. If it causes actual problems for actual setuptools users, it will be actually fixed or actually removed. :) (Meanwhile, the separately-distributed setuptools package is not a part of Python, any more than the Optik is, despite Python including the 'optparse' package spun off from Optik.)

I'm talking about the setuptools package which does apply monkey patching and is needed to manage the download and installation of plugin eggs, AFAIK.

In which case the monkeypatching is needed, to handle situations like building eggs for third-party non-setuptools libraries listed as dependencies. You can't have it both ways; either you need setuptools or you don't. If you do, you will almost certainly need the monkeypatching, and you'll definitely need it to build eggs for non-setuptools-based packages.

What's good about it, is that this approach doesn't modify anything inside distutils at run-time, but does these modifications on a per-setup()-call basis.

What's bad about it, is that it provides no options for virtualizing the execution of other packages' setup scripts. The 'run_setup' mechanism was horribly broken in the versions of the distutils that I wrestled with. (I also don't see anybody stepping up to provide alternative implementations or patches to third-party setup scripts or distutils extensions to deal with this issue.)

There aren't all that many things that are wrong in setuptools, but some of them are essential:

* setuptools should not monkey patch distutils on import

Please propose an alternative, or better still, produce a patch. Be sure that it still allows distutils extensions like py2exe to work. The only real alternative to monkeypatching that I'm aware of for that issue is to actually merge setuptools with the distutils, but my guess is that you'll like that even less. :)

* the standard "python setup.py install" should continue to work as documented in the Python documentation; any new install command should use a different name, e.g. "install_egg"

* placing too many ZIP files on PYTHONPATH is a bad idea since it slows down import searches for all Python applications, not just ones relying on eggs; a solution to this would be to have a PYTHONEGGPATH which is then only scanned by egg-aware modules/applications

* the user should have freedom of choice in whether to have her Python installation rely on eggs or not (and not only --by-using-some-complicated-options)

These questions have been hashed to death on the distutils-sig already, but I will try to summarize my responses here as briefly as practical while still covering the full spectrum of issues. I would begin by saying that the tradeoffs I've made favor inexperienced users at the expense of experts (who I assume are capable of learning to use --root or --single-version-externally managed in order to have finer-grained control). Your proposals, however, generally favor experts at the expense of the average user, and treat it as axiomatic that the benefits provided by setuptools are not worth having, no matter how small the cost. By contrast, users who *want* to use setuptools for either distribution or installation of packages, would rather it "just work", no matter how much complexity is required behind-the-scenes. They don't care about my cost to implement, they just care that they type "easy_install foo" to get a package from PyPI or "setup.py sdist bdist_egg upload" to put it there. Therefore, it makes no sense to apply your design approach to setuptools, since by your criteria it wouldn't exist in the first place! After all, expert users can munge distutils extensions to their hearts' content (and you certainly have done so already). Next, your own measurements posted to the distutils-sig debunked the PYTHONPATH performance question, IMO: if you installed *every Python package listed on PyPI at that time* as a zip file (assuming that the average zipfile directory size was the same as in your tests, around 1100 total packages on PyPI, and linear degradation), Python startup time would have increased by little more than half a second. To me, that says the performance is pretty darn good, since the average user isn't going to install anywhere near that many packages. Note also that installing eggs with -m (or --multi-version) doesn't put them on sys.path to begin with unless they're needed at runtime, so if you care about performance tweaking, you can just use -m for everything, or put it in your distutils.cfg or ~/.pydistutils.cfg file so it's automatic. Since scripts that depend on eggs can automatically find and add the eggs to sys.path at runtime, they don't need the eggs to be on sys.path. The reason they are put on sys.path by default is to make the default case of installation "just work". Specifically, it's so that "setup.py install" works the same way as it does with the distutils (in terms of what the users can *do* with a package after installation, not in terms of how the files are laid out). As far as I can tell, your remaining arguments can be reduced to this: Users who care about how stuff works inside (i.e. experts) will have to use command-line options to get the old behaviors or to tweak the new ones, when working with packages that *explicitly chose* to use setuptools. And that is a perfectly valid concern -- for an audience that's quite capable of learning to do something new, however. Also note that nothing stops people from creating packages that warp distutils into installing things in weird places, so arguing that "setup.py install" should always do the same thing is moot anyway; you would not get your wish even if setuptools had never existed. May I suggest that perhaps a better solution to the actual practical problem here is to make the distutils "install" command accept the options that setuptools supports? Then, people who want it can just use --single-version-externally-managed with any package under Python 2.5 and be guaranteed to get the old-fashioned behavior (that can't be cleanly upgraded or uninstalled without a package manager). As far as I know, however, most tools needing the old installation layout use "install --root" anyway (including bdist_wininst, bdist_msi, bdist_rpm, etc.), and setuptools automatically switches to compatibility mode in that case. So setuptools "just works" with these tools already.

Fredrik Lundh wrote:

it's still listed under "possible additions" in the release PEP, and there don't seem to be a PEP or any other easily located document that explains exactly how it works, what's required from library developers, how it affects existing toolchains, etc. is this really ready for inclusion ? does anyone but Phillip understand how it works ?

I don't understand it. My concern is that it appears to involve a lot of magic. This magic might do the "right thing" in many cases, and it might indeed help the user that the magic is present, yet I still do believe that magic is inherently evil: explicit is better than implicit. Regards, Martin

...

I don't understand it.

Have you read the manuals?

You mean, http://peak.telecommunity.com/DevCenter/setuptools Yes, I did. However, this would only enable me to use it (should I ever find a need). What I really need to understand is how all this stuff works inside. Now, I haven't actually *tried* to learn it so far. Fredrik's question was not "who thinks he could learn it" (I certainly could), I read his question as "who does understand it, as of today". I can't faithfully answer "yes" to that question. So far, I believe nobody has said "yes, I do understand it, and could maintain it if Phillip Eby became unavailable".

Please define "magic".

"Functionality with no obvious implementation, or functionality implemented in an unobvious way".

Better, please point to the API functions or classes, or the setup commands documented in the manual, to show me what these things are that appear to be magic.

Here are some that I either already knew, or discovered just now: 1. setuptools manages to replace pydoc on earlier Python versions. MAL assumed it does so by overwriting the files. This would be bad practice, but is the obvious solution. You then told him that this is not what actually happens (so it's already magic); instead, you arrange that Python finds your version before it finds its own. Again, this is still magic: How does it do that? If you append to sys.path, Python would still find its own version before it finds yours. So perhaps you insert another path at sys.path[0]? This would also be bad practice, but it would be the next-obvious approach. But I fully expect that you are now telling me that this is still not how it works. 2. setuptools manages to install code deep inside a subdirectory of site-python, and manages to install multiple versions simultaneously. How does that work? The first part (packages outside sys.path) can be solved with a .pth file (which I already consider bad practice, as it predates the package concept of Python); but that can't work for multiple versions: magic. I (think I) know the answer: there is a single .pth file, and that is *edited* on package installation time. This is a completely non-obvious approach. If Python needs to support multiple versions of a package (and there is apparently a need), that should be supported "properly", with a clear design, in a PEP. 3. namespace packages. From the documentation, I see that you pass namespace_packages= to setup, and that you put __import__('pkg_resources').declare_namespace(__name__) into the namespace package. How does that work? The documentation only says "This code ensures that the namespace package machinery is operating and that the current package is registered as a namespace package." Also, why not import pkg_resources pkg_resources.declare_namespace(__name__) What does declare_namespace actually do? 4. zip_safe. Documentation says "If this argument is not supplied, the bdist_egg command will have to analyze all of your project's contents for possible problems each time it buids an egg." How? Sounds like it is solving the halting problem, if it manages to find possible problems.

There do exist implementation details that are not included in user documentation, but this is going to be true of any system. These details are sometimes necessarily complex due to distutils limitations, behavioral quirks of deployed packages using distutils, and the sometimes baroque variations in sys.path resolution across platforms, Python versions, and invocation methods.

I trust that there is a good reason for each of them. However, for inclusion in the standard library, some of these should go away: - if you have distutils limitations, remove them - differences in Python versions shouldn't matter: you always know what the Python version is - if there are baroque variations in sys.path resolution across platforms, they should be removed, or modernized. Not sure what "invocation methods" are.

...

This magic might do the "right thing" in many cases, and it might indeed help the user that the magic is present, yet I still do believe that magic is inherently evil: explicit is better than implicit.

Are documented defaults "implicit" or "magic"?

Documented defaults are explicit.

With respect to you and MAL, I think that some of your judgments regarding setuptools may have perhaps been largely formed at a time last year when, among other things:

* That documentation section I just referenced didn't exist yet

I don't think I ever complained about the lack of documentation. I would (personally) not trust the documentation, anyway, but always look at the code to find out how it actually works. I read documentation primarily to find out what the *intent* was.

These are significant changes that are directly relevant to the objections that you and he raised (regarding startup time, path length, tools compatibility, etc.), and which I added because of those objections. I think that they are appropriate responses to the issues raised, in that they allow the audience that cares about them (you, MAL, and system packagers in general) to avoid the problems that those features' absence caused.

And I like these changes indeed - my (current) complaints are not about the functionality provided. Instead, I'm concerned about maintainability of this code.

It would probably helpful if you would both be as specific as possible in your objections so that they can be addressed individually. If you don't want setuptools in 2.5, let's be clear either on the specific objections. If the objection is to the very *idea* of setuptools, that's fine too, as long as we're clear that's what the objection is.

setuptools combines several independent aspects, as far as I can tell: 1. automatic download of packages (actually ez_setup). This is a nice feature, and should be included. +1 2. enhancements to the development process, and improvements to the extensibility of distutils. Also nice, but should be part of distutils itself. +0 in its current form, +1 if it were distutils features. 3. "package resources". I dislike the term resources (it is not about natural gas, water, main memory, or processor cycles, right?); instead, this seems to provide access to "embedded" data files. Apparently, there is a need for it, so be it. -0 4. .egg files. -1 5. entry points. I don't understand that concept 6. package dependencies. +1 7. namespace packages. Apparently, there is a need, although I believe that flat is better than nested. -0 However, these are all besides my original objections: - I disliked the fact that nobody explicitly approved inclusion of setuptools. Now that Anthony Baxter did, I'm fine. - I still fear that the code of setuptools will turn out to be a maintenance nightmare. Of course, this doesn't have to concern me - I could just ignore all bug reports about that, and ignore it in the next revision of my book. However, I know that I can't take that position, hence my concerns.

So I would ask that you please make a list of what you would have to see changed in setuptools before you would consider it viable for stdlib inclusion, or simply admit that there are no changes that would satisfy you, or that you don't know enough about it to say, or that you'd like it to be kicked back to distutils-sig for more discussion ad infinitum, or whatever your actual objections are.

See above: now that Anthony approved it, I can accept it. There is nothing you could possibly do to ease my concerns about maintainability, or my dislike of a Python-specific package format.

Meanwhile, this discussion has used up the time that I otherwise would have spent writing 2.5 documentation today (e.g., for the pkgutil additions).

Same here. I spent several hours now typing setuptools email. Still, I don't think this is wasted time (or else I would not have typed this message). Regards, Martin

Anthony Baxter wrote:

...

4. .egg files. -1

As far as I understand it, an egg file is just a zipimport format zip file with some additional metadata. You can also install the egg files in an unpacked way, if you prefer that. I don't understand the objection here - it's no better or worse than countless packages in site-packages, and if it gives us multiple versions of the same code, all the better.

It is worse: each .egg file makes an additional sys.path entry (if I understand correctly); every import statement will traverse every every package. I'm not sure precisely how all this works, but I wouldn't be surprised if the zip directory is read over and over again. Compare that to countless packages in site-packages: "import foo" will *just* look for foo.py, foo.so, and the directory foo. I understand there is a second API to importing, some kind of "require()" call. I consider that even worse: it obsoletes the language's mechanism for modules, and defines its own modularization. However, this isn't really my objection to .egg files. I dislike them because they compete with platform packages: .rpm, .msi, .deb. Package authors will refuse to produce them, putting the burden of package maintenance (what packages are installed, what are their dependencies, when should I remove a package) onto the the end user/system administrator. Regards, Martin

Ronald Oussoren wrote:

As far as I understand the issues they compete up to a point, but should also make it easier to create platform packages that contain proper the proper dependencies because those are part of machine-readable meta-data instead of being written down in readme files. Oddly enough that was also the objection from one linux distribution maintainer: somehow his opinion was that the author of a package couldn't possibly know the right depedencies for it.

What he can't possibly know is the *name* of the package he depends on. For example, a distutils package might be called 'setuptools', so developers of additional packages might depend on 'setuptools'. However, on Debian, the dependency should be different: The package should depend on either 'python-setuptools' or 'python2.3-setuptools', depending on details which are off-topic here.

As for platform packages: not all platforms have useable packaging systems. MacOSX is one example of those, the system packager is an installer and doesn't include an uninstaller. Eggs make it a lot easier to manage python software in such an environment (and please don't point me to Fink or DarwinPorts on OSX, those have serious problems of their own).

Isn't uninstallation just a matter of deleting a directory? If I know that I want to uninstall the Python package 'foo', I just delete its directory. Now, with setuptools, I might have multiple versions installed, so I have to chose (in Finder) which of them I want to delete. That doesn't require Eggs to be single-file zipfiles; deleting a directory would work just as will (and I believe it will work with ez_install, too).

...

Package authors will refuse to produce them, putting the burden of package maintenance (what packages are installed, what are their dependencies, when should I remove a package) onto the the end user/system administrator.

Philip has added specific support for them: it is possible to install packages in the tradition way but with some additional files that tell setuptools about installed packages.

As a system administrator, I don't *want* to learn how to install Python packages. I know how to install RPMs (or MSIs, or whatever system I manage); this should be good enough. If "this Python junk" comes with its own installer procedure, I will hate it. Regards, Martin

Ronald Oussoren wrote:

...

That doesn't require Eggs to be single-file zipfiles; deleting a directory would work just as will (and I believe it will work with ez_install, too).

Egg directories (which are basically just the same as packages using .pth files) also work for this and that's what I usually install. Setuptools can work with python extension inside .zip files, but I'm not entirely comfortable with that.

It's primarily the .egg *files* that I dislike. I'm can accept the .egg directories. Regards, Martin

Guido van Rossum wrote:

The "resources" name is actually quite a common meme;

I believe it goes back to the original Macintosh, which was the first and only computer in the world to have files with something called a "resource fork". The resource fork contained pieces of data called "resources". Then Microsoft stole the name, and before you knew, everyone was using it. It's all been downhill from there. :-) -- Greg

At 11:26 PM 4/20/2006 +0200, Martin v. Löwis wrote:

Greg Ewing wrote:

...

...

The "resources" name is actually quite a common meme;

I believe it goes back to the original Macintosh, which was the first and only computer in the world to have files with something called a "resource fork". The resource fork contained pieces of data called "resources".

I can believe that history. Still, I thought a resource is something you can exhaust; the fork should have been named "data fork" or just "second fork".

I suspect that the distinction is that "data" sounds too much like something belonging to the *user* of the program, whereas "resources" can reasonably be construed to be something that belongs to the program itself. By now, however, the term is popularly used with GUI toolkits of all kinds to mean essentially read-only data files that are required by a program or library to function, but which are not directly part of the code. Interestingly enough, there is another "resource" tool for Python out there, that actually works by converting resource files to strings in .py files, so that you can then just import them.

On 4/20/06, "Martin v. Löwis" <martin@v.loewis.de> wrote:

I thought a resource is something you can exhaust; the fork should have been named "data fork" or just "second fork".

Hmm... I don't think that the English word "resource" necessarily implies that it can be exhausted. In US businesses, people are often also referred to as resources. I've come to think of it as simply "what you need". The definition at http://www.answers.com/resource has this: """ 1. Something that can be used for support or help: The local library is a valuable resource. 2. An available supply that can be drawn on when needed. Often used in the plural. 3. The ability to deal with a difficult or troublesome situation effectively; initiative: a person of resource. 4. Means that can be used to cope with a difficult situation. Often used in the plural: needed all my intellectual resources for the exam. 5. 1. resources The total means available for economic and political development, such as mineral wealth, labor force, and armaments. 2. resources The total means available to a company for increasing production or profit, including plant, labor, and raw material; assets. 3. Such means considered individually. """ #1 and #2 especially explain the usage we're considering. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

"Guido van Rossum wrote:

...

- I still fear that the code of setuptools will turn out to be a maintenance nightmare.

AFAICT Phillip has explicitly promised he will maintain it (or if he doesn't, I expect that he would gladly do so if you asked). This has always been sufficient as a "guarantee" that a new module isn't orphaned.

I'm not that worried about maintenance of the tool itself, but supporting "it's magic! (as long as you use the correct combination of one-letter options)" stuff is no fun at all. </F>

On 4/20/06, "Martin v. Löwis" <martin@v.loewis.de> wrote:

1. don't load packages out of .zip files. It's not that bad if software on the user's disk occupies multiple files, as long as there is a convenient way to get rid of them at once. Many problems go away if you just say no to zipimport.

You forget that importing from a zip file is often much faster than importing from the filesystem. That's why zipimport was added in the first place.

2. standardize on file names, not API. If I want to deploy read-only data files, I put them into /usr/share. If I need read-write files, I put them into /var.

What about Windows? Putting data files in the same directory as source files solves a lot of problems of making the data follow the source no matter how/where it is installed (or if it's not installed at all, like when running code straight out of svn). Anyway, perhaps it's a matter of choice. It's clear to me that many developers prefer to do it this way. You don't. This is an area that has enough external constraints that I'm uncomfortable telling developers they can't do it that way. A standard API to access resources by name (with perhaps a registry for defining additional ways to find them) makes a lot of sense to me, and I don't think we're exactly inventing something novel here. [...]

Some libraries (not necessarily in Python) have gone the path of providing a "unified" API for all kinds of file stream access, e.g. in KDE, any tool can open files over many protocols, without the storage being mounted locally. I consider this approach flawed: once I leave the realm of KDE programs, this all stops working. It is the operating system's job to provide unified access to storage, not the programming language's or the job of a library.

I don't see it that way. All operating system access is mediated via the Python library anyway; and in many cases the Python library provides additional abstractions over what the operating system natively provides (e.g. many APIs in the os and os.path modules). You can't blame KDE for providing mechanisms that only work in the KDE world. It's fine for Python to provide Python-specific solutions for issues that have no cross-platform native solution. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:

On 4/20/06, "Martin v. Löwis" <martin@v.loewis.de> wrote:

...

1. don't load packages out of .zip files. It's not that bad if software on the user's disk occupies multiple files, as long as there is a convenient way to get rid of them at once. Many problems go away if you just say no to zipimport.

You forget that importing from a zip file is often much faster than importing from the filesystem. That's why zipimport was added in the first place.

This is true if you only have a single ZIP file, e.g. python24.zip on sys.path. It is not if you have dozens of ZIP files on sys.path, since you have to take the initial scan time plus the memory for caching the ZIP file contents into account. Each ZIp file adds to Python startup time, since the ZIP files on sys.path are scanned for every Python startup. Here's an example of the effect of having just 20 eggs on sys.path: tmp/eggs> time python -S -c '0' 0.014u 0.006s 0:00.02 50.0% 0+0k 0+0io 0pf+0w tmp/eggs> unsetenv PYTHONPATH tmp/eggs> time python -S -c '0' 0.006u 0.003s 0:00.01 0.0% 0+0k 0+0io 0pf+0w (Reference: http://mail.python.org/pipermail/distutils-sig/2005-December/005739.html) Startup time effectively doubles, even if you don't use any of the installed eggs in your script. The main reason for adding zip imports was to simplify deploying (complete) applications to users, e.g. have the complete Python stdlib in one file. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 20 2006)

...

...

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

Greg Ewing wrote:

Guido van Rossum wrote:

...

You can't blame KDE for providing mechanisms that only work in the KDE world. It's fine for Python to provide Python-specific solutions for issues that have no cross-platform native solution.

Also keep in mind that we're talking about resources used internally by the application. They don't *need* to be accessible outside the application. So I don't think the KDE argument applies here.

They might need to be available outside "Python". Two use cases: 1. The application embeds an sqlite database. Even though it knows how to get at the data, it can't use it, because the sqlite3 library won't accept .../foo-1.0.egg/resources/the_data (or some such) as a database name, if foo-1.0.egg is a ZIP file. If the installed application was a set of files, that would work. 2. The application embeds an SGML DTD (say, HTML). In order to perform validation, it invokes nsgmls on the host system. It cannot pass the SGML catalog to nsgmls (using the -C option) since you can't refer to DTD files inside ZIP files inside an SGML catalog. If this was a bunch of installed files, it would work. 3. The application includes an SSL certificate. It can't pass it to socket.ssl, since OpenSSL expects a host system file name, not a fragment of a zip file. If this was installed as files, it would work. This is precisely what happens in KDE: you have konqueror happily browse an SMB directory, double-click on a .xls file, OpenOffice starts and can't access the file it was started with. It doesn't matter to the user that there is "a good reason" for that. Regards, Martin

Martin v. Löwis wrote:

Some libraries (not necessarily in Python) have gone the path of providing a "unified" API for all kinds of file stream access, e.g. in KDE, any tool can open files over many protocols, without the storage being mounted locally.

Maybe a compromise would be to provide an extended version of open() that knew about zip files. This could be a stand-alone facility to complement zip importing, and needn't be tied to distutils or setuptools or anything like that. Code which finds things using module __file__ attributes would work exactly the way it does now as long as it used the special open(). -- Greg

At 11:55 PM 4/18/2006 +0200, Fredrik Lundh wrote:

who decided that setuptools should be added to 2.5, btw?

Guido proposed it on Python-dev when the 2.5 schedule was first being discussed. I discussed it with him off-list, to ensure that it could be done in a way that wouldn't interfere with existing setuptools users or affect Python itself in a negative way. (For example, it needed to be upgradeable in the field in case users wanted/needed a later version than the one included in 2.5.) He then mentioned it in his 2.5 slideshow at PyCon. This is the first anyone's objected to it, however, at least that I'm aware of.

it's still listed under "possible additions" in the release PEP,

I imagine that might be why nobody raised any objections sooner, although I understood the possibility to mean "if nobody objects". :) I also posted on Python-dev repeatedly in recent weeks, referring to how the various PEP 302 fixes and updates would interact with setuptools when it got in for 2.5. Also, Neal emailed me the week before last, asking when I would be getting setuptools checked in, and I told him April 17th - i.e., yesterday. So, I was under the impression this was all a done deal.

and there don't seem to be a PEP or any other easily located document that explains exactly how it works, what's required from library developers, how it affects existing toolchains, etc.

The setuptools manual is currently at: http://peak.telecommunity.com/DevCenter/setuptools pending conversion to the standard Pythondoc format. I posted earlier today asking about how it, and the other related manuals should be included in the overall Python documentation structure: http://mail.python.org/pipermail/python-dev/2006-April/063846.html The reST source of these manuals is in trunk/sandbox/setuptools, where it has been evolving over the last year.

is this really ready for inclusion ?

Please define "ready". I don't mean that in a flippant way, I just don't know what you mean.

does anyone but Phillip understand how it works ?

Does anybody besides Thomas understand how ctypes works? ;) Rhetorical jokes aside, every time I've made a significant change to how setuptools works, I've posted it to the distutils-sig -- and usually I make proposals in advance to get feedback or stimulate discussion. I regularly post explanations there in response to questions from people who are integrating it with system packaging tools, or creating various other customizations. And there are other people on distutils-sig who can answer questions about it. The TurboGears community is proficient enough with it that it's only once every few months now that a question gets kicked upstairs to me to answer. A number of people have contributed patches, including Ian Bicking and Tres Seaver. Bob Ippolito was a significant participant in the original design and wrote some of the initial code for the runtime. A *considerable* number of distutils-sig participants have had design input, either through direct suggestions, or through their giving more use case examples that I needed to make "just work". So, I'm not too pleased by insinuations that setuptools is anything other than a Python community project. But MAL and MvL are the only folks from Python-Dev who I've seen over there arguing for changes to setuptools -- and I actually made changes based on their input, although they rarely got 100% of what they asked for. The --really-long-option MAL is complaining about was put in to provide a feature that *he and MvL wanted me to include*; I just don't want that behavior to be the default behavior for setuptools. (And neither do package developers who have to support "non-root" users on virtual hosting systems, or other environments where system packaging tools aren't available.) So, it seems to me that MAL claiming that nobody got to participate in the design process is rather misleading. It's like somebody who wanted decorators in Python and then gripes about the '@' syntax. Everybody's got to compromise a bit. I put their feature in months ago, and it is even the default when you use --root with install.

At 07:08 PM 4/18/2006 -0400, Fred L. Drake, Jr. wrote:

Saw that; hopefully I'll have a chance to look at it soon. I wonder, generally, if it should be merged into the distutils documentation. Those documents happen to be distutils-centric now, because that's what's been provided. Their titles should be the guide to their content, however.

No doubt that's the proper thing to do in the long term, when/if setuptools is the official One Obvious Way To Do It. I was wondering, however, if perhaps Python 2.5 should include them as "Building and Distributing Python Eggs" (for what's now setuptools.txt) and "Installing Python Eggs" (for what's now EasyInstall.txt).

I've no doubt about that at all, FWIW. I think you've put a lot of effort into discussing it with the community, and applaud you for that as well as your implementation efforts.

Thank you.

At 02:12 AM 4/19/2006 +0200, Giovanni Bajo wrote:

But, why can't setuptools be gradually merged into distutils, instead of being kept as a separate package? Let's take a real example: setuptools' sdist is much enhanced, has integration with CVS/SVN, uses MANIFEST in a way that it really works, etc. Why can't it be merged into the original distutils? Is it just for backward compatibility?

This specific issue was discussed last year on the distutils-sig, and the issue is indeed one of compatibility. Setuptools' behavior for MANIFEST generation definitely matches new or infrequent users' expectations 1000% better than the distutils, and requires much less work to get right, even for experts. But for anybody who has extended the distutils using external tools, it would not necessarily work. MAL gave the example of someone who has written other scripts or Makefile rules to add things to MANIFEST or use it to do other things. They might be relying on quirks of the existing behaviors, in other words, and thus it should not be changed without explicit action on their part. And I agree with his reasoning, although I also think that any "distutils 2" should have only One Obvious Way to send input to that process, and it should be via MANIFEST.in, not MANIFEST. Likewise, it should have only one way to get the output. However, unless somebody explicitly chooses to use "distutils 2", they should get the old behavior. This unfortunately means that we can't backport most of setuptools' enhancements to the existing distutils without breaking backward compatibility for people who may have made extensive investment in integrating with the distutils. (Of course, how many of these people exist I don't know; in my personal experience it seems rare for people to integrate with external tools in this fashion, versus simply subclassing things in Python or abandoning distutils altogether. But that's a separate question.)

If so, can't we have some kind of versioning system?

We do: "import setuptools". We could perhaps rename it to "import distutils2" if you prefer, but it would mean essentially the same thing. :)

At 08:11 AM 4/19/2006 +0200, Giovanni Bajo wrote:

Then, about new commands. Why should I need to do "import distutils2" to do, eg, "setup.py develop"? This doesn't break backward compatibility.

The develop command uses the egg_info command. egg_info uses the setuptools enhanced MANIFEST scheme. Both make use of extended setup() arguments, and the entry points feature that allows distutils plugins to co-operate. Develop also uses easy_install... and so on. I'm not saying it would be impossible to merge this stuff into the distutils, just that it's not a trivial undertaking.

Neal Norwitz wrote:

I was also working under the assumption that people would complain if they didn't like something. What do people think should happen for the "Possible features" section? Should I ask if there are any objections to each item?

some discussion on python-dev for each non-trivial item should be required. larger items may need discussion + grace period + discussion before they've checked in. the increasing amount of "but I've discussed this in some other forum" worries me. </F>

At 08:33 PM 4/19/2006 +0200, Ronald Oussoren wrote:

Have you considered such a merger? It's rather odd to include a package in the stdlib that monkeypatches another part of the stdlib.

I assumed that it would be more controversial to merge setuptools into distutils, than to provide it as an enhanced alternative. Also, there are two practical issues: 1. How to activate or deactivate backward compatibility for packages or people relying on intimate details of current distutils behaviors 2. Maintaining the existing version of setuptools to work with Python 2.3 and 2.4, which would not have this integration with the distutils.

Martin> c) ask for consent in advance to making a potentially-breaking Martin> change. Doesn't that potentially extend the release time for an enhanced distutils across multiple Python releases? With both distutils and setuptools available simultaneously, setuptools can be designed and implemented to suit the needs of its constituency while distutils remains avilable and compatible for those people using it. Skip

[removing the python-checkins CC] Phillip J. Eby wrote:

At 09:02 PM 4/18/2006 +0200, M.-A. Lemburg wrote:

...

Phillip J. Eby wrote:

...

...

Why should a 3rd party extension be hot-fixing the standard Python distribution ? Because setuptools installs things in zip files, and older versions of

At 07:15 PM 4/18/2006 +0200, M.-A. Lemburg wrote: pydoc don't work for packages zip files. That doesn't answer my question.

That is the answer to the question you asked: "why hot-fix?" Because setuptools uses zip files, and older pydocs crash when trying to display the documentation for a package (not module; modules were fine) that is in a zip file.

I fail to see the relationship between setuptools and pydoc. Please don't follow this route - you are putting the integrity of the Python stdlib at risk. The setuptools distribution is not the authoritative source for these kinds of fixes and that should be made clear by separating those parts out into a different package and making the installation an explicit user option. You should also note that users won't benefit from bug fixed versions of e.g. such modules in patch level releases.

...

Third-party extensions *should not do this* !

If you install setuptools, you presumably would like for things to work, and the hot fix eliminates a bug that interferes with them working.

I'm not sure, however, what you believe any of that has to do with python-checkins or python-dev. The version of setuptools that will do this is not yet released, and the hotfix aspect will be documented. If it causes actual problems for actual setuptools users, it will be actually fixed or actually removed. :) (Meanwhile, the separately-distributed setuptools package is not a part of Python, any more than the Optik is, despite Python including the 'optparse' package spun off from Optik.)

It's an issue to discuss on python-dev because it touches things that python-dev folks develop. However, it's not related to the rest of this discussion, so I'm changing the subject line and will respond to the rest in a separate email. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 19 2006)

...

...

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

On Apr 19, 2006, at 3:19 PM, Phillip J. Eby wrote:

At 08:45 PM 4/19/2006 +0200, M.-A. Lemburg wrote:

...

Phillip J. Eby wrote:

...

At 09:02 PM 4/18/2006 +0200, M.-A. Lemburg wrote:

...

Phillip J. Eby wrote:

...

...

Why should a 3rd party extension be hot-fixing the standard Python distribution ? Because setuptools installs things in zip files, and older versions of

At 07:15 PM 4/18/2006 +0200, M.-A. Lemburg wrote: pydoc don't work for packages zip files. That doesn't answer my question.

That is the answer to the question you asked: "why hot-fix?" Because setuptools uses zip files, and older pydocs crash when trying to display the documentation for a package (not module; modules were fine) that is in a zip file.

I fail to see the relationship between setuptools and pydoc.

People blame setuptools when pydoc doesn't work on packages in zip files. Rather than refer to some theoretical argument why it's not my fault and I shouldn't be the one to fix it, I prefer to fix the problem.

FWIW, I see very little harm from setuptools replacing the old version of pydoc with a later official python version which causes it to actually work with setuptools. Twisted has also done some of of this kind of thing in the past, so that the code can be written to use modules/features in current versions of python yet still run on earlier versions where that functionality doesn't exist/doesn't work. I don't recall ever hearing of a user complaining about said monkeypatching. Users want compatibility with older versions of python, and sometimes that may mean you have to patch said older version's stdlib to make it work. James

Phillip J. Eby wrote:

...

...

...

...

...

Why should a 3rd party extension be hot-fixing the standard Python distribution ? Because setuptools installs things in zip files, and older versions of pydoc don't work for packages zip files. That doesn't answer my question. That is the answer to the question you asked: "why hot-fix?" Because setuptools uses zip files, and older pydocs crash when trying to display the documentation for a package (not module; modules were fine) that is in a zip file. I fail to see the relationship between setuptools and pydoc.

People blame setuptools when pydoc doesn't work on packages in zip files. Rather than refer to some theoretical argument why it's not my fault and I shouldn't be the one to fix it, I prefer to fix the problem.

And people come to python-dev to complain about why a Python patch level release doesn't seem to fix some other problem in that same code that is documented to be fixed in the release. Don't you realize the general problem with this kind of approach ? BTW: if setuptools would stop defaulting to using .egg files as means of installation and instead default to using their unzipped form, the problem would go away and setuptools would again be in line with the Python standard of installing modules and packages. (For the interested: see my reply to Anthony for details or the fruitless discussions on distutils-sig on this matter.)

...

The setuptools distribution is not the authoritative source for these kinds of fixes and that should be made clear by separating those parts out into a different package and making the installation an explicit user option.

...which nobody will use, following which they will still blame setuptools for pydoc's failure.

But I do agree that it might be a good idea to:

1) tell people about the issue 2) tell people that the fix is being installed 3) make it easy to remove the fix

However, maybe I should just create 'pydoc-hotfix' and 'doctest-hotfix' packages on PyPI and then tell people to "easy_install pydoc-hotfix doctest-hotfix".

Yes, please ! Please use a different name for these hot-fixes, though, so that it's clear that they only apply to specific Python versions and are in fact, back-ports of Python 2.5 versions of those modules.

Or perhaps just create a "pep302-hotfixes" package that includes all of the PEP 302 support changes for linecache, traceback, and so on, although I'm not sure how many of those can be installed in such a way that the fixes would be active prior to the stdlib versions being imported.

I suppose I could handle the "nobody will use it" problem by having easy_install nag about the hotfixes not being installed when running under 2.3 or 2.4.

...

You should also note that users won't benefit from bug fixed versions of e.g. such modules in patch level releases.

The pydoc fixes to support zip files are too extensive to justify backporting, as they rely on features provided elsewhere. So if bug fixes occur on the 2.5 trunk, I would have to update the hotfix versions

Fair enough. directly. Right. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 20 2006)

...

...

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::