[Python-Dev] magic in setuptools (Was: setuptools in the stdlib)

["Martin v. Löwis"]

>> I don't understand it.
> 
> Have you read the manuals?

You mean,

http://peak.telecommunity.com/DevCenter/setuptools

Yes, I did. However, this would only enable me to use it (should I
ever find a need). What I really need to understand is how all this
stuff works inside.

Now, I haven't actually *tried* to learn it so far. Fredrik's question
was not "who thinks he could learn it" (I certainly could), I
read his question as "who does understand it, as of today". I can't
faithfully answer "yes" to that question.

So far, I believe nobody has said "yes, I do understand it, and could
maintain it if Phillip Eby became unavailable".

> Please define "magic". 

"Functionality with no obvious implementation, or functionality
implemented in an unobvious way".

> Better, please point to the API functions or 
> classes, or the setup commands documented in the manual, to show me what 
> these things are that appear to be magic.

Here are some that I either already knew, or discovered just now:

1. setuptools manages to replace pydoc on earlier Python versions.
   MAL assumed it does so by overwriting the files. This would be
   bad practice, but is the obvious solution.
   You then told him that this is not what actually happens (so
   it's already magic); instead, you arrange that Python finds
   your version before it finds its own.
   Again, this is still magic: How does it do that? If you append
   to sys.path, Python would still find its own version before
   it finds yours. So perhaps you insert another path at
   sys.path[0]? This would also be bad practice, but it would
   be the next-obvious approach.
   But I fully expect that you are now telling me that this is
   still not how it works.

2. setuptools manages to install code deep inside a subdirectory
   of site-python, and manages to install multiple versions
   simultaneously.
   How does that work? The first part (packages outside sys.path)
   can be solved with a .pth file (which I already consider bad
   practice, as it predates the package concept of Python);
   but that can't work for multiple versions: magic.

   I (think I) know the answer: there is a single .pth file,
   and that is *edited* on package installation time. This
   is a completely non-obvious approach. If Python needs
   to support multiple versions of a package (and there
   is apparently a need), that should be supported
   "properly", with a clear design, in a PEP.

3. namespace packages. From the documentation, I see that
   you pass namespace_packages= to setup, and that you
   put

   __import__('pkg_resources').declare_namespace(__name__)

   into the namespace package. How does that work?
   The documentation only says

   "This code ensures that the namespace package machinery
   is operating and that the current package is registered
   as a namespace package."

   Also, why not

   import pkg_resources
   pkg_resources.declare_namespace(__name__)

   What does declare_namespace actually do?

4. zip_safe. Documentation says

   "If this argument is not supplied, the bdist_egg command
   will have to analyze all of your project's contents
   for possible problems each time it buids an egg."

   How? Sounds like it is solving the halting problem, if
   it manages to find possible problems.


> There do exist implementation details that are not included in user 
> documentation, but this is going to be true of any system.  These details 
> are sometimes necessarily complex due to distutils limitations, behavioral 
> quirks of deployed packages using distutils, and the sometimes baroque 
> variations in sys.path resolution across platforms, Python versions, and 
> invocation methods.

I trust that there is a good reason for each of them. However,
for inclusion in the standard library, some of these should go away:
- if you have distutils limitations, remove them
- differences in Python versions shouldn't matter: you always
  know what the Python version is
- if there are baroque variations in sys.path resolution across
  platforms, they should be removed, or modernized.
Not sure what "invocation methods" are.

>> This magic might do the "right thing" in many cases, and it might
>> indeed help the user that the magic is present, yet I still do believe
>> that magic is inherently evil: explicit is better than implicit.
> 
> Are documented defaults "implicit" or "magic"?

Documented defaults are explicit.

> With respect to you and MAL, I think that some of your judgments regarding 
> setuptools may have perhaps been largely formed at a time last year when, 
> among other things:
> 
> * That documentation section I just referenced didn't exist yet

I don't think I ever complained about the lack of documentation. I would
(personally) not trust the documentation, anyway, but always look at the
code to find out how it actually works. I read documentation primarily
to find out what the *intent* was.

> These are significant changes that are directly relevant to the objections 
> that you and he raised (regarding startup time, path length, tools 
> compatibility, etc.), and which I added because of those objections.  I 
> think that they are appropriate responses to the issues raised, in that 
> they allow the audience that cares about them (you, MAL, and system 
> packagers in general) to avoid the problems that those features' absence 
> caused.

And I like these changes indeed - my (current) complaints are not about
the functionality provided.

Instead, I'm concerned about maintainability of this code.

> It would probably helpful if you would both be as specific as possible in 
> your objections so that they can be addressed individually.  If you don't 
> want setuptools in 2.5, let's be clear either on the specific 
> objections.  If the objection is to the very *idea* of setuptools, that's 
> fine too, as long as we're clear that's what the objection is.

setuptools combines several independent aspects, as far as I can tell:

1. automatic download of packages (actually ez_setup). This is a nice
   feature, and should be included. +1
2. enhancements to the development process, and improvements to the
   extensibility of distutils. Also nice, but should be part of
   distutils itself. +0 in its current form, +1 if it were distutils
   features.
3. "package resources". I dislike the term resources (it is not about
   natural gas, water, main memory, or processor cycles, right?);
   instead, this seems to provide access to "embedded" data files.
   Apparently, there is a need for it, so be it. -0
4. .egg files. -1
5. entry points. I don't understand that concept
6. package dependencies. +1
7. namespace packages. Apparently, there is a need, although
   I believe that flat is better than nested. -0

However, these are all besides my original objections:

- I disliked the fact that nobody explicitly approved inclusion
  of setuptools. Now that Anthony Baxter did, I'm fine.

- I still fear that the code of setuptools will turn out to be
  a maintenance nightmare. Of course, this doesn't have to concern
  me - I could just ignore all bug reports about that, and ignore
  it in the next revision of my book. However, I know that I can't
  take that position, hence my concerns.


> So I would ask that you please make a list of what you would have to see 
> changed in setuptools before you would consider it viable for stdlib 
> inclusion, or simply admit that there are no changes that would satisfy 
> you, or that you don't know enough about it to say, or that you'd like it 
> to be kicked back to distutils-sig for more discussion ad infinitum, or 
> whatever your actual objections are.

See above: now that Anthony approved it, I can accept it. There is
nothing you could possibly do to ease my concerns about maintainability,
or my dislike of a Python-specific package format.

> Meanwhile, this discussion has used 
> up the time that I otherwise would have spent writing 2.5 documentation 
> today (e.g., for the pkgutil additions).

Same here. I spent several hours now typing setuptools email. Still,
I don't think this is wasted time (or else I would not have typed
this message).

Regards,
Martin