[Python-Dev] Internal documentation for egg formats now available

[M.-A. Lemburg]

Phillip J. Eby wrote:
> At 09:54 PM 4/27/2006 +0200, M.-A. Lemburg wrote:
>> Note that I was talking about the .pth file being
>> writable, not the directory.
> 
> Please stop this vague, handwaving FUD.  You have yet to explain how this 
> situation is supposed to arise.  Is there some platform on which files with 
> a .pth extension are automatically writable by users *when .py files are 
> not also*?
> 
> If not, then you are talking out of your... um, hat.  If files are writable 
> by other users by default, then .py files are too.  Once again: *no new 
> vector*.
> 
>> Even if they are not
>> writable by non-admins, the .pth files can point
>> to directories which are.
> 
> Uh huh.  And how does that happen, exactly?  Um, the user puts them 
> there?  What is your point, exactly?  That people can do things insecurely 
> if they're allowed near a computer?

You don't seem to want to see the problem:

By modifying site.py to take .pth files on all directories
in PYTHONPATH into account, you extend the possibility to
add "symbolic" links to a lot more directories than those
which are normally monitored by the sysadmin.

Every entry in a .pth file is added to sys.path, it even
allows imports to happen at .pth file scanning time.

A single malicious .pth file on PYTHONPATH could trick
the user into running Python modules that he isn't really
aware of.

Adding directories to PYTHONPATH which are writable by
others is easy, just add '/tmp' as entry in some .pth
file. Then put e.g. a modified os.py into /tmp and have
it execute 'rm -rf /' at import time.

In summary: I don't think that allowing .pth files on all
PYTHONPATH directories is the right way to go.

In order to solve your problem with users not being able to
install user local copies of packages, we should instead
discuss the possibility of having a standard directory
on PYTHONPATH that is being searched for such packages
and then allow .pth files only in that directory (which
is under user control).

Please consider discussing such things on python-dev before
implementing a hack that works around standard Python
behavior.

>> Here's a HOWTO to optimize startup time, without loosing
>> convenience:
> 
> I meant, a HOWTO for setuptools users who care about this, although at the 
> moment I have heard only from one -- and you're not, AFAIK, actually a 
> setuptools user.

You asked for a HOWTO, I gave you one :-)

I have played with setuptools and it lots of interesting
things, most of which are great and I applaud you for
taking the effort in creating the tool.

However, as you know, I do have a few issues with it that I'd like
to see  resolved, which is why I keep trying to convince you of
considering them (ever since you started in 2004 with this).

>> No, I'm talking about a format which has the same if not
>> more benefits as what you're trying to achieve with the
>> .egg file approach, but without all the magic and hacks.
>>
>> It's not like this wouldn't be possible to achieve.
> 
> That may or may not be true.  Perhaps if you had participated in the 
> original call to the distutils-sig for developing such a format (back in 
> December 2004), perhaps the design would've been more to your liking.
> 
> Oh wait...  you did:
> 
> http://mail.python.org/pipermail/distutils-sig/2004-December/004351.html

Indeed. And I suggested that you reconsider the idea to
use ZIP files for installation (rather than just distribution):

http://mail.python.org/pipermail/distutils-sig/2004-December/004349.html

> And if you replace 'syspathtools.use()' in that email, with 
> 'pkg_resources.require()', then it describes *exactly how setuptools 
> works  with .egg directories today*.

Interesting that you used that idea, because back then you didn't
reply to the email. Looks like I deserve some credit ;-)

If you've already implemented this (which I wasn't aware of, since
when I played with setuptools it kept installing .egg ZIP files),
then why don't you make .egg *directories* the standard installation
scheme, instead of insisting on having .egg ZIP files in
site-packages/ ?

I've now played with it again and found that for some
packages (e.g. kid and Paste) it installs these as .egg
directories (horray!).

For other packages such as elementtree which are not available
as egg files, it still creates egg files (by first downloading the
source package, then creating an egg file and installing that).

It also uses the egg files for quite a few packages that are
distributed as egg files.

So far, I've not found a pattern to this.

I wonder why you don't always create .egg directories.

> Apparently, you hadn't yet thought of any the objections that you are now 
> raising to *the very scheme that you yourself proposed*, until somebody 
> else took the trouble to actually implement it.

Where am I doing that ?

> And now you want to say that I never listen to or implement your 
> proposals?  Please.  Your email is the first documentation on record of how 
> this system works!

I'll repeat it again (hopefully the last time):

* I'm not against using directories for managing packages

* I'm not against using ZIP files of whatever format for
  the distribution of packages.

* I'm not against adding all kinds of new and useful commands
  to distutils, including bdist_egg and a new install_as_egg
  command

* I am against using ZIP as medium for package installation.

* I am against using standard distutils commands to mean
  different things depending on which set of packages you
  have installed (in this case the "install" command)

I believe that these requests are very reasonable.

>> Not really.
> 
> Then I won't bother adding it, since nobody else asked for it.  But don't 
> ever say that I didn't offer you a real solution to the behavior you 
> complained about.
> 
> Meanwhile, I will take your choice as prima facie evidence that the things 
> you are griping about have no real impact on you, and you are simply trying 
> to make other people conform to your views of how things "should" be, 
> rather than being interested in solving actual problems.
> 
> It also makes it clear that your opinion about setuptools default 
> installation behavior isn't relevant to any future Python-Dev discussion 
> about setuptools' inclusion in the standard library, because it's:
> 
> 1. Obviously not a real problem for you (else you'd accept the offer of a 
> feature that would permanently solve it for you)
> 
> 2. Not something that anybody else has complained about since I made --root 
> automatically activate distutils compatibility
> 
> In short, the credibility of your whining about this point and my supposed 
> failure to accomodate you is now thoroughly debunked.  I added an option to 
> enable this behavior, I made other options enable the behavior where it 
> could be reasonably assumed valid, and I offered you an option you could 
> use to globally disable it for *any* package using setuptools, so that it 
> would never affect you again.
> 
> (And all of this... to disable the behavior that implements a scheme that 
> you yourself proposed as the best way to do this sort of thing!)

Phillip, you're always side-stepping answering questions
(and cutting away quoted email text doesn't help either).

Adding dozens of options to your system doesn't help remedy
the problem, it only makes things worse.

Of course, there are always ways for me to have my way of
doing things - from hacking setuptools to simply avoiding
setuptools altogether.

But that's not what I want.

I would like setuptools to play nice with the rest of the
Python world and become part of the standard distribution.

However, the level of magic introduced by setuptools is
simply not what we're used to in the Python world, at least
not around here (meaning python-dev).

The few suggestions I made would not hurt setuptools much,
but would help put it back into the "explicit is better than
implicit" camp, simply because a lot of magic could be
removed.

As an example, you could avoid all the problems you currently
have with C extensions in .egg ZIP files. You currently unzip
just the C extension to a temporary cache directory and add
proxies for in the original packages (a strategy that may
or may not work depending on how the C extension is tied
to the rest of the package).

By installing the .egg ZIP file as .egg directory, you'd
avoid this completely.

Another example: the code that tries to find packages on the
web. Screen scrapping is not really what I'd consider a safe
way to find a package - the user might end up installing a
completely different package.

Your argument is that you'd like things to work without
the user having to pop up a browser and find the right
URL for a package (your usually "just works" argument).

You should be aware that both package authors and users
are capable of learning and adapting.

If we tell package authors to put a link to their eggs on
PyPI along with an MD5 sum, then the users be in a much
better position and could feel more secure about what they
download and install.

It may take a while, but in the end will make the Python
world better and safer for everyone, while still making
the user experience a good one.

Plus, you could rip out all the code that currently
tries to read web-sites and find packages that resemble
the package name, reducing magic.

Have a nice weekend,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Apr 28 2006)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::