[Python-Dev] Internal documentation for egg formats now available

[Phillip J. Eby]

At 06:20 PM 4/28/2006 +0200, M.-A. Lemburg wrote:
>A single malicious .pth file on PYTHONPATH could trick
>the user into running Python modules that he isn't really
>aware of.
>
>Adding directories to PYTHONPATH which are writable by
>others is easy, just add '/tmp' as entry in some .pth
>file. Then put e.g. a modified os.py into /tmp and have
>it execute 'rm -rf /' at import time.

You still haven't explained *how* these malicious .pth files or entries 
come into being *without the user's permission*.  How does this malicious 
user get write access to the .pth file to put /tmp into it?  Whatever your 
answer is, *that* is the security hole, not the existence of usable .pth files.

So this is still just handwaving and FUD.


> > And if you replace 'syspathtools.use()' in that email, with
> > 'pkg_resources.require()', then it describes *exactly how setuptools
> > works  with .egg directories today*.
>
>Interesting that you used that idea, because back then you didn't
>reply to the email. Looks like I deserve some credit ;-)

In truth, I completely forgot about that post and only rediscovered it 
yesterday.  When Bob and I were doing the initial design, he said he 
thought that the naively-unzipped form of an egg should be usable as-is, 
and I agreed that it made sense to do that.


>If you've already implemented this (which I wasn't aware of, since
>when I played with setuptools it kept installing .egg ZIP files),

See the --always-unzip option to easy_install, which has been there since 
0.5a9 -- i.e., since July 2005.  Package authors can also mark their 
packages "zip_safe=False" to force it to be installed unzipped (unless the 
user forces it to be zipped using --zip-ok).


>then why don't you make .egg *directories* the standard installation
>scheme, instead of insisting on having .egg ZIP files in
>site-packages/ ?

Because zip files add *less* import overhead over the life of a program 
(even a relatively short program) than directories do!

(Which, by the way, I believe I've explained to you on the distutils-SIG on 
more than one occasion, during some of the previous N times you brought 
this up.)


>I wonder why you don't always create .egg directories.

Performance, plain and simple.

What you *should* be benchmarking when you test performance impact, is 
twenty directories at the front of sys.path vs. twenty zipfiles at the 
front, both for "python -c pass" and for a program that does a few stdlib 
imports.  That will show you why EasyInstall never unzips eggs if it can 
help it.


>* I am against using ZIP as medium for package installation.

Then add this to your distutils.cfg or ~/.pydistutils.cfg and be happy 
forevermore:

[easy_install]
always_unzip = 1

If, however, you mean that you want *everyone else* to do as you prefer, 
forget about it, 'cause it's not going to happen.  You aren't setuptools' 
target user, so the defaults aren't going to be catering to your prejudices 
-- especially not the ones based on ignorance, like this one.


>* I am against using standard distutils commands to mean
>   different things depending on which set of packages you
>   have installed (in this case the "install" command)

And as I just pointed out, you've failed to show that there is any 
practical purpose to this.  I've already made the behavior an option, and 
it's the default behavior when you use --root.  You haven't shown any 
actual use case for it being the default behavior, and practicality beats 
purity.  What's more, you said you don't want the option to make it the 
default behavior for you.  Which means that this is only about you deciding 
what is "right" for *other* people than you to have, so I don't see any 
reason to consider that subject any further.  If other people want that 
behavior, they can certainly speak up for themselves about what *actual* 
(not hypothetical) problems it creates for them.  (And when they do speak 
up, as some system packagers did, I made --root enable the old behavior and 
that addressed the *actual* problems the new behavior created for that 
audience.)


>Phillip, you're always side-stepping answering questions
>(and cutting away quoted email text doesn't help either).

Repeating arguments I've already shown to be spurious helps even less.  I 
don't see much point to chasing red herrings and straw men, either.


>You should be aware that both package authors and users
>are capable of learning and adapting.

Then they can certainly adapt to "install" doing the right thing for most 
users and most packages: to install things in such a way that they can be 
uninstalled or upgraded cleanly, by default.


>If we tell package authors to put a link to their eggs on
>PyPI along with an MD5 sum, then the users be in a much
>better position and could feel more secure about what they
>download and install.

People who want to do that can easily use:

easy_install --allow-hosts=*.python.org

or put it in their configuration files to make it the default behavior.  See:

http://peak.telecommunity.com/DevCenter/EasyInstall#restricting-downloads-with-allow-hosts



>Have a nice weekend,

It'd be an even nicer one if people objecting to things would read the 
documentation first, to find out whether their objections actually 
apply.  Your zip vs. directory arguments would've been moot if you'd read 
things like this:

http://peak.telecommunity.com/DevCenter/EasyInstall#compressed-installation

Or for that matter, if you'd read the documentation that was linked to in 
the post that started this thread, which explains that .egg can be a 
zipfile or a directory.  Or if you'd read the parts of the EasyInstall 
documentation where it explains that OS X already processes .pth files from 
~/Library/Python2.x/site-packages.

If you don't have time to read the documentation, please have the courtesy 
to *ask questions* to try to alleviate your concerns, preferably *before* 
you launch attacks using nothing but vague premonitions based on ignorance.