[Distutils] distlib - installer support vs runtime support

[PJ Eby]

On Sat, Mar 9, 2013 at 8:14 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
> On 10 Mar 2013 10:16, "Vinay Sajip" <vinay_sajip at yahoo.co.uk> wrote:
>>
>> Paul Moore <p.f.moore <at> gmail.com> writes:
>>
>> > Would it be worth considering splitting distlib into two separate
>> > parts - one that is intended solely for writers of installers and
>> > similar tools, and another for "runtime support" functions that end
>> > users would use? It may not be a practical thing to achieve, but it
>> > would be worth at least understanding the trade-offs involved.
>>
>> While this could be done, it would not exactly be elegant, and IMO it
>> would be
>> the wrong way to address the valid concerns you mention. It would make
>> more
>> sense for pip to *contain* a specific version of distlib for its use (e.g.
>> as a
>> .zip) so that it never worries about conflicts with another copy. This is
>> the
>> approach that Django takes and it seems to work reasonably well for that
>> project.
>>
>> Re. the size of distlib - it's larger than pip, because pip relies on a
>> external dependency (setuptools/distribute) to do a lot of its work,
>> whereas
>> distlib is self-contained. So, direct size comparisons can be misleading
>> (e.g.
>> distlib contains a _backport package for 2.6 support, which is not tiny;
>> distlib's tests contain tests for the backports plus a complete copy of
>> unittest2, again for 2.6 support).
>>
>> The concerns about stability (in terms of API stability, as well as the
>> presence of bugs) are more valid than size. Given distlib's youth, we can
>> expect to see some API changes, as well as to see some bugs flushed out of
>> the
>> woodwork. (Obviously, I would aim to keep API changes to a minimum.)
>>
>> A good level of stability is generally achieved after a period of usage in
>> anger followed by feedback based on that usage. Until then, the test
>> suite,
>> coverage results and docs will need to be used to give an indication of
>> quality.
>
> pip vendoring its own copy of distlib sounds like the best workaround for
> now, as it addresses both the bootstrapping problem and the API stability
> question.
>
> Longer term, something like the import engine PEP may let us implement a
> cleaner solution.

I've been giving the bootstrapping issue a bit more thought, though,
and I think there's a way to take the pain out of multi-file
bootstraps of things like pip and setuptools and whatnot.

Suppose you make a .py file that contains (much like ez_setup.py) a
list of files to download, along with their checksums.  And suppose
this .py file basically just has some code that checks a local cache
directory for those files, adds them to sys.path if they're found,
downloads them if they're not (validating the checksum), and then
proceeds to import a specific entry point and run it?

At that point, you could distribute a Python application (like pip)
"in" a single .py file.  The file is for a specific version and a
specific set of dependencies, but if someone wants to update it, they
just download the new .py file and discard the old one.  (Technically,
it needn't have a .py extension, and for Windows you might want to
ship it as a .exe+"-script.py/pyw" wrapper pair.)

The really interesting bit is that you could generate the .py part
using nothing more than a Metadata 2.0 file for the target app, access
to the /simple index, and a specification of which script/entry-point
the .py file is supposed to represent.

We could call this concept PyStart, as it's a little bit like Java
WebStart.  It would work equally well with wheels or eggs, though with
wheels you'd need to unpack any "impure" ones to a subdirectory.  The
main idea, though, is that if properly done, the PyStart script for an
application wouldn't need to understand version parsing or comparison,
PyPI APIs, environment markers, or any of that crud.  (It *might* need
to understand platform tags to some extent, if "impure" code is
involved.)

Heck, it wouldn't even care about SSL cert verification, as it'll be
relying on its hardcoded hashes for verification purposes.  So as long
as you download the PyStart script from a trusted source, you're ready
to go.  (This is an important requirement for setuptools going
forward, because it may have to depend on more than one other project
(e.g. Requests + SSL backport) in order to implement full SSL
security.)

This is far from a finished idea, but I think it has some merit as an
approach for distributing "zero-install" Python apps in general, while
working around the hairy problem of things like, "so, to install pip,
you need to have setuptools installed.  But you can't use setuptools
to install pip, because that wouldn't be secure.  So, make sure you
download *both* things by hand, oh, and don't forget to compile the
SSL backport..."

Thoughts?