On Sat, Mar 9, 2013 at 8:14 PM, Nick Coghlan wrote:

On 10 Mar 2013 10:16, "Vinay Sajip" wrote:

...

Paul Moore writes:

...

Would it be worth considering splitting distlib into two separate parts - one that is intended solely for writers of installers and similar tools, and another for "runtime support" functions that end users would use? It may not be a practical thing to achieve, but it would be worth at least understanding the trade-offs involved.

While this could be done, it would not exactly be elegant, and IMO it would be the wrong way to address the valid concerns you mention. It would make more sense for pip to *contain* a specific version of distlib for its use (e.g. as a .zip) so that it never worries about conflicts with another copy. This is the approach that Django takes and it seems to work reasonably well for that project.

Re. the size of distlib - it's larger than pip, because pip relies on a external dependency (setuptools/distribute) to do a lot of its work, whereas distlib is self-contained. So, direct size comparisons can be misleading (e.g. distlib contains a _backport package for 2.6 support, which is not tiny; distlib's tests contain tests for the backports plus a complete copy of unittest2, again for 2.6 support).

The concerns about stability (in terms of API stability, as well as the presence of bugs) are more valid than size. Given distlib's youth, we can expect to see some API changes, as well as to see some bugs flushed out of the woodwork. (Obviously, I would aim to keep API changes to a minimum.)

A good level of stability is generally achieved after a period of usage in anger followed by feedback based on that usage. Until then, the test suite, coverage results and docs will need to be used to give an indication of quality.

pip vendoring its own copy of distlib sounds like the best workaround for now, as it addresses both the bootstrapping problem and the API stability question.

Longer term, something like the import engine PEP may let us implement a cleaner solution.

I've been giving the bootstrapping issue a bit more thought, though, and I think there's a way to take the pain out of multi-file bootstraps of things like pip and setuptools and whatnot. Suppose you make a .py file that contains (much like ez_setup.py) a list of files to download, along with their checksums. And suppose this .py file basically just has some code that checks a local cache directory for those files, adds them to sys.path if they're found, downloads them if they're not (validating the checksum), and then proceeds to import a specific entry point and run it? At that point, you could distribute a Python application (like pip) "in" a single .py file. The file is for a specific version and a specific set of dependencies, but if someone wants to update it, they just download the new .py file and discard the old one. (Technically, it needn't have a .py extension, and for Windows you might want to ship it as a .exe+"-script.py/pyw" wrapper pair.) The really interesting bit is that you could generate the .py part using nothing more than a Metadata 2.0 file for the target app, access to the /simple index, and a specification of which script/entry-point the .py file is supposed to represent. We could call this concept PyStart, as it's a little bit like Java WebStart. It would work equally well with wheels or eggs, though with wheels you'd need to unpack any "impure" ones to a subdirectory. The main idea, though, is that if properly done, the PyStart script for an application wouldn't need to understand version parsing or comparison, PyPI APIs, environment markers, or any of that crud. (It *might* need to understand platform tags to some extent, if "impure" code is involved.) Heck, it wouldn't even care about SSL cert verification, as it'll be relying on its hardcoded hashes for verification purposes. So as long as you download the PyStart script from a trusted source, you're ready to go. (This is an important requirement for setuptools going forward, because it may have to depend on more than one other project (e.g. Requests + SSL backport) in order to implement full SSL security.) This is far from a finished idea, but I think it has some merit as an approach for distributing "zero-install" Python apps in general, while working around the hairy problem of things like, "so, to install pip, you need to have setuptools installed. But you can't use setuptools to install pip, because that wouldn't be secure. So, make sure you download *both* things by hand, oh, and don't forget to compile the SSL backport..." Thoughts?

On Mon, Mar 11, 2013 at 11:36 AM, Erik Bray wrote:

On Sat, Mar 9, 2013 at 9:33 PM, PJ Eby wrote:

...

On Sat, Mar 9, 2013 at 8:14 PM, Nick Coghlan wrote:

...

Longer term, something like the import engine PEP may let us implement a cleaner solution.

I've been giving the bootstrapping issue a bit more thought, though, and I think there's a way to take the pain out of multi-file bootstraps of things like pip and setuptools and whatnot.

Suppose you make a .py file that contains (much like ez_setup.py) a list of files to download, along with their checksums. And suppose this .py file basically just has some code that checks a local cache directory for those files, adds them to sys.path if they're found, downloads them if they're not (validating the checksum), and then proceeds to import a specific entry point and run it?

I've been thinking about something like this lately too. A simple script like your proposed pystart could be used to generate these files. I've gone one further and considered a format that includes dependencies (or at the very least install-time dependencies right in the .py file itself as a base64 string. This would work fine at least for small dependencies--no downloads necessary and you're guaranteed to get the right version that actually installs the damn package.

You could, but it makes the initial download bigger and adds more moving parts. If you have to download some of them, might as well do all of them. (Besides, the approach I outlined allows for sharing distributions and avoiding repeated downloads.)

For packages using setuptools it's also possible to just include a dependency as an egg or tar.gz right in your source dist and add something like "[easy_install]\nfind_links = ." to your setup.cfg and it'll go. But I'm not sure that's exactly the use case you're after here.

No. A particular goal behind this idea is that a pystart script should be cross-platform and ideally cross-Python-version as well, using environment markers to determine what should be downloaded. So, syntactically, the script would be written so that it runs on as many Python versions as possible, which shouldn't be too hard since it will basically be just looking for a series of files, downloading and hashing them. You'd have one function that does a "check for this file, download and extract if needed (w/hash validation), add to sys.path", and a bunch of if/then blocks calling that function with different name/url/hash triplets according to the current platform/python version markers. Then the bottom of the script would import the script entry point and run it. (And the top would have a preamble to import stuff from the right places based on Python version.) Alternatively, for a simpler code generation model, it could just stick everything in a giant data structure at the top, and append standard code that checks the environment markers. But I'd rather not have to include full environment marker interpretation, so translating the markers to Python seems advisable, even if it's as lambdas in the data structure.

The pkg_resources script entry point is there so the right eggs can be added to sys.path based on solving dependencies for the invoked package. On Mar 10, 2013 7:30 AM, "Paul Moore" wrote:

On 10 March 2013 00:15, Vinay Sajip wrote:

...

Paul Moore writes:

...

Would it be worth considering splitting distlib into two separate parts - one that is intended solely for writers of installers and similar tools, and another for "runtime support" functions that end users would use? It may not be a practical thing to achieve, but it would be worth at least understanding the trade-offs involved.

While this could be done, it would not exactly be elegant, and IMO it

would be

...

the wrong way to address the valid concerns you mention. It would make more sense for pip to *contain* a specific version of distlib for its use (e.g. as a .zip) so that it never worries about conflicts with another copy. This is the approach that Django takes and it seems to work reasonably well for that project.

Thanks for the comments. It looks like pip will indeed contain a copy of distlib, at least for now. And I think you're right, that is the best solution there.

My other interest was with regard to virtualenv. Here, the particular "runtime support" issue that bothers me is the way that setuptools wrapper scripts use entry points. As a result, something like nosetests.exe will not work without setuptools being present, simply because it looks up the code to run using the entry point mechanisms (the actual code itself does not need setuptools). So virtualenv pretty much *has* to preinstall setuptools (or at least pkg_resources), as pip uses exe-wrappers, and those won't use an embedded copy.

But looking at the code generated by distlib's script wrappers, I see that it does not use the exports functionality of distlib, and as a result distlib-generated wrappers can be used without distlib being present. So my apologies here - it looks like my concern was unfounded.

Paul. _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig