On Mon, Mar 29, 2010 at 9:30 AM, anatoly techtonik wrote:

So, there won't be any package management tool shipped with Python 2.7 and users will have to download and install `setuptools` manually as before:

 "search" -> "download" -> "unzip" -> "cmd" -> "cd" -> "python setup.py install"

Therefore I still propose shipping bootstrap package that instruct user how to download and install an actual package  management tool when users tries to use it. So far I know only one stable tool - `easy_install` - a part of `setuptools` package.

There are other tools to install something in a vanilla Python : - pip (which also have an uninstall feature) - distutils - distribute

The required behavior for very basic user friendliness: 1. user installs Python 2.7 2. user issues `python -m easy_install something` 3. user gets message 'easy_install' tool is not installed on this system. To make it available, download and install `setuptools` package from http://pypi.python.org/pypi/setuptools/

Are you thinking about something generic ? Like, being able to call : $ python -m ANYTHING And get an error message saying that the ANYTHING script is not installed ? Then have a registry somewhere to get the name of the project that owns the ANYTHING script ? [..]

So, do we need a PEP for that? How else can I know if consensus is reached? Anybody is willing to elaborate on implementation?

I see two paths here: 1 - do you want to define a general mechanism for Python to install scripts ? 2 - are you just suggesting to have this mechanism only for Package Managers ? In case of 2), what you would need to do is propose an extension to PEP 376, we are currently working on, then we can add it there once people have discussed it on distutils-SIG, or maybe create a new PEP if the topic is too big to fit in 376.

P.S. Please be careful to reply to relevant lists

Yes, so if it's 2) let's keep this thread in Distutils-SIG. Cross-posting like this makes it hard to follow. Tarek -- Tarek Ziadé | http://ziade.org

On Mon, Mar 29, 2010 at 10:55 AM, Tarek Ziadé wrote:

...

Therefore I still propose shipping bootstrap package that instruct user how to download and install an actual package  management tool when users tries to use it. So far I know only one stable tool - `easy_install` - a part of `setuptools` package.

There are other tools to install something in a vanilla Python :

- pip  (which also have an uninstall feature) - distutils - distribute

distutils is not a `package management` tool, because it doesn't know anything even about installed packages, not saying anything about dependencies. `pip` and `distribute` are unknown for a vast majority of Python users, so if you have a perspective replacement for `easy_install` - it can be said in bootstrap package message. There is no problem with packages that require `setuptools` either - they will require `setuptools` as dependency anyway. For now there are two questions: 1. Are they stable enough for the replacement of user command line `easy_install` tool? 2. Which one is the recommended? P.S. Should there be an accessible FAQ in addition to ML?

...

The required behavior for very basic user friendliness: 1. user installs Python 2.7 2. user issues `python -m easy_install something` 3. user gets message 'easy_install' tool is not installed on this system. To make it available, download and install `setuptools` package from http://pypi.python.org/pypi/setuptools/

Are you thinking about something generic ?

Like, being able to call :

$ python -m ANYTHING

And get an error message saying that the ANYTHING script is not installed ? Then have a registry somewhere to get the name of the project that owns the ANYTHING script ?

No. To get something in 2.7 I would refrain from developing into this direction for now.

[..]

...

So, do we need a PEP for that? How else can I know if consensus is reached? Anybody is willing to elaborate on implementation?

I see two paths here: 1 - do you want to define a general mechanism for Python to install scripts ? 2 - are you just suggesting to have this mechanism only for Package Managers ?

3 - I want current "best practice" for installing Python modules with dependencies from PyPI to be shipped with Python 2.7 by default Answering your questions: 1. I want to have some user-friendly way for installing Python scripts, but I am more concerned that I will miss `easy_install` in Python 2.7 2. Bootstrap script is aimed at users. Package managers are assumed to already have their package at PyPI and provide install instructions that involve `easy_install`, so nothing is required to be done on their part.

In case of 2), what you would need to do is propose an extension to PEP 376, we are currently working on, then we can add it there once people have discussed it on distutils-SIG, or maybe create a new PEP if the topic is too big to fit in 376.

PEP 376 is completely irrelevant to user side boot package proposal. This proposal is to recommend whatever package managing tool you think user need and instruct how to get the tool. The boot package doesn't know anything at all about how packages are managed.

...

P.S. Please be careful to reply to relevant lists

Yes, so if it's 2) let's keep this thread in Distutils-SIG. Cross-posting like this makes it hard to follow.

My vision is that decision about having bootstrap package or not in 2.7 should be in python-dev and specific packaging, implementation and pip/distutils/distribute questions in distutils-sig. -- anatoly t.

On Mon, Mar 29, 2010 at 11:15 AM, Tarek Ziadé wrote: [..]

Depending on how you call a Python user, I disagree here. Many people use pip and distribute.

s/how/who :)

2010/3/29 anatoly techtonik :

2010/3/29 Lennart Regebro :

...

On Mon, Mar 29, 2010 at 09:30, anatoly techtonik wrote:

...

Therefore I still propose shipping bootstrap package that instruct user how to download and install an actual package  management tool when users tries to use it. So far I know only one stable tool - `easy_install` - a part of `setuptools` package.

We can't ship *A* bootstrap script until there is *A* package management tool in the Python world. Currently there are three. :)

In short: The tools that exist are not ready yet.

Ok. How about shipping bootstrap script only for `easy_install` tool for now?

-1 I don't see how this will improve the current state. It will make it worse since the future of packaging for the stldib is being built in distutils2, and also in Pip. People that want to use easy_install can install it. It's very simple and doesn't require more work than what you have described through a -m call. The instructions you want to add in that bootstrap script to inform people belong to the documentation imho. And that's one of the goal of the packaging guide we are building. Another suggestion could be to add in the Python installer a notice about the guide, saying : "Hey, we are working on packaging right now. Python currently doesn't ship with a full-featured package manager, you can have a look in this guide though, for guidance" I still think you should remove python-dev from the cc list ;) Tarek -- Tarek Ziadé | http://ziade.org

anatoly techtonik wrote:

So, there won't be any package management tool shipped with Python 2.7 and users will have to download and install `setuptools` manually as before:

Until the discussed package management tools support a robust inventory and uninstallation system that plays well with directly installed Python packages, you won't find widespread support on python-dev for endorsing any of them. Yes, the people who use them love them for good and valid reasons, but those who absolutely detest them also do so for good and valid reasons. While this is still the case, it would be highly inappropriate for python-dev to include bootstrap scripts that direct users to these systems, as they're all flawed in their current incarnations (this isn't the packaging systems' fault - the flaws are largely due to a lack of supporting infrastructure in the standard library). The distutils2 work and the various metadata PEPs that have been approved recently are all about addressing those limitations in the infrastructure. With those in place and flowing through the Python package management ecosystem, bootstrapping interoperable package management tools is likely to become a reasonable option in the future. But we aren't there yet, and won't be for 2.7 or 3.2. From an outsider's perspective, the 3.3 time frame appears to be very possible though. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

anatoly techtonik wrote:

So, there won't be any package management tool shipped with Python 2.7 and users will have to download and install `setuptools` manually as before:

"search" -> "download" -> "unzip" -> "cmd" -> "cd" -> "python setup.py install"

Therefore I still propose shipping bootstrap package that instruct user how to download and install an actual package management tool when users tries to use it. So far I know only one stable tool - `easy_install` - a part of `setuptools` package.

The required behavior for very basic user friendliness: 1. user installs Python 2.7 2. user issues `python -m easy_install something` 3. user gets message 'easy_install' tool is not installed on this system. To make it available, download and install `setuptools` package from http://pypi.python.org/pypi/setuptools/

4. the screen is paused before exit (for windows systems)

Other design notes: 1. if package tries to import `easy_install` module used for bootstrap, it gets the same ImportException as if there were no `easy_install` at all 2. bootstrap module is overwritten by actual package when users installs it

So, do we need a PEP for that? How else can I know if consensus is reached? Anybody is willing to elaborate on implementation?

P.S. Please be careful to reply to relevant lists

An even lighter option would be to add an item to pythons 'help' feature. Currently help("PACKAGES") == help("import") It may be enough at this time to add a "PACKAGES" help entry that gives an overview of packages and hints on installing them. Then "import" can be a related help topic for "PACKAGES". Ron

On Mon, Mar 29, 2010 at 10:45 PM, anatoly techtonik wrote:

On Mon, Mar 29, 2010 at 12:15 PM, Tarek Ziadé wrote:

...

[..]

...

distutils is not a `package management` tool, because it doesn't know anything even about installed packages, not saying anything about dependencies.

At this point, no one knows anything about installed packages at the Python level.

Users do not care about this, and `distutils` doesn't know this even at package level.

...

Keeping track of installed projects is a feature done within each package managment system.

And the whole purpose of PEP 376 is to define a database of what's installed, for the sake of interoperability.

That's great. When it will be ready everybody would be happy to make their package management tool compliant.

...

...

`pip` and `distribute` are unknown for a vast majority of Python users, so if you have a perspective replacement for `easy_install` -

Depending on how you call a Python user, I disagree here. Many people use pip and distribute.

The first one because it has an uninstall feature among other things. The second one because it fixes some bugs of setuptools and provides Python 3 support

I do not mind if we can distribute three stubs, they will also serve as pointers for a better way of packaging when an ultimate tool is finally born. Me personally is willing to elaborate for `easy_install` stub in 2.7.

...

...

For now there are two questions: 1. Are they stable enough for the replacement of user command line `easy_install` tool? 2. Which one is the recommended?

P.S. Should there be an accessible FAQ in addition to ML?

This FAQ work has been started in th "HitchHicker's guide to Packaging" you can find here:

http://guide.python-distribute.org

I can see any FAQ. To me the FAQ is something that could be posted to distutils ML once a month to reflect current state of packaging. It should also carry version number. So anybody can comment on the FAQ, ask another question or ask to make a change.

...

Again, any new code work will not happen because 2.7 is due in less than a week. Things are happening in Distutils2.

That doesn't solve the problem. Bootstrap script can be written in one day. What we need is a consensus whatever this script is welcomed in 2.7 or not? Who is the person to make the decision?

...

Now, for the "best practice" documentation, I think the guide is the best plce to look at.

Let's refer to original user story: "I installed Python and need a quick way to install my packages on top of it."

python setup.py install works well, and has for almost a decade. If you need setuptools, you can include ez_setup.py, which does exactly what you want, without adding a hugely controversial feature to python proper. You do something like: try: import setuptools except ImportError: print "Run ez_setup.py first" .... And you're done, cheers, David

On Mon, Mar 29, 2010 at 1:42 PM, Paul Moore wrote:

...

I still think you should remove python-dev from the cc list  ;)

I don't know what happened with the change of subject and crossposting, but I'm now seeing 2 copies of every mail, and both copies are appearing in both python-dev and distutils-sig (probably, some of that is gmail's fault, but other crosspostings don't make things this bad...)

The same here. This time the thread was hijacked by software. That's why I prefer reading Google Groups from the web with optional subscriptions.

Given that nothing will happen in Python 2.7, I agree that python-dev should be removed.

Given that easy_install is NOT the "one, true answer" that Anatoly seems to think it is, I'm dubious that anything can or should be done until distutils2 is published, so I'd suggest that the whole subject is dropped until distutils2 is available.

It is NOT about UPPMS (Universal Python Package Management Solution) - it is about small bootstrap Script that Guides User how to install the package management system the user is trying to execute. -- anatoly t.

2010/3/29 Nick Coghlan :

anatoly techtonik wrote:

...

So, there won't be any package management tool shipped with Python 2.7 and users will have to download and install `setuptools` manually as before:

Until the discussed package management tools support a robust inventory and uninstallation system that plays well with directly installed Python packages, you won't find widespread support on python-dev for endorsing any of them.

This wasn't the question, but the summary. I would greatly appreciate if you could provide your feedback on the second part of my letter that started with "Therefore..."

Yes, the people who use them love them for good and valid reasons, but those who absolutely detest them also do so for good and valid reasons. While this is still the case, it would be highly inappropriate for python-dev to include bootstrap scripts that direct users to these

Scripts do not _direct_ users - they _help_ users, already directed by some installation instruction or tutorial, to find and install package management system they are _trying_ to use.

The distutils2 work and the various metadata PEPs that have been approved recently are all about addressing those limitations in the infrastructure. With those in place and flowing through the Python package management ecosystem, bootstrapping interoperable package management tools is likely to become a reasonable option in the future.

Bootstrap tools evolve together with packaging situation. You may deprecate package management tool in your future bootstrap scripts if it is "incompatible" with this Python release. It is just user message that is a flexible as the mind of its author.

But we aren't there yet, and won't be for 2.7 or 3.2. From an outsider's perspective, the 3.3 time frame appears to be very possible though.

Just a thought about user story my customers would likely write if I shipped Python as a product: "As a user, I think Python is suxx, because it makes its users suffer for a long time from packaging disorder". -- anatoly t.

On Mon, Mar 29, 2010 at 5:18 PM, Tarek Ziadé wrote:

...

...

Again, any new code work will not happen because 2.7 is due in less than a week. Things are happening in Distutils2.

That doesn't solve the problem. Bootstrap script can be written in one day. What we need is a consensus whatever this script is welcomed in 2.7 or not? Who is the person to make the decision?

Guido makes all final decisions.

But before he does, in general, we work all together to find a consensus, because he can't take part in all discussions. In this process, I am the guy in charge for packaging matters in the stdlib, and I am trying to write the PEPs that reflects those consensus. But anyone is welcome to champion a PEP as long as we discuss it a bit here, and we agree that a new PEP is required.

Feel free to make more points here in distutils-SIG about your idea, to explain the advantages of doing it and gain traction.

So far I haven't seen any counter arguments against _bootstrap script_ itself. I see that the difference between "package management tool" and "bootstrap script" should be defined more clearly. But I do not know how.

But there are no chance at all this will be accepted for 2.7. It's a long process (not writing a script, but deciding if it's the best way to go)

Ok. It seemed so simple, so that I do not even know if I able to propose anything at all. It takes an enormous amount of time and effort to keep discussion focused.

...

...

Now, for the "best practice" documentation, I think the guide is the best plce to look at.

Let's refer to original user story: "I installed Python and need a quick way to install my packages on top of it." "I execute `easy_install something` as said in installation manual, but nothing/error happens."

Have you read : http://guide.python-distribute.org/installation.html ? How could we change it to help this user story ?

1. Provide a less "boring" version for users who are not interested in details of package management history 2. It doesn't describe what command on windows should I use (and I am on windows) to "wget" distribute_setup.py 3. Too complicated for a user To completely get the picture - imagine a typical student who learns English as a second language. If the person has weak language skill - he will skip directly to examples, believe me. Even I do this. And your examples are not crossplatform, and this complicates things even more. "packaging ecosystem manipulation", "combination of tool used for ecosystem isolation and two other tools", argumentation that "Distribute was created because the Setuptools package is no longer maintained". It is all aimed at Python developers, probably with bachelor's degree. I speak on behalf of user that doesn't know anything about Python besides that it should be installed along with some API package to turn on support for youtube video sharing.

...

The scope of my proposal is a bootstrap script that instructs user how to install the package management tool of user's choice if this tool is not yet installed on user system (this is the case with new Python installation).  If you'll come up with a better way of package management - you will update this bootstrap script with relevant information in future Python releases.

Sounds like a simple installation instruction, I can't see the benefit of adding a script for that.

Of course! Because you are a developer. You need to be a user or at least QA engineer to see it. =) For now in its simplest minimal form that is possible to create in a day for 2.7 it is an simple text instruction. This will surely can be ready for 2.7. But in future distributions, i.e. in Python 2.7 non-beta, it may be able to download required package management tool automatically if user agrees, unpack it to %TEMP% and execute "setup.py install".

Plus, we can't make a choice for Python 2.7. Some people use easy_install, some people use pip, some use another tool. You use easy_install, fine, install it.

Some people just blindly follow the instructions and they are disappointed if: 1. it doesn't work 2. if the thing need to be done quickly and instruction is too long 3. if instruction is too long and it still doesn't work

Once PEP 376 is finished and accepted, we might provide a basic script in the stdlib though.

I can see immediate benefit in this script now, but I can not predict if it will be actual 5 years later. =)

[..]

...

If there will be no bootstrap script in 2.7, I won't have any motivation to continue discussion until 2.8 deadline is near. Of course I will be disappointed, because I will have to explain everything to 2.7 users again and again instead of letting them execute one command and follow its instructions.

Frankly, I am not understanding why you are pushing so hard to include this script one week before 2.7b1 without any previous discussion.  I am sorry if you feel disappointed about this.

I can't comment anything about it. It is too subjective, like no time, no Python 2.7 installation until alpha 4 (because I want to see yet another trivial patch with hijacked discussion included there), not being able to get back to problem, because installing `setuptools` for me as a developer is faster than describing user problems. I could also add that mailing lists are archaic to my taste, Python lists are too flooded, it is hard to follow up on threads, that there are always much more ideas that I able to type at any given moment. -- anatoly t.

On Tue, Mar 30, 2010 at 12:32 AM, anatoly techtonik wrote:

On Mon, Mar 29, 2010 at 5:18 PM, Tarek Ziadé wrote:

...

...

...

Again, any new code work will not happen because 2.7 is due in less than a week. Things are happening in Distutils2.

That doesn't solve the problem. Bootstrap script can be written in one day. What we need is a consensus whatever this script is welcomed in 2.7 or not? Who is the person to make the decision?

Guido makes all final decisions.

But before he does, in general, we work all together to find a consensus, because he can't take part in all discussions. In this process, I am the guy in charge for packaging matters in the stdlib, and I am trying to write the PEPs that reflects those consensus. But anyone is welcome to champion a PEP as long as we discuss it a bit here, and we agree that a new PEP is required.

Feel free to make more points here in distutils-SIG about your idea, to explain the advantages of doing it and gain traction.

So far I haven't seen any counter arguments against _bootstrap script_ itself.

You certainly have not provided any argument *for* one. Again, what's the point compared to adding the bootstrap script ez_setup.py in your package. I also find the easiness argument rather dubious: especially on windows, the thing to give a user is an installer (.exe/.msi), not sources. That will solve more problems than any buggy bootstrap script requiring a working internet connection will ever do, cheers, David

anatoly techtonik wrote:

So, there won't be any package management tool shipped with Python 2.7 and users will have to download and install `setuptools` manually as before:

"search" -> "download" -> "unzip" -> "cmd" -> "cd" -> "python setup.py install"

Therefore I still propose shipping bootstrap package that instruct user how to download and install an actual package management tool when users tries to use it.

For what it's worth, Guido prototyped something similar in March of 2008, but his was an actual bootstrapping tool for package management: http://mail.python.org/pipermail/python-dev/2008-March/077837.html His tool knew how to download a tar file, untar it, and run "python setup.py install" on it. No version numbers, no dependency management, simple enough that it should be easy to get right. Only appropriate for bootstrapping into a real package management tool. The thread ends with him saying "I don't have time to deal with this further this week", and I dunno, maybe it just fell off the radar? I'd been thinking about resurrecting the discussion but I didn't have time either. I fear it is too late for 2.7, //larry//

anatoly techtonik wrote:

So, there won't be any package management tool shipped with Python 2.7 and users will have to download and install `setuptools` manually as before: [...] So, do we need a PEP for that? How else can I know if consensus is reached? Anybody is willing to elaborate on implementation?

Notice that the first beta release of Python 2.7 will be made on April 3. Any new feature that you propose must be committed by then. I suggest focusing on Python 3.2 instead. Regards, Martin

anatoly techtonik wrote:

Let's refer to original user story: ... "I execute `easy_install something` as said in installation manual, but nothing/error happens."

Which installation manual did the user see that in? It can't be anything that came with Python, because easy_install is not a standard part of Python. If a package is telling the user to install it with a non-standard installation tool, without explaining this fact and how to get the tool, then the package's documentation is lacking. -- Greg

On Tue, Mar 30, 2010 at 12:45 AM, Ian Bicking wrote:

On Mon, Mar 29, 2010 at 11:26 AM, Larry Hastings wrote:

...

anatoly techtonik wrote:

...

So, there won't be any package management tool shipped with Python 2.7 and users will have to download and install `setuptools` manually as before:

 "search" -> "download" -> "unzip" -> "cmd" -> "cd" -> "python setup.py install"

Therefore I still propose shipping bootstrap package that instruct user how to download and install an actual package  management tool when users tries to use it.

For what it's worth, Guido prototyped something similar in March of 2008, but his was an actual bootstrapping tool for package management:

  http://mail.python.org/pipermail/python-dev/2008-March/077837.html

His tool knew how to download a tar file, untar it, and run "python setup.py install" on it.  No version numbers, no dependency management, simple enough that it should be easy to get right.  Only appropriate for bootstrapping into a real package management tool.

The thread ends with him saying "I don't have time to deal with this further this week", and I dunno, maybe it just fell off the radar?  I'd been thinking about resurrecting the discussion but I didn't have time either.

I would consider this bootstrap to be quite workable, though I would add that any extra option to the bootstrap script should be passed to setup.py install, and the download should be cached (so you can do -h and not have to re-download the package once you figure out the extra options -- at least a --user option is reasonable here for people without root).  Specifically targeting this bootstrap for tools like pip and virtualenv is no problem.

I think looking around PyPI etc is kind of more than I'd bother with.  Those things change, this bootstrap code won't change, it could cause unnecessary future pain.  Maybe (*maybe*) it could look in http://pypi.python.org/well-known-packages/PACKAGE_NAME and so we can have it install a certain small number of things quickly that way -- if the URL it looks to is targeted only for the bootstrap script itself then we don't have to worry about compatibility problems as much.

Oh... then i can think of a half dozen other options it could take, and then it becomes an installer.  Blech.  OK, I'd be willing to cut off the options at --user (which I think is a minimum... maybe --prefix too), and maybe some simple package detection so people could write "python -m boostrap Setuptools --user" -- entirely based on some well-known URL baked into bootstrap.py, where the URL is independent of any other service (and so is least likely to cause future problems or ambiguities).

An advantage to this kind of bootstrapper is that as future packaging systems are developed there's a clear way to get started with them, without prematurely baking anything in to Python.

The bootstrap should only be a temporary package that will propose user to automatically install package management tool user is trying to use if he doesn't have any. It is not Ubuntu-like "missing package autodetection" - only few most popular non-conflicting packaging solutions should be bootstrapped in this way:

python -m easy_install review `easy_install` is not installed on this system. Do you want to fetch it to proceed (Y/n)?

Think of it as of one used oriented "1-Click" technology. -- anatoly t.