Mailman 3 can someone explain why I have started to get these pip failures - Distutils-SIG

newer
Re: [Distutils] Outdated packages...

can someone explain why I have started to get these pip failures

Robin Becker

14 Jul 2016 14 Jul '16

9:37 a.m.

(myenv) rptlab@app0:~/myenv $ python Python 2.7.11+ (default, Apr 17 2016, 14:00:29) [GCC 5.3.1 20160413] on linux2 Type "help", "copyright", "credits" or "license" for more information.

...
...
...

...

(myenv) rptlab@app0:~/myenv $ pip --version pip 8.1.2 from /home/rptlab/myenv/local/lib/python2.7/site-packages (python 2.7)

...

(myenv) rptlab@app0:~/myenv pip install -r requirements.txt Collecting MySQL-python<1.3,>=1.2.5 (from -r requirements.txt (line 1)) Downloading MySQL-python-1.2.5.zip (108kB) 100% |################################| 112kB 2.1MB/s Could not import setuptools which is required to install from a source distribution. Traceback (most recent call last): File "/home/rptlab/> (myenv) rptlab@app0:~/myenv/local/lib/python2.7/site-packages/pip/req/req_install.py", line 372, in setup_py import setuptools # noqa File "/home/rptlab/> (myenv) rptlab@app0:~/myenv/local/lib/python2.7/site-packages/setuptools/__init__.py", line 11, in <module> from setuptools.extern.six.moves import filterfalse, map File "/home/rptlab/> (myenv) rptlab@app0:~/myenv/local/lib/python2.7/site-packages/setuptools/extern/__init__.py", line 1, in <module> from pkg_resources.extern import VendorImporter ImportError: No module named pkg_resources.extern (myenv) rptlab@app0:~/myenv $ pip install setuptools -U Requirement already up-to-date: setuptools in ./lib/python2.7/site-packages

This is on a unbuntu 16.04lts system

...

$ uname -a Linux app0 4.4.0-28-generic #47-Ubuntu SMP Fri Jun 24 10:09:13 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

-- Robin Becker

Show replies by date

Ian Cordasco

14 Jul 14 Jul

10:41 a.m.

Try: pip install --force-reinstall setuptools -U On Thu, Jul 14, 2016 at 8:37 AM, Robin Becker wrote:

...

...
(myenv) rptlab@app0:~/myenv $ python Python 2.7.11+ (default, Apr 17 2016, 14:00:29) [GCC 5.3.1 20160413] on linux2 Type "help", "copyright", "credits" or "license" for more information.

...
...
...
...
(myenv) rptlab@app0:~/myenv

$ pip --version pip 8.1.2 from /home/rptlab/myenv/local/lib/python2.7/site-packages (python 2.7)

...
(myenv) rptlab@app0:~/myenv

pip install -r requirements.txt Collecting MySQL-python<1.3,>=1.2.5 (from -r requirements.txt (line 1)) Downloading MySQL-python-1.2.5.zip (108kB) 100% |################################| 112kB 2.1MB/s Could not import setuptools which is required to install from a source distribution. Traceback (most recent call last): File "/home/rptlab/> (myenv) rptlab@app0:~/myenv/local/lib/python2.7/site-packages/pip/req/req_install.py", line 372, in setup_py import setuptools # noqa File "/home/rptlab/> (myenv) rptlab@app0:~/myenv/local/lib/python2.7/site-packages/setuptools/__init__.py", line 11, in <module> from setuptools.extern.six.moves import filterfalse, map File "/home/rptlab/> (myenv) rptlab@app0:~/myenv/local/lib/python2.7/site-packages/setuptools/extern/__init__.py", line 1, in <module> from pkg_resources.extern import VendorImporter ImportError: No module named pkg_resources.extern

...
(myenv) rptlab@app0:~/myenv

$ pip install setuptools -U Requirement already up-to-date: setuptools in ./lib/python2.7/site-packages

This is on a unbuntu 16.04lts system

...
$ uname -a Linux app0 4.4.0-28-generic #47-Ubuntu SMP Fri Jun 24 10:09:13 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

-- Robin Becker _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

Robin Becker

11:01 a.m.

On 14/07/2016 15:41, Ian Cordasco wrote:

...

Try:

pip install --force-reinstall setuptools -U

I didn't do the force-reinstall and for some reason when I cleaned both ~/.cache/pip and ~/.pip the pip install -r requirements.txt did work. I have tried various solutions proposed in the past eg sudo apt-get install --reinstall python-pkg-resources but nothing seems to work. I did the cache cleanups in desperation mode. I did try pip install -U setuptools, but it says it is up to date. If this might be a case of the underlying python changing on the server I have turned off automatic security updates. I would like to try and understand this happens as then I might have some wya of fixing it.

...

On Thu, Jul 14, 2016 at 8:37 AM, Robin Becker wrote:

...
...
(myenv) rptlab@app0:~/myenv ......... Robin Becker -- Robin Becker

Jeremy Stanley

11:14 a.m.

On 2016-07-14 16:01:23 +0100 (+0100), Robin Becker wrote:

...

On 14/07/2016 15:41, Ian Cordasco wrote:

...
Try:

pip install --force-reinstall setuptools -U

I didn't do the force-reinstall and for some reason when I cleaned both ~/.cache/pip and ~/.pip the pip install -r requirements.txt did work.

I have tried various solutions proposed in the past eg

sudo apt-get install --reinstall python-pkg-resources

but nothing seems to work.

I did the cache cleanups in desperation mode.

I did try pip install -U setuptools, but it says it is up to date.

If this might be a case of the underlying python changing on the server I have turned off automatic security updates.

I would like to try and understand this happens as then I might have some wya of fixing it.

You really should avoid mixing pip-installed packages in the system context with distro-provided Python libraries, otherwise you will run into these sorts of issues constantly. I help maintain some very, very large test infrastructure for Python-based tools and libraries: in scenarios where we use pip to install anything system-wide we first make sure to scrub every last distro-provided Python library from the system along with any other Python-based applications that might depend on them, and only then we bootstrap pip completely independent of distro packaging (downloading and running get-pip.py). Also whenever possible, we instead rely on pip install within virtualenvs _without_ --system-site-packages, so that there's no risk of interaction with any Python libraries that might somehow get subsequently installed on the system. -- Jeremy Stanley

Robin Becker

12:23 p.m.

On 14/07/2016 16:14, Jeremy Stanley wrote:

...

On 2016-07-14 16:01:23 +0100 (+0100), Robin Becker wrote:

...
On 14/07/2016 15:41, Ian Cordasco wrote:

...
Try: .............

I would like to try and understand this happens as then I might have some wya of fixing it.

You really should avoid mixing pip-installed packages in the system context with distro-provided Python libraries, otherwise you will run into these sorts of issues constantly. I help maintain some

not really sure why this is an issue. All of my problems are in virtual environments and I never use the --system-site-packages flag. I have never used pip to install anything system wide (so far as I know). Step 0 in after setting up an environment is pip install -U pip setuptools Of course you are right in that the python is a distro provided one; and also the pip and virtualenv etc etc.

...

very, very large test infrastructure for Python-based tools and libraries: in scenarios where we use pip to install anything system-wide we first make sure to scrub every last distro-provided Python library from the system along with any other Python-based applications that might depend on them, and only then we bootstrap pip completely independent of distro packaging (downloading and running get-pip.py). Also whenever possible, we instead rely on pip install within virtualenvs _without_ --system-site-packages, so that there's no risk of interaction with any Python libraries that might somehow get subsequently installed on the system.

I used always to build python from source in older ubuntus, but that was because we wanted the latest python 2.x etc etc. Using a local copy prevents the os from smashing stuff, but means more work whenever a serious upgrade is required. When I run python -mvirtualenv I end up with an environment that has pip and setuptools already. Are you saying I should do /usr/bin/python -mvirtualenv --no-pip --no-setuptools myenv myenv/bin/python get_pip.py and then proceed from there? Or is it foolish to rely on the system python at all? I haven't seen this problem in ubuntu 14.04, but that may be just luck. I certainly notice some new behaviour ie the system pip seems to want to assert --user whereas it used to fail for lack of rights in installing into /usr/lib/python.... -- Robin Becker

Paul Moore

12:39 p.m.

On 14 July 2016 at 17:23, Robin Becker wrote:

...

I used always to build python from source in older ubuntus, but that was because we wanted the latest python 2.x etc etc. Using a local copy prevents the os from smashing stuff, but means more work whenever a serious upgrade is required.

When I run python -mvirtualenv I end up with an environment that has pip and setuptools already. Are you saying I should do

/usr/bin/python -mvirtualenv --no-pip --no-setuptools myenv myenv/bin/python get_pip.py

and then proceed from there? Or is it foolish to rely on the system python at all?

I haven't seen this problem in ubuntu 14.04, but that may be just luck.

I certainly notice some new behaviour ie the system pip seems to want to assert --user whereas it used to fail for lack of rights in installing into /usr/lib/python....

Yes, apparently Ubuntu have patched pip to make --user the default. I've seen some bug reports caused by this patch, so it's possible that it's what is causing you problems here. Unfortunately, as that is an Ubuntu patch you'd need to report it to them. Paul

Daniel Holth

12:45 p.m.

It's from Debian. They had time to break pip, but they don't have time to fix it again. On Thu, Jul 14, 2016 at 12:39 PM Paul Moore wrote:

...

...
I used always to build python from source in older ubuntus, but that was because we wanted the latest python 2.x etc etc. Using a local copy

...
the os from smashing stuff, but means more work whenever a serious upgrade is required.

When I run python -mvirtualenv I end up with an environment that has pip and setuptools already. Are you saying I should do

/usr/bin/python -mvirtualenv --no-pip --no-setuptools myenv myenv/bin/python get_pip.py

and then proceed from there? Or is it foolish to rely on the system

On 14 July 2016 at 17:23, Robin Becker wrote: prevents python

...
at all?

I haven't seen this problem in ubuntu 14.04, but that may be just luck.

I certainly notice some new behaviour ie the system pip seems to want to assert --user whereas it used to fail for lack of rights in installing into /usr/lib/python....

Yes, apparently Ubuntu have patched pip to make --user the default. I've seen some bug reports caused by this patch, so it's possible that it's what is causing you problems here. Unfortunately, as that is an Ubuntu patch you'd need to report it to them.

Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

Jeremy Stanley

1:30 p.m.

On 2016-07-14 17:23:24 +0100 (+0100), Robin Becker wrote:

...

not really sure why this is an issue. All of my problems are in virtual environments and I never use the --system-site-packages flag.

I have never used pip to install anything system wide (so far as I know).

Ahh, whoops, after rereading back through some of your earlier messages in the thread I see this is indeed all in the scope of a virtualenv.

...

Step 0 in after setting up an environment is pip install -U pip setuptools

Of course you are right in that the python is a distro provided one; and also the pip and virtualenv etc etc. [...]

I've noticed in the past that for some reason the system pkg_resources can end up getting used by setuptools (on Debian/Ubuntu it's unvendored and split into a separate python-pkg-resources deb). I haven't had an opportunity to track it down. You might try myenv/bin/python -c 'import sys;print sys.path' and then check the results in order to see which the first one is to provide pkg_resources. Hunting around a bit, this looks similar to https://github.com/pypa/setuptools/issues/497 so see if any of the suggestions mentioned there help at all.

...

When I run python -mvirtualenv I end up with an environment that has pip and setuptools already. Are you saying I should do

/usr/bin/python -mvirtualenv --no-pip --no-setuptools myenv myenv/bin/python get_pip.py [...]

No, but you might try bootstrapping a newer version of virtualenv into its own virtualenv, like: virtualenv foo foo/bin/pip install -U pip setuptools virtualenv foo/bin/virtualenv myenv Doing that on some systems where I'm forced to start from distro-provided pip/setuptools/virtualenv has worked out consistently well. -- Jeremy Stanley

2840

Age (days ago)

2840

Last active (days ago)

List overview

Download

7 comments

5 participants

participants (5)

Daniel Holth
Ian Cordasco
Jeremy Stanley
Paul Moore
Robin Becker

can someone explain why I have started to get these pip failures

tags

participants (5)