How to get pip to really, really, I mean it -- rebuild this damn package!
Context: I'm maintaining a number of conda packages of various packages, some of which are mine, some others, some pure python, some extensions, etc. The way conda build works is you specify some meta data, and a build script(s), and conda: sets up an isolated environment in which to build. installs the build dependencies runs teh build script see's what got installed, and makes a package of it. (there are complications, but that's the idea) so what to do in the build script for a python package? the simple anser is: $PYTHON setup.py install But then you get those god- awful eggs, or if it's not a setuptools built package, you don't get the right meta data for pip, etc. to resolve dependencies. [NOTE: I do want all the pip compatible meta data, otherwise, you have pip trying to re-instll stuff, etc if someone does install something with pip, or pip in editable mode, or...] so some of us have started doing: $PYTHON setup.py install --single-version-externally-managed --record record.txt Which mostly seems to work -- though that is a God-awful command line to remember.... And it fails if the package has a plain old distuitls-based setup.py so I started going with: $PYTHON -m pip install ./ and that seemed to work for awhile for me. However, I've been having problems lately with pip not bulding and re-installing the package. This is really weird, as the conda build environment is a clean environment, there really isn't a package already installed. here is the log: + /Users/chris.barker/miniconda2/envs/_build/bin/python -m pip install -v ./ Processing /Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3 Running setup.py (path:/tmp/pip-umxsOD-build/setup.py) egg_info for package from file:///Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3 Running command python setup.py egg_info Source in /tmp/pip-umxsOD-build has version 3.0.3, which satisfies requirement gsw==3.0.3 from file:///Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3 Requirement already satisfied (use --upgrade to upgrade): gsw==3.0.3 from file:///Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3 in /Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3 Requirement already satisfied (use --upgrade to upgrade): numpy in /Users/chris.barker/miniconda2/envs/_build/lib/python2.7/site-packages (from gsw==3.0.3) Requirement already satisfied (use --upgrade to upgrade): nose in /Users/chris.barker/miniconda2/envs/_build/lib/python2.7/site-packages (from gsw==3.0.3) Building wheels for collected packages: gsw Running setup.py bdist_wheel for gsw ... Destination directory: /tmp/tmprPhOYkpip-wheel- Running command /Users/chris.barker/miniconda2/envs/_build/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-umxsOD-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bdist_wheel -d /tmp/tmprPhOYkpip-wheel- --python-tag cp27 done Stored in directory: /Users/chris.barker/Library/Caches/pip/wheels/51/4e/d7/b4cfa75866df9da00f4e4f8a9c5c35cfacfa9e92c4885ec5c4 Removing source in /tmp/pip-umxsOD-build Successfully built gsw Cleaning up... You are using pip version 8.0.1, however version 8.0.2 is available. You should consider upgrading via the 'pip install --upgrade pip' command. So it seems to think it's already installed -- huh? what? IN any case, it doesn't install anything. It looks like it's referencing some cache, or manifest or something outside of the python environment itself. So if I installed it in a different Anaconda environment, it gets confused here. (BTW, I can replicate this behavior outside of conda build by creating a new conda environment by hand, and trying to ues pip to build a package locally) So I tried various command-line options: $PYTHON -m pip install -I -v --upgrade --no-deps ./ but no dice. I also tried --no-cache-dir -- no change. So how can I tell pip that I really do want it to bulid and install this dran package from source, damn it! Other option -- go back to: $PYTHON setup.py install --single-version-externally-managed --record record.txt And have to fight with pip only for the non-setuptools packages. Does the --single-version-externally-managedcommand do ayting different than pip? Thanks, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
The pip wheel cache is in ~/Library/Caches/pip/wheels (OS X) or
~/.cache/pip/wheels (Linux). I'm not sure about Windows. You might have
some luck deleting files from there.
-Robert
On Thu, Jan 28, 2016 at 4:28 PM, Chris Barker
Context:
I'm maintaining a number of conda packages of various packages, some of which are mine, some others, some pure python, some extensions, etc.
The way conda build works is you specify some meta data, and a build script(s), and conda:
sets up an isolated environment in which to build. installs the build dependencies runs teh build script see's what got installed, and makes a package of it. (there are complications, but that's the idea)
so what to do in the build script for a python package? the simple anser is:
$PYTHON setup.py install
But then you get those god- awful eggs, or if it's not a setuptools built package, you don't get the right meta data for pip, etc. to resolve dependencies.
[NOTE: I do want all the pip compatible meta data, otherwise, you have pip trying to re-instll stuff, etc if someone does install something with pip, or pip in editable mode, or...]
so some of us have started doing:
$PYTHON setup.py install --single-version-externally-managed --record record.txt
Which mostly seems to work -- though that is a God-awful command line to remember....
And it fails if the package has a plain old distuitls-based setup.py
so I started going with:
$PYTHON -m pip install ./
and that seemed to work for awhile for me. However, I've been having problems lately with pip not bulding and re-installing the package. This is really weird, as the conda build environment is a clean environment, there really isn't a package already installed.
here is the log:
+ /Users/chris.barker/miniconda2/envs/_build/bin/python -m pip install -v ./
Processing /Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3
Running setup.py (path:/tmp/pip-umxsOD-build/setup.py) egg_info for package from file:///Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3
Running command python setup.py egg_info
Source in /tmp/pip-umxsOD-build has version 3.0.3, which satisfies requirement gsw==3.0.3 from file:///Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3
Requirement already satisfied (use --upgrade to upgrade): gsw==3.0.3 from file:///Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3 in /Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3
Requirement already satisfied (use --upgrade to upgrade): numpy in /Users/chris.barker/miniconda2/envs/_build/lib/python2.7/site-packages (from gsw==3.0.3)
Requirement already satisfied (use --upgrade to upgrade): nose in /Users/chris.barker/miniconda2/envs/_build/lib/python2.7/site-packages (from gsw==3.0.3)
Building wheels for collected packages: gsw
Running setup.py bdist_wheel for gsw ... Destination directory: /tmp/tmprPhOYkpip-wheel-
Running command /Users/chris.barker/miniconda2/envs/_build/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-umxsOD-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bdist_wheel -d /tmp/tmprPhOYkpip-wheel- --python-tag cp27
done
Stored in directory: /Users/chris.barker/Library/Caches/pip/wheels/51/4e/d7/b4cfa75866df9da00f4e4f8a9c5c35cfacfa9e92c4885ec5c4
Removing source in /tmp/pip-umxsOD-build
Successfully built gsw
Cleaning up...
You are using pip version 8.0.1, however version 8.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
So it seems to think it's already installed -- huh? what? IN any case, it doesn't install anything. It looks like it's referencing some cache, or manifest or something outside of the python environment itself. So if I installed it in a different Anaconda environment, it gets confused here.
(BTW, I can replicate this behavior outside of conda build by creating a new conda environment by hand, and trying to ues pip to build a package locally)
So I tried various command-line options:
$PYTHON -m pip install -I -v --upgrade --no-deps ./ but no dice.
I also tried --no-cache-dir -- no change.
So how can I tell pip that I really do want it to bulid and install this dran package from source, damn it!
Other option -- go back to:
$PYTHON setup.py install --single-version-externally-managed --record record.txt And have to fight with pip only for the non-setuptools packages. Does the --single-version-externally-managedcommand do ayting different than pip?
Thanks,
-Chris
--
Christopher Barker, Ph.D. Oceanographer
Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
-- -Robert
On Thu, Jan 28, 2016 at 4:28 PM, Chris Barker
Source in /tmp/pip-umxsOD-build has version 3.0.3, which satisfies requirement gsw==3.0.3 from file:///Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3
Requirement already satisfied (use --upgrade to upgrade): gsw==3.0.3 from file:///Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3 in /Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3
I think this is saying that pip thinks it has found an already-installed version of gsw 3.0.3 in sys.path, and that the directory in your sys.path where it's already installed is /Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3 I think this means that that directory is (a) in sys.path, and (b) contains a .egg-info/.dist-info directory for gsw 3.0.3. Part (a) seems weird and broken. Do you have "." in your PYTHONPATH or anything like that?
Requirement already satisfied (use --upgrade to upgrade): numpy in /Users/chris.barker/miniconda2/envs/_build/lib/python2.7/site-packages (from gsw==3.0.3)
Requirement already satisfied (use --upgrade to upgrade): nose in /Users/chris.barker/miniconda2/envs/_build/lib/python2.7/site-packages (from gsw==3.0.3)
Building wheels for collected packages: gsw
Running setup.py bdist_wheel for gsw ... Destination directory: /tmp/tmprPhOYkpip-wheel-
Running command /Users/chris.barker/miniconda2/envs/_build/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-umxsOD-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bdist_wheel -d /tmp/tmprPhOYkpip-wheel- --python-tag cp27
Don't know why it seems to be building a wheel for it, if it already thinks that it's installed... this is also odd.
(BTW, I can replicate this behavior outside of conda build by creating a new conda environment by hand, and trying to ues pip to build a package locally)
So I tried various command-line options:
$PYTHON -m pip install -I -v --upgrade --no-deps ./
but no dice.
I also tried --no-cache-dir -- no change.
Maybe $PYTHON -m pip install --no-cache-dir --upgrade --force-reinstall ./ ? Though I'd think that -I would have the same affect as --force-reinstall... (It doesn't look like the cache dir is your problem here, but you do probably want to use --no-cache-dir anyway just as good practice, just because you don't want to accidentally package up a stale version of the software that got pulled out of your cache instead of the version you thought you were packaging in the tree in front of you. Also, I think it's a bug in pip that it caches builds of source trees -- PyPI can enforce the rule that each (package name, version number) sdist is unique, but for a work-in-progress VCS checkout it's just not true that (package name, version number) uniquely identifies a snapshot of the whole tree. So in something like 'pip install .', then requirement resolution code should treat this as a special requirement that it wants *this exact tree*, not just any package that has the same (package name, version number) as this tree; and the resulting wheel should not be cached. I don't know if there are any bugs filed in pip on this...) -n -- Nathaniel J. Smith -- https://vorpus.org
Requirement already satisfied (use --upgrade to upgrade): gsw==3.0.3 from file:///Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3 in /Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3
I think this is saying that pip thinks it has found an already-installed version of gsw 3.0.3 in sys.path, and that the directory in your sys.path where it's already installed is
/Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3
That is the temp dir conda sets up to unpack downloaded files, and do its work in -- hence the name. I'll look and see what's there. I'm pretty sure conda build starts out with an empty dir, however. And that dir should not be on sys.path.
I think this means that that directory is (a) in sys.path, and (b) contains a .egg-info/.dist-info directory for gsw 3.0.3. Part (a) seems weird and broken.
Indeed. And I get the same symptoms with a clean environment that I've set up outside conda build. Though with the same source dir. But with conda build, it's a fresh unpack of the tarball.
Do you have "." in your PYTHONPATH or anything like that?
God no!
Don't know why it seems to be building a wheel for it, if it already thinks that it's installed... this is also odd.
Yes it is. But it doesn't install it :-(
$PYTHON -m pip install --no-cache-dir --upgrade --force-reinstall ./
? Though I'd think that -I would have the same affect as --force-reinstall...
So did I, and I think I tried --force-reinstall already, but I will again.
(It doesn't look like the cache dir is your problem here, but you do probably want to use --no-cache-dir anyway just as good practice, just because you don't want to accidentally package up a stale version of the software that got pulled out of your cache instead of the version you thought you were packaging in the tree in front of you.
Exactly. Doesn't seem to make a difference, though.
Also, I think it's a bug in pip that it caches builds of source trees -- PyPI can enforce the rule that each (package name, version number) sdist is unique, but for a work-in-progress VCS checkout it's just not true that (package name, version number) uniquely identifies a snapshot of the whole tree. So in something like 'pip install .', then requirement resolution code should treat this as a special requirement that it wants *this exact tree*, not just any package that has the same (package name, version number) as this tree; and the resulting wheel should not be cached.
Absolutely! In fact, I'll bet that approach is the source of the problem here. If not automagically, there should be a flag, at least. However, what seems to be happening is that pip is looking outside the current Python environment somewhere to see if this package needs to be installed. It may be something that works with virtualenv, but doesn't with conda environments for some reason. I guess on some level pip simply isn't designed to build and install from local source :-( In the end, I'm still confused: does pip install give me anything that: setup.py install single-version-externally-managed Doesn't? Other that support for non-setuptools installs, anyway. CHB
I don't know if there are any bugs filed in pip on this...)
-n
-- Nathaniel J. Smith -- https://vorpus.org
On Jan 29, 2016 7:48 AM, "Chris Barker - NOAA Federal" < chris.barker@noaa.gov> wrote:
Requirement already satisfied (use --upgrade to upgrade): gsw==3.0.3
from
file:///Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3 in /Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3
I think this is saying that pip thinks it has found an already-installed version of gsw 3.0.3 in sys.path, and that the directory in your sys.path where it's already installed is
/Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3
That is the temp dir conda sets up to unpack downloaded files, and do its work in -- hence the name. I'll look and see what's there. I'm pretty sure conda build starts out with an empty dir, however. And that dir should not be on sys.path.
This is the point where I'd probably start adding print statements to my local copy of pip. E.g. grep for that "requirement already satisfied" message and then have it print out sys.path as well. To me this still smells like some simple bug in one of conda, your environment, or pip, so it's worth figuring out...
I think this means that that directory is (a) in sys.path, and (b) contains a .egg-info/.dist-info directory for gsw 3.0.3. Part (a) seems weird and broken.
Indeed. And I get the same symptoms with a clean environment that I've set up outside conda build. Though with the same source dir. But with conda build, it's a fresh unpack of the tarball.
Do you have "." in your PYTHONPATH or anything like that?
God no!
Too bad, it would have been an easy explanation ;-)
Don't know why it seems to be building a wheel for it, if it already thinks that it's installed... this is also odd.
Yes it is. But it doesn't install it :-(
$PYTHON -m pip install --no-cache-dir --upgrade --force-reinstall ./
? Though I'd think that -I would have the same affect as
--force-reinstall...
So did I, and I think I tried --force-reinstall already, but I will again.
(It doesn't look like the cache dir is your problem here, but you do probably want to use --no-cache-dir anyway just as good practice, just because you don't want to accidentally package up a stale version of the software that got pulled out of your cache instead of the version you thought you were packaging in the tree in front of you.
Exactly. Doesn't seem to make a difference, though.
Also, I think it's a bug in pip that it caches builds of source trees -- PyPI can enforce the rule that each (package name, version number) sdist is unique, but for a work-in-progress VCS checkout it's just not true that (package name, version number) uniquely identifies a snapshot of the whole tree. So in something like 'pip install .', then requirement resolution code should treat this as a special requirement that it wants *this exact tree*, not just any package that has the same (package name, version number) as this tree; and the resulting wheel should not be cached.
Absolutely! In fact, I'll bet that approach is the source of the problem here. If not automagically, there should be a flag, at least.
Maybe... it looks like at least @dstufft agrees that the current behavior is a bug: https://github.com/pypa/pip/issues/536 and if this were fixed then it would at least hide your problem. But the underlying bug here is that pip seems to think that a package is installed when it wasn't. Hacking it to ignore whether a package is installed would just be papering over this. -n
On Fri, Jan 29, 2016 at 2:43 PM, Nathaniel Smith
This is the point where I'd probably start adding print statements to my local copy of pip. E.g. grep for that "requirement already satisfied" message and then have it print out sys.path as well. To me this still smells like some simple bug in one of conda, your environment, or pip, so it's worth figuring out...
yeah, I was hoping not to have to debug pip :-) But Robert indicate that this may well be a pipi bug -- I'll try an older version first, then maybe dig into the pip code... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
participants (4)
-
Chris Barker -
Chris Barker - NOAA Federal
-
Nathaniel Smith -
Robert T. McGibbon