[Distutils] Module from install breaks subsequent install of different distribution (was: Python module for use in ‘setup.py’ but not to install)

[Marius Gedminas]

On Mon, Jan 19, 2015 at 07:37:35PM +1100, Ben Finney wrote:
> Ben Finney <ben+python at benfinney.id.au> writes:
> > That has worked for me. But it appears to cause problems for some
> > others, related to this module which should not be installed.
> 
> This discussion was prompted, for me, by this bug report
> <URL:https://alioth.debian.org/tracker/index.php?func=detail&aid=314948&group_id=100328&atid=413098>.
> Can someone please inspect the log output from Pip and help me
> understand why an unrelated package is breaking when installing
> ‘python-daemon’ 2.0.3?

(It would be a bit easier if Alioth didn't require creating a guest
account and logging in just to read a public bug report.)

> The ‘python-daemon’ 2.0.3 distribution declares ‘docutils’ as a
> dependency with ‘setup_requires’ and ‘install_requires’. So it's
> expected that asking Pip to install ‘python-daemon’ will attempt to
> install ‘docutils’.
> 
> But why does the dependency install of ‘docutils’ then fail with::
> 
> =====
> Downloading/unpacking docutils (from python-daemon)
>   […]
>   Downloading from URL https://pypi.python.org/packages/source/d/docutils/docutils-0.12.tar.gz#md5=4622263b62c5c771c03502afa3157768 (from https://pypi.python.org/simple/docutils/)
>   Running setup.py (path:/home/pwj/.virtualenvs/venv/build/docutils/setup.py) egg_info for package docutils
>   […]
> Installing collected packages: python-daemon, lockfile, docutils
>   Running setup.py install for docutils
>     Running command /home/pwj/.virtualenvs/venv/bin/python -c "import setuptools, tokenize;__file__='/home/pwj/.virtualenvs/venv/build/docutils/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-7iBtrf-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/pwj/.virtualenvs/venv/include/site/python2.7
>     […]
>     Running docutils-0.12/setup.py -q bdist_egg --dist-dir /tmp/easy_install-XO8spm/docutils-0.12/egg-dist-tmp-0EsqLD
>     Traceback (most recent call last):
>       […]
>       File "/home/pwj/.virtualenvs/venv/local/lib/python2.7/site-packages/pkg_resources.py", line 2147, in load
>         ['__name__'])
>     ImportError: No module named version

I've reproduced this error at a different stage:

   virtualenv /tmp/sandbox
   /tmp/sandbox/bin/pip install --no-use-wheel python-daemon==2.0.3 -I

This tries to reinstall setuptools and fails while running 'python
setup.py egg_info' in /tmp/pip-build-0DrVHy/setuptools, with the same
ImportError: No module named version inside pkg_resources.py

Very interesting.

If I attempt pip install without the -I, I get a different (but also
very weird error):

    Collecting python-daemon==2.0.3
      Using cached python-daemon-2.0.3.tar.gz
        Traceback (most recent call last):
          File "<string>", line 20, in <module>
          File "/tmp/pip-build-ADeQ8n/python-daemon/setup.py", line 101, in <module>
    ...
          File "/tmp/sandbox/lib/python2.7/site-packages/setuptools/dist.py", line 311, in fetch_build_eggs

    ...
    distutils.errors.DistutilsError: Setup script exited with error: [Errno 2] No such file or directory: u'ChangeLog'

AFAICT it also happens during the setup_requires processing.

I was unable to reproduce either error if I tried

  pip install /path/to/extracted/python-daemon

but perhaps I wasn't careful enough to keep all the other variables the
same?

> =====
> 
> Why on earth would the presence of ‘version.py’ in this distribution
> cause the install of ‘docutils’ to fail? Or is some other correlation
> going on? I have had other reports that subsequent attempts to install
> packages also fail with the same error.

My gut instinct is to say 'setup_requires is horrible, is there any way
to avoid using it'?  Of course that's not a reasonable long-term
strategy.  It'd be good to get to the bottom of this.

I tried to debug the 'ChangeLog' error with 

  virtualenv /tmp/sandbox
  /tmp/sandbox/bin/pip install --no-use-wheel --no-clean python-daemon=2.0.3
  cd /tmp/pip-build-zRzNwu/python-daemon
  strace -e chdir,open,execve -f -o /tmp/crash.log /tmp/sandbox/bin/python setup.py egg_info

and it seems to me that the error happens because setuptools changes the
working directory to /tmp/easy_install-qcgHIl/docutils-0.12, and then
again to /tmp/easy_install-qcgHIl/docutils-0.12/temp/easy_install-5RFCuY/docutils-0.12,
right before it gives control back to python-daemon's code that attempts
to read ChangeLog from the current working directory.

Hmm, I think this bit in your setup.py

        entry_points={
            "distutils.setup_keywords": [
                "release_date = version:validate_distutils_release_date_value",
                ],
            "egg_info.writers": [
                "{filename} = version:generate_egg_info_metadata".format(
                    filename=metadata.version_info_filename),
                ],
            },

causes your code to run while setup_requires is busy installing docutils
(in the same process!), and your code makes assumptions about the
package it's used with ('there's a version.py and a ChangeLog in the
current working directory') that are false when it's used with other
packages, installed via setup_requires.

HTH,
Marius Gedminas
-- 
I think one of the enduring tragedies of the 22nd century will be that during
the 20th and 21st centuries we persistently treat nuclear reactors as if
they're nuclear weapons, and nuclear weapons as if they're nuclear reactors.
        -- Charlie Stoss
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 173 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150119/61679434/attachment-0001.sig>