[Numpy-discussion] numpy distutils log error with easy_install

[Matthew Brett]

Hi,

On Thu, Oct 4, 2012 at 7:35 AM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>
>
> On Thu, Oct 4, 2012 at 12:14 AM, Matthew Brett <matthew.brett at gmail.com>
> wrote:
>>
>> Hi,
>>
>> On Wed, Oct 3, 2012 at 10:51 PM, Ralf Gommers <ralf.gommers at gmail.com>
>> wrote:
>> >
>> >
>> > On Wed, Oct 3, 2012 at 11:29 PM, Matthew Brett <matthew.brett at gmail.com>
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> On Wed, Oct 3, 2012 at 10:17 PM, Ralf Gommers <ralf.gommers at gmail.com>
>> >> wrote:
>> >> >
>> >> >
>> >> > On Mon, Oct 1, 2012 at 10:47 PM, Matthew Brett
>> >> > <matthew.brett at gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> Hi,
>> >> >>
>> >> >> On Mon, Oct 1, 2012 at 9:42 PM, Matthew Brett
>> >> >> <matthew.brett at gmail.com>
>> >> >> wrote:
>> >> >> > Hi,
>> >> >> >
>> >> >> > One of our kind users pointed out an error when using easy_install
>> >> >> > to
>> >> >> > install our package nipy.   I've reproduced it now on a bare
>> >> >> > package
>> >> >> > using numpy distutils and having a trivial extension:
>> >> >> >
>> >> >> > https://github.com/matthew-brett/apkg
>> >> >> >
>> >> >> > To reproduce:
>> >> >> >
>> >> >> > git clone git://github.com/mathew-brett/apkg.git
>> >> >> >
>> >> >> > easy_install apkg
>> >> >> >
>> >> >> > You should get something like this:
>> >> >> >
>> >> >> > Processing apkg
>> >> >> > Running setup.py -q bdist_egg --dist-dir
>> >> >> > /home/mb312/tmp/apkg/egg-dist-tmp-T5yjuB
>> >> >> > Appending apkg configuration to
>> >> >> > Ignoring attempt to set 'name' (from '' to 'apkg')
>> >> >> > zip_safe flag not set; analyzing archive contents...
>> >> >> > Adding apkg 0.1 to easy-install.pth file
>> >> >> >
>> >> >> > Installed
>> >> >> >
>> >> >> >
>> >> >> > /home/mb312/.virtualenvs/np-1.6.2/lib/python2.6/site-packages/apkg-0.1-py2.6-linux-i686.egg
>> >> >> > Processing dependencies for apkg==0.1
>> >> >> > Finished processing dependencies for apkg==0.1
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > /home/mb312/.virtualenvs/np-1.6.2/lib/python2.6/site-packages/numpy/distutils/misc_util.py:252:
>> >> >> > RuntimeWarning: Parent module 'numpy.distutils' not found while
>> >> >> > handling absolute import
>> >> >> >   from numpy.distutils import log
>> >> >> >
>> >> >> > Note the last error.
>> >> >>
>> >> >> Sorry, correcting myself - it's (obviously) a Warning rather than an
>> >> >> error, but still distracting, and it would be good to avoid it if
>> >> >> possible...
>> >> >
>> >> >
>> >> > The combination of two or all of atexit.register, easy_install and
>> >> > virtualenv seems to be causing this. Unless someone feels like
>> >> > digging
>> >> > into
>> >> > that (I certainly don't), there are two easy solutions:
>> >> > 1. Silence the warning.
>> >>
>> >> Sorry - I am not sure what you mean.  The problem here is the user who
>> >> assumes that something bad happened when running easy_install - which
>> >> is what happened in the case of nipy.  Is there some way of silencing
>> >> this (specific) warning from within setup.py?
>> >>
>> >> > 2. Remove the offending import and the logging. This will only remove
>> >> > the
>> >> > line "removing: _configtest.c _configtest.o" from the build log
>> >> > (x20).
>> >>
>> >> Which import did you mean?  I think I need all the imports I'm using
>> >> in the example minimal package.  I'm not explicitly importing logging
>> >> for example.
>> >
>> >
>> > The import that's indicated in the warning, on line 252 of
>> > numpy/distutils/misc_util.py. Relevant code:
>> >
>> > _temporary_directory = None
>> > def clean_up_temporary_directory():
>> >     from numpy.distutils import log    # CAUSES RUNTIME WARNING
>> >     global _temporary_directory
>> >     if not _temporary_directory:
>> >         return
>> >     log.debug('removing %s', _temporary_directory)
>> >     try:
>> >         shutil.rmtree(_temporary_directory)
>> >     except OSError:
>> >         pass
>> >     _temporary_directory = None
>> >
>> > def make_temp_file(suffix='', prefix='', text=True):
>> >     global _temporary_directory
>> >     if not _temporary_directory:
>> >         _temporary_directory = tempfile.mkdtemp()
>> >         atexit.register(clean_up_temporary_directory)
>>
>> Sorry - I still don't understand.  You mean I should (in my package -
>> say nipy or 'apkg') monkey-patch numpy distutils.misc_util ?
>
>
> No, I was proposing to fix this in numpy.distutils.misc_util directly. Sorry
> for not being clear.
>
>>
>>
>> Another option would be to move the import outside the callback function
>> thus:
>
>
> Should work too I guess, but since I don't understand why the import is
> where it is now I'm not 100% sure. Removing the import on line 252 and the
> log.debug call would be the safest fix.

The import is in the function because the following outside the
function (at the top of misc_util):

from numpy.distutils import log

leads to the following on `python setup.py install`:

Running from numpy source directory.
Traceback (most recent call last):
  File "setup.py", line 214, in <module>
    setup_package()
  File "setup.py", line 191, in setup_package
    from numpy.distutils.core import setup
  File "/home/mb312/dev_trees/numpy/numpy/distutils/__init__.py", line
7, in <module>
    import ccompiler
  File "/home/mb312/dev_trees/numpy/numpy/distutils/ccompiler.py",
line 14, in <module>
    from numpy.distutils import log
  File "/home/mb312/dev_trees/numpy/numpy/distutils/log.py", line 9, in <module>
    from misc_util import red_text, default_text, cyan_text,
green_text, is_sequence, is_string
  File "/home/mb312/dev_trees/numpy/numpy/distutils/misc_util.py",
line 21, in <module>
    from numpy.distutils import log
ImportError: cannot import name log

I'm guessing this is a problem of circular imports, because in
numpy.distutils.log we see that numpy distutils is importing
misc_util.

It so happens that:

from numpy.distutils.log import debug as log_debug

works in this situation, I don't understand why.  I suppose also that
it may rely on something complex and / or undocumented in the details
of python's import mechanism, and so might be better avoided.

So, is the 'remove log call in function' reasonable for a patch to
trunk?  For the upcoming release?

Cheers,

Matthew