[ python-Bugs-1537167 ] 2nd issue with need for speed patch

[SourceForge.net]

Bugs item #1537167, was opened at 2006-08-09 07:01
Message generated for change (Comment added) made by robinbryce2
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1537167&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Interpreter Core
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Robin Bryce (robinbryce2)
Assigned to: Phillip J. Eby (pje)
Summary: 2nd issue with need for speed patch

Initial Comment:
This is not a duplicate of the "realease manager
pronouncement on 302 Fix needed" issue raised on pydev.

If a custom importer is present, import.c skips the
builtin import machinery if the find_module method of
that  importer returns None. For python 2.4.3 if
find_module returns none the normal builtin machinery
gets a lookin. The relevent change was the addition of
a continue statement with svn commit r46372 (at around
line 1283 of import.c on the trunk). 

I don't understand, in the face of this change, how pep
302 importers are expected to cascade. returning None
from find_module is the way an importer says "no I
can't load this module but I cant say for certain this
means ImportError" isnt it ?

One (unintended?) consequence of this change is the
following corner case:

As __import__ allows non dotted module names
__import__('fakemod.a/b') *will* succede on python
2.4.3 provided b is a directory under the package a
that contains an __init__.py. In python 2.5b3 this fails.

I've atatched a detailed repro case of this particular
corner case.




----------------------------------------------------------------------

>Comment By: Robin Bryce (robinbryce2)
Date: 2006-08-11 06:06

Message:
Logged In: YES 
user_id=1547259

>It never was the intention of the entire machinery that such
>a fallback is implemented.

Ah. And having just reread pep 302 the wording on this is
actualy quite specifif. From: "specification part 1: the
importer protocol"

"It should return a loader object if the module was found".
Wich precludes the trickery I was thinking of anyway.

Appologies for the bandwidth this has taken up. I have no
objection to this bug being closed. Thanks for your time.

Cheers,
Robin

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2006-08-10 00:33

Message:
Logged In: YES 
user_id=21627

The patch is originally mine; it does not have to do much
with the need-for-speed sprint. The rationale is to reduce
the number of stat/open calls when loading a module and the
directory doesn't even exist (e.g. for the first sys.path
entry, which is python25.zip). It originally put True/False
on sys.path_importer_cache.

Philipp Eby changed it to put the NullImporter on
path_importer_cache, and not fall back to the builtin import
if the path importer returns None.

It never was the intention of the entire machinery that such
a fallback is implemented. Instead, it always should have
continued with the next sys.path entry instead.

If a path import claims responsibility for a sys.path entry,
and then finds it cannot fulfill the responsibility, and
wants to fall back to the traditional file-based lookup, it
needs to implement that itself. I would advise against doing
so, though, and make the path import reject responsibility
for the sys.path entry in the first place if that entry
really is an on-disk directory.

----------------------------------------------------------------------

Comment By: Robin Bryce (robinbryce2)
Date: 2006-08-09 16:50

Message:
Logged In: YES 
user_id=1547259

I've tried the attatched test case patch against
release24-maint and it passes.

----------------------------------------------------------------------

Comment By: Robin Bryce (robinbryce2)
Date: 2006-08-09 15:33

Message:
Logged In: YES 
user_id=1547259

The 'illeagal' module name is a red herring. The problem
exists  with leagal paths also::

Python 2.5b3 (trunk:51136, Aug  9 2006, 15:17:14)
[GCC 4.0.3 (Ubuntu 4.0.3-1ubuntu5)] on linux2
Type "help", "copyright", "credits" or "license" for more
information.
>>> from fakemod import urlpathimport
>>> urlpathimport.install()
>>> m=__import__('fakemod.a')
*** fullname='fakemod.a' initpath='fakemod' ***
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named a
>>>
robin at rsb2-ubuntu:~/devel/blackmile$ python
Python 2.4.3 (#2, Apr 27 2006, 14:43:58)
[GCC 4.0.3 (Ubuntu 4.0.3-1ubuntu5)] on linux2
Type "help", "copyright", "credits" or "license" for more
information.
>>> from fakemod import urlpathimport
>>> urlpathimport.install()
>>> m=__import__('fakemod.a')
*** fullname='fakemod.a' initpath='fakemod' ***
>>>

Working on a test case. At present I think it is imposible
for a 2.5 custom importer to choose *not* to import a
standard python module by returning None from find_module.
Because if it returns None the standard import is skipped.

gbrandl, I think it was your commit that added the
'continue' statement, what is the reasoning behind making
that optimisation ?

Cheers,
Robin

----------------------------------------------------------------------

Comment By: Georg Brandl (gbrandl)
Date: 2006-08-09 14:28

Message:
Logged In: YES 
user_id=849994

Guido agreed that the 2.4 behavior is to be regarded as a
bug:
http://mail.python.org/pipermail/python-dev/2006-May/065174.html

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2006-08-09 13:27

Message:
Logged In: YES 
user_id=21627

I'd say it is a bug that Python 2.4 allows non-dotted module
names for __import__. Can you come up with a change in
behaviour for "regular" module names?

As for cascading: path importers doe not cascade. Per
sys.path item, there can be at most one path importer. They
"cascade" in the sense that search continues with other
sys.path items if it wasn't found in one sys.path entry.
This cascading continues to work with 2.5.

Phillip, can you take a look?

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1537167&group_id=5470