[Import-SIG] My objections to implicit package directories

[Guido van Rossum]

On Thu, Mar 15, 2012 at 4:26 AM, Calvin Spealman <ironfroggy at gmail.com> wrote:
>
> On Mar 14, 2012 11:56 PM, "Guido van Rossum" <guido at python.org> wrote:
>>
>> On Wed, Mar 14, 2012 at 8:33 PM, PJ Eby <pje at telecommunity.com> wrote:
>> > On Mon, Mar 12, 2012 at 11:49 PM, Guido van Rossum <guido at python.org>
>> > wrote:
>> >>
>> >> So the only backwards incompatibility is that "import foo" may succeed
>> >> where it previously failed if there is a directory foo/ somewhere on
>> >> sys.path but no foo.py and no foo/__init__.py anywhere. I don't think
>> >> this is a big deal.
>> >
>> >
>> > Actually, it *is* a big deal.  An early draft of PEP 402 was like the
>> > newly-proposed approach, and it was shot down by an *actual code sample
>> > from
>> > a real package* that somebody posted to show the problem.  The
>> > motivating
>> > example is actually presented in the PEP: trying to "import json" with a
>> > json/ directory present -- and the "json" package *not* present -- and
>> > then
>> > trying to use its contents.  The PEP works around this by taking the
>> > namespace concept to its logical extreme and saying you can only import
>> > the
>> > *contents* of the namespace, because the namespace itself doesn't exist.
>> >  (i.e., it's virtual.)
>> >
>> > tl;dr version: the killer problem with allowing "import foo" to succeed
>> > is
>> > that code which does try/except ImportError to test for a package's
>> > presence
>> > will get false positives.  That's the motivating reason for only
>> > allowing
>> > sub-namespace package imports to succeed: it prevents the immediate
>> > breakage
>> > of real code, the moment the feature is introduced.  :-(
>>
>> Well, too bad. It's too much of a wart. Such try/except clauses make
>> me sad anyway -- and we're trying to get rid of them by e.g.
>> encouraging the pattern where heapq.py tries to import _heapq rather
>> than having user code try first _heapq and then heapq. (And the idiom
>> used in heapq.py doesn't mind if _heapq exists as an empty package.)
>
> While I agree it is a wart, the heapq.py example might not be the best. A
> more real-world example might be the situation is PEP 417, where we'll be
> seeing people write
>
> try:
>     import unittest.mock
> except ImportError:
>     import mock
>
> And owning neither the two modules, the _ trick is not available.
>
> I like the idea that if a package is only an implied namespace package, with
> no contents of it's own, it cannot itself be directly imported. This still
> allows the simple check, but without seriously modifying the new feature
> itself.

How would you implement that anyway? The import logic always tries to
import the parent module before importing the child module. So the
import attempt for "foo" has no idea whether it is imported as *part*
of "import foo.bar", or as plain "import foo", or perhaps as part of
"from foo import bar".

It would also be odd to find that

  import foo
  import foo.bar

would fail, whereas

  import foo.bar
  import foo

would succeed, because as a side effect of "import foo.bar", a module
object for foo is created and inserted as sys.modules['foo'].

I really don't want to have to maintain and explain such complexity --
nor the complexity that follows from attempts to "fix" this objection.

Finally, in your example, why on earth would unittest/mock/ exist as
an empty directory???

-- 
--Guido van Rossum (python.org/~guido)