[Python-ideas] My objections to implicit package directories

[Guido van Rossum]

On Mon, Mar 12, 2012 at 11:35 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Tue, Mar 13, 2012 at 4:07 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> It still seems to me that moving to a marker *suffix* (rather than a
>> marker file) as PEP 382 proposes brings all the real practical
>> benefits of implicit package directories (i.e. no empty __init__.py
>> files wasting space) and absolutely *none* of the pain (i.e. no
>> backwards compatibility concerns, no ambiguity in the filesystem to
>> module hierarchy mapping, still able to fix direct execution of
>> modules inside packages rather than having to explain forevermore why
>> it doesn't work), but Guido clearly feels otherwise.
>
> I think this paragraph really gets to the heart of what I'm objecting to.
>
> I agree wholeheartedly with the objective of eliminating __init__.py
> files, there's no need to convince me of that.
>
> However, *two* proposals were made to that end:
> PEP 382 kept the explicit marker, simply changing it to a directory
> suffix rather than a separate file. Simple, clean, straightforward,
> minimalist, effective.
> PEP 402 threw away the marker entirely, and then had to patch the
> package finding algorithm with a whole series of complications to
> avoid breaking backwards compatibility with Python 3.2. It also has
> the side effect of eliminating the 1:1 mapping between the filesystem
> and the module hierarchy. Once we lose that, there's no going back.
>
> What I really want out of the new PEP is a clear rationale for why the
> horrible package finding algorithm hacks needed to make the PEP 402
> approach work in a backwards compatible way are to be preferred to the
> explicitly marked PEP 382 approach which *doesn't pose a backwards
> compatibility problem in the first place*.
>
> The other thing to keep in mind is that, if, for whatever reason, we
> decided further down the road that the explicit directory suffix
> solution wasn't good enough, then *we could change our minds* and
> allow implicit package directories after all (just as the formats for
> valid C extension module names have changed over time).
>
> There's no such freedom with implicit package directories - once
> they're in, they're in and we can never introduce a requirement for an
> explicit marker again without breaking working packages.
>
> Is it so bad that I want us to take baby steps here, rather than
> jumping straight to the implicit solution?

I think it comes down to this: I really, really, really hate
directories with a suffix. I'd like to point out that the suffix is
also introducing a backwards incompatibility: everybody will have to
teach their tools, IDEs, and brains about .pyp directories, and they
will also have to *rename* their directories (*if* they want to
benefit from the new feature). Renaming directories is a huge pain --
I counted over 400 directories in Django, so that would mean renaming
over 400 renames. In my experience renaming a directory is a huge pain
no matter which version control system you use -- yes, it can be done,
and modern VCSes have some support for renaming, but it's still a huge
mess. Importing patches will be painful. Producing diffs across the
renames will be hugely painful. I just think there are too many tools
that won't know how to deal with this.

(I just did a little experiment: I cloned a small project using Hg and
renamed one directory. Then I made a small change to one of the files
whose parent was renamed. I have not figured out how to get a diff
between the latest version of that file and any version before the
mass renaming; the renaming is shown as a delete of the entire old
file and an add of the entire new file. Even if you can tell me how to
do this, my point stays: it's not easy to figure out. Similarly for
logs: by default, "hg log" stops at the rename. You must add --follow
to see logs across the rename.)

And regardless of which PEP we adopt, there will still be two types of
package directories: PEP 382 still maintains backwards compatibility
with directories that don't have a suffix but do have an __init__.py.
So the unification still remains elusive.

And at the end of the day I still really, really, really hate
directories with a suffix.

-- 
--Guido van Rossum (python.org/~guido)