Mailman 3 Dropping __init__.py requirement for subpackages - Python-Dev

Dropping init.py requirement for subpackages

Guido van Rossum

26 Apr 2006 26 Apr '06

11:16 p.m.

(Context: There's a large crowd with pitchforks and other sharp pointy farm implements just outside the door of my office at Google. They are making an unbelievable racket. It appears they are Google engineers who have been bitten by a misfeature of Python, and they won't let me go home before I have posted this message.) The requirement that a directlry must contain an __init__.py file before it is considered a valid package has always been controversial. It's designed to prevent the existence of a directory with a common name like "time" or "string" from preventing fundamental imports to work. But the feature often trips newbie Python programmers (of which there are a lot at Google, at our current growth rate we're probably training more new Python programmers than any other organization worldwide :-). One particular egregious problem is that *subpackage* are subject to the same rule. It so happens that there is essentially only one top-level package in the Google code base, and it already has an __init__.py file. But developers create new subpackages at a frightening rate, and forgetting to do "touch __init__.py" has caused many hours of lost work, not to mention injuries due to heads banging against walls. So I have a very simple proposal: keep the __init__.py requirement for top-level pacakages, but drop it for subpackages. This should be a small change. I'm hesitant to propose *anything* new for Python 2.5, so I'm proposing it for 2.6; if Neal and Anthony think this would be okay to add to 2.5, they can do so. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Show replies by thread

André Malo

26 Apr 26 Apr

11:25 p.m.

* Guido van Rossum wrote:

...

So I have a very simple proposal: keep the __init__.py requirement for top-level pacakages, but drop it for subpackages. This should be a small change. I'm hesitant to propose *anything* new for Python 2.5, so I'm proposing it for 2.6; if Neal and Anthony think this would be okay to add to 2.5, they can do so.

Not that it would count in any way, but I'd prefer to keep it. How would I mark a subdirectory as "not-a-package" otherwise? echo "raise ImportError" >__init__.py ? nd -- Das, was ich nicht kenne, spielt stückzahlmäßig *keine* Rolle. -- Helmut Schellong in dclc

Benji York

11:33 p.m.

Guido van Rossum wrote:

...

So I have a very simple proposal: keep the __init__.py requirement for top-level pacakages, but drop it for subpackages.

So this would mean that current non-package subdirectories in a package (that contain things like data files or configuration info) would become packages with no modules in them? sharpening-my-farm-implements-ly y'rs, -- Benji York

Guido van Rossum

11:44 p.m.

On 4/26/06, André Malo wrote:

...

* Guido van Rossum wrote:

...
So I have a very simple proposal: keep the __init__.py requirement for top-level pacakages, but drop it for subpackages. This should be a small change. I'm hesitant to propose *anything* new for Python 2.5, so I'm proposing it for 2.6; if Neal and Anthony think this would be okay to add to 2.5, they can do so.

Not that it would count in any way, but I'd prefer to keep it. How would I mark a subdirectory as "not-a-package" otherwise?

What's the use case for that? Have you run into this requirement? And even if you did, was there a requirement that the subdirectory's name be the same as a standard library module? If the subdirectory's name is not constrained, the easiest way to mark it as a non-package is to put a hyphen or dot in its name; if you can't do that, at least name it something that you don't need to import. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum

11:46 p.m.

On 4/26/06, Benji York wrote:

...

Guido van Rossum wrote:

...
So I have a very simple proposal: keep the __init__.py requirement for top-level pacakages, but drop it for subpackages.

So this would mean that current non-package subdirectories in a package (that contain things like data files or configuration info) would become packages with no modules in them?

Yup. Of course unless you try to import from them that wouldn't particularly hurt, except if the subdir name happens to be the same as a module name. Note that absolute import (which will be turned on for all in 2.6) will solve the ambiguity; the only ambiguity left would be if you had a module foo.py and also a non-package subdirectory foo. But that's just asking for trouble. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Barry Warsaw

11:50 p.m.

On Wed, 2006-04-26 at 10:16 -0700, Guido van Rossum wrote:

...

So I have a very simple proposal: keep the __init__.py requirement for top-level pacakages, but drop it for subpackages. This should be a small change. I'm hesitant to propose *anything* new for Python 2.5, so I'm proposing it for 2.6; if Neal and Anthony think this would be okay to add to 2.5, they can do so.

But if it's there, then nothing changes, right? IOW, if we want to expose names in the subpackage's namespace, we can still do it in the subpackage's __init__.py. It's just that otherwise empty subpackage __init__.py files won't be required. What would the following print? import toplevel.sub1.sub2 print toplevel.sub1.sub2.__file__ If it's "<path>/sub1/sub2" then that kind of breaks a common idiom of using os.path.dirname() on a module's __file__ to find co-located resources. Or at least, you have to know whether to dirname its __file__ or not (which might not be too bad, since you'd probably know how that package dir is organized anyway). I dunno. Occasionally it trips me up too, but it's such an obvious and easy fix that it's never bothered me enough to care. I can't think of an example, but I suppose it's still possible that lifting this requirement could cause some in-package located data directories to be mistakenly importable. I'd be somewhat more worried about frameworks that dynamically import things having to be more cautious about cleansing their __import__() arguments now. I'd be -1 but the remote possibility of you being burned at the stake by your fellow Googlers makes me -0 :). -Barry

Guido van Rossum

11:59 p.m.

On 4/26/06, Barry Warsaw wrote:

...

On Wed, 2006-04-26 at 10:16 -0700, Guido van Rossum wrote:

...
So I have a very simple proposal: keep the __init__.py requirement for top-level pacakages, but drop it for subpackages. This should be a small change. I'm hesitant to propose *anything* new for Python 2.5, so I'm proposing it for 2.6; if Neal and Anthony think this would be okay to add to 2.5, they can do so.

But if it's there, then nothing changes, right? IOW, if we want to expose names in the subpackage's namespace, we can still do it in the subpackage's __init__.py. It's just that otherwise empty subpackage __init__.py files won't be required.

Correct. This is an important clarification.

...

What would the following print?

import toplevel.sub1.sub2 print toplevel.sub1.sub2.__file__

If it's "<path>/sub1/sub2" then that kind of breaks a common idiom of using os.path.dirname() on a module's __file__ to find co-located resources. Or at least, you have to know whether to dirname its __file__ or not (which might not be too bad, since you'd probably know how that package dir is organized anyway).

Oh, cool gray area. I propose that if there's no __init__.py it prints '<path>/sub1/sun2/' i.e. with a trailing slash; that causes dirname to just strip the '/'. (It would be a backslash on Windows of course).

...

I dunno. Occasionally it trips me up too, but it's such an obvious and easy fix that it's never bothered me enough to care.

But you're not a newbie. for a newbie who's just learned about packages, is familiar with Java, and missed one line in the docs, it's an easy mistake to make and a tough one to debug.

...

I can't think of an example, but I suppose it's still possible that lifting this requirement could cause some in-package located data directories to be mistakenly importable. I'd be somewhat more worried about frameworks that dynamically import things having to be more cautious about cleansing their __import__() arguments now.

But (assuming 2.6 and absolute import) what would be the danger of importing such a package? Presumably it contains no *.py or *.so files so there's no code there; but even so you'd have to go out of your way to import it (since if the directory exists it can't also be a subpackage or submodule name that's in actual use).

...

I'd be -1 but the remote possibility of you being burned at the stake by your fellow Googlers makes me -0 :).

I'm not sure I understand what your worry is. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

André Malo

27 Apr 27 Apr

12:04 a.m.

* Guido van Rossum wrote:

...

On 4/26/06, André Malo wrote:

...
* Guido van Rossum wrote:

...
So I have a very simple proposal: keep the __init__.py requirement for top-level pacakages, but drop it for subpackages. This should be a small change. I'm hesitant to propose *anything* new for Python 2.5, so I'm proposing it for 2.6; if Neal and Anthony think this would be okay to add to 2.5, they can do so.

Not that it would count in any way, but I'd prefer to keep it. How would I mark a subdirectory as "not-a-package" otherwise?

What's the use case for that? Have you run into this requirement? And even if you did, was there a requirement that the subdirectory's name be the same as a standard library module? If the subdirectory's name is not constrained, the easiest way to mark it as a non-package is to put a hyphen or dot in its name; if you can't do that, at least name it something that you don't need to import.

Actually I have no problems with the change from inside python, but from the POV of tools, which walk through the directories, collecting/separating python packages and/or supplemental data directories. It's an explicit vs. implicit issue, where implicit would mean "kind of heuristics" from now on. IMHO it's going to break existing stuff [1] and should at least not be done in such a rush. nd [1] Well, it does break some of mine ;-) -- Da fällt mir ein, wieso gibt es eigentlich in Unicode kein "i" mit einem Herzchen als Tüpfelchen? Das wär sooo süüss! -- Björn Höhrmann in darw

Thomas Wouters

12:22 a.m.

On 4/26/06, Guido van Rossum wrote:

...

On 4/26/06, Benji York wrote:

...
Guido van Rossum wrote:

...
So I have a very simple proposal: keep the __init__.py requirement for top-level pacakages, but drop it for subpackages.

I don't particularly like it. You still need __init__.py's for 'import *' to work (not that I like or use 'import *' :). The first question that pops into my mind when I think file-less modules is "where does the package-module come from". That question is a lot easier to answer (not to mention explain) when all packages have an __init__.py. It also adds to Python's consistency (which means people learn something from it that they can apply to other things later; in that case, removing it just hampers their growth.) And besides, it's just not that big a deal. I don't feel strongly enough about it to object, though. However, I would suggest adding a warning for existing, __init__.py-less directories that would-have-been imported in 2.5. (There's an alpha3 scheduled, so it doesn't have to go in alpha2 tonight, and it could probably be last-minuted into beta1 too.) That should fix both Google's problems and that of everyone having existing non-package subdirs :-) Then, if it really matters, we can change the import in 2.6. Note that absolute import (which will be turned on for all in 2.6) 2.7, see the PEP. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

Guido van Rossum

12:36 a.m.

On 4/26/06, André Malo wrote:

...

* Guido van Rossum wrote:

...
On 4/26/06, André Malo wrote:

...
* Guido van Rossum wrote:

...
So I have a very simple proposal: keep the __init__.py requirement for top-level pacakages, but drop it for subpackages. This should be a small change. I'm hesitant to propose *anything* new for Python 2.5, so I'm proposing it for 2.6; if Neal and Anthony think this would be okay to add to 2.5, they can do so.

Not that it would count in any way, but I'd prefer to keep it. How would I mark a subdirectory as "not-a-package" otherwise?

What's the use case for that? Have you run into this requirement? And even if you did, was there a requirement that the subdirectory's name be the same as a standard library module? If the subdirectory's name is not constrained, the easiest way to mark it as a non-package is to put a hyphen or dot in its name; if you can't do that, at least name it something that you don't need to import.

Actually I have no problems with the change from inside python, but from the POV of tools, which walk through the directories, collecting/separating python packages and/or supplemental data directories. It's an explicit vs. implicit issue, where implicit would mean "kind of heuristics" from now on. IMHO it's going to break existing stuff [1] and should at least not be done in such a rush.

nd

[1] Well, it does break some of mine ;-)

Can you elaborate? You could always keep the __init__.py files, you know... -- --Guido van Rossum (home page: http://www.python.org/~guido/)

M.-A. Lemburg

12:59 a.m.

Guido van Rossum wrote:

...

On 4/26/06, Barry Warsaw wrote:

...
On Wed, 2006-04-26 at 10:16 -0700, Guido van Rossum wrote:

...
So I have a very simple proposal: keep the __init__.py requirement for top-level pacakages, but drop it for subpackages. This should be a small change. I'm hesitant to propose *anything* new for Python 2.5, so I'm proposing it for 2.6; if Neal and Anthony think this would be okay to add to 2.5, they can do so.

I'm not really sure what this would buy us. Newbies would still forget the __init__.py in top-level packages (not all newbies work for Google). Oldies would have trouble recognizing a directory as being a Python package, rather than just a collection of modules - you wouldn't go hunting up the path to find the top-level __init__.py file which identifies the directory as being a sub-package of some top-level package (not all oldies work for Google either where you only have a single top-level package). -1. It doesn't appear to make things easier and breaks symmetry.

...

...
But if it's there, then nothing changes, right? IOW, if we want to expose names in the subpackage's namespace, we can still do it in the subpackage's __init__.py. It's just that otherwise empty subpackage __init__.py files won't be required.

Correct. This is an important clarification.

...
What would the following print?

import toplevel.sub1.sub2 print toplevel.sub1.sub2.__file__

If it's "<path>/sub1/sub2" then that kind of breaks a common idiom of using os.path.dirname() on a module's __file__ to find co-located resources. Or at least, you have to know whether to dirname its __file__ or not (which might not be too bad, since you'd probably know how that package dir is organized anyway).

Oh, cool gray area. I propose that if there's no __init__.py it prints '<path>/sub1/sun2/' i.e. with a trailing slash; that causes dirname to just strip the '/'. (It would be a backslash on Windows of course).

...
I dunno. Occasionally it trips me up too, but it's such an obvious and easy fix that it's never bothered me enough to care.

But you're not a newbie. for a newbie who's just learned about packages, is familiar with Java, and missed one line in the docs, it's an easy mistake to make and a tough one to debug.

Why not make the ImportError's text a little smarter instead, e.g. let the import mechanism check for this particular gotcha ? This would solve the newbie problem without any changes to the import scheme.

...

...
I can't think of an example, but I suppose it's still possible that lifting this requirement could cause some in-package located data directories to be mistakenly importable. I'd be somewhat more worried about frameworks that dynamically import things having to be more cautious about cleansing their __import__() arguments now.

But (assuming 2.6 and absolute import) what would be the danger of importing such a package? Presumably it contains no *.py or *.so files so there's no code there; but even so you'd have to go out of your way to import it (since if the directory exists it can't also be a subpackage or submodule name that's in actual use).

Wasn't there agreement on postponing the absolute imports to Py3k due to the common use-case of turning e.g. 3rd party top-level packages into sub-packages of an application ? Without absolute imports your proposed scheme is going to break, since can easily mask top-level packages or modules.

...

...
I'd be -1 but the remote possibility of you being burned at the stake by your fellow Googlers makes me -0 :).

I'm not sure I understand what your worry is.

-- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 26 2006)

...

...
...
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

André Malo

1:05 a.m.

* Guido van Rossum wrote: [me]

...

...
Actually I have no problems with the change from inside python, but from the POV of tools, which walk through the directories, collecting/separating python packages and/or supplemental data directories. It's an explicit vs. implicit issue, where implicit would mean "kind of heuristics" from now on. IMHO it's going to break existing stuff [1] and should at least not be done in such a rush.

nd

[1] Well, it does break some of mine ;-)

[Guido]

...

Can you elaborate? You could always keep the __init__.py files, you know...

Okay, here's an example. It's about a non-existant __init__.py, though ;-) I have a test system which collects the test suites from one or more packages automatically by walking through the tree. Now there are subdirectories which explicitly are not packages (no __init__.py), but do contain some python files (helper scripts, spawned for particular tests). The test collector doesn't consider now these subdirectories at all, but in future it would need to (because it should search like python itself). Another point is that one can even hide supplementary packages within such a subdirectory. It's only visible to scripts inside the dir (I admit, that the latter is not a real usecase, just a thought that came up while writing this up). Anyway: sure, one could tweak the naming - just not for existing a.k.a. already released stuff. It's not very nice to force that, too ;-) nd -- If God intended people to be naked, they would be born that way. -- Oscar Wilde

Fred L. Drake, Jr.

1:26 a.m.

On Wednesday 26 April 2006 15:05, André Malo wrote:

...

Another point is that one can even hide supplementary packages within such a subdirectory. It's only visible to scripts inside the dir (I admit, that the latter is not a real usecase, just a thought that came up while writing this up).

I have tests that do this. This is a very real use case. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org>

Donovan Baarda

1:41 a.m.

Guido van Rossum wrote:

...

On 4/26/06, Barry Warsaw wrote:

...
On Wed, 2006-04-26 at 10:16 -0700, Guido van Rossum wrote:

...
So I have a very simple proposal: keep the __init__.py requirement for top-level pacakages, but drop it for subpackages. This should be a small change. I'm hesitant to propose *anything* new for Python 2.5, so I'm proposing it for 2.6; if Neal and Anthony think this would be okay to add to 2.5, they can do so. [...]

...
I'd be -1 but the remote possibility of you being burned at the stake by your fellow Googlers makes me -0 :).

I'm not sure I understand what your worry is.

I happen to be a Googler too, but I was a Pythonista first... I'm -1 for minor mainly subjective reasons; 1) explicit is better than implicit. I prefer to be explicit about what is and isn't a module. I have plenty of "doc" and "test" and other directories inside python module source tree's that I don't want to be python modules. 2) It feels more consistant to always require it. /foo/ is a python package because it contains an __init__.py... so package /foo/bar/ should have one one too. 3) It changes things for what feels like very little gain. I've never had problems with it, and don't find the import exception hard to diagnose. Note that I think the vast majority of "newbie missing __init__.py" problems within google occur because people are missing __init__.py at the root of package import tree. This change would not not solve that problem. It wouldn't surprise me if this change would introduce a slew of newbies complaining that "I have /foo on my PYTHONPATH, why can't I import foo/bar/" because they're forgotten the (now) rarely required __init__.py -- Donovan Baarda

Wolfgang Langner

1:57 a.m.

On 4/26/06, Guido van Rossum wrote:

...

So I have a very simple proposal: keep the __init__.py requirement for top-level pacakages, but drop it for subpackages. This should be a small change. I'm hesitant to propose *anything* new for Python 2.5, so I'm proposing it for 2.6; if Neal and Anthony think this would be okay to add to 2.5, they can do so.

-1 from me. I had never a problem with __init__.py to mark a package or subpackage. Better add __namespace__.py to state a package dir as namespace package. And support multiple occurrences of "on_python_path/namespace_name/package". -- bye by Wolfgang

Joe Smith

2:33 a.m.

"Guido van Rossum" wrote in message news:ca471dc20604261016g14854274i970d6f4fc72561c7@mail.gmail.com...

...

(Context: There's a large crowd with pitchforks and other sharp pointy farm implements just outside the door of my office at Google. They are making an unbelievable racket. It appears they are Google engineers who have been bitten by a misfeature of Python, and they won't let me go home before I have posted this message.)

One particular egregious problem is that *subpackage* are subject to the same rule. It so happens that there is essentially only one top-level package in the Google code base, and it already has an __init__.py file. But developers create new subpackages at a frightening rate, and forgetting to do "touch __init__.py" has caused many hours of lost work, not to mention injuries due to heads banging against walls.

It seems to me that the right way to fix this is to simply make a small change to the error message. On a failed import, have the code check if there is a directory that would have been the requested package if it had contained an __init__ module. If there is then append a message like "You might be missing an __init__.py file". It might also be good to check that the directory actually contained python modules.

Guido van Rossum

2:49 a.m.

OK, forget it. I'll face the pitchforks. I'm disappointed though -- it sounds like we can never change anything about Python any more because it will upset the oldtimers. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Phillip J. Eby

2:52 a.m.

New subject: Dropping __init__.py requirement for subpackages

At 04:33 PM 4/26/2006 -0400, Joe Smith wrote:

...

It seems to me that the right way to fix this is to simply make a small change to the error message. On a failed import, have the code check if there is a directory that would have been the requested package if it had contained an __init__ module. If there is then append a message like "You might be missing an __init__.py file".

It might also be good to check that the directory actually contained python modules.

This is a great idea, but might be hard to implement in practice with the current C implementation of import, at least for the general case. But if we're talking about subpackages only, the common case is a one-element __path__, and for that case there might be something we could do. (The number of path items is relevant because the existence of a correctly-named but init-less directory should not stop later path items from being searched, so the actual "error" occurs far from the point where the empty directory would've been detected.)

James Y Knight

3:12 a.m.

On Apr 26, 2006, at 4:49 PM, Guido van Rossum wrote:

...

OK, forget it. I'll face the pitchforks.

I'm disappointed though -- it sounds like we can never change anything about Python any more because it will upset the oldtimers.

No, you can not make a change which has a tiny (and arguably negative) advantage but large compatibility risks. Like it or not, Python is a stable, widely deployed program. Making almost arbitrary sideways changes is not something you can do to such a system without a lot of pushback. Breaking things requires (and *should* require) well thought out reasoning as to why the new way is actually better enough to justify the breakage. James

Terry Reedy

3:23 a.m.

New subject: Dropping __init__.py requirement for subpackages

"Phillip J. Eby" wrote in message news:5.1.1.6.0.20060426164925.01e33228@mail.telecommunity.com...

...

...
It might also be good to check that the directory actually contained python modules.

This is a great idea, but might be hard to implement in practice with the current C implementation of import, at least for the general case.

Perhaps checking and autogeneration of __init__.py should better be part of a Python-aware editor/IDE. A File menu could have a New Package entry to create a directory + empty __init__.py. Terry Jan Reedy

Ian Bicking

4:35 a.m.

Joe Smith wrote:

...

It seems to me that the right way to fix this is to simply make a small change to the error message. On a failed import, have the code check if there is a directory that would have been the requested package if it had contained an __init__ module. If there is then append a message like "You might be missing an __init__.py file".

+1. It's not that putting an __init__.py file in is hard, it's that people have a hard time realizing when they've forgotten to do it. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org

Thomas Wouters

4:40 a.m.

On 4/26/06, Guido van Rossum wrote:

...

OK, forget it. I'll face the pitchforks.

Maybe this'll help: http://python.org/sf/1477281 (You can call it 'oldtimer-repellant' if you want to use it to convince people there isn't any *real* backward-compatibility issue.) I'm disappointed though -- it sounds like we can never change anything

...

about Python any more because it will upset the oldtimers.

That's a bit unfair, Guido. There are valid reasons not to change Python's behaviour in this respect, regardless of upset old-timers. Besides, you're the BDFL; if you think the old-timers are wrong, I implore you to put their worries aside (after dutiful contemplation.) I've long since decided that any change what so ever will have activist luddites opposing it. I think most of them would stop when you make a clear decision -- how much whining have you had about the if-else syntax since you made the choice? I've heard lots of people gripe about it in private (at PyCon, of course, I never see Pythonistas anywhere else :-P), but I haven't seen any python-dev rants about it. I certainly hate PEP-308's guts, but if if-else is your decision, if-else is what we'll do. And so it is, I believe, with this case. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

Guido van Rossum

4:58 a.m.

On 4/26/06, Thomas Wouters wrote:

...

On 4/26/06, Guido van Rossum wrote:

...
OK, forget it. I'll face the pitchforks.

Maybe this'll help:

http://python.org/sf/1477281

(You can call it 'oldtimer-repellant' if you want to use it to convince people there isn't any *real* backward-compatibility issue.)

I'd worry that it'll cause complaints when the warning is incorrect and a certain directory is being skipped intentionally. E.g. the "string" directory that someone had. Getting a warning like this can be just as upsetting to newbies!

...

...
I'm disappointed though -- it sounds like we can never change anything about Python any more because it will upset the oldtimers.

That's a bit unfair, Guido. There are valid reasons not to change Python's behaviour in this respect, regardless of upset old-timers.

Where are the valid reasons? All I see is knee-jerk -1, -1, -1, and "this might cause tools to do the wrong thing". Not a single person attempted to see it from the newbie POV; several people explicitly rejected the newbie POV as invalid. I still don't know the name of any tool that would break due to this *and where the breakage wouldn't be easy to fix by adjusting the tool's behavior*. Yes, fixing tools is a pain. But they have to be fixed for every new Python version anyway -- new keywords, new syntax, new bytecodes, etc.

...

Besides, you're the BDFL; if you think the old-timers are wrong, I implore you to put their worries aside (after dutiful contemplation.)

I can only do that so many times before I'm no longer the BDFL. It's one thing to break a tie when there is widespread disagreement amongst developers (like about the perfect decorator syntax). It's another to go against a see of -1's.

...

I've long since decided that any change what so ever will have activist luddites opposing it. I think most of them would stop when you make a clear decision -- how much whining have you had about the if-else syntax since you made the choice? I've heard lots of people gripe about it in private (at PyCon, of course, I never see Pythonistas anywhere else :-P), but I haven't seen any python-dev rants about it. I certainly hate PEP-308's guts, but if if-else is your decision, if-else is what we'll do. And so it is, I believe, with this case.

OK. Then I implore you, please check in that patch (after adding error checking for PyErr_Warn() -- and of course after a2 hs been shipped), and damn the torpedoes. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Thomas Wouters

5:10 a.m.

On 4/27/06, Guido van Rossum wrote:

...

On 4/26/06, Thomas Wouters wrote:

...
On 4/26/06, Guido van Rossum wrote:

...
OK, forget it. I'll face the pitchforks.

Maybe this'll help:

http://python.org/sf/1477281

(You can call it 'oldtimer-repellant' if you want to use it to convince people there isn't any *real* backward-compatibility issue.)

I'd worry that it'll cause complaints when the warning is incorrect and a certain directory is being skipped intentionally. E.g. the "string" directory that someone had. Getting a warning like this can be just as upsetting to newbies!

I don't think getting a spurious warning is as upsetting as getting no warning but the damned thing just not working. At least you have something to google for. And the warning includes the original line of source that triggered it *and* the directory (or directories) it's complaining about, which is quite a lot of helpful hints. The clashes with directories that aren't intended to be packages *and* a module of the same name is imported, yes, that's a real problem. It's not any worse than if we change package imports the way you originally proposed, though, and I think the actual number of spurious errors is very small (which self-respecting module still does 'import string', eh? :-) I don't think the fix for such a warning is going to be non-trivial. Where are the valid reasons? Of course, I only consider *my* reasons to be valid, and mine weren't knee-jerk or tool-related. I don't think Python should be going "Oh, what you wanted wasn't possible, but I think I know what you wanted, let me do it for you", first of all because it's not very Pythonic, and second of all because it doesn't lower the learning curve, it just delays some upward motion a little (meaning the curve may become steeper, later.) A clear warning, on the other hand, can be a helpful nudge towards the 'a-HA' moment.

...

Then I implore you, please check in that patch (after adding error checking for PyErr_Warn() -- and of course after a2 hs been shipped), and damn the torpedoes.

Alrighty then. The list has about 12 hours to convince me (and you) that it's a bad idea to generate that warning. I'll be asleep by the time the trunk un-freezes, and I have a string of early meetings tomorrow. I'll get to it somewhere in the afternoon :) -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

Guido van Rossum

5:24 a.m.

On 4/26/06, Thomas Wouters wrote:

...

Of course, I only consider *my* reasons to be valid, and mine weren't knee-jerk or tool-related. I don't think Python should be going "Oh, what you wanted wasn't possible, but I think I know what you wanted, let me do it for you", first of all because it's not very Pythonic, and second of all because it doesn't lower the learning curve, it just delays some upward motion a little (meaning the curve may become steeper, later.) A clear warning, on the other hand, can be a helpful nudge towards the 'a-HA' moment.

That still sounds like old-timer reasoning. Long ago we were very close to defining a package as "a directory" -- with none of this "must contain __init__.py or another *.py file" nonsense. IIRC the decision to make __init__.py mandatory faced opposition too, since people were already doing packages with just directories (which is quite clean and elegant, and that's also how it was in Java), but I added it after seeing a few newbies tear out their hair. I believe that if at that time __init__.py had remained optional, and today I had proposed to require it, the change would have been derided as unpythonic as well. There's nothing particularly unpythonic about optional behavior; e.g. classes may or may not provide an __init__ method. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Anthony Baxter

8:22 a.m.

On Thursday 27 April 2006 06:49, Guido van Rossum wrote:

...

OK, forget it. I'll face the pitchforks.

I'm disappointed though -- it sounds like we can never change anything about Python any more because it will upset the oldtimers.

I'm not averse to changing this - just not to changing it on short notice for 2.5 and causing me pain from having to cut new releases to fix some breakage in the stdlib caused by this <wink> -- Anthony Baxter It's never too late to have a happy childhood.

Guido van Rossum

8:39 a.m.

On 4/26/06, Anthony Baxter wrote:

...

On Thursday 27 April 2006 06:49, Guido van Rossum wrote:

...
OK, forget it. I'll face the pitchforks.

I'm disappointed though -- it sounds like we can never change anything about Python any more because it will upset the oldtimers.

I'm not averse to changing this - just not to changing it on short notice for 2.5 and causing me pain from having to cut new releases to fix some breakage in the stdlib caused by this <wink>

I wasn't proposing it for 2.5 (for this very reason) -- AFAICT most responses were again st this for 2.6 or 2.7. Hopefully you're okay with Thomas's patch going into 2.5a3 -- it *warns* about directories without __init__.py that otherwise match the requested name. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Barry Warsaw

8:42 a.m.

Boy, threads here sure move fast when there's work to be done :). Although largely moot now, I'll follow up for posterity's sake. On Wed, 2006-04-26 at 10:59 -0700, Guido van Rossum wrote:

...

Oh, cool gray area. I propose that if there's no __init__.py it prints '<path>/sub1/sun2/' i.e. with a trailing slash; that causes dirname to just strip the '/'. (It would be a backslash on Windows of course).

Yep, that's the right thing to do. Given all the other traffic in this thread, I don't have much more to add except: I probably don't remember things accurately, but ISTR that __init__.py was added as a way to expose names in the package's namespace first. We probably went from that to requiring __init__.py as a way to be explicit about what directories are packages and which aren't. So I suspect you're right when you say that if the rule had already been relaxed and you were now proposing to tighten the rules, we probably get just as many complaints. alternative-universe-ly y'rs, -Barry

Fred L. Drake, Jr.

8:59 a.m.

On Wednesday 26 April 2006 22:42, Barry Warsaw wrote:

...

So I suspect you're right when you say that if the rule had already been relaxed and you were now proposing to tighten the rules, we probably get just as many complaints.

Indeed. I think the problem many of us have with the proposal isn't the new behavior, but the change in the behavior. That's certainly it for me. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org>

Guido van Rossum

9:02 a.m.

On 4/26/06, Fred L. Drake, Jr. wrote:

...

Indeed. I think the problem many of us have with the proposal isn't the new behavior, but the change in the behavior. That's certainly it for me.

Which is why I said earlier that I felt disappointed that we can't change anything any more. But I'm fine with the warning -- it should be enough to keep Google's newbies from wasting their time. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Aahz

9:48 a.m.

On Wed, Apr 26, 2006, Guido van Rossum wrote:

...

Which is why I said earlier that I felt disappointed that we can't change anything any more.

I've been here since Python 1.5.1. I don't understand why this issue in particular makes you feel disappointed. I also think your statement is just plain untrue. Looking at the What's New for Python 2.5, we have changes to the import machinery, the with statement, new functionality for generators, conditional expressions, ssize_t, exceptions as new-style classes, deleted modules (regex, regsub, and whrandom), and on and on and on. Some of these changes are going to cause moderate breakage for some people; they cumulatively represent a *lot* of change. Python 2.5 is gonna be a *huge* release. Quite honestly, I think you've let yourself get emotionally attached to this issue for some reason. On the other hand, I think that a lot of the blowback you got comes from you bringing up this issue right before 2.5a2. My suggestion is that you let this sit and make yourself a calendar entry to think about this again in October. If it still seems like a good idea, go ahead and bring it up. (Or just punt to 3.0.) -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "Argue for your limitations, and sure enough they're yours." --Richard Bach

Paul Moore

1:47 p.m.

On 4/27/06, Guido van Rossum wrote:

...

On 4/26/06, Thomas Wouters wrote:

...
Of course, I only consider *my* reasons to be valid, and mine weren't knee-jerk or tool-related. I don't think Python should be going "Oh, what you wanted wasn't possible, but I think I know what you wanted, let me do it for you", first of all because it's not very Pythonic, and second of all because it doesn't lower the learning curve, it just delays some upward motion a little (meaning the curve may become steeper, later.) A clear warning, on the other hand, can be a helpful nudge towards the 'a-HA' moment.

That still sounds like old-timer reasoning. Long ago we were very close to defining a package as "a directory" -- with none of this "must contain __init__.py or another *.py file" nonsense. IIRC the decision to make __init__.py mandatory faced opposition too, since people were already doing packages with just directories (which is quite clean and elegant, and that's also how it was in Java), but I added it after seeing a few newbies tear out their hair.

OK. After due consideration, I'm happy to accept the change. (Call that +0, it doesn't bother me much personally either way). Although reading the above paragraph, I get the impression that you are saying that __init__.py was originally added to help newbies, and yet you are now saying the exact opposite. I'll leave you to resolve that particular contradiction, though... FWIW, I still have every confidence in your judgement about features. However, I'd have to say that your timing sucks :-) Your initial message read to me as "Quick! I'm about to get lynched here - can I have the OK to shove this change in before a2 goes out?" (OK, 2.5 isn't feature frozen until a3, so maybe you only meant a3, but you clearly wanted a quick response). So it's hard to expect anything other than immediate knee-jerk responses. And those are usually driven by personal experience (implying less consideration of newbie mistakes from this type of audience) and unfocused fear of breakage. Paul.

Paul Moore

1:55 p.m.

On 4/27/06, Paul Moore wrote:

...

However, I'd have to say that your timing sucks :-) Your initial message read to me as "Quick! I'm about to get lynched here - can I have the OK to shove this change in before a2 goes out?"

And this just proves that my response wasn't anywhere near as considered as I thought. You explicitly said 2.6, which I'd forgotten. I have no problem with going with your instinct in 2.6. But I do believe that there may be more breakage than you have considered (PEP 302 experience talking here :-)) so get it in early rather than late! Of course this is all a bit moot now, as the decision seems to have been made. Sigh. Paul.

Anthony Baxter

2:06 p.m.

On Thursday 27 April 2006 17:47, Paul Moore wrote:

...

FWIW, I still have every confidence in your judgement about features. However, I'd have to say that your timing sucks :-) Your initial message read to me as "Quick! I'm about to get lynched here - can I have the OK to shove this change in before a2 goes out?" (OK, 2.5 isn't feature frozen until a3, so maybe you only meant a3, but you clearly wanted a quick response).

Feature freeze is the day we release beta1. New features go in until then - after that, not without approval and a general concensus that the changes have a substantial cost/benefit for breaking the feature freeze. Or if Guido gets Google developers parading him in effigy around the office and needs to get them off his back. <wink> Anthony -- Anthony Baxter It's never too late to have a happy childhood.

Talin

2:44 p.m.

New subject: [Python-Dev] Dropping __init__.py requirement for subpackages

Guido van Rossum writes:

...

The requirement that a directlry must contain an __init__.py file before it is considered a valid package has always been controversial. It's designed to prevent the existence of a directory with a common name like "time" or "string" from preventing fundamental imports to work. But the feature often trips newbie Python programmers (of which there are a lot at Google, at our current growth rate we're probably training more new Python programmers than any other organization worldwide .

Might I suggest an alternative? I too find it cumbersome to have to litter my directory tree with __init__.py iles. However, rather than making modules be implicit ("Explicit is better than implicit"), I would rather see a more powerful syntax for declaring modules. Specifically, what I would like to see is a way for a top-level __init__.py to explicitly list which subdirectories are modules. Rather than spreading that information over a bunch of different __init__.py files, I would much rather have the information be centralized in a single text file for the whole package. Just as we have an __all__ variable that indicates which symbols are to be exported, we could have a '__submodules__' array which explicitly calls out the list of submodule directory names. Or perhaps more simply, just have some code in the top-level __init__.py that creates (but does not load) the module objects for the various sub-modules. The presence of "__init__.py" could perhaps also indicate the root of a *standalone* module tree; submodules that don't have their own __init__.py, but which are declared indirectly via an ancestor are assumed by convention to be 'component' modules which are not intended to operate outside of their parent. (In other words, the presence of the __init__.py signals that that tree is separately exportable.) I'm sure that someone who is familiar with the import machinery could whip up something like this in a matter of minutes. As far as the "compatibility with tools" argument goes, I say, break em :) Those tool programmers need a hobby anyway :) -- Talin

Fredrik Lundh

3:30 p.m.

Guido van Rossum wrote:

...

So I have a very simple proposal: keep the __init__.py requirement for top-level pacakages, but drop it for subpackages. This should be a small change. I'm hesitant to propose *anything* new for Python 2.5, so I'm proposing it for 2.6; if Neal and Anthony think this would be okay to add to 2.5, they can do so.

I'm going to skip the discussion thread (or is it a flame war? cannot tell from the thread pattern), but here are my votes: +0 on dropping the requirement for subpackages in 2.X (this will break tools, but probably not break much code, and fixing the tools should be straightforward) (+1 on dropping it in 3.X) -1 on dropping the requirement for packages in 2.X (-0 on dropping it in 3.X) +0 on making this change in 2.5 </F>

Nick Coghlan

4:43 p.m.

Guido van Rossum wrote:

...

So I have a very simple proposal: keep the __init__.py requirement for top-level pacakages, but drop it for subpackages. This should be a small change. I'm hesitant to propose *anything* new for Python 2.5, so I'm proposing it for 2.6; if Neal and Anthony think this would be okay to add to 2.5, they can do so.

I haven't scanned this whole thread yet, but my first thought was to just try to find a way to give a better error message if we find a candidate package directory, but there's no __init__.py file. i.e. something like: ImportError: __init__.py not found in directory '<whatever>/foo' for package 'foo' I like the fact that __init__.py documents, right there in the file system directory listing, that the current directory is a Python package. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

skip＠pobox.com

7:18 p.m.

Guido> One particular egregious problem is that *subpackage* are subject Guido> to the same rule. It so happens that there is essentially only Guido> one top-level package in the Google code base, and it already has Guido> an __init__.py file. But developers create new subpackages at a Guido> frightening rate, and forgetting to do "touch __init__.py" has Guido> caused many hours of lost work, not to mention injuries due to Guido> heads banging against walls. That's why God created make: install: touch __init__.py blah blah blah Skip

skip＠pobox.com

7:21 p.m.

>> Not that it would count in any way, but I'd prefer to keep it. How >> would I mark a subdirectory as "not-a-package" otherwise? Guido> What's the use case for that? Have you run into this requirement? Yes, we run into it. We typically install a package with any resources in a resources subdirectory. Now, those resources subdirectories generally don't contain Python files (Glade files are the most frequent occupants), but there's no reason they couldn't contain Python files. Skip

Gustavo Carneiro

7:42 p.m.

On 4/26/06, Guido van Rossum wrote: [...]

...

So I have a very simple proposal: keep the __init__.py requirement for top-level pacakages, but drop it for subpackages. This should be a small change. I'm hesitant to propose *anything* new for Python 2.5, so I'm proposing it for 2.6; if Neal and Anthony think this would be okay to add to 2.5, they can do so.

Damn these threads are so quick they are born and die off in 24 hours and don't give enough time for people to comment :( I'm a bit late in the thread, but I'm +1 (for 2.5) for the following reason. There are mainly two use cases for having a couple of modules foo.bar, and foo.zbr: 1. foo.bar and foo.zbr are both part of the same _package_, and are distributed together; 2. foo.bar and foo.zbr are two independent modules, distributed separately, but which share a common 'foo' _namespace_, to denote affiliation with a project. The use case #1 is arguably more common, but use case #2 is also very relevant. It happens for a lot of GNOME python bindings, for example, where we used to have gnome, gnome.ui, gnome.vfs, gnome.applet, etc. Now the problem. Suppose you have the source package python-foo-bar, which installs $pythondir/foo/__init__.py and $pythondir/foo/bar.py. This would make a module called "foo.bar" available. Likewise, you can have the source package python-foo-zbr, which installs $pythondir/foo/__init__.py and $pythondir/foo/zbr.py. This would make a module called "foo.zbr" available. The two packages above install the file $pythondir/foo/__init__.py. If one of them adds some content to __init__.py, the other one will overwrite it. Packaging these two packages for e.g. debian would be extremely difficult, because no two .deb packages are allowed to intall the same file. One solution is to generate the __init__.py file with post-install hooks and shell scripts. Another solution would be for example to have only python-foo-bar install the __init__.py file, but then python-foo-zbr would have to depend on python-foo-bar, while they're not really related. I hope I made the problem clear enough, and I hope people find this a compelling argument in favour of eliminating the need for __init__.py. Regards, Gustavo Carneiro.

Bernhard Herzog

7:48 p.m.

"Gustavo Carneiro" writes:

...

Now the problem. Suppose you have the source package python-foo-bar, which installs $pythondir/foo/__init__.py and $pythondir/foo/bar.py. This would make a module called "foo.bar" available. Likewise, you can have the source package python-foo-zbr, which installs $pythondir/foo/__init__.py and $pythondir/foo/zbr.py. This would make a module called "foo.zbr" available.

The two packages above install the file $pythondir/foo/__init__.py. If one of them adds some content to __init__.py, the other one will overwrite it. Packaging these two packages for e.g. debian would be extremely difficult, because no two .deb packages are allowed to intall the same file.

One solution is to generate the __init__.py file with post-install hooks and shell scripts. Another solution would be for example to have only python-foo-bar install the __init__.py file, but then python-foo-zbr would have to depend on python-foo-bar, while they're not really related.

Yet another solution would be to put foo/__init__.py into a third package, e.g. python-foo, on which both python-foo-bar and python-foo-zbr depend. Bernhard -- Intevation GmbH http://intevation.de/ Skencil http://skencil.org/ Thuban http://thuban.intevation.org/

Thomas Wouters

8:39 p.m.

On 4/27/06, Thomas Wouters wrote:

...

Alrighty then. The list has about 12 hours to convince me (and you) that it's a bad idea to generate that warning. I'll be asleep by the time the trunk un-freezes, and I have a string of early meetings tomorrow. I'll get to it somewhere in the afternoon :)

I could check it in, except the make-testall I ran overnight showed a small problem: the patch would generate a number of spurious warnings in the trunk: /home/thomas/python/python/trunk/Lib/gzip.py:9: ImportWarning: Not importing directory '/home/thomas/python/python/trunk/Modules/zlib': missing __init__.py /home/thomas/python/python/trunk/Lib/ctypes/__init__.py:8: ImportWarning: Not importing directory '/home/thomas/python/python/trunk/Modules/_ctypes': missing __init__.py (and a few more zlib ones.) The reason for that is that ./Modules is added to the import path, by a non-installed Python. This is because of the pre-distutils Modules/Setup-style build method of modules (which is still sometimes used.) I can't find where Modules is added to sys.path, though, even if I wanted to remove it :) So, do we: a) forget about the warning because of the layout of the svn tree (bad, imho) 2) rename Modules/zlib and Modules/_ctypes to avoid the warning (inconvenient, but I don't know how inconvenient) - fix the build procedure so Modules isn't added to sys.path unless it absolutely has to (which is only very rarely the case, I believe) or lastly, make regrtest.py ignore those specific warnings? -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

Phillip J. Eby

8:50 p.m.

New subject: Dropping __init__.py requirement for subpackages

At 03:48 PM 4/27/2006 +0200, Bernhard Herzog wrote:

...

"Gustavo Carneiro" writes:

...
Now the problem. Suppose you have the source package python-foo-bar, which installs $pythondir/foo/__init__.py and $pythondir/foo/bar.py. This would make a module called "foo.bar" available. Likewise, you can have the source package python-foo-zbr, which installs $pythondir/foo/__init__.py and $pythondir/foo/zbr.py. This would make a module called "foo.zbr" available.

The two packages above install the file $pythondir/foo/__init__.py. If one of them adds some content to __init__.py, the other one will overwrite it. Packaging these two packages for e.g. debian would be extremely difficult, because no two .deb packages are allowed to intall the same file.

One solution is to generate the __init__.py file with post-install hooks and shell scripts. Another solution would be for example to have only python-foo-bar install the __init__.py file, but then python-foo-zbr would have to depend on python-foo-bar, while they're not really related.

Yet another solution would be to put foo/__init__.py into a third package, e.g. python-foo, on which both python-foo-bar and python-foo-zbr depend.

Or you can package them with setuptools, and declare foo to be a namespace package. If installing in the mode used for building RPMs and debs, there will be no __init__.py. Instead, each installs a .pth file that ensures a dummy package object is created in sys.modules with an appropriate __path__. This solution is packaging-system agnostic and doesn't require any special support from the packaging tool. (The downside, however, is that neither foo.bar nor foo.zbr's __init__.py will be allowed to have any content, since in some installation scenarios there will be no __init__.py at all.)

Gustavo Carneiro

9:14 p.m.

On 4/27/06, Phillip J. Eby wrote:

...

...
"Gustavo Carneiro" writes:

...
Now the problem. Suppose you have the source package

At 03:48 PM 4/27/2006 +0200, Bernhard Herzog wrote: python-foo-bar,

...
...
which installs $pythondir/foo/__init__.py and $pythondir/foo/bar.py. This would make a module called "foo.bar" available. Likewise, you can have the source package python-foo-zbr, which installs $pythondir/foo/__init__.py and $pythondir/foo/zbr.py. This would make a module called "foo.zbr" available.

The two packages above install the file $pythondir/foo/__init__.py. If one of them adds some content to __init__.py, the other one will overwrite it. Packaging these two packages for e.g. debian would be extremely difficult, because no two .deb packages are allowed to intall the same file.

One solution is to generate the __init__.py file with post-install hooks and shell scripts. Another solution would be for example to have only python-foo-bar install the __init__.py file, but then python-foo-zbr would have to depend on python-foo-bar, while they're not really related.

Yet another solution would be to put foo/__init__.py into a third package, e.g. python-foo, on which both python-foo-bar and python-foo-zbr depend.

You can't be serious. One package just to install a __init__.py file? Or you can package them with setuptools, and declare foo to be a namespace

...

package.

Let's not assume setuptools are always used, shall we? If installing in the mode used for building RPMs and debs, there

...

will be no __init__.py. Instead, each installs a .pth file that ensures a dummy package object is created in sys.modules with an appropriate __path__. This solution is packaging-system agnostic and doesn't require any special support from the packaging tool.

I don't understand this solution. How can a .pth file create a 'dummy package'? Remember that the objective is to have "foo.bar" and "foo.zbr" modules, not just "bar" and "zbr". But in any case, it already sounds like a convoluted solution. No way it can beat the obvious/simple solution: to remove the need to have __init__.py in the first place. (The downside, however, is that neither foo.bar nor foo.zbr's __init__.py

...

will be allowed to have any content, since in some installation scenarios there will be no __init__.py at all.)

That's ok in the context of this proposal (not having __init__.py at all).

Thomas Wouters

9:31 p.m.

On 4/27/06, Gustavo Carneiro wrote:

...

On 4/27/06, Phillip J. Eby wrote:

...
...
"Gustavo Carneiro" writes:

...
Now the problem. Suppose you have the source package

At 03:48 PM 4/27/2006 +0200, Bernhard Herzog wrote: python-foo-bar,

...
...
which installs $pythondir/foo/__init__.py and $pythondir/foo/bar.py. This would make a module called "foo.bar" available. Likewise, you can have the source package python-foo-zbr, which installs $pythondir/foo/__init__.py and $pythondir/foo/zbr.py. This would make a module called "foo.zbr" available.

The two packages above install the file $pythondir/foo/__init__.py. If one of them adds some content to __init__.py, the other one will overwrite it. Packaging these two packages for e.g. debian would be extremely difficult, because no two .deb packages are allowed to intall the same file.

Yet another solution would be to put foo/__init__.py into a third package, e.g. python-foo, on which both python-foo-bar and python-foo-zbr depend.

You can't be serious. One package just to install a __init__.py file?

Sure. Have you counted the number of 'empty' packages in Debian lately? Besides, Guido's original proposal is not a fix for your problem, either; he only proposes to change the requirement for *sub*packages. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

Gustavo Carneiro

9:42 p.m.

On 4/27/06, Thomas Wouters wrote:

...

On 4/27/06, Gustavo Carneiro wrote:

...
On 4/27/06, Phillip J. Eby < pje@telecommunity.com> wrote:

...
At 03:48 PM 4/27/2006 +0200, Bernhard Herzog wrote: "Gustavo Carneiro" writes:

...
Now the problem. Suppose you have the source package python-foo-bar, which installs $pythondir/foo/__init__.py and $pythondir/foo/bar.py. This would make a module called "foo.bar" available. Likewise, you can have the source package python-foo-zbr, which installs $pythondir/foo/__init__.py and $pythondir/foo/zbr.py. This would make a module called "foo.zbr" available.

The two packages above install the file $pythondir/foo/__init__.py. If one of them adds some content to __init__.py, the other one will overwrite it. Packaging these two packages for e.g. debian would be extremely difficult, because no two .deb packages are allowed to intall the same file.

Yet another solution would be to put foo/__init__.py into a third package, e.g. python-foo, on which both python-foo-bar and python-foo-zbr depend.

You can't be serious. One package just to install a __init__.py file?

Sure. Have you counted the number of 'empty' packages in Debian lately?

Sure. That is already a problem; let's not make it a worse problem for no good reason; I haven't heard a good reason to keep the __init__.py file besides backward compatibility concerns. Besides, Guido's original proposal is not a fix for your problem, either; he

...

only proposes to change the requirement for *sub*packages.

It *is* a solution for my problem. I don't need the __init__.py file for anything, since I don't need anything defined in the the 'foo' namespace, only the subpackages foo.bar and foo.zbr.

Thomas Wouters

9:49 p.m.

On 4/27/06, Gustavo Carneiro wrote:

...

Besides, Guido's original proposal is not a fix for your problem, either;

...
he only proposes to change the requirement for *sub*packages.

It *is* a solution for my problem. I don't need the __init__.py file for anything, since I don't need anything defined in the the 'foo' namespace, only the subpackages foo.bar and foo.zbr .

... No. Guido's original proposal is not a fix for your problem, because *it doesn't affect the 'foo' namespace*. Guido's original proposal still requires foo/__init__.py for your namespace to work, it just makes foo/bar/__init__.py and foo/zbr/__init__.py optional. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

Gustavo Carneiro

10:20 p.m.

On 4/27/06, Thomas Wouters wrote:

...

On 4/27/06, Gustavo Carneiro wrote:

...
Besides, Guido's original proposal is not a fix for your problem,

...
either; he only proposes to change the requirement for *sub*packages.

It *is* a solution for my problem. I don't need the __init__.py file for anything, since I don't need anything defined in the the 'foo' namespace, only the subpackages foo.bar and foo.zbr .

... No. Guido's original proposal is not a fix for your problem, because *it doesn't affect the 'foo' namespace*. Guido's original proposal still requires foo/__init__.py for your namespace to work, it just makes foo/bar/__init__.py and foo/zbr/__init__.py optional.

Damn, you're right, I confused subpackage with submodule :P In that case, can I counter-propose to stop requiring the __init__.py file in [foo/__init__.py, foo/bar.py] ? ;-) The proposal would mean that a directory 'foo' with a single file bar.pywould make the module ' foo.bar' available if the parent directory of 'foo' is in sys.path. /me faces the pitchforks.

Guido van Rossum

11:25 p.m.

On 4/27/06, Thomas Wouters wrote:

...

I could check it in, except the make-testall I ran overnight showed a small problem: the patch would generate a number of spurious warnings in the trunk:

/home/thomas/python/python/trunk/Lib/gzip.py:9: ImportWarning: Not importing directory '/home/thomas/python/python/trunk/Modules/zlib': missing __init__.py

/home/thomas/python/python/trunk/Lib/ctypes/__init__.py:8: ImportWarning: Not importing directory '/home/thomas/python/python/trunk/Modules/_ctypes': missing __init__.py

(and a few more zlib ones.) The reason for that is that ./Modules is added to the import path, by a non-installed Python. This is because of the pre-distutils Modules/Setup-style build method of modules (which is still sometimes used.) I can't find where Modules is added to sys.path, though, even if I wanted to remove it :)

So, do we: a) forget about the warning because of the layout of the svn tree (bad, imho) 2) rename Modules/zlib and Modules/_ctypes to avoid the warning (inconvenient, but I don't know how inconvenient) - fix the build procedure so Modules isn't added to sys.path unless it absolutely has to (which is only very rarely the case, I believe) or lastly, make regrtest.py ignore those specific warnings?

I'd say the latter. That's how we deal with warnings during the test suite in general don't we? -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Thomas Heller

28 Apr 28 Apr

9:07 p.m.

Thomas Wouters wrote:

...

On 4/27/06, Thomas Wouters wrote:

...
Alrighty then. The list has about 12 hours to convince me (and you) that it's a bad idea to generate that warning. I'll be asleep by the time the trunk un-freezes, and I have a string of early meetings tomorrow. I'll get to it somewhere in the afternoon :)

I could check it in, except the make-testall I ran overnight showed a small problem: the patch would generate a number of spurious warnings in the trunk:

/home/thomas/python/python/trunk/Lib/gzip.py:9: ImportWarning: Not importing directory '/home/thomas/python/python/trunk/Modules/zlib': missing __init__.py

/home/thomas/python/python/trunk/Lib/ctypes/__init__.py:8: ImportWarning: Not importing directory '/home/thomas/python/python/trunk/Modules/_ctypes': missing __init__.py

(and a few more zlib ones.) The reason for that is that ./Modules is added to the import path, by a non-installed Python. This is because of the pre-distutils Modules/Setup-style build method of modules (which is still sometimes used.) I can't find where Modules is added to sys.path, though, even if I wanted to remove it :)

So, do we: a) forget about the warning because of the layout of the svn tree (bad, imho) 2) rename Modules/zlib and Modules/_ctypes to avoid the warning (inconvenient, but I don't know how inconvenient) - fix the build procedure so Modules isn't added to sys.path unless it absolutely has to (which is only very rarely the case, I believe) or lastly, make regrtest.py ignore those specific warnings?

Would not another way be to make sure Modules is moved *behind* the setup.py build directory on sys.path? Thomas

Thomas Wouters

9:39 p.m.

On 4/28/06, Thomas Heller wrote:

...

Would not another way be to make sure Modules is moved *behind* the setup.py build directory on sys.path?

Indeed! I hadn't realized that, although I might've if I'd been able to find where Modules is put on sys.path. And, likewise, I would do as you suggest (which feels like the right thing) if I could only find out where Modules is put on sys.path :) I don't have time to search for it today nor, probably, this weekend (which is a party weekend in the Netherlands.) I'll get to it eventually, although a helpful hint from an old-timer who remembers as far back as Modules/Setup would be welcome. :) -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

"Martin v. Löwis"

10:11 p.m.

Thomas Wouters wrote:

...

Indeed! I hadn't realized that, although I might've if I'd been able to find where Modules is put on sys.path. And, likewise, I would do as you suggest (which feels like the right thing) if I could only find out where Modules is put on sys.path :) I don't have time to search for it today nor, probably, this weekend (which is a party weekend in the Netherlands.) I'll get to it eventually, although a helpful hint from an old-timer who remembers as far back as Modules/Setup would be welcome. :)

With some debugging, I found it out: search_for_exec_prefix looks for the presence of Modules/Setup; if that is found, it strips off /Setup, leaving search_for_exec_prefix with -1. This then gets added to sys.path with /* Finally, on goes the directory for dynamic-load modules */ strcat(buf, exec_prefix); I wasn't following exactly, so I might have mixed something up, but... it appears that this problem here comes from site.py adding the build directory for the distutils dynamic objects even after Modules. The site.py code is # XXX This should not be part of site.py, since it is needed even when # using the -S option for Python. See http://www.python.org/sf/586680 def addbuilddir(): """Append ./build/lib.<platform> in case we're running in the build dir (especially for Guido :-)""" from distutils.util import get_platform s = "build/lib.%s-%.3s" % (get_platform(), sys.version) s = os.path.join(os.path.dirname(sys.path[-1]), s) sys.path.append(s) I would suggest fixing #586680: Add build/lib.* before adding dynamic-load modules, by moving the code into Modules/getpath.c. You should be able to use efound < 0 as an indication that this is indeed the build directory. Regards, Martin

Clark C. Evans

20 Jun 20 Jun

8:36 p.m.

+1 Excellent Change +1 Minimal Backward Compatibility Difficulties I think this would also help quite a bit with newbie adoption of Python. I've had to explain this un-feature on numerous occassions and it given how smart Python is, I've wondered why it has this requirement. If you look in various open source packages, you'll find that 95% of these __init__.py files are empty. The ones at my work actually say: # stupid Python requirement, don't remove this file Why? Someone decided to remove files of length 0 in our repository without realizing the consequences. Since it had the __init__.pyc file around, it still worked... till one brought down a fresh copy of the repository and then it just "stopped" working. Quite a bit of hair pulling that one caused us. The only case where this might cause a problem is with "resource" directories that only contain ".html", ".jpg" and other files. So, perhpas this feature would only turn a directory into a package if it didn't have any .py files. It could also trigger only when the package is explicitly imported? Good luck /w the pitch-fork wielding users and telling the old-timers where they can keep their backward compatibility. Clark

6514

Age (days ago)

6569

Last active (days ago)

List overview

Download

52 comments

26 participants

participants (26)

"Martin v. Löwis"
Aahz
André Malo
Anthony Baxter
Barry Warsaw
Benji York
Bernhard Herzog
Clark C. Evans
Donovan Baarda
Fred L. Drake, Jr.
Fredrik Lundh
Guido van Rossum
Gustavo Carneiro
Ian Bicking
James Y Knight
Joe Smith
M.-A. Lemburg
Nick Coghlan
Paul Moore
Phillip J. Eby
skip＠pobox.com
Talin
Terry Reedy
Thomas Heller
Thomas Wouters
Wolfgang Langner

Dropping __init__.py requirement for subpackages

Donovan Baarda

Wolfgang Langner

Joe Smith

Talin

tags

participants (26)

Dropping init.py requirement for subpackages