I was just testing Zope with the latest CVS python and ran into trouble with the pickle module. The module has grown an __all__ attribute: __all__ = ["PickleError", "PicklingError", "UnpicklingError", "Pickler", "Unpickler", "dump", "dumps", "load", "loads"] This definition excludes a lot of other names defined at the module level, like all of the constants for the pickle format, e.g. MARK, STOP, POP, PERSID, etc. It also excludes format_version and compatible_formats. I don't understand why these names were excluded from __all__. The Zope code uses "from pickle import *" and writes a custom pickler extension. It needs to have access to these names to be compatible, and I can't think of a good reason to forbid it. What's the right solution? Zap the __all__ attribute; the namespace pollution that results is fairly small (marshal, sys, struct, the contents of tupes). Make __all__ a really long list? I wonder how much breakage we should impose on people who use "from ... import *" for Python 2.1. As you know, I was an early advocate of the it's-sloppy-so-let-em-suffer philosophy, but I have learned the error of my ways. I worry that people will be unhappy with __all__ if other modules suffer from similar code breakage. Has __all__ been described by a PEP? If so, it ought to be posted to c.l.py for discussion. If not, we should probably write a short PEP. It would probably be a page of text, but it would help clarify that confusion that persists about what __all__ is for and what its consequences are. Jeremy
[Jeremy Hylton]
... Has __all__ been described by a PEP?
No. IIRC, it popped up when Guido approved of a bulletproof __exports__ patch, and subsequent complaints revealed that was controversial. Then __all__ somehow made it in without opposition, in analogy with the special __all__ attribute of __init__.py files (which doesn't appear to have made it into the Lang or Lib refs, but is documented in Guido's package essay on python.org, and in the Tutorial(?!)).
... If not, we should probably write a short PEP. It would probably be a page of text, but it would help clarify that confusion that persists about what __all__ is for and what its consequences are.
I agree, but if someone can make time for that I'd much rather see Guido's package essay folded into the Lang Ref first. Packages have been part of the language since 1.5 ...
Jeremy Hylton wrote:
I was just testing Zope with the latest CVS python and ran into trouble with the pickle module.
The module has grown an __all__ attribute:
__all__ = ["PickleError", "PicklingError", "UnpicklingError", "Pickler", "Unpickler", "dump", "dumps", "load", "loads"]
This definition excludes a lot of other names defined at the module level, like all of the constants for the pickle format, e.g. MARK, STOP, POP, PERSID, etc. It also excludes format_version and compatible_formats.
I don't understand why these names were excluded from __all__. The Zope code uses "from pickle import *" and writes a custom pickler extension. It needs to have access to these names to be compatible, and I can't think of a good reason to forbid it.
I guess it was a simple oversight. Why not add the constants to the above list ?!
What's the right solution? Zap the __all__ attribute; the namespace pollution that results is fairly small (marshal, sys, struct, the contents of tupes). Make __all__ a really long list?
The latter, I guess. Some lambda hackery ought to fix this elegantly.
I wonder how much breakage we should impose on people who use "from ... import *" for Python 2.1. As you know, I was an early advocate of the it's-sloppy-so-let-em-suffer philosophy, but I have learned the error of my ways. I worry that people will be unhappy with __all__ if other modules suffer from similar code breakage.
IMHO, we should try to get this right now, rather than later. The betas will get enough testing to reduce the breakage below the noise level. If there's still serious breakage somewhere, then patches will be simple: just comment out the __all__ definition -- even newbies will be able to do this (and feel great about it ;-). Besides, the __all__ attribute is a good way to define a module API and certainly can be put to good use in future Python versions, e.g. by declaring those module attribute read-only and pre-fetching them into function locals...
Has __all__ been described by a PEP? If so, it ought to be posted to c.l.py for discussion. If not, we should probably write a short PEP. It would probably be a page of text, but it would help clarify that confusion that persists about what __all__ is for and what its consequences are.
No, there's no PEP for it. The reason is that __all__ has been in existence for quite a few years already. Previously it was just used for packages -- the patch just extended it's scope to simple modules. It is documented in the tutorial and the API docs, plus in Guido's essays. -- Marc-Andre Lemburg ______________________________________________________________________ Company: http://www.egenix.com/ Consulting: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
Jeremy> I was just testing Zope with the latest CVS python and ran into Jeremy> trouble with the pickle module. Jeremy> The module has grown an __all__ attribute: Jeremy> __all__ = ["PickleError", "PicklingError", "UnpicklingError", "Pickler", Jeremy> "Unpickler", "dump", "dumps", "load", "loads"] Jeremy> This definition excludes a lot of other names defined at the Jeremy> module level, like all of the constants for the pickle format, Jeremy> e.g. MARK, STOP, POP, PERSID, etc. It also excludes Jeremy> format_version and compatible_formats. In deciding what to include in __all__ up to this point I have only had my personal experience with the modules and the documentation to help me decide what to include. My initial assumption was that undocumented module-level constants were not to be exported. I just added the following to my version of pickle: __all__.extend([x for x in dir() if re.match("[A-Z][A-Z0-9_]*$",x)]) That seems to catch all the defined constants. Let me know if that's sufficient in this case. Skip
[Skip Montanaro]
In deciding what to include in __all__ up to this point I have only had my personal experience with the modules and the documentation to help me decide what to include. My initial assumption was that undocumented module-level constants were not to be exported.
And it's been a very educational exercise! Thank you for pursuing it. The fact is we often don't know what authors intended to export, and it's Good to try to make that explicit. I'm still not sure I've got any use for __all__, though <wink>. sure-"a-problem"-has-been-identified-but-not-sure-the-solution- has-been-ly y'rs - tim
Tim> I'm still not sure I've got any use for __all__, though <wink>. That may be true. I think the canonical case that is being defended against is a module-level symbol in one module obscuring a builtin, e.g.: # a.py def list(s): return s # b.py from a import * ... l = list(('a','b','c')) I suspect in the long-run there's a better way to accomplish this than adding __all__ to most Python modules, perhaps pylint. Which reminds me... I did write something once upon a time to catch symbols that hide builtins, only at more than the module level: http://musi-cal.mojam.com/~skip/python/hiding.py Skip
participants (4)
-
Jeremy Hylton -
M.-A. Lemburg -
Skip Montanaro -
Tim Peters