<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, May 7, 2013 at 8:03 AM, Nick Coghlan <span dir="ltr"><<a href="mailto:ncoghlan@gmail.com" target="_blank">ncoghlan@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On Tue, May 7, 2013 at 11:34 PM, Eli Bendersky <<a href="mailto:eliben@gmail.com">eliben@gmail.com</a>> wrote:<br>
> One of the contended issues with PEP 435 on which Guido pronounced was the<br>
> functional API, that allows created enumerations dynamically in a manner<br>
> similar to namedtuple:<br>
><br>
> Color = Enum('Color', 'red blue green')<br>
><br>
> The biggest complaint reported against this API is interaction with pickle.<br>
> As promised, I want to discuss here how we're going to address this concern.<br>
><br>
> At this point, the pickle docs say that module-top-level classes can be<br>
> pickled. This obviously works for the normal Enum classes, but is a problem<br>
> with the functional API because the class is created dynamically and has no<br>
> __module__.<br>
><br>
> To solve this, the reference implementation is used the same approach as<br>
> namedtuple (*). In the metaclass's __new__ (this is an excerpt, the real<br>
> code has some safeguards):<br>
><br>
> module_name = sys._getframe(1).f_globals['__name__']<br>
> enum_class.__module__ = module_name<br>
><br>
> According to an earlier discussion, this is works on CPython, PyPy and<br>
> Jython, but not on IronPython. The alternative that works everywhere is to<br>
> define the Enum like this:<br>
><br>
> Color = Enum('the_module.Color', 'red blue green')<br>
><br>
> The reference implementation supports this as well.<br>
><br>
> Some points for discussion:<br>
><br>
> 1) We can say that using the functional API when pickling can happen is not<br>
> recommended, but maybe a better way would be to just explain the way things<br>
> are and let users decide?<br>
<br>
</div></div>It's probably worth creating a section in the pickle docs and<br>
explaining the vagaries of naming things and the dependency on knowing<br>
the module name. The issue comes up with defining classes in __main__<br>
and when implementing pseudo-modules as well (see PEP 395).<br>
<div><br></div></blockquote><div><br></div><div>Any pickle-expert volunteers to do this? I guess we can start by creating a documentation issue.<br></div><div><br> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">
> 2) namedtuple should also support the fully qualified name syntax. If this<br>
> is agreed upon, I can create an issue.<br>
<br>
</div>Yes, I think that part should be done.<br></blockquote><div><br><br></div><div>OK, I'll create an issue.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im"><br>
> 3) Antoine mentioned that work is being done in 3.4 to enable pickling of<br>
> nested classes (<a href="http://www.python.org/dev/peps/pep-3154/" target="_blank">http://www.python.org/dev/peps/pep-3154/</a>). If that gets<br>
> implemented, I don't see a reason why Enum and namedtuple can't be adjusted<br>
> to find the __qualname__ of the class they're internal to. Am I missing<br>
> something?<br>
<br>
</div>The class based form should still work (assuming only classes are<br>
involved), the stack inspection will likely fail.<br>
</blockquote><div><br></div><div>I can probably be made to work with a bit more effort than the current "hack", but I don't see why it wouldn't be doable.<br></div><div><br> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">
> 4) Using _getframe(N) here seems like an overkill to me.<br>
<br>
</div>It's not just overkill, it's fragile - it only works if you call the<br>
constructor directly. If you use a convenience function in a utility<br>
module, it will try to load your pickles from there rather than<br>
wherever you bound the name.<br></blockquote><div><br></div><div>In theory you can climb the frame stack until the desired place, but this is specifically what my proposal of adding a function tries to avoid.</div><div><br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im"><br>
> What we really need<br>
> is just the module in which the current execution currently is (i.e. the<br>
> metaclass's __new__ in our case). Would it make sense to add a new function<br>
> somewhere in the stdlib of 3.4 (in sys or inspect or ...) that just provides<br>
> the current module name? It seems that all Pythons should be able to easily<br>
> provide it, it's certainly a very small subset of the functionality provided<br>
> by walking the callframe stack. This function can then be used for build<br>
> fully qualified names for pickling of Enum and namedtuple. Moreover, it can<br>
> be general even more widely - dynamic class building is quite common in<br>
> Python code, and as Nick mentioned somewhere earlier, the extra power of<br>
> metaclasses in the recent 3.x's will probably make it even more common.<br>
<br>
</div>Yes, I've been thinking along these lines myself, although in a<br>
slightly more expanded form that also touches on the issues that<br>
stalled PEP 406 (the import engine API that tries to better<br>
encapsulate the import state). It may also potentially address some<br>
issues with initialisation of C extensions (I don't remember the exact<br>
details off the top of my head, but there's some info we want to get<br>
from the import machinery to modules initialised from Cython, but the<br>
loader API and the C module initialisation API both get in the way).<br>
<br>
Specifically, what I'm talking about is some kind of implicit context<br>
similar to the approach the decimal module uses to control operations<br>
on Decimal instances. In this case, what we're trying to track is the<br>
"active module", either __main__ (if the code has been triggered<br>
directly through an operation in that module), or else the module<br>
currently being imported (if the import machinery has been invoked).<br>
<br>
The bare minimum would just be to store the __name__ (using<br>
sys.modules to get access to the full module if needed) in a way that<br>
adequately handles nested, circular and threaded imports, but there<br>
may be a case for tracking a richer ModuleContext object instead.<br>
<br>
However, there's also a separate question of whether implicitly<br>
tracking the active module is really what we want. Do we want that, or<br>
is what we actually want the ability to define an arbitrary "naming<br>
context" in order to use functional APIs to construct classes without<br>
losing the pickle integration of class statements?<br>
<br>
What if there was a variant of the class statement that bound the<br>
result of a function call rather than using the normal syntax:<br>
<br>
class Animal from enum.Enum(members="dog cat bear")<br>
<br>
And it was only class statements in that form which manipulated the<br>
naming context? (you could also use the def keyword rather than class)<br>
<br>
Either form would essentially be an ordinary assignment statement,<br>
*except* that they would manipulate the naming context to record the<br>
name being bound *and* relevant details of the active module.<br>
<br>
Regardless, I think the question is not really well enough defined to<br>
be a topic for python-dev, even though it came up in a python-dev<br>
discussion - it's more python-ideas territory.<br></blockquote><div><br></div><div>Wait... I agree that having a special syntax for this is a novel idea that's not well defined and can be discussed on python-ideas. But the utility function I was mentioning is a pretty simple idea, and it's well defined. It can be very useful in contexts where code is created dynamically, by removing the amount of explicit-frame-walking hacks.<br>
<br>Eli<br><br></div><div><br><br> </div></div></div></div>