PyPy, Jython, & IronPython: Enum convenience function and pickleablity
In order for the Enum convenience function to be pickleable, we have this line of code in the metaclass: enum_class.__module__ = sys._getframe(1).f_globals['__name__'] This works fine for Cpython, but what about the others? -- ~Ethan~
2013/5/2 Ethan Furman <ethan@stoneleaf.us>:
In order for the Enum convenience function to be pickleable, we have this line of code in the metaclass:
enum_class.__module__ = sys._getframe(1).f_globals['__name__']
This works fine for Cpython, but what about the others?
Regardless of that, perhaps we should come up with better ways to do this. -- Regards, Benjamin
On Thu, 2 May 2013 15:48:14 -0400 Benjamin Peterson <benjamin@python.org> wrote:
2013/5/2 Ethan Furman <ethan@stoneleaf.us>:
In order for the Enum convenience function to be pickleable, we have this line of code in the metaclass:
enum_class.__module__ = sys._getframe(1).f_globals['__name__']
This works fine for Cpython, but what about the others?
Regardless of that, perhaps we should come up with better ways to do this.
Two things that were suggested in private: 1) ask users to pass the module name to the convenience function explicitly (i.e. pass "seasonmodule.Season" instead of "Season" as the class "name"). Guido doesn't like it :-) 2) dicth the "convenience function" and replace it with a regular class-based syntax. Ethan doesn't like it :-) Regards Antoine.
On Thu, May 2, 2013 at 1:10 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
2013/5/2 Ethan Furman <ethan@stoneleaf.us>:
In order for the Enum convenience function to be pickleable, we have
line of code in the metaclass:
enum_class.__module__ = sys._getframe(1).f_globals['__name__']
This works fine for Cpython, but what about the others?
Regardless of that, perhaps we should come up with better ways to do
On Thu, 2 May 2013 15:48:14 -0400 Benjamin Peterson <benjamin@python.org> wrote: this this.
Two things that were suggested in private:
1) ask users to pass the module name to the convenience function explicitly (i.e. pass "seasonmodule.Season" instead of "Season" as the class "name"). Guido doesn't like it :-)
2) dicth the "convenience function" and replace it with a regular class-based syntax. Ethan doesn't like it :-)
Re (2), we already have the hack in stdlib in namedtuple, so not allowing it for an enum is a step backwards. If sys._getframe(1).f_globals['__name__'] feels hackish, maybe it can be shortened to a convenience function the stdlib provides? Are there conditions where it doesn't produce what we expect from it? The point at which the enumeration is defined resides in *some* module, no? Eli
On Thu, 2 May 2013 13:15:00 -0700 Eli Bendersky <eliben@gmail.com> wrote:
Two things that were suggested in private:
1) ask users to pass the module name to the convenience function explicitly (i.e. pass "seasonmodule.Season" instead of "Season" as the class "name"). Guido doesn't like it :-)
2) dicth the "convenience function" and replace it with a regular class-based syntax. Ethan doesn't like it :-)
Re (2), we already have the hack in stdlib in namedtuple, so not allowing it for an enum is a step backwards.
That's a fallacy. There is no step backwards if you adopt a class-based syntax, which is just as convenient as the proposed "convenience function". I have a hard time understanding that calling a function to declare a class is suddenly considered "convenient".
If sys._getframe(1).f_globals['__name__'] feels hackish, maybe it can be shortened to a convenience function the stdlib provides?
It's not the notation which is hackish, it's the fact that you are inspecting the frame stack in the hope of getting the right information. What if someone wants to write another convenience function that wraps your convenience function? What if your code is executing from some kind of step-by-step debugger which inserts an additional frame in the call stack? What if someone wants the enum to be nested inside another class (rather than reside at the module top-level)? Regards Antoine.
On Thu, May 2, 2013 at 1:22 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Thu, 2 May 2013 13:15:00 -0700 Eli Bendersky <eliben@gmail.com> wrote:
Two things that were suggested in private:
1) ask users to pass the module name to the convenience function explicitly (i.e. pass "seasonmodule.Season" instead of "Season" as the class "name"). Guido doesn't like it :-)
2) dicth the "convenience function" and replace it with a regular class-based syntax. Ethan doesn't like it :-)
Re (2), we already have the hack in stdlib in namedtuple, so not allowing it for an enum is a step backwards.
That's a fallacy. There is no step backwards if you adopt a class-based syntax, which is just as convenient as the proposed "convenience function". I have a hard time understanding that calling a function to declare a class is suddenly considered "convenient".
If sys._getframe(1).f_globals['__name__'] feels hackish, maybe it can be shortened to a convenience function the stdlib provides?
It's not the notation which is hackish, it's the fact that you are inspecting the frame stack in the hope of getting the right information.
What if someone wants to write another convenience function that wraps your convenience function? What if your code is executing from some kind of step-by-step debugger which inserts an additional frame in the call stack? What if someone wants the enum to be nested inside another class (rather than reside at the module top-level)?
Would nesting the non-convenience Enum in a function or a class allow one to pickle it? I think programmers who want their libraries to be pickle-able already have to be aware of some restrictions about what can and cannot be pickled. Eli
On Thu, 2 May 2013 13:33:21 -0700 Eli Bendersky <eliben@gmail.com> wrote:
On Thu, May 2, 2013 at 1:22 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Thu, 2 May 2013 13:15:00 -0700 Eli Bendersky <eliben@gmail.com> wrote:
Two things that were suggested in private:
1) ask users to pass the module name to the convenience function explicitly (i.e. pass "seasonmodule.Season" instead of "Season" as the class "name"). Guido doesn't like it :-)
2) dicth the "convenience function" and replace it with a regular class-based syntax. Ethan doesn't like it :-)
Re (2), we already have the hack in stdlib in namedtuple, so not allowing it for an enum is a step backwards.
That's a fallacy. There is no step backwards if you adopt a class-based syntax, which is just as convenient as the proposed "convenience function". I have a hard time understanding that calling a function to declare a class is suddenly considered "convenient".
If sys._getframe(1).f_globals['__name__'] feels hackish, maybe it can be shortened to a convenience function the stdlib provides?
It's not the notation which is hackish, it's the fact that you are inspecting the frame stack in the hope of getting the right information.
What if someone wants to write another convenience function that wraps your convenience function? What if your code is executing from some kind of step-by-step debugger which inserts an additional frame in the call stack? What if someone wants the enum to be nested inside another class (rather than reside at the module top-level)?
Would nesting the non-convenience Enum in a function or a class allow one to pickle it?
Once PEP 3154 is implemented (Alexandre is on it :-)), nested classes should be picklable. As for classes inside functions, it sounds quite impossible (how do you instantiate the function namespace without calling the function?). Regards Antoine.
Two things that were suggested in private:
1) ask users to pass the module name to the convenience function explicitly (i.e. pass "seasonmodule.Season" instead of "Season" as
On Thu, 2 May 2013 13:15:00 -0700 Eli Bendersky <eliben@gmail.com> wrote: the
class "name"). Guido doesn't like it :-)
2) dicth the "convenience function" and replace it with a regular class-based syntax. Ethan doesn't like it :-)
Re (2), we already have the hack in stdlib in namedtuple, so not allowing it for an enum is a step backwards.
That's a fallacy. There is no step backwards if you adopt a class-based syntax, which is just as convenient as the proposed "convenience function". I have a hard time understanding that calling a function to declare a class is suddenly considered "convenient".
If sys._getframe(1).f_globals['__name__'] feels hackish, maybe it can be shortened to a convenience function the stdlib provides?
It's not the notation which is hackish, it's the fact that you are inspecting the frame stack in the hope of getting the right information.
What if someone wants to write another convenience function that wraps your convenience function? What if your code is executing from some kind of step-by-step debugger which inserts an additional frame in the call stack? What if someone wants the enum to be nested inside another class (rather than reside at the module top-level)?
Would nesting the non-convenience Enum in a function or a class allow one to pickle it?
Once PEP 3154 is implemented (Alexandre is on it :-)), nested classes should be picklable.
Interesting, I did not know that.
As for classes inside functions, it sounds quite impossible (how do you instantiate the function namespace without calling the function?).
True. Back to my question from before, though - do we have a real technical limitation of having something like inspect.what_module_am_i_now_in() that's supposed to work for all Python code? Eli
On 05/02/2013 01:52 PM, Eli Bendersky wrote:
Back to my question from before, though - do we have a real technical limitation of having something like inspect.what_module_am_i_now_in() that's supposed to work for all Python code?
By which you really mean inspect.what_module_was_I_called_from() ? -- ~Ethan~
On Thu, May 2, 2013 at 2:05 PM, Ethan Furman <ethan@stoneleaf.us> wrote:
On 05/02/2013 01:52 PM, Eli Bendersky wrote:
Back to my question from before, though - do we have a real technical limitation of having something like inspect.what_module_am_i_now_**in() that's supposed to work for all Python code?
By which you really mean inspect.what_module_was_I_called_from() ?
Yes, I guess this is what I meant by "now_in" part. Let's be precise: Animal = Enum('Animal', '...........') The call to Enum is the interesting here. In happens in some library and Animal members can then be passed around. But we really want the module where Enum() was invoked to create Animal in the first place. Eli
On Thu, 2 May 2013 13:52:29 -0700 Eli Bendersky <eliben@gmail.com> wrote:
Back to my question from before, though - do we have a real technical limitation of having something like inspect.what_module_am_i_now_in() that's supposed to work for all Python code?
I already gave an answer (e.g. the debugger case), but you are free to consider it not reasonable :) In any case, I just find the argument for a function-based syntax non-existent compared to a similarly compact class-based syntax. Regards Antoine.
On Thu, May 2, 2013 at 2:10 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Thu, 2 May 2013 13:52:29 -0700 Eli Bendersky <eliben@gmail.com> wrote:
Back to my question from before, though - do we have a real technical limitation of having something like inspect.what_module_am_i_now_in() that's supposed to work for all Python code?
I already gave an answer (e.g. the debugger case), but you are free to consider it not reasonable :)
Sorry, but I do find the argument "let's not have a convenience syntax because enums created with such syntax won't pickle properly from within a debugger" not convincing enough :-) It may be just me though, and I'm open to other opinions. Eli
On Thu, 2 May 2013 14:15:40 -0700 Eli Bendersky <eliben@gmail.com> wrote:
Sorry, but I do find the argument "let's not have a convenience syntax because enums created with such syntax won't pickle properly from within a debugger" not convincing enough :-)
Eli, it would be nice if you stopped with this claim. I'm not advocating "not having a convenience syntax", I'm advocating having a convenience syntax which is *class-based* rather than function-based. Debuggers are beside the point: there are two kinds of "convenience syntax" on the table; one allows pickling by construction, one requires an ugly hack which may not solve all cases (and which may apparently make Jython / IronPython mildly unhappy). Why you insist on ignoring the former and imposing the latter is beyond me. Regards you Antoine.
Eli, it would be nice if you stopped with this claim.
I'm not advocating "not having a convenience syntax", I'm advocating having a convenience syntax which is *class-based* rather than function-based.
Debuggers are beside the point: there are two kinds of "convenience syntax" on the table; one allows pickling by construction, one requires an ugly hack which may not solve all cases (and which may apparently make Jython / IronPython mildly unhappy). Why you insist on ignoring the former and imposing the latter is beyond me.
I'm not trying to belittle our class-based suggestion. I just think there are two separate issues here, and I was focusing on just one of them for now. The one I've been focusing on is how to make the function-based convenience syntax work with pickling in the vast majority of interesting cases. This appears to be possible by using the same pattern used by namedtuple, and even better by encapsulating this pattern formally in stdlib so it stops being a hack (and may actually be useful for other code too). The other issue is your proposal to have a class-based convenience syntax akin to (correct me if I got this wrong): class Animal(Enum): __values__ = 'cat dog' This is obviously a matter of preference (and hence bikeshedding), but this still looks better to me: Animal = Enum('Animal', 'cat dog') It has two advantages: 1. Shorter 2. Parallels namedtuple, which is by now a well known and widely used construct On the other hand, your proposal has the advantage that it allows pickles without hacks in the implementation. Did I sum up the issues fairly? I don't know what to decide here. There's no clear technical merit to decide on one against the other (IMHO!), it's a matter of preference. Hopefully Guido will step in and save us from our misery ;-) Eli
On 3 May 2013 08:00, "Eli Bendersky" <eliben@gmail.com> wrote:
Eli, it would be nice if you stopped with this claim.
I'm not advocating "not having a convenience syntax", I'm advocating having a convenience syntax which is *class-based* rather than function-based.
Debuggers are beside the point: there are two kinds of "convenience syntax" on the table; one allows pickling by construction, one requires an ugly hack which may not solve all cases (and which may apparently make Jython / IronPython mildly unhappy). Why you insist on ignoring the former and imposing the latter is beyond me.
I'm not trying to belittle our class-based suggestion. I just think there
are two separate issues here, and I was focusing on just one of them for now. The one I've been focusing on is how to make the function-based convenience syntax work with pickling in the vast majority of interesting cases. This appears to be possible by using the same pattern used by namedtuple, and even better by encapsulating this pattern formally in stdlib so it stops being a hack (and may actually be useful for other code too).
The other issue is your proposal to have a class-based convenience syntax
akin to (correct me if I got this wrong):
class Animal(Enum): __values__ = 'cat dog'
I would suggest moving the field names into the class header for a class based convenience API: class Animal(Enum, members='cat dog'): pass Cheers, Nick.
This is obviously a matter of preference (and hence bikeshedding), but
this still looks better to me:
Animal = Enum('Animal', 'cat dog')
It has two advantages:
1. Shorter 2. Parallels namedtuple, which is by now a well known and widely used
construct
On the other hand, your proposal has the advantage that it allows pickles
without hacks in the implementation.
Did I sum up the issues fairly?
I don't know what to decide here. There's no clear technical merit to
decide on one against the other (IMHO!), it's a matter of preference. Hopefully Guido will step in and save us from our misery ;-)
Eli
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
On Thu, May 2, 2013 at 4:14 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I would suggest moving the field names into the class header for a class based convenience API:
class Animal(Enum, members='cat dog'): pass
Would you propose the same for namedtuple? -- --Guido van Rossum (python.org/~guido)
On May 03, 2013, at 09:14 AM, Nick Coghlan wrote:
The other issue is your proposal to have a class-based convenience syntax akin to (correct me if I got this wrong):
class Animal(Enum): __values__ = 'cat dog'
I would suggest moving the field names into the class header for a class based convenience API:
class Animal(Enum, members='cat dog'): pass
Wait, what is this trying to solve? "Convenience API" is really a shorthand for "functional API". Two very different use cases that the above suggestion doesn't address. IMHO, it's not worth giving up the functional API for picklability if the technical problems cannot be resolved, especially given we already have the same problem for namedtuples. -Barry
Le Fri, 3 May 2013 09:14:22 +1000, Nick Coghlan <ncoghlan@gmail.com> a écrit :
The other issue is your proposal to have a class-based convenience syntax
akin to (correct me if I got this wrong):
class Animal(Enum): __values__ = 'cat dog'
I would suggest moving the field names into the class header for a class based convenience API:
class Animal(Enum, members='cat dog'): pass
This looks good to me (assuming some people don't like the special attribute scheme). Regards Antoine.
On 03/05/13 18:42, Antoine Pitrou wrote:
Le Fri, 3 May 2013 09:14:22 +1000, Nick Coghlan <ncoghlan@gmail.com> a écrit :
I would suggest moving the field names into the class header for a class based convenience API:
class Animal(Enum, members='cat dog'): pass
This looks good to me (assuming some people don't like the special attribute scheme).
The problem is that this is not an expression, it is a statement. The advantage of the convenience function is not just that it is shorter, but that it is an expression. -- Steven
Le Fri, 03 May 2013 19:40:21 +1000, Steven D'Aprano <steve@pearwood.info> a écrit :
On 03/05/13 18:42, Antoine Pitrou wrote:
Le Fri, 3 May 2013 09:14:22 +1000, Nick Coghlan <ncoghlan@gmail.com> a écrit :
I would suggest moving the field names into the class header for a class based convenience API:
class Animal(Enum, members='cat dog'): pass
This looks good to me (assuming some people don't like the special attribute scheme).
The problem is that this is not an expression, it is a statement. The advantage of the convenience function is not just that it is shorter, but that it is an expression.
What does that change exactly? Regards Antoine.
On May 03, 2013, at 07:40 PM, Steven D'Aprano wrote:
The problem is that this is not an expression, it is a statement. The advantage of the convenience function is not just that it is shorter, but that it is an expression.
Exactly right, but let's stop calling it the "convenience API" and instead call it the "functional API". I probably started the perpetuation of this problem; let's update the PEP. BTW, I made a suggestion elsewhere that the first argument could accept, but not require dotted names in the first argument. If provided, rsplit the string and use the prefix as __module__. If not given, fallback to the _getframe() hack for those implementations where it's available. The same could probably be done to namedtuples. -Barry
On Fri, May 3, 2013 at 9:08 AM, Barry Warsaw <barry@python.org> wrote:
On May 03, 2013, at 07:40 PM, Steven D'Aprano wrote:
The problem is that this is not an expression, it is a statement. The advantage of the convenience function is not just that it is shorter, but that it is an expression.
Exactly right, but let's stop calling it the "convenience API" and instead call it the "functional API". I probably started the perpetuation of this problem; let's update the PEP.
BTW, I made a suggestion elsewhere that the first argument could accept, but not require dotted names in the first argument. If provided, rsplit the string and use the prefix as __module__. If not given, fallback to the _getframe() hack for those implementations where it's available.
The same could probably be done to namedtuples.
All sounds good to me. -- --Guido van Rossum (python.org/~guido)
2013/5/3 Barry Warsaw <barry@python.org>:
On May 03, 2013, at 07:40 PM, Steven D'Aprano wrote:
The problem is that this is not an expression, it is a statement. The advantage of the convenience function is not just that it is shorter, but that it is an expression.
Exactly right, but let's stop calling it the "convenience API" and instead call it the "functional API". I probably started the perpetuation of this problem; let's update the PEP.
BTW, I made a suggestion elsewhere that the first argument could accept, but not require dotted names in the first argument. If provided, rsplit the string and use the prefix as __module__. If not given, fallback to the _getframe() hack for those implementations where it's available.
What about adding simple syntax that allows get rid of those ugly hacks, something like: def name = expression which would be rough equivalent for: name = expression name.__name__ = 'name' name.__module__ = __name__ -- 闇に隠れた黒い力 弱い心を操る
On 5/3/2013 12:08 PM, Barry Warsaw wrote:
Exactly right, but let's stop calling it the "convenience API" and instead call it the "functional API". I probably started the perpetuation of this problem; let's update the PEP.
Please do. To me, a 'convenience function' is something like the timeit functions or subprocess.call that create a class instance, call a method (or two) on the instance, and then discard the instance while returning the result of calling methods. For the common case handled by the function, the implementation via a class with methods is a detail that the user hardly need know about. Using a function interface to create and return a class is something else. -- Terry Jan Reedy
Am 03.05.2013 11:40, schrieb Steven D'Aprano:
On 03/05/13 18:42, Antoine Pitrou wrote:
Le Fri, 3 May 2013 09:14:22 +1000, Nick Coghlan <ncoghlan@gmail.com> a écrit :
I would suggest moving the field names into the class header for a class based convenience API:
class Animal(Enum, members='cat dog'): pass
This looks good to me (assuming some people don't like the special attribute scheme).
The problem is that this is not an expression, it is a statement. The advantage of the convenience function is not just that it is shorter, but that it is an expression.
But using that expression in any form other than NAME = Enum('NAME', ...) will again result in an unpicklable enum, which was the point of this thread. Georg
On 4 May 2013 05:17, "Georg Brandl" <g.brandl@gmx.net> wrote:
Am 03.05.2013 11:40, schrieb Steven D'Aprano:
On 03/05/13 18:42, Antoine Pitrou wrote:
Le Fri, 3 May 2013 09:14:22 +1000, Nick Coghlan <ncoghlan@gmail.com> a écrit :
I would suggest moving the field names into the class header for a
based convenience API:
class Animal(Enum, members='cat dog'): pass
This looks good to me (assuming some people don't like the special attribute scheme).
The problem is that this is not an expression, it is a statement. The advantage of the convenience function is not just that it is shorter, but that it is an expression.
But using that expression in any form other than
NAME = Enum('NAME', ...)
will again result in an unpicklable enum, which was the point of this
class thread. Right, if all we want is a functional API that doesn't support pickling of the resulting class, that's trivial. What I'm after is a convenience API that supports *autonumbering*, as a trivial replacement for code that currently uses "range(n)". A class statement is perfectly acceptable to me for that purpose. Independently of that, I do like the notion of a "types.set_name(cls, dotted_name)" API that alters __name__ and __module__, while leaving __qualname__ alone. Cheers, Nick.
Georg
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
Am 02.05.2013 23:57, schrieb Eli Bendersky:
Eli, it would be nice if you stopped with this claim.
I'm not advocating "not having a convenience syntax", I'm advocating having a convenience syntax which is *class-based* rather than function-based.
Debuggers are beside the point: there are two kinds of "convenience syntax" on the table; one allows pickling by construction, one requires an ugly hack which may not solve all cases (and which may apparently make Jython / IronPython mildly unhappy). Why you insist on ignoring the former and imposing the latter is beyond me.
I'm not trying to belittle our class-based suggestion. I just think there are two separate issues here, and I was focusing on just one of them for now. The one I've been focusing on is how to make the function-based convenience syntax work with pickling in the vast majority of interesting cases. This appears to be possible by using the same pattern used by namedtuple, and even better by encapsulating this pattern formally in stdlib so it stops being a hack (and may actually be useful for other code too).
The other issue is your proposal to have a class-based convenience syntax akin to (correct me if I got this wrong):
class Animal(Enum): __values__ = 'cat dog'
This is obviously a matter of preference (and hence bikeshedding), but this still looks better to me:
Animal = Enum('Animal', 'cat dog')
It has two advantages:
1. Shorter 2. Parallels namedtuple, which is by now a well known and widely used construct
Not to forget 3. Has to specify the class name twice for good measure ;) Georg
Le Thu, 2 May 2013 14:57:35 -0700, Eli Bendersky <eliben@gmail.com> a écrit :
class Animal(Enum): __values__ = 'cat dog'
This is obviously a matter of preference (and hence bikeshedding), but this still looks better to me:
Animal = Enum('Animal', 'cat dog')
It has two advantages:
1. Shorter
You're gaining one line of code. I suppose it's significant if you write ten enums a day, otherwise... ;-)
2. Parallels namedtuple, which is by now a well known and widely used construct
namedtuple is the exception, not the rule. I don't know of another popular type which follows a similar scheme. On the other hand, well-known ORMs (SQLAlchemy, Django ORM) use a class-based syntax despite their declarative nature and the fact that they allow you to set "meta" options (e.g. the name of the reflected table). As an egoistical data point, I always subclass namedtuples, because I minimally want to add a docstring, and sometimes I also want to add behaviour (e.g. alternate constructors, serialization). Which means namedtuple's declarative conciseness is generally lost for me :-) Note that besides ORMs, the proposed __values__ has built-in precedent with __slots__. Regards Antoine.
2013/5/2 Eli Bendersky <eliben@gmail.com>:
On Thu, May 2, 2013 at 1:10 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Thu, 2 May 2013 15:48:14 -0400 Benjamin Peterson <benjamin@python.org> wrote:
2013/5/2 Ethan Furman <ethan@stoneleaf.us>:
In order for the Enum convenience function to be pickleable, we have this line of code in the metaclass:
enum_class.__module__ = sys._getframe(1).f_globals['__name__']
This works fine for Cpython, but what about the others?
Regardless of that, perhaps we should come up with better ways to do this.
Two things that were suggested in private:
1) ask users to pass the module name to the convenience function explicitly (i.e. pass "seasonmodule.Season" instead of "Season" as the class "name"). Guido doesn't like it :-)
2) dicth the "convenience function" and replace it with a regular class-based syntax. Ethan doesn't like it :-)
Re (2), we already have the hack in stdlib in namedtuple, so not allowing it for an enum is a step backwards. If sys._getframe(1).f_globals['__name__'] feels hackish, maybe it can be shortened to a convenience function the stdlib provides? Are there conditions where it doesn't produce what we expect from it? The point at which the enumeration is defined resides in *some* module, no?
I disagree that not allowing code smell to spread is a step backwards. Rather we should realize that this is a common problem and find a proper solution rather than further propogating this hack. -- Regards, Benjamin
Am 02.05.2013 22:10, schrieb Antoine Pitrou:
On Thu, 2 May 2013 15:48:14 -0400 Benjamin Peterson <benjamin@python.org> wrote:
2013/5/2 Ethan Furman <ethan@stoneleaf.us>:
In order for the Enum convenience function to be pickleable, we have this line of code in the metaclass:
enum_class.__module__ = sys._getframe(1).f_globals['__name__']
This works fine for Cpython, but what about the others?
Regardless of that, perhaps we should come up with better ways to do this.
Two things that were suggested in private:
1) ask users to pass the module name to the convenience function explicitly (i.e. pass "seasonmodule.Season" instead of "Season" as the class "name"). Guido doesn't like it :-)
2) dicth the "convenience function" and replace it with a regular class-based syntax. Ethan doesn't like it :-)
5) accept that convenience-created enums have restrictions such as no picklability and point them out in the docs? Georg
On May 02, 2013, at 10:18 PM, Georg Brandl wrote:
5) accept that convenience-created enums have restrictions such as no picklability and point them out in the docs?
That would work fine for me, but ultimately I'm with Guido. I just don't want to have to pass the module name in. -Barry
On Thu, May 2, 2013 at 1:39 PM, Barry Warsaw <barry@python.org> wrote:
On May 02, 2013, at 10:18 PM, Georg Brandl wrote:
5) accept that convenience-created enums have restrictions such as no picklability and point them out in the docs?
That would work fine for me, but ultimately I'm with Guido. I just don't want to have to pass the module name in.
The problem with (5) is this: you use some library that exports an enumeration, and you want to use pickling. Now you depend on the way the library implemented - if it used the convenience API, you can't pickle. If it used the class API, you can. Eli
On Thu, 2 May 2013 13:48:24 -0700 Eli Bendersky <eliben@gmail.com> wrote:
On Thu, May 2, 2013 at 1:39 PM, Barry Warsaw <barry@python.org> wrote:
On May 02, 2013, at 10:18 PM, Georg Brandl wrote:
5) accept that convenience-created enums have restrictions such as no picklability and point them out in the docs?
That would work fine for me, but ultimately I'm with Guido. I just don't want to have to pass the module name in.
The problem with (5) is this: you use some library that exports an enumeration, and you want to use pickling. Now you depend on the way the library implemented - if it used the convenience API, you can't pickle. If it used the class API, you can.
A good reason to ditch the function-based syntax. Regards Antoine.
On May 02, 2013, at 10:57 PM, Antoine Pitrou wrote:
On Thu, 2 May 2013 13:48:24 -0700
The problem with (5) is this: you use some library that exports an enumeration, and you want to use pickling. Now you depend on the way the library implemented - if it used the convenience API, you can't pickle. If it used the class API, you can.
A good reason to ditch the function-based syntax.
Why? Not everything is picklable. Oh well. -Barry
On Thu, 2 May 2013 14:16:34 -0700 Barry Warsaw <barry@python.org> wrote:
On May 02, 2013, at 10:57 PM, Antoine Pitrou wrote:
On Thu, 2 May 2013 13:48:24 -0700
The problem with (5) is this: you use some library that exports an enumeration, and you want to use pickling. Now you depend on the way the library implemented - if it used the convenience API, you can't pickle. If it used the class API, you can.
A good reason to ditch the function-based syntax.
Why? Not everything is picklable. Oh well.
Then why insist on the _getframe hack? You are losing me: are you bothered by picklability or not? ;-) If you are not, then fine, let's just make the function-based version *documentedly* unpicklable, and move along. Regards Antoine.
On Thu, May 2, 2013 at 12:07 PM, Ethan Furman <ethan@stoneleaf.us> wrote:
In order for the Enum convenience function to be pickleable, we have this line of code in the metaclass:
enum_class.__module__ = sys._getframe(1).f_globals['__name__']
This works fine for Cpython, but what about the others? This should work for Jython, but I can't say I like it. I believe IronPython has a sort of speedup mode that disallows the use of _getframe, and I'd like to add this to Jython someday.
-Frank
On Thu, May 2, 2013 at 1:18 PM, fwierzbicki@gmail.com <fwierzbicki@gmail.com> wrote:
On Thu, May 2, 2013 at 12:07 PM, Ethan Furman <ethan@stoneleaf.us> wrote:
In order for the Enum convenience function to be pickleable, we have this line of code in the metaclass:
enum_class.__module__ = sys._getframe(1).f_globals['__name__']
This works fine for Cpython, but what about the others? This should work for Jython, but I can't say I like it. I believe IronPython has a sort of speedup mode that disallows the use of _getframe, and I'd like to add this to Jython someday.
This particular function is typically only called at module load time, so speeding it up isn't worth it. FWIW, as Eli pointed out, namedtuple() does the same thing (since Python 2.6), so we'll just copy that code (refactoring it doesn't have to hold up the PEP). The only other alternative I find acceptable is not to have the convenience API at all. That's Eli's call. [Eli]
Would nesting the non-convenience Enum in a function or a class allow one to pickle it? I think programmers who want their libraries to be pickle-able already have to be aware of some restrictions about what can and cannot be pickled.
Apparently it hasn't been a problem for namedtuple. Calling namedtuple() or Enum() in another function is similar to a class statement inside a function -- the resulting class isn't picklable. (But from this, don't conclude that it's not important for namedtuple() or Enum() to return a picklable class. It is important. It is just not important to try to make it work when they are called through some other wrapper -- there's just not much use for such a pattern.) -- --Guido van Rossum (python.org/~guido)
On Thu, May 2, 2013 at 1:39 PM, Guido van Rossum <guido@python.org> wrote:
On Thu, May 2, 2013 at 12:07 PM, Ethan Furman <ethan@stoneleaf.us> wrote:
In order for the Enum convenience function to be pickleable, we have
On Thu, May 2, 2013 at 1:18 PM, fwierzbicki@gmail.com <fwierzbicki@gmail.com> wrote: this
line of code in the metaclass:
enum_class.__module__ = sys._getframe(1).f_globals['__name__']
This works fine for Cpython, but what about the others? This should work for Jython, but I can't say I like it. I believe IronPython has a sort of speedup mode that disallows the use of _getframe, and I'd like to add this to Jython someday.
This particular function is typically only called at module load time, so speeding it up isn't worth it.
FWIW, as Eli pointed out, namedtuple() does the same thing (since Python 2.6), so we'll just copy that code (refactoring it doesn't have to hold up the PEP). The only other alternative I find acceptable is not to have the convenience API at all. That's Eli's call.
I really prefer having the convenience API and acknowledging that it has some limitations (i.e. picking enums that were created with the convenience API and are nested in classes).
[Eli]
Would nesting the non-convenience Enum in a function or a class allow one to pickle it? I think programmers who want their libraries to be pickle-able already have to be aware of some restrictions about what can and cannot be pickled.
Apparently it hasn't been a problem for namedtuple. Calling namedtuple() or Enum() in another function is similar to a class statement inside a function -- the resulting class isn't picklable.
(But from this, don't conclude that it's not important for namedtuple() or Enum() to return a picklable class. It is important. It is just not important to try to make it work when they are called through some other wrapper -- there's just not much use for such a pattern.)
I agree. Eli
2013/5/2 Guido van Rossum <guido@python.org>
On Thu, May 2, 2013 at 12:07 PM, Ethan Furman <ethan@stoneleaf.us> wrote:
In order for the Enum convenience function to be pickleable, we have
On Thu, May 2, 2013 at 1:18 PM, fwierzbicki@gmail.com <fwierzbicki@gmail.com> wrote: this
line of code in the metaclass:
enum_class.__module__ = sys._getframe(1).f_globals['__name__']
This works fine for Cpython, but what about the others? This should work for Jython, but I can't say I like it. I believe IronPython has a sort of speedup mode that disallows the use of _getframe, and I'd like to add this to Jython someday.
This particular function is typically only called at module load time, so speeding it up isn't worth it.
It works fine on PyPy as well. It probably also kills any JIT optimization, but it's not an issue since classes are not usually created in tight loops. -- Amaury Forgeot d'Arc
On Thu, May 2, 2013 at 1:18 PM, fwierzbicki@gmail.com <fwierzbicki@gmail.com> wrote:
On Thu, May 2, 2013 at 12:07 PM, Ethan Furman <ethan@stoneleaf.us> wrote:
In order for the Enum convenience function to be pickleable, we have this line of code in the metaclass:
enum_class.__module__ = sys._getframe(1).f_globals['__name__']
This works fine for Cpython, but what about the others? This should work for Jython, but I can't say I like it. I believe IronPython has a sort of speedup mode that disallows the use of _getframe, and I'd like to add this to Jython someday.
It's not just a "speedup mode", it's the default. IronPython requires frames to explicitly enabled because tracking them is about a 10% performance hit (or so Dino told me once upon a time). If you must use it, please copy the code block from namedtuple that ignores it on IronPython. - Jeff
On Thu, May 2, 2013 at 9:07 PM, Ethan Furman <ethan@stoneleaf.us> wrote:
In order for the Enum convenience function to be pickleable, we have this line of code in the metaclass:
enum_class.__module__ = sys._getframe(1).f_globals['__name__']
This works fine for Cpython, but what about the others?
It's ugly as hell, but it's not a performance problem for PyPy, since this is executed at module load time (you probably won't jit that code anyway) Cheers, fijal
Ethan Furman, 02.05.2013 21:07:
In order for the Enum convenience function to be pickleable, we have this line of code in the metaclass:
enum_class.__module__ = sys._getframe(1).f_globals['__name__']
What a hack. And fragile, too.
This works fine for Cpython, but what about the others?
This doesn't work when used from Cython compiled code due to the lack of frames. They are only created for exception tracebacks and not for normal code by default (just for profiling, coverage etc.). My guess is that no-one noticed the problem for namedtuples so far because using them is still uncommon enough in general, let alone pickling them, and the module name hack only leads to an error when someone tries to pickle such an object. I think that this will be more of a problem for enums than for namedtuples, because enums are more likely to appear in data structures that people want to pickle. The most simple work-around seems to be this, once you know about it: """ ttuple = namedtuple('ttuple', 'a b c') ttuple.__module__ = __name__ # enable pickle support """ Not any worse than the hack above, IMHO, but at least guaranteed to work. For enums, a regular class based declaration can easily avoid this hack, so my vote is for getting rid of the "convenience" API before it starts doing any harm. Or document it explicitly as generating unpicklable objects, as Antoine suggests. Stefan
participants (16)
-
Amaury Forgeot d'Arc -
Antoine Pitrou -
Barry Warsaw -
Benjamin Peterson -
Eli Bendersky -
Ethan Furman -
fwierzbicki＠gmail.com -
Georg Brandl -
Guido van Rossum -
Jeff Hardy -
Maciej Fijalkowski -
Nick Coghlan -
Piotr Duda -
Stefan Behnel -
Steven D'Aprano -
Terry Jan Reedy