PyPy, Jython, & IronPython: Enum convenience function and pickleablity
In order for the Enum convenience function to be pickleable, we have this line of code in the metaclass: enum_class.__module__ = sys._getframe(1).f_globals['__name__'] This works fine for Cpython, but what about the others? -- ~Ethan~
On Thu, 2 May 2013 15:48:14 -0400 Benjamin Peterson <benjamin@python.org> wrote:
Two things that were suggested in private: 1) ask users to pass the module name to the convenience function explicitly (i.e. pass "seasonmodule.Season" instead of "Season" as the class "name"). Guido doesn't like it :-) 2) dicth the "convenience function" and replace it with a regular class-based syntax. Ethan doesn't like it :-) Regards Antoine.
On Thu, May 2, 2013 at 1:10 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Re (2), we already have the hack in stdlib in namedtuple, so not allowing it for an enum is a step backwards. If sys._getframe(1).f_globals['__name__'] feels hackish, maybe it can be shortened to a convenience function the stdlib provides? Are there conditions where it doesn't produce what we expect from it? The point at which the enumeration is defined resides in *some* module, no? Eli
On Thu, 2 May 2013 13:15:00 -0700 Eli Bendersky <eliben@gmail.com> wrote:
That's a fallacy. There is no step backwards if you adopt a class-based syntax, which is just as convenient as the proposed "convenience function". I have a hard time understanding that calling a function to declare a class is suddenly considered "convenient".
It's not the notation which is hackish, it's the fact that you are inspecting the frame stack in the hope of getting the right information. What if someone wants to write another convenience function that wraps your convenience function? What if your code is executing from some kind of step-by-step debugger which inserts an additional frame in the call stack? What if someone wants the enum to be nested inside another class (rather than reside at the module top-level)? Regards Antoine.
On Thu, May 2, 2013 at 1:22 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Would nesting the non-convenience Enum in a function or a class allow one to pickle it? I think programmers who want their libraries to be pickle-able already have to be aware of some restrictions about what can and cannot be pickled. Eli
On Thu, 2 May 2013 13:33:21 -0700 Eli Bendersky <eliben@gmail.com> wrote:
Once PEP 3154 is implemented (Alexandre is on it :-)), nested classes should be picklable. As for classes inside functions, it sounds quite impossible (how do you instantiate the function namespace without calling the function?). Regards Antoine.
Interesting, I did not know that.
True. Back to my question from before, though - do we have a real technical limitation of having something like inspect.what_module_am_i_now_in() that's supposed to work for all Python code? Eli
On 05/02/2013 01:52 PM, Eli Bendersky wrote:
Back to my question from before, though - do we have a real technical limitation of having something like inspect.what_module_am_i_now_in() that's supposed to work for all Python code?
By which you really mean inspect.what_module_was_I_called_from() ? -- ~Ethan~
On Thu, May 2, 2013 at 2:05 PM, Ethan Furman <ethan@stoneleaf.us> wrote:
Yes, I guess this is what I meant by "now_in" part. Let's be precise: Animal = Enum('Animal', '...........') The call to Enum is the interesting here. In happens in some library and Animal members can then be passed around. But we really want the module where Enum() was invoked to create Animal in the first place. Eli
On Thu, 2 May 2013 13:52:29 -0700 Eli Bendersky <eliben@gmail.com> wrote:
I already gave an answer (e.g. the debugger case), but you are free to consider it not reasonable :) In any case, I just find the argument for a function-based syntax non-existent compared to a similarly compact class-based syntax. Regards Antoine.
On Thu, May 2, 2013 at 2:10 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Sorry, but I do find the argument "let's not have a convenience syntax because enums created with such syntax won't pickle properly from within a debugger" not convincing enough :-) It may be just me though, and I'm open to other opinions. Eli
On Thu, 2 May 2013 14:15:40 -0700 Eli Bendersky <eliben@gmail.com> wrote:
Eli, it would be nice if you stopped with this claim. I'm not advocating "not having a convenience syntax", I'm advocating having a convenience syntax which is *class-based* rather than function-based. Debuggers are beside the point: there are two kinds of "convenience syntax" on the table; one allows pickling by construction, one requires an ugly hack which may not solve all cases (and which may apparently make Jython / IronPython mildly unhappy). Why you insist on ignoring the former and imposing the latter is beyond me. Regards you Antoine.
Eli, it would be nice if you stopped with this claim.
I'm not trying to belittle our class-based suggestion. I just think there are two separate issues here, and I was focusing on just one of them for now. The one I've been focusing on is how to make the function-based convenience syntax work with pickling in the vast majority of interesting cases. This appears to be possible by using the same pattern used by namedtuple, and even better by encapsulating this pattern formally in stdlib so it stops being a hack (and may actually be useful for other code too). The other issue is your proposal to have a class-based convenience syntax akin to (correct me if I got this wrong): class Animal(Enum): __values__ = 'cat dog' This is obviously a matter of preference (and hence bikeshedding), but this still looks better to me: Animal = Enum('Animal', 'cat dog') It has two advantages: 1. Shorter 2. Parallels namedtuple, which is by now a well known and widely used construct On the other hand, your proposal has the advantage that it allows pickles without hacks in the implementation. Did I sum up the issues fairly? I don't know what to decide here. There's no clear technical merit to decide on one against the other (IMHO!), it's a matter of preference. Hopefully Guido will step in and save us from our misery ;-) Eli
On 3 May 2013 08:00, "Eli Bendersky" <eliben@gmail.com> wrote:
are two separate issues here, and I was focusing on just one of them for now. The one I've been focusing on is how to make the function-based convenience syntax work with pickling in the vast majority of interesting cases. This appears to be possible by using the same pattern used by namedtuple, and even better by encapsulating this pattern formally in stdlib so it stops being a hack (and may actually be useful for other code too).
The other issue is your proposal to have a class-based convenience syntax
akin to (correct me if I got this wrong):
class Animal(Enum): __values__ = 'cat dog'
I would suggest moving the field names into the class header for a class based convenience API: class Animal(Enum, members='cat dog'): pass Cheers, Nick.
This is obviously a matter of preference (and hence bikeshedding), but
this still looks better to me:
construct
On the other hand, your proposal has the advantage that it allows pickles
without hacks in the implementation.
Did I sum up the issues fairly?
I don't know what to decide here. There's no clear technical merit to
decide on one against the other (IMHO!), it's a matter of preference. Hopefully Guido will step in and save us from our misery ;-)
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
On May 03, 2013, at 09:14 AM, Nick Coghlan wrote:
Wait, what is this trying to solve? "Convenience API" is really a shorthand for "functional API". Two very different use cases that the above suggestion doesn't address. IMHO, it's not worth giving up the functional API for picklability if the technical problems cannot be resolved, especially given we already have the same problem for namedtuples. -Barry
On 03/05/13 18:42, Antoine Pitrou wrote:
Le Fri, 3 May 2013 09:14:22 +1000, Nick Coghlan <ncoghlan@gmail.com> a écrit :
The problem is that this is not an expression, it is a statement. The advantage of the convenience function is not just that it is shorter, but that it is an expression. -- Steven
On May 03, 2013, at 07:40 PM, Steven D'Aprano wrote:
Exactly right, but let's stop calling it the "convenience API" and instead call it the "functional API". I probably started the perpetuation of this problem; let's update the PEP. BTW, I made a suggestion elsewhere that the first argument could accept, but not require dotted names in the first argument. If provided, rsplit the string and use the prefix as __module__. If not given, fallback to the _getframe() hack for those implementations where it's available. The same could probably be done to namedtuples. -Barry
2013/5/3 Barry Warsaw <barry@python.org>:
What about adding simple syntax that allows get rid of those ugly hacks, something like: def name = expression which would be rough equivalent for: name = expression name.__name__ = 'name' name.__module__ = __name__ -- 闇に隠れた黒い力 弱い心を操る
On 5/3/2013 12:08 PM, Barry Warsaw wrote:
Please do. To me, a 'convenience function' is something like the timeit functions or subprocess.call that create a class instance, call a method (or two) on the instance, and then discard the instance while returning the result of calling methods. For the common case handled by the function, the implementation via a class with methods is a detail that the user hardly need know about. Using a function interface to create and return a class is something else. -- Terry Jan Reedy
On 4 May 2013 05:17, "Georg Brandl" <g.brandl@gmx.net> wrote:
class thread. Right, if all we want is a functional API that doesn't support pickling of the resulting class, that's trivial. What I'm after is a convenience API that supports *autonumbering*, as a trivial replacement for code that currently uses "range(n)". A class statement is perfectly acceptable to me for that purpose. Independently of that, I do like the notion of a "types.set_name(cls, dotted_name)" API that alters __name__ and __module__, while leaving __qualname__ alone. Cheers, Nick.
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
Le Thu, 2 May 2013 14:57:35 -0700, Eli Bendersky <eliben@gmail.com> a écrit :
You're gaining one line of code. I suppose it's significant if you write ten enums a day, otherwise... ;-)
2. Parallels namedtuple, which is by now a well known and widely used construct
namedtuple is the exception, not the rule. I don't know of another popular type which follows a similar scheme. On the other hand, well-known ORMs (SQLAlchemy, Django ORM) use a class-based syntax despite their declarative nature and the fact that they allow you to set "meta" options (e.g. the name of the reflected table). As an egoistical data point, I always subclass namedtuples, because I minimally want to add a docstring, and sometimes I also want to add behaviour (e.g. alternate constructors, serialization). Which means namedtuple's declarative conciseness is generally lost for me :-) Note that besides ORMs, the proposed __values__ has built-in precedent with __slots__. Regards Antoine.
2013/5/2 Eli Bendersky <eliben@gmail.com>:
I disagree that not allowing code smell to spread is a step backwards. Rather we should realize that this is a common problem and find a proper solution rather than further propogating this hack. -- Regards, Benjamin
On May 02, 2013, at 10:18 PM, Georg Brandl wrote:
5) accept that convenience-created enums have restrictions such as no picklability and point them out in the docs?
That would work fine for me, but ultimately I'm with Guido. I just don't want to have to pass the module name in. -Barry
On Thu, May 2, 2013 at 1:39 PM, Barry Warsaw <barry@python.org> wrote:
The problem with (5) is this: you use some library that exports an enumeration, and you want to use pickling. Now you depend on the way the library implemented - if it used the convenience API, you can't pickle. If it used the class API, you can. Eli
On Thu, 2 May 2013 14:16:34 -0700 Barry Warsaw <barry@python.org> wrote:
Then why insist on the _getframe hack? You are losing me: are you bothered by picklability or not? ;-) If you are not, then fine, let's just make the function-based version *documentedly* unpicklable, and move along. Regards Antoine.
On Thu, May 2, 2013 at 1:18 PM, fwierzbicki@gmail.com <fwierzbicki@gmail.com> wrote:
This particular function is typically only called at module load time, so speeding it up isn't worth it. FWIW, as Eli pointed out, namedtuple() does the same thing (since Python 2.6), so we'll just copy that code (refactoring it doesn't have to hold up the PEP). The only other alternative I find acceptable is not to have the convenience API at all. That's Eli's call. [Eli]
Apparently it hasn't been a problem for namedtuple. Calling namedtuple() or Enum() in another function is similar to a class statement inside a function -- the resulting class isn't picklable. (But from this, don't conclude that it's not important for namedtuple() or Enum() to return a picklable class. It is important. It is just not important to try to make it work when they are called through some other wrapper -- there's just not much use for such a pattern.) -- --Guido van Rossum (python.org/~guido)
On Thu, May 2, 2013 at 1:39 PM, Guido van Rossum <guido@python.org> wrote:
I really prefer having the convenience API and acknowledging that it has some limitations (i.e. picking enums that were created with the convenience API and are nested in classes).
I agree. Eli
2013/5/2 Guido van Rossum <guido@python.org>
It works fine on PyPy as well. It probably also kills any JIT optimization, but it's not an issue since classes are not usually created in tight loops. -- Amaury Forgeot d'Arc
On Thu, May 2, 2013 at 1:18 PM, fwierzbicki@gmail.com <fwierzbicki@gmail.com> wrote:
It's not just a "speedup mode", it's the default. IronPython requires frames to explicitly enabled because tracking them is about a 10% performance hit (or so Dino told me once upon a time). If you must use it, please copy the code block from namedtuple that ignores it on IronPython. - Jeff
On Thu, May 2, 2013 at 9:07 PM, Ethan Furman <ethan@stoneleaf.us> wrote:
It's ugly as hell, but it's not a performance problem for PyPy, since this is executed at module load time (you probably won't jit that code anyway) Cheers, fijal
Ethan Furman, 02.05.2013 21:07:
What a hack. And fragile, too.
This works fine for Cpython, but what about the others?
This doesn't work when used from Cython compiled code due to the lack of frames. They are only created for exception tracebacks and not for normal code by default (just for profiling, coverage etc.). My guess is that no-one noticed the problem for namedtuples so far because using them is still uncommon enough in general, let alone pickling them, and the module name hack only leads to an error when someone tries to pickle such an object. I think that this will be more of a problem for enums than for namedtuples, because enums are more likely to appear in data structures that people want to pickle. The most simple work-around seems to be this, once you know about it: """ ttuple = namedtuple('ttuple', 'a b c') ttuple.__module__ = __name__ # enable pickle support """ Not any worse than the hack above, IMHO, but at least guaranteed to work. For enums, a regular class based declaration can easily avoid this hack, so my vote is for getting rid of the "convenience" API before it starts doing any harm. Or document it explicitly as generating unpicklable objects, as Antoine suggests. Stefan
On Thu, 2 May 2013 15:48:14 -0400 Benjamin Peterson <benjamin@python.org> wrote:
Two things that were suggested in private: 1) ask users to pass the module name to the convenience function explicitly (i.e. pass "seasonmodule.Season" instead of "Season" as the class "name"). Guido doesn't like it :-) 2) dicth the "convenience function" and replace it with a regular class-based syntax. Ethan doesn't like it :-) Regards Antoine.
On Thu, May 2, 2013 at 1:10 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Re (2), we already have the hack in stdlib in namedtuple, so not allowing it for an enum is a step backwards. If sys._getframe(1).f_globals['__name__'] feels hackish, maybe it can be shortened to a convenience function the stdlib provides? Are there conditions where it doesn't produce what we expect from it? The point at which the enumeration is defined resides in *some* module, no? Eli
On Thu, 2 May 2013 13:15:00 -0700 Eli Bendersky <eliben@gmail.com> wrote:
That's a fallacy. There is no step backwards if you adopt a class-based syntax, which is just as convenient as the proposed "convenience function". I have a hard time understanding that calling a function to declare a class is suddenly considered "convenient".
It's not the notation which is hackish, it's the fact that you are inspecting the frame stack in the hope of getting the right information. What if someone wants to write another convenience function that wraps your convenience function? What if your code is executing from some kind of step-by-step debugger which inserts an additional frame in the call stack? What if someone wants the enum to be nested inside another class (rather than reside at the module top-level)? Regards Antoine.
On Thu, May 2, 2013 at 1:22 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Would nesting the non-convenience Enum in a function or a class allow one to pickle it? I think programmers who want their libraries to be pickle-able already have to be aware of some restrictions about what can and cannot be pickled. Eli
On Thu, 2 May 2013 13:33:21 -0700 Eli Bendersky <eliben@gmail.com> wrote:
Once PEP 3154 is implemented (Alexandre is on it :-)), nested classes should be picklable. As for classes inside functions, it sounds quite impossible (how do you instantiate the function namespace without calling the function?). Regards Antoine.
Interesting, I did not know that.
True. Back to my question from before, though - do we have a real technical limitation of having something like inspect.what_module_am_i_now_in() that's supposed to work for all Python code? Eli
On 05/02/2013 01:52 PM, Eli Bendersky wrote:
Back to my question from before, though - do we have a real technical limitation of having something like inspect.what_module_am_i_now_in() that's supposed to work for all Python code?
By which you really mean inspect.what_module_was_I_called_from() ? -- ~Ethan~
On Thu, May 2, 2013 at 2:05 PM, Ethan Furman <ethan@stoneleaf.us> wrote:
Yes, I guess this is what I meant by "now_in" part. Let's be precise: Animal = Enum('Animal', '...........') The call to Enum is the interesting here. In happens in some library and Animal members can then be passed around. But we really want the module where Enum() was invoked to create Animal in the first place. Eli
On Thu, 2 May 2013 13:52:29 -0700 Eli Bendersky <eliben@gmail.com> wrote:
I already gave an answer (e.g. the debugger case), but you are free to consider it not reasonable :) In any case, I just find the argument for a function-based syntax non-existent compared to a similarly compact class-based syntax. Regards Antoine.
On Thu, May 2, 2013 at 2:10 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Sorry, but I do find the argument "let's not have a convenience syntax because enums created with such syntax won't pickle properly from within a debugger" not convincing enough :-) It may be just me though, and I'm open to other opinions. Eli
On Thu, 2 May 2013 14:15:40 -0700 Eli Bendersky <eliben@gmail.com> wrote:
Eli, it would be nice if you stopped with this claim. I'm not advocating "not having a convenience syntax", I'm advocating having a convenience syntax which is *class-based* rather than function-based. Debuggers are beside the point: there are two kinds of "convenience syntax" on the table; one allows pickling by construction, one requires an ugly hack which may not solve all cases (and which may apparently make Jython / IronPython mildly unhappy). Why you insist on ignoring the former and imposing the latter is beyond me. Regards you Antoine.
Eli, it would be nice if you stopped with this claim.
I'm not trying to belittle our class-based suggestion. I just think there are two separate issues here, and I was focusing on just one of them for now. The one I've been focusing on is how to make the function-based convenience syntax work with pickling in the vast majority of interesting cases. This appears to be possible by using the same pattern used by namedtuple, and even better by encapsulating this pattern formally in stdlib so it stops being a hack (and may actually be useful for other code too). The other issue is your proposal to have a class-based convenience syntax akin to (correct me if I got this wrong): class Animal(Enum): __values__ = 'cat dog' This is obviously a matter of preference (and hence bikeshedding), but this still looks better to me: Animal = Enum('Animal', 'cat dog') It has two advantages: 1. Shorter 2. Parallels namedtuple, which is by now a well known and widely used construct On the other hand, your proposal has the advantage that it allows pickles without hacks in the implementation. Did I sum up the issues fairly? I don't know what to decide here. There's no clear technical merit to decide on one against the other (IMHO!), it's a matter of preference. Hopefully Guido will step in and save us from our misery ;-) Eli
On 3 May 2013 08:00, "Eli Bendersky" <eliben@gmail.com> wrote:
are two separate issues here, and I was focusing on just one of them for now. The one I've been focusing on is how to make the function-based convenience syntax work with pickling in the vast majority of interesting cases. This appears to be possible by using the same pattern used by namedtuple, and even better by encapsulating this pattern formally in stdlib so it stops being a hack (and may actually be useful for other code too).
The other issue is your proposal to have a class-based convenience syntax
akin to (correct me if I got this wrong):
class Animal(Enum): __values__ = 'cat dog'
I would suggest moving the field names into the class header for a class based convenience API: class Animal(Enum, members='cat dog'): pass Cheers, Nick.
This is obviously a matter of preference (and hence bikeshedding), but
this still looks better to me:
construct
On the other hand, your proposal has the advantage that it allows pickles
without hacks in the implementation.
Did I sum up the issues fairly?
I don't know what to decide here. There's no clear technical merit to
decide on one against the other (IMHO!), it's a matter of preference. Hopefully Guido will step in and save us from our misery ;-)
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
On May 03, 2013, at 09:14 AM, Nick Coghlan wrote:
Wait, what is this trying to solve? "Convenience API" is really a shorthand for "functional API". Two very different use cases that the above suggestion doesn't address. IMHO, it's not worth giving up the functional API for picklability if the technical problems cannot be resolved, especially given we already have the same problem for namedtuples. -Barry
On 03/05/13 18:42, Antoine Pitrou wrote:
Le Fri, 3 May 2013 09:14:22 +1000, Nick Coghlan <ncoghlan@gmail.com> a écrit :
The problem is that this is not an expression, it is a statement. The advantage of the convenience function is not just that it is shorter, but that it is an expression. -- Steven
On May 03, 2013, at 07:40 PM, Steven D'Aprano wrote:
Exactly right, but let's stop calling it the "convenience API" and instead call it the "functional API". I probably started the perpetuation of this problem; let's update the PEP. BTW, I made a suggestion elsewhere that the first argument could accept, but not require dotted names in the first argument. If provided, rsplit the string and use the prefix as __module__. If not given, fallback to the _getframe() hack for those implementations where it's available. The same could probably be done to namedtuples. -Barry
2013/5/3 Barry Warsaw <barry@python.org>:
What about adding simple syntax that allows get rid of those ugly hacks, something like: def name = expression which would be rough equivalent for: name = expression name.__name__ = 'name' name.__module__ = __name__ -- 闇に隠れた黒い力 弱い心を操る
On 5/3/2013 12:08 PM, Barry Warsaw wrote:
Please do. To me, a 'convenience function' is something like the timeit functions or subprocess.call that create a class instance, call a method (or two) on the instance, and then discard the instance while returning the result of calling methods. For the common case handled by the function, the implementation via a class with methods is a detail that the user hardly need know about. Using a function interface to create and return a class is something else. -- Terry Jan Reedy
On 4 May 2013 05:17, "Georg Brandl" <g.brandl@gmx.net> wrote:
class thread. Right, if all we want is a functional API that doesn't support pickling of the resulting class, that's trivial. What I'm after is a convenience API that supports *autonumbering*, as a trivial replacement for code that currently uses "range(n)". A class statement is perfectly acceptable to me for that purpose. Independently of that, I do like the notion of a "types.set_name(cls, dotted_name)" API that alters __name__ and __module__, while leaving __qualname__ alone. Cheers, Nick.
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
Le Thu, 2 May 2013 14:57:35 -0700, Eli Bendersky <eliben@gmail.com> a écrit :
You're gaining one line of code. I suppose it's significant if you write ten enums a day, otherwise... ;-)
2. Parallels namedtuple, which is by now a well known and widely used construct
namedtuple is the exception, not the rule. I don't know of another popular type which follows a similar scheme. On the other hand, well-known ORMs (SQLAlchemy, Django ORM) use a class-based syntax despite their declarative nature and the fact that they allow you to set "meta" options (e.g. the name of the reflected table). As an egoistical data point, I always subclass namedtuples, because I minimally want to add a docstring, and sometimes I also want to add behaviour (e.g. alternate constructors, serialization). Which means namedtuple's declarative conciseness is generally lost for me :-) Note that besides ORMs, the proposed __values__ has built-in precedent with __slots__. Regards Antoine.
2013/5/2 Eli Bendersky <eliben@gmail.com>:
I disagree that not allowing code smell to spread is a step backwards. Rather we should realize that this is a common problem and find a proper solution rather than further propogating this hack. -- Regards, Benjamin
On May 02, 2013, at 10:18 PM, Georg Brandl wrote:
5) accept that convenience-created enums have restrictions such as no picklability and point them out in the docs?
That would work fine for me, but ultimately I'm with Guido. I just don't want to have to pass the module name in. -Barry
On Thu, May 2, 2013 at 1:39 PM, Barry Warsaw <barry@python.org> wrote:
The problem with (5) is this: you use some library that exports an enumeration, and you want to use pickling. Now you depend on the way the library implemented - if it used the convenience API, you can't pickle. If it used the class API, you can. Eli
On Thu, 2 May 2013 14:16:34 -0700 Barry Warsaw <barry@python.org> wrote:
Then why insist on the _getframe hack? You are losing me: are you bothered by picklability or not? ;-) If you are not, then fine, let's just make the function-based version *documentedly* unpicklable, and move along. Regards Antoine.
On Thu, May 2, 2013 at 1:18 PM, fwierzbicki@gmail.com <fwierzbicki@gmail.com> wrote:
This particular function is typically only called at module load time, so speeding it up isn't worth it. FWIW, as Eli pointed out, namedtuple() does the same thing (since Python 2.6), so we'll just copy that code (refactoring it doesn't have to hold up the PEP). The only other alternative I find acceptable is not to have the convenience API at all. That's Eli's call. [Eli]
Apparently it hasn't been a problem for namedtuple. Calling namedtuple() or Enum() in another function is similar to a class statement inside a function -- the resulting class isn't picklable. (But from this, don't conclude that it's not important for namedtuple() or Enum() to return a picklable class. It is important. It is just not important to try to make it work when they are called through some other wrapper -- there's just not much use for such a pattern.) -- --Guido van Rossum (python.org/~guido)
On Thu, May 2, 2013 at 1:39 PM, Guido van Rossum <guido@python.org> wrote:
I really prefer having the convenience API and acknowledging that it has some limitations (i.e. picking enums that were created with the convenience API and are nested in classes).
I agree. Eli
2013/5/2 Guido van Rossum <guido@python.org>
It works fine on PyPy as well. It probably also kills any JIT optimization, but it's not an issue since classes are not usually created in tight loops. -- Amaury Forgeot d'Arc
On Thu, May 2, 2013 at 1:18 PM, fwierzbicki@gmail.com <fwierzbicki@gmail.com> wrote:
It's not just a "speedup mode", it's the default. IronPython requires frames to explicitly enabled because tracking them is about a 10% performance hit (or so Dino told me once upon a time). If you must use it, please copy the code block from namedtuple that ignores it on IronPython. - Jeff
On Thu, May 2, 2013 at 9:07 PM, Ethan Furman <ethan@stoneleaf.us> wrote:
It's ugly as hell, but it's not a performance problem for PyPy, since this is executed at module load time (you probably won't jit that code anyway) Cheers, fijal
Ethan Furman, 02.05.2013 21:07:
What a hack. And fragile, too.
This works fine for Cpython, but what about the others?
This doesn't work when used from Cython compiled code due to the lack of frames. They are only created for exception tracebacks and not for normal code by default (just for profiling, coverage etc.). My guess is that no-one noticed the problem for namedtuples so far because using them is still uncommon enough in general, let alone pickling them, and the module name hack only leads to an error when someone tries to pickle such an object. I think that this will be more of a problem for enums than for namedtuples, because enums are more likely to appear in data structures that people want to pickle. The most simple work-around seems to be this, once you know about it: """ ttuple = namedtuple('ttuple', 'a b c') ttuple.__module__ = __name__ # enable pickle support """ Not any worse than the hack above, IMHO, but at least guaranteed to work. For enums, a regular class based declaration can easily avoid this hack, so my vote is for getting rid of the "convenience" API before it starts doing any harm. Or document it explicitly as generating unpicklable objects, as Antoine suggests. Stefan
participants (16)
-
Amaury Forgeot d'Arc -
Antoine Pitrou -
Barry Warsaw -
Benjamin Peterson -
Eli Bendersky -
Ethan Furman -
fwierzbicki＠gmail.com -
Georg Brandl -
Guido van Rossum -
Jeff Hardy -
Maciej Fijalkowski -
Nick Coghlan -
Piotr Duda -
Stefan Behnel -
Steven D'Aprano -
Terry Jan Reedy