Pickle improvement: global qualnames for local classes
Only objects of globally defined classes are picklable: class Global: pass picklable = Global() def f(): class Local: pass return Local Local_here = f() unpicklable = Local_here() However, instances become picklable if we assign the local class to some globally reachable identifier. Local_here.__qualname__ = 'Local_here' now_picklable = Local_here() Why not make it a feature? Let's define module level global identifier, for instance `__local_classes__`. It will be a weak dictionary of class objects. Whenever a local class is created, a weak ref is made and reflected in the new classes' qualname. def f(): class Local: pass # weak ref is created with some unique id return Local Local_here = f() print(Local_here.__qualname__) # '__local_classes__["uniqueidxxx"]' assert Local_here is __local_classes__["uniqueidxxx"] Likewise for local functions. This would make many objects picklable, which also positively affects modules that rely on `pickle`, like `multiprocessing`.
On Thu, 3 Sep 2020 at 18:06, haael <haael@interia.pl> wrote:
Only objects of globally defined classes are picklable:
class Global: pass
picklable = Global()
def f(): class Local: pass return Local
Local_here = f() unpicklable = Local_here()
However, instances become picklable if we assign the local class to some globally reachable identifier.
Local_here.__qualname__ = 'Local_here' now_picklable = Local_here()
Why not make it a feature? Let's define module level global identifier, for instance `__local_classes__`. It will be a weak dictionary of class objects. Whenever a local class is created, a weak ref is made and reflected in the new classes' qualname.
def f(): class Local: pass # weak ref is created with some unique id return Local
Local_here = f()
print(Local_here.__qualname__) # '__local_classes__["uniqueidxxx"]' assert Local_here is __local_classes__["uniqueidxxx"]
Likewise for local functions.
This would make many objects picklable, which also positively affects modules that rely on `pickle`, like `multiprocessing`.
I'm not sure that would work in practice. If I pickle Local_here, then run a new instance of the program that loads in the pickle file, but has never even run f() in the new instance, then the class Local doesn't exist. So you can't unpikcle an instance of it, as you don't know how to create the class... Apologies if I'm misunderstanding your point here, but it feels like what you're proposing would only work if you pickled and unpickled the data in the *same* running instance of the program. Paul
Yes, but that could also be said about ordinary global classes. if sys.args[1] == "a": class A: pass a = A() print(pickle.dumps(a)) else: a = pickle.loads(input()) # no definition of class A in this run
I'm not sure that would work in practice. If I pickle Local_here, then run a new instance of the program that loads in the pickle file, but has never even run f() in the new instance, then the class Local doesn't exist. So you can't unpikcle an instance of it, as you don't know how to create the class...
Apologies if I'm misunderstanding your point here, but it feels like what you're proposing would only work if you pickled and unpickled the data in the *same* running instance of the program.
Paul
On Thu, 3 Sep 2020 at 18:06, haael <haael@interia.pl> wrote:
Only objects of globally defined classes are picklable:
class Global: pass
picklable = Global()
def f(): class Local: pass return Local
Local_here = f() unpicklable = Local_here()
However, instances become picklable if we assign the local class to some globally reachable identifier.
Local_here.__qualname__ = 'Local_here' now_picklable = Local_here()
Why not make it a feature? Let's define module level global identifier, for instance `__local_classes__`. It will be a weak dictionary of class objects. Whenever a local class is created, a weak ref is made and reflected in the new classes' qualname.
def f(): class Local: pass # weak ref is created with some unique id return Local
Local_here = f()
print(Local_here.__qualname__) # '__local_classes__["uniqueidxxx"]' assert Local_here is __local_classes__["uniqueidxxx"]
Likewise for local functions.
This would make many objects picklable, which also positively affects modules that rely on `pickle`, like `multiprocessing`.
On Fri, Sep 4, 2020 at 7:19 PM haael <haael@interia.pl> wrote:
Yes, but that could also be said about ordinary global classes.
(Please don't top-post, it makes it very difficult to usefully quote multiple people.)
if sys.args[1] == "a": class A: pass
a = A()
print(pickle.dumps(a)) else: a = pickle.loads(input()) # no definition of class A in this run
Apologies if I'm misunderstanding your point here, but it feels like what you're proposing would only work if you pickled and unpickled the data in the *same* running instance of the program.
Of course you COULD do that, but generally, any run of the program will create equivalent classes with the same name. What you're proposing would have some sort of unique ID generated for every class, so how do you ensure that the correct class has the same ID? If you're able to correctly match a class to its identifier, you could far better do this yourself - or alternatively, have them all be the SAME class with some sort of parameterization. ChrisA
participants (3)
-
Chris Angelico -
haael -
Paul Moore