Auto-pickle progress?
There is a proposal for auto-pickling of Cython extension types: http://wiki.cython.org/enhancements/pickling I was wondering whether there has been any progress on this front? Auto-pickling would be tremendously helpful as pickling and unpickling is one of the most annoying features of working with threads and processes in python. Regards, Ian
On Thu, Sep 20, 2012 at 12:11 PM, Ian Bell <ian.h.bell@gmail.com> wrote:
There is a proposal for auto-pickling of Cython extension types: http://wiki.cython.org/enhancements/pickling
I was wondering whether there has been any progress on this front?
Auto-pickling would be tremendously helpful as pickling and unpickling is one of the most annoying features of working with threads and processes in python.
Though I agree this would be really useful, some people are less enthusiastic. In any case, no work has been done on this front, but we accept contributions. - Robert
On 20.09.2012 21:11, Ian Bell wrote:
Auto-pickling would be tremendously helpful as pickling and unpickling is one of the most annoying features of working with threads and processes in python.
How should Cython interfere how to pickle a C pointer? cdef class foobar: cdef double *data A C object can be anything. Cython does not know anything about size, offset or strides, or even if it's safe to take a copy. Example: How to pickle a shared memory buffer? Surely we cannot take a copy, because that would defeat the purpose of "shared" memory. And even if could take a copy, how many bytes should be copied? Do you think an autopickler could have figured this out? https://github.com/sturlamolden/sharedmem-numpy/blob/master/sharedmem/shared... https://github.com/sturlamolden/sharedmem-numpy/blob/master/sharedmem/shared... On yes, the code is different on Unix and Windows, something the auto-pickler could not possibly know either. Auto-pickling cdef classes is not doable, IMHO. And by the way, implementing a __reduce__ method manually is not very difficult either. Sturla Molden
I agree that some of the more powerful elements of Cython would be/are difficult or impossible to pickle in a general way. I have a counter use-case where auto-pickling would be very useful. I have cdef classes that have many (think 50-100) members of double and long types, as well as other cdef classes that are also only using Cython-friendly data types. I have implemented pickling for these classes and it sucks. Trying to maintain the list of parameters that need to be pickled is a major hassle. The biggest problem is that if you add add a parameter to the class but forget to add it to the pickling function, you end up with the parameter not getting reloaded on unpickling. This is not good! Yes it is up the programmer to do this right, but I could use some help. What I have ended up doing for my other classes is define a __cdict__ function that returns a dictionary with the values and parameters as defined in the cdef class. In this way, pickling and unpickling is a little bit more pleasant. But it would be fantastic if the __cdict__ could be baked into the class automatically. It would be fine for me if only the things that could be easily handled would be returned, and a second list of attributes that require more work to be serialized so that the user/coder could be sure that all the attributes are being handled and none get forgotten. Also, the docs on pickling of cdef classes is very sparse, a simple example would be very beneficial. The is one in the examples in the source, but it is not easy to find Ian On Tue, Sep 25, 2012 at 6:44 AM, Sturla Molden <sturla@molden.no> wrote:
On 20.09.2012 21:11, Ian Bell wrote:
Auto-pickling would be tremendously helpful as pickling and unpickling
is one of the most annoying features of working with threads and processes in python.
How should Cython interfere how to pickle a C pointer?
cdef class foobar: cdef double *data
A C object can be anything. Cython does not know anything about size, offset or strides, or even if it's safe to take a copy.
Example: How to pickle a shared memory buffer? Surely we cannot take a copy, because that would defeat the purpose of "shared" memory. And even if could take a copy, how many bytes should be copied? Do you think an autopickler could have figured this out?
https://github.com/**sturlamolden/sharedmem-numpy/**blob/master/sharedmem/ **sharedmemory_sysv.pyx<https://github.com/sturlamolden/sharedmem-numpy/blob/master/sharedmem/shared...>
https://github.com/**sturlamolden/sharedmem-numpy/**blob/master/sharedmem/ **sharedmemory_win.pyx<https://github.com/sturlamolden/sharedmem-numpy/blob/master/sharedmem/shared...>
On yes, the code is different on Unix and Windows, something the auto-pickler could not possibly know either.
Auto-pickling cdef classes is not doable, IMHO.
And by the way, implementing a __reduce__ method manually is not very difficult either.
Sturla Molden
______________________________**_________________ cython-devel mailing list cython-devel@python.org http://mail.python.org/**mailman/listinfo/cython-devel<http://mail.python.org/mailman/listinfo/cython-devel>
On Tue, Sep 25, 2012 at 5:41 AM, Ian Bell <ian.h.bell@gmail.com> wrote:
I agree that some of the more powerful elements of Cython would be/are difficult or impossible to pickle in a general way. I have a counter use-case where auto-pickling would be very useful.
I have cdef classes that have many (think 50-100) members of double and long types, as well as other cdef classes that are also only using Cython-friendly data types. I have implemented pickling for these classes and it sucks. Trying to maintain the list of parameters that need to be pickled is a major hassle. The biggest problem is that if you add add a parameter to the class but forget to add it to the pickling function, you end up with the parameter not getting reloaded on unpickling. This is not good! Yes it is up the programmer to do this right, but I could use some help.
What I have ended up doing for my other classes is define a __cdict__ function that returns a dictionary with the values and parameters as defined in the cdef class. In this way, pickling and unpickling is a little bit more pleasant. But it would be fantastic if the __cdict__ could be baked into the class automatically. It would be fine for me if only the things that could be easily handled would be returned, and a second list of attributes that require more work to be serialized so that the user/coder could be sure that all the attributes are being handled and none get forgotten.
Exactly. Even if we handled only cdef classes with only (transitively) picklable fields, this would go a long way. The only question to have is how to handle changes in the class struct between pickling and unpickling. Eventually we could come up with syntax/mechanisms for manually specifying serialization of more complicated members like double*, but that's the kind of class you might have to write a __reduce__ method on anyways.
Also, the docs on pickling of cdef classes is very sparse, a simple example would be very beneficial. The is one in the examples in the source, but it is not easy to find
Agreed.
On Tue, Sep 25, 2012 at 6:44 AM, Sturla Molden <sturla@molden.no> wrote:
On 20.09.2012 21:11, Ian Bell wrote:
Auto-pickling would be tremendously helpful as pickling and unpickling is one of the most annoying features of working with threads and processes in python.
How should Cython interfere how to pickle a C pointer?
cdef class foobar: cdef double *data
A C object can be anything. Cython does not know anything about size, offset or strides, or even if it's safe to take a copy.
Example: How to pickle a shared memory buffer? Surely we cannot take a copy, because that would defeat the purpose of "shared" memory. And even if could take a copy, how many bytes should be copied? Do you think an autopickler could have figured this out?
https://github.com/sturlamolden/sharedmem-numpy/blob/master/sharedmem/shared...
https://github.com/sturlamolden/sharedmem-numpy/blob/master/sharedmem/shared...
On yes, the code is different on Unix and Windows, something the auto-pickler could not possibly know either.
Auto-pickling cdef classes is not doable, IMHO.
And by the way, implementing a __reduce__ method manually is not very difficult either.
Sturla Molden
_______________________________________________ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
participants (3)
-
Ian Bell -
Robert Bradshaw -
Sturla Molden