Do not block threads when pickle/unpickle
Hi, the old and slow python implementation of pickle didn't block background thread. But the newer C-implementation blocks other threads while dump/load is running. Wouldn't it be possible to allow other threads during this time? Especially could load/loads release the GIL, because Python objects are not available to the Python code until these functions have finished? Regards, Martin
The GIL must be held to allocate memory for Python objects and to invoke the Python code to deserialize user defined picklable objects. I don't think there is a long span of time where the code could leave the GIL released. The Python implementation is just pausing to let other Python threads run, but it is not actually able to parallelize. The same would be true of the C implementation, is there a reason to want to pause the thread to let another thread run? On Mon, Jul 16, 2018 at 1:56 PM, Martin Bammer <mrbm74@gmail.com> wrote:
If you've for example an application with a GUI. Every time bigger objects are pickled/unpickled the complete GUI blocks. Or if you've a server application with multiple network connections and the application should have a guaranteed response time, then it is impossible if one single client could lead the server to block for a longer time if it sends a big object which takes long to unpickle. Just 2 simple examples. Am 2018-07-16 um 20:05 schrieb Joseph Jevnik:
If you've for example an application with a GUI. Every time bigger objects are pickled/unpickled the complete GUI blocks. Or if you've a server application with multiple network connections and the application should have a guaranteed response time, then it is impossible if one single client could lead the server to block for a longer time if it sends a big object which takes long to unpickle. Just 2 simple examples. Am 2018-07-16 um 20:05 schrieb Joseph Jevnik:
Hi, On Mon, 16 Jul 2018 19:56:34 +0200 Martin Bammer <mrbm74@gmail.com> wrote:
This is a fair comment. Please open an issue on the bug tracker at https://bugs.python.org/ Regards Antoine.
The GIL must be held to allocate memory for Python objects and to invoke the Python code to deserialize user defined picklable objects. I don't think there is a long span of time where the code could leave the GIL released. The Python implementation is just pausing to let other Python threads run, but it is not actually able to parallelize. The same would be true of the C implementation, is there a reason to want to pause the thread to let another thread run? On Mon, Jul 16, 2018 at 1:56 PM, Martin Bammer <mrbm74@gmail.com> wrote:
If you've for example an application with a GUI. Every time bigger objects are pickled/unpickled the complete GUI blocks. Or if you've a server application with multiple network connections and the application should have a guaranteed response time, then it is impossible if one single client could lead the server to block for a longer time if it sends a big object which takes long to unpickle. Just 2 simple examples. Am 2018-07-16 um 20:05 schrieb Joseph Jevnik:
If you've for example an application with a GUI. Every time bigger objects are pickled/unpickled the complete GUI blocks. Or if you've a server application with multiple network connections and the application should have a guaranteed response time, then it is impossible if one single client could lead the server to block for a longer time if it sends a big object which takes long to unpickle. Just 2 simple examples. Am 2018-07-16 um 20:05 schrieb Joseph Jevnik:
Hi, On Mon, 16 Jul 2018 19:56:34 +0200 Martin Bammer <mrbm74@gmail.com> wrote:
This is a fair comment. Please open an issue on the bug tracker at https://bugs.python.org/ Regards Antoine.
participants (3)
-
Antoine Pitrou -
Joseph Jevnik -
Martin Bammer