[Python-Dev] Extendability of C vs Python pickle

[Antoine Pitrou]

On Mon, 10 Jul 2017 02:35:37 +0200
Victor Stinner <victor.stinner at gmail.com> wrote:
> 
> I already proposed to remove this benchmark:
> https://mail.python.org/pipermail/speed/2017-April/000554.html
> 
> *but* Antoine Pitrou mentionned that the cloudpickle project uses it.
> 
> Maybe we should try to understand what's wrong with _pickle (C module)
> for cloudpickle?

That's a good question, Victor.  cloudpickle uses three hooks inside
pickle.py's Pickler:
- the "dispatch" dictionary
- overriding the "save_global" method to support saving more objects
  (such as closures, etc.)
- overriding the "save_reduce" method; this one doesn't seem really
  necessary, perhaps some leftover from previous attempts

_pickle.c's Pickler does seem to allow a custom "dispatch_table", but
it doesn't allow overriding "save_global".

Of course, if _pickle.c were improved to allow such extensions, it
would suddenly allow cloudpickle to be much more performant!

Regards

Antoine.