[New-bugs-announce] [issue33652] Improve pickling of typing types

[Serhiy Storchaka]

New submission from Serhiy Storchaka <storchaka+cpython at gmail.com>:

The following PR makes pickles for typing types more portable. Type variables no longer use _find_name() and can be unpickled in 3.6. Subscripted generics no longer expose internals and can be unpickled in 3.6 and future versions with changed internal implementation.

Before this PR:

>>> import pickle, pickletools, typing                                            
>>> pickletools.dis(pickletools.optimize(pickle.dumps(typing.T, 4)))              
    0: \x80 PROTO      4
    2: \x95 FRAME      30
   11: \x8c SHORT_BINUNICODE 'typing'
   19: \x94 MEMOIZE    (as 0)
   20: \x8c SHORT_BINUNICODE '_find_name'
   32: \x93 STACK_GLOBAL
   33: h    BINGET     0
   35: \x8c SHORT_BINUNICODE 'T'
   38: \x86 TUPLE2
   39: R    REDUCE
   40: .    STOP
highest protocol among opcodes = 4
>>> pickletools.dis(pickletools.optimize(pickle.dumps(typing.Union[int, str], 4)))
    0: \x80 PROTO      4
    2: \x95 FRAME      198
   11: \x8c SHORT_BINUNICODE 'copyreg'
   20: \x8c SHORT_BINUNICODE '_reconstructor'
   36: \x93 STACK_GLOBAL
   37: \x8c SHORT_BINUNICODE 'typing'
   45: \x94 MEMOIZE    (as 0)
   46: \x8c SHORT_BINUNICODE '_GenericAlias'
   61: \x93 STACK_GLOBAL
   62: \x8c SHORT_BINUNICODE 'builtins'
   72: \x94 MEMOIZE    (as 1)
   73: \x8c SHORT_BINUNICODE 'object'
   81: \x93 STACK_GLOBAL
   82: N    NONE
   83: \x87 TUPLE3
   84: R    REDUCE
   85: }    EMPTY_DICT
   86: (    MARK
   87: \x8c     SHORT_BINUNICODE '_inst'
   94: \x88     NEWTRUE
   95: \x8c     SHORT_BINUNICODE '_special'
  105: \x89     NEWFALSE
  106: \x8c     SHORT_BINUNICODE '_name'
  113: N        NONE
  114: \x8c     SHORT_BINUNICODE '__origin__'
  126: h        BINGET     0
  128: \x8c     SHORT_BINUNICODE 'Union'
  135: \x93     STACK_GLOBAL
  136: \x8c     SHORT_BINUNICODE '__args__'
  146: h        BINGET     1
  148: \x8c     SHORT_BINUNICODE 'int'
  153: \x93     STACK_GLOBAL
  154: h        BINGET     1
  156: \x8c     SHORT_BINUNICODE 'str'
  161: \x93     STACK_GLOBAL
  162: \x86     TUPLE2
  163: \x8c     SHORT_BINUNICODE '__parameters__'
  179: )        EMPTY_TUPLE
  180: \x8c     SHORT_BINUNICODE '__slots__'
  191: N        NONE
  192: \x8c     SHORT_BINUNICODE '__module__'
  204: h        BINGET     0
  206: u        SETITEMS   (MARK at 86)
  207: b    BUILD
  208: .    STOP
highest protocol among opcodes = 4

With this PR:

>>> import pickle, pickletools, typing                                                              
>>> pickletools.dis(pickletools.optimize(pickle.dumps(typing.T, 4)))                       
    0: \x80 PROTO      4
    2: \x95 FRAME      13
   11: \x8c SHORT_BINUNICODE 'typing'
   19: \x8c SHORT_BINUNICODE 'T'
   22: \x93 STACK_GLOBAL
   23: .    STOP
highest protocol among opcodes = 4
>>> pickletools.dis(pickletools.optimize(pickle.dumps(typing.Union[int, str], 4)))       
    0: \x80 PROTO      4
    2: \x95 FRAME      66
   11: \x8c SHORT_BINUNICODE '_operator'
   22: \x8c SHORT_BINUNICODE 'getitem'
   31: \x93 STACK_GLOBAL
   32: \x8c SHORT_BINUNICODE 'typing'
   40: \x8c SHORT_BINUNICODE 'Union'
   47: \x93 STACK_GLOBAL
   48: \x8c SHORT_BINUNICODE 'builtins'
   58: \x94 MEMOIZE    (as 0)
   59: \x8c SHORT_BINUNICODE 'int'
   64: \x93 STACK_GLOBAL
   65: h    BINGET     0
   67: \x8c SHORT_BINUNICODE 'str'
   72: \x93 STACK_GLOBAL
   73: \x86 TUPLE2
   74: \x86 TUPLE2
   75: R    REDUCE
   76: .    STOP
highest protocol among opcodes = 4

If there is a chance it would be nice to merge these changes into 3.7. Otherwise either pickles created in 3.7.0 will be incompatible not only with older Python versions, but with future Python versions too, or we will need to add complex code for supporting specific 3.7.0 pickles in future versions.

----------
components: Library (Lib)
messages: 317727
nosy: gvanrossum, levkivskyi, ned.deily, serhiy.storchaka
priority: normal
severity: normal
status: open
title: Improve pickling of typing types
versions: Python 3.7, Python 3.8

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue33652>
_______________________________________