[pypy-issue] Issue #2551: struct.Struct has weird __new__ arguments (pypy/pypy)

Steven Noonan issues-reply at bitbucket.org
Wed May 3 08:08:33 EDT 2017

New issue 2551: struct.Struct has weird __new__ arguments

Steven Noonan:

For some reason, CPython and PyPy differ here:

$ pypy -c "import struct; help(struct.Struct.__new__)"
Help on function __new__:

__new__(subtype, format)

$ python2 -c "import struct; help(struct.Struct.__new__)"
Help on built-in function __new__:

    T.__new__(S, ...) -> a new object with type S, a subtype of T

This difference breaks two Python modules I use (livestreamer and streamlink). Both of those modules have some common code which creates several classes that inherit from struct.Struct. The derived classes can't be constructed properly due to the `__new__` difference:

Traceback (most recent call last):
  File "/opt/pypy/bin/streamlink", line 11, in <module>
    load_entry_point('streamlink==0.5.0', 'console_scripts', 'streamlink')()
  File "/opt/pypy/site-packages/pkg_resources/__init__.py", line 565, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/opt/pypy/site-packages/pkg_resources/__init__.py", line 2631, in load_entry_point
    return ep.load()
  File "/opt/pypy/site-packages/pkg_resources/__init__.py", line 2291, in load
    return self.resolve()
  File "/opt/pypy/site-packages/pkg_resources/__init__.py", line 2297, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/opt/pypy/site-packages/streamlink_cli/main.py", line 17, in <module>
    from streamlink.stream import StreamProcess
  File "/opt/pypy/site-packages/streamlink/stream/__init__.py", line 4, in <module>
    from streamlink.stream.akamaihd import AkamaiHDStream
  File "/opt/pypy/site-packages/streamlink/stream/akamaihd.py", line 15, in <module>
    from ..packages.flashmedia import FLV, FLVError
  File "/opt/pypy/site-packages/streamlink/packages/flashmedia/__init__.py", line 4, in <module>
    from .amf import *
  File "/opt/pypy/site-packages/streamlink/packages/flashmedia/amf.py", line 3, in <module>
    from .types import AMF0String, AMF0Value, U8, U16BE, U32BE
  File "/opt/pypy/site-packages/streamlink/packages/flashmedia/types.py", line 298, in <module>
    U24BE = HighLowCombo(">HB", 8, True)
TypeError: __new__() takes exactly 2 arguments (4 given)
The arguments for `__init__` are apparently being passed into `__new__` and  `Struct.__new__` only accepts two arguments: the first is the class to create an instance of (as with all `__new__` functions) and the second is the format string.

Sometimes the derived classes take other kinds of arguments for `__init__` too (such as instances of other derived classes). So in order to get the livestreamer and streamlink modules to work, I came up with this hack:

diff --git a/src/streamlink/packages/flashmedia/types.py b/src/streamlink/packages/flashmedia/types.py
index cc4fcca..731bff1 100644
--- a/src/streamlink/packages/flashmedia/types.py
+++ b/src/streamlink/packages/flashmedia/types.py
@@ -30,6 +30,19 @@ AMF3_MAX_INTEGER = 268435455
 class PrimitiveType(Struct):
+    def __new__(classtype, *args):
+        try:
+            return Struct.__new__(classtype, *args)
+        except:
+            pass
+        try:
+            return Struct.__new__(classtype, args[0])
+        except:
+            pass
+        return Struct.__new__(classtype, '')
     def __call__(self, *args):
         return self.pack(*args)

This change makes the modules *work properly*, but it feels like this may be a PyPy library bug.

So either:

* PyPy needs to make the Struct.__new__ function match CPython's
* or, the streamlink and livestreamer modules shouldn't have classes inheriting from Struct

Which makes more sense?

More information about the pypy-issue mailing list