[pypy-issue] Issue #3014: JIT Issue inlining `struct.unpack('<HH', bytearray(...))` produces incorrect results (pypy/pypy)

[Jason at bitbucket.org]

New issue 3014: JIT Issue inlining `struct.unpack('<HH', bytearray(...))` produces incorrect results
https://bitbucket.org/pypy/pypy/issues/3014/jit-issue-inlining-structunpack-hh

Jason Madden:

Versions:

* Python 2.7.13 \(8cdda8b8cdb8, Apr 14 2019, 14:06:58\)  
  \[PyPy 7.1.1 with GCC 4.2.1 Compatible Apple LLVM 10.0.0 \(clang-1000.11.45.5\)\] \(macOS 10.14\)
* Python 3.6.1 \(784b254d6699, Apr 14 2019, 10:22:55\)  
  \[PyPy 7.1.1-beta0 with GCC 4.2.1 Compatible Apple LLVM 10.0.0 \(clang-1000.11.45.5\)\]
* Python 2.7.13 \(8cdda8b8cdb8, Apr 14 2019, 14:06:58\)  
  \[PyPy 7.1.1 with GCC 4.2.1 Compatible Apple LLVM 10.0.0 \(clang-1000.11.45.5\)\]

‌

I’ve been working on getting RelStorage to run on PyPy with \(pure-Python\) mysql-connector-python at [https://github.com/zodb/relstorage/issues/228](https://github.com/zodb/relstorage/issues/228#issuecomment-492423284), but was running into a very strange, non-sensical error only on PyPy partway through the test sequence.

In a nutshell,  the value of the `status_flag` in the OK result packet from the MySQL server is coming back with the `ServerFlag.MORE_RESULTS_EXIST` bit set, leading to the error. But that’s not actually the packet that the server is sending.

`protocol.py` has `parse_ok` that parses the `status_flag` from a packet \(a `bytearray`\) like this:

```python
    def parse_ok(self, packet):
        """Parse a MySQL OK-packet"""
        if not packet[4] == 0:
            raise errors.InterfaceError("Failed parsing OK packet (invalid).")

        ok_packet = {}
        ok_packet['orig_data'] = bytes(packet)
        try:
            ok_packet['field_count'] = struct_unpack('<xxxxB', packet[0:5])[0]
            (packet, ok_packet['affected_rows']) = utils.read_lc_int(packet[5:])
            (packet, ok_packet['insert_id']) = utils.read_lc_int(packet)
            (ok_packet['status_flag'], # This is the bad value
             ok_packet['warning_count']) = struct_unpack('<HH', packet[0:4]) 
            packet = packet[4:]
            if packet:
                (packet, ok_packet['info_msg']) = utils.read_lc_string(packet)
                ok_packet['info_msg'] = ok_packet['info_msg'].decode('utf-8')
        except ValueError:
            raise errors.InterfaceError("Failed parsing OK packet.")
        return ok_packet
```

\( `struct_unpack` is a small helper function for Python 2 that wraps a `bytearray` into a `buffer` and uses `struct.unpack_from`; I get the same results if I modify the code to use `struct.unpack` directly. Python 3 uses struct.unpack directly already.\)

I tweaked the error message to report the packet value, and to also unpack the data at that time. \(This is a cold branch of code that’s never been taken before\)

```python
 if self._have_next_result: # The status_flag was not what we wanted.
     from .catch23 import PY2, struct_unpack
     raise errors.InterfaceError(
         'Use cmd_query_iter for statements with multiple queries.'
         '\n%s\n%r\n%r\nUnpacks to:%r' % (
             query, result, raw_result, 
             struct_unpack('<HH', raw_result[0:4])))
```

```
InterfaceError: Use cmd_query_iter for statements with multiple queries.
    INSERT INTO temp_pack_visit (zoid, keep_tid)
    SELECT zoid, 0
    FROM current_object
    WHERE tid > 274639514178723874
{'field_count': 0, 'affected_rows': 0, 'insert_id': 0,
 'status_flag': 3480, 'warning_count': 0, 'info_msg': u'Records: 0  Duplicates: 0  Warnings: 0'}

bytearray(b'.\x00\x00\x01\x00\x00\x00\x02\x00\x00\x00&Records: 0  Duplicates: 0  Warnings: 0')
    
Unpacks to: (46, 256)
```

You can see that protocol.py's `parse_ok` method gets a `status_flag` of `3480`, while performing the same unpacking on the same data in this function gets a value of `46` \(which is what CPython gets too\).

\*\*Running with `--jit off` solves the problem and lets the tests complete successfully. Also, just running with `--jit inlining=0` solves the problem too.\*\*

How can I help with this? I imagine you’ll want a JIT log of some sort?