I'm working on an ongoing task to test our existing code on PyPy. My latest exploration was is using PyPy 5.1 along with lxml 3.6.0 and of partuclar relevance to my question, requests 2.9.1 and betamax 0.7.0 Our test suite completes without incident on Python 2.7.9, but on PyPy 5.1.0 we encounter errors trying to match an http request to a recorded session in betamax. This process fails when trying to match a request against the correct recorded session due to a failure in the header comparison when we include a cookie string. This is a comparison of 2 dicts and the contents are the same, except for the Cookie string. We created the Cookie using cookielib.Cookie and add the various items via the Cookie.__init__() method. The resulting strings comes from Cookie.__str__ I believe and the comparison in requests is a simple dict1 == dict2. Any thoughts on where this difference might lie (dictionary comparison issues, string comparison issues, or ...)? I'd like to avoid having to have a different set of tests for PyPy vs CPython as that defeats the idea of verifying compatibility. Thanks for any input. - Jeff Doran This comes from cookielib.Cookie under PyPY. request.headers: %s { 'User-Agent': 'python-requests/2.9.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Cookie': 'NextGenCSO=8MWACYK5f24yllbpMwVpocfsyGYIBUXl4aWQvo7nUcWz2OwU7p4Dy40bxiyjGS5es8hBv5LxPT8PqBnWNzaBNo0k0PGffpQTDI4xBGc9WwQevnzyUCmq7WaXMTOTSpKM; PacerClientCode=dev; PacerSession=8MWACYK5f24yllbpMwVpocfsyGYIBUXl4aWQvo7nUcWz2OwU7p4Dy40bxiyjGS5es8hBv5LxPT8PqBnWNzaBNo0k0PGffpQTDI4xBGc9WwQevnzyUCmq7WaXMTOTSpKM; domain=.uscourts.gov; path=/; PacerPref="receipt=Y"' } This was recorded previously using Python 2.7.9 recorded headers: %s { u'Connection': u'keep-alive', u'Cookie': u'PacerClientCode=dev; path=/; NextGenCSO=8MWACYK5f24yllbpMwVpocfsyGYIBUXl4aWQvo7nUcWz2OwU7p4Dy40bxiyjGS5es8hBv5LxPT8PqBnWNzaBNo0k0PGffpQTDI4xBGc9WwQevnzyUCmq7WaXMTOTSpKM; domain=.uscourts.gov; PacerSession=8MWACYK5f24yllbpMwVpocfsyGYIBUXl4aWQvo7nUcWz2OwU7p4Dy40bxiyjGS5es8hBv5LxPT8PqBnWNzaBNo0k0PGffpQTDI4xBGc9WwQevnzyUCmq7WaXMTOTSpKM; PacerPref="receipt=Y"', u'Accept-Encoding': u'gzip, deflate', u'Accept': u'*/*', u'User-Agent': u'python-requests/2.9.1' }
Hi Jeff, On 27 May 2016 at 23:45, Jeff Doran <jdoran@lexmachina.com> wrote:
The resulting strings comes from Cookie.__str__ I believe and the comparison in requests is a simple dict1 == dict2.
If what you're printing are two dicts, then they don't compare equal, because of this value:
'NextGenCSO=8MWACYK5f24yllbpMwVpocfsyGYIBUXl4aWQvo7nUcWz2OwU7p4Dy40bxiyjGS5es8hBv5LxPT8PqBnWNzaBNo0k0PGffpQTDI4xBGc9WwQevnzyUCmq7WaXMTOTSpKM; PacerClientCode=dev;PacerSession=8MWACYK5f24yllbpMwVpocfsyGYIBUXl4aWQvo7nUcWz2OwU7p4Dy40bxiyjGS5es8hBv5LxPT8PqBnWNzaBNo0k0PGffpQTDI4xBGc9WwQevnzyUCmq7WaXMTOTSpKM; domain=.uscourts.gov; path=/; PacerPref="receipt=Y"'
which is just a long string, which is different from this other long unicode string:
u'PacerClientCode=dev; path=/;NextGenCSO=8MWACYK5f24yllbpMwVpocfsyGYIBUXl4aWQvo7nUcWz2OwU7p4Dy40bxiyjGS5es8hBv5LxPT8PqBnWNzaBNo0k0PGffpQTDI4xBGc9WwQevnzyUCmq7WaXMTOTSpKM;domain=.uscourts.gov;PacerSession=8MWACYK5f24yllbpMwVpocfsyGYIBUXl4aWQvo7nUcWz2OwU7p4Dy40bxiyjGS5es8hBv5LxPT8PqBnWNzaBNo0k0PGffpQTDI4xBGc9WwQevnzyUCmq7WaXMTOTSpKM;PacerPref="receipt=Y"'
This is likely because the string was built following some dictionary order, which is (in CPython) nondeterministic. It is often roughly stable, in that the same dictionaries often end up in the same order, which is why such errors in the tests are not immediately apparent. PyPy has a different order (which actually guarantees that dicts work like OrderedDicts). You should check that theory by verifying if your tests pass or break with "python -R". My guess is that they do not. Then you can fix the test---but maybe a better idea would be to fix the code to sort the keys (I see that Cookie.py does sort the keys, so I'm guessing there is more code involved here). A bientôt, Armin.
participants (2)
-
Armin Rigo
-
Jeff Doran