Visual Studio 2015 build strange behavior
Hi, I managed to build pypy with visual studio 2015 ( win32 build on window 10/64 ) with both python and by pypy itself but I have one strange behavior it manifest in a first steps using new pypy by running pip pypy-c -mpip install -U wheel and an error reads Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x042cd170>: Failed to establish a new connection: [Errno 10035] A non-blocking socket operation could not be completed immediately',)': /simple/whe exactly error is: [Errno 10035] A non-blocking socket operation could not be completed immediately' ( the same error [Errno 10035] A non-blocking socket operation could not be completed immediately', also appears if I try to use PyMySQL driver to connect to mysql , so this is not that something happened to pip repository) the reason I built with Visual Studio 2015 was that stock pypy for windows would crash on my projects - and I have visual studio 2015. the build I obtained also seems somewhat more stable to my eyes, so it worth the try. but now I have seemingly more stable pypy, but at the same time that network error, any hints what might be involved here and how to debug? Regards Sergey
Hi Sergey, On 1 April 2017 at 13:57, Sergey Kurdakov <sergey.forum@gmail.com> wrote:
but now I have seemingly more stable pypy, but at the same time that network error, any hints what might be involved here and how to debug?
As an aside, we'd love to get any precise information about "unstabilities" of the standard PyPy. We need precise information, though. We can't help much with what you said so far. Maybe some issue like the C-level connect() function that behaves differently when built with another version of Visual Studio? As a starting point, try to reproduce the problem in a few lines of code, which is usually a good way to get a hint about what is wrong. A bientôt, Armin.
Hi Armin,
As an aside, we'd love to get any precise information about "unstabilities" of the standard PyPy.
the precise instability is that stock installation crashes on my project ( unfortunately it is too big to catch for me - so that I try to compile some custom version and find out )
We need precise information, though. We can't help much with what you said so far.
so far I narrowed that I reimplemented _PyVerify_fd() ( which does not compile in VS2015 ) and did it wrong. fortunately I found some code in ruby which implements requesting pioinfo in VS2015 - will try that. Then will report of progress if I have some. Regards Sergey On Sun, Apr 2, 2017 at 11:03 AM, Armin Rigo <armin.rigo@gmail.com> wrote:
Hi, for anyone interested here are my changes to compile with VS2015 compiler in attachment (over pypy version 5.7.1) I compiled with stock python cd pypy/goalpython ../../rpython/bin/rpython --opt=jit compilation options in files are changed manually to reflect new compiler rpython\translator\platform\windows.py in _get_msvc_env(vsver, x64flag) toolsdir = os.environ['VS140COMNTOOLS'] vcvars = "C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\Common7\\Tools\\vsvars32.bat"#os.path.join(toolsdir, 'vsvars32.bat') otherwise - I followed http://doc.pypy.org/en/latest/windows.html few caveats: VS2015 now loads ucrtbase.dll in place of msvcrt.dll changed _PyVerify_fd implementation as it won't compile, fixed few other compilation issues. the compilation works, but all network calls end with [Errno 10035] A non-blocking socket operation could not be completed immediately ( but _PyVerify_fd works correctly ) did not figure out so far why it happens. while changes are not production ready - they could be a hint what might be needed to change in pypy to compile with new ms compilers ( ucrtbase vs msvcrt etc, new _PyVerify_fd etc ) attachment is a zip file with changed extension ( gmail checks for py files .... ) and contains only changed files (pypy 5.7.1) Now I will try older compilers to see if I could catch my initial crash with python project, but in a few weeks time. Still think, that it is ok to share changes - someone could try and see what is needed for future support of vs2015/vs2017 Best regards Sergey
Hi, btw seems I got why error 10035 appears in VS 2015 build network calls, in pypy code there is check for EWOULDBLOCK and required steps are made to deal with situation. but in VS 2015 EWOULDBLOCK is defined as #define EWOULDBLOCK 140 while 10035 error is defined as #define WSAEWOULDBLOCK 10035L so all code logic checking real EWOULDBLOCK is misused by pypy compiled with VS 2015 now will try to play with that new definition, possibly code will start to make socket calls in VS 2015. Regards Sergey On Tue, Apr 4, 2017 at 8:43 AM, Sergey Kurdakov <sergey.forum@gmail.com> wrote:
Hi, with additionally changed pypy\rpython\rlib\_rsocket_rffi.py file (see attachment) pypy now compiles with vs2015 and work on simple tests ( see previous explanations after this mail text ) ( one more note - I had to change expat to compile to dll - static lib won't pass test when compiled with python.exe and while pypy would pass this step it would later report memory errors ( I even tried setting allocations from higher addresses (as using MEM_TOP_DOWN in VirtualAlloc via registry setting HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Memory Management\AllocationPreference to 0x100000 ) ) even with jit disabled and memory block set PYPY_GC_MAX_DELTA=200MB --jit loop_longevity=300 ) now I have an opportunity to track my project problems. Regards Sergey On Wed, Apr 5, 2017 at 3:58 PM, Sergey Kurdakov <sergey.forum@gmail.com> wrote:
Hi Sergey, On 5 April 2017 at 16:56, Sergey Kurdakov <sergey.forum@gmail.com> wrote:
with additionally changed pypy\rpython\rlib\_rsocket_rffi.py file (see attachment)
Cool! But it would be more useful for us if you could open a pull request on the bitbucket project page, https://bitbucket.org/pypy/pypy . Failing that, I could attempt to turn your attached files into a pull request. A bientôt, Armin.
Hi Armin, ok, I will prepare pull request, though there are other things to change. somehow openssl crashes on callback openssl is compiled by default with __cdecl, but export win32 functions in pypy are defined as __stdcall so callback to openssl is sent as stdcall but it expects __cdecl ( maybe I could just change that callback declaration to __cdecl) but maybe I just miss something, but I will look into issue, and when I'm fine with it I will make pull request then. Regards Sergey On Wed, Apr 5, 2017 at 10:08 PM, Armin Rigo <armin.rigo@gmail.com> wrote:
On 05/04/17 22:54, Sergey Kurdakov wrote:
How are you coping with third party packages? Were these directions at all useful http://pypy.readthedocs.io/en/latest/windows.html#nonabrided-method-building... or are you using the cpython build scripts with your compiler? Matti
Hi Matti,
How are you coping with third party packages?
I built them myself,
Were these directions at all useful
they were useful to the extent, for example openssl won't build with these instructions ( nasm step was missing ) also opensssl will use /MT library option by default after ms/do_ms.bat call, but all libs are recommended to be built with /MD so I changed flag in generated mak file.
or are you using the cpython build scripts with your compiler
no just build them as recommended by each library (and t setting /MD etc for consistensy) Regards Sergey On Wed, Apr 5, 2017 at 11:04 PM, Matti Picus <matti.picus@gmail.com> wrote:
Hi Armin, concerning pull request. there are few issues: the new PyVerify_fd code will not work with vs 2013 as that compiler has a bug: it reports console fd s as positive numbers ( earlier ms compilers and vs 2015 are correct to return negative values ) so both versions ( previous and new one ) of the code ( now lacking in my changes - only new code ) had to be included will add it to my changes. the other new C code also would be ifdef ed for the same reasons. So I will include older code to pypy code - but ifdef ed. there is another issue with current changes cfunc = get_on_lib(ctypes.cdll.ucrtbase, funcname) in addition to cfunc = get_on_lib(ctypes.cdll.msvcrt, funcname) in /rpython/rtyper/lltypesystem/ll2ctypes.py in previews versions of compiler there was no ucrtbase, so some checks will be needed ( so far I have no idea how to make it better ) as for my last error I observed. also to Matti interestingly, debug build of pypy does not exhibit the problem with openssl and the only difference between builds I could observe so far - is that debug allocates more stack /rpython/translator/platform/windows.py stack_size = 8 << 20 # 8 Mb self.link_flags = self.link_flags + ('/STACK:%d' % stack_size,) ( so that function call fail which I reported in this thread might be not calling convention fail but lack of enough stack memory might be a culprit ) , thus I will try to play with this in release build as I do this research is spare time. I hope that I will appear with code in a week time. Best regards Sergey
Hi Armin and Matti, now my project works. So concerning current stock installation of pypy seems that the only problem was a lack of sufficient stack space on windows, in debug pypy is configured to increase stack, but in release it is not. And as stack space is insufficient - some code might crash on windows (on linux stack is by default is bigger - so no problem occurs ). while I'm preparing my changes, is it possible, that current code is changed that pypy on windows is compiled in release build with increased stack, possibly even some more, than currently in debug mode? another point. while compiling pypy lack of memory is an issue even on machines with more memory that required for compilation. Basically pypy will crash when achieves 2 Gb memory limit even with setting pypy to use extended memory I suppose it starts to interfere with system memory addresses which are located exactly at 2 Gb starting point, but VirtualAlloc allows explicitly to allocate memory from higher addresses and avoid system memory clashes when memory is continuously allocated. Here - one change would be required ( which I would not attempt now as have insufficient knowledge of pypy internals ) to use not a stock malloc, but a redefined (wrapper) malloc which will call to windows VirtualAlloc with appropriate MEM_TOP_DOWN flag. The flag could be passed to pypy in command line to indicate, that memory should be allocated from top. Then on machines which have enough memory ( I have 8 Gb - but pypy will still report memory error when compiling itself ) there will not be memory problems. Best regards Sergey
Hi Matti,
Are you using 64-bit windows? Then using the editbin /largeaddressaware trick as mentioned on the windows build >page you should be able to access all your memory from the 32-bit python executable used for translation.
yes I'm on windows 64 ( win 10/64 ) and I did, but to no avail ( btw pypy builds with largeaddressaware flag too - so it is there either way ) so the memory error appears in spite of having /largeaddressaware trick, Regards Sergey On Sat, Apr 8, 2017 at 9:53 PM, Matti Picus <matti.picus@gmail.com> wrote:
Hi Matti, after double checking pypy-c.exe headers with dumpbin /headers pypy-c.exe (actually while objects are compiled with /LARGEADDRESSAWARE flag in windows.py - final exe does not contain required flag so possibly here - either this flag should be removed or set correctly such that produced exe has needed headers as might be expected from setting /LARGEADDRESSAWARE flag in code - will look into this in my merge request) and running editbin /LARGEADDRESSAWARE pypy-c.exe I achieved compilation of pypy by pypy but I also set BCDEDIT /Set IncreaseUserVa 3072 so my intent before pull request is a) to check that /LARGEADDRESSAWARE flag in windows.py start to work b) will check several times to be sure if BCDEDIT /Set IncreaseUserVa 3072 is really needed to work. when I'm done will submit code and if it actually needed - also docs to compile pypy on windows 64 could be changed such that it it does not require additional efforts to investigate. Regards Sergey On Sat, Apr 8, 2017 at 10:36 PM, Sergey Kurdakov <sergey.forum@gmail.com> wrote:
Hi Sergey, On 10 April 2017 at 19:45, Sergey Kurdakov <sergey.forum@gmail.com> wrote:
BCDEDIT /Set IncreaseUserVa 3072
According to the documentation here: https://msdn.microsoft.com/en-us/library/windows/hardware/bb613473(v=vs.85).... this line has no effect on Windows 64 (even if the running program is 32-bit), but is required on a 32-bit Windows in order to configure the system in a way that after reboot(!) it will let all(!) its applications use more than 2GB. A bientôt, Armin.
Hi Sergey, On 1 April 2017 at 13:57, Sergey Kurdakov <sergey.forum@gmail.com> wrote:
but now I have seemingly more stable pypy, but at the same time that network error, any hints what might be involved here and how to debug?
As an aside, we'd love to get any precise information about "unstabilities" of the standard PyPy. We need precise information, though. We can't help much with what you said so far. Maybe some issue like the C-level connect() function that behaves differently when built with another version of Visual Studio? As a starting point, try to reproduce the problem in a few lines of code, which is usually a good way to get a hint about what is wrong. A bientôt, Armin.
Hi Armin,
As an aside, we'd love to get any precise information about "unstabilities" of the standard PyPy.
the precise instability is that stock installation crashes on my project ( unfortunately it is too big to catch for me - so that I try to compile some custom version and find out )
We need precise information, though. We can't help much with what you said so far.
so far I narrowed that I reimplemented _PyVerify_fd() ( which does not compile in VS2015 ) and did it wrong. fortunately I found some code in ruby which implements requesting pioinfo in VS2015 - will try that. Then will report of progress if I have some. Regards Sergey On Sun, Apr 2, 2017 at 11:03 AM, Armin Rigo <armin.rigo@gmail.com> wrote:
Hi, for anyone interested here are my changes to compile with VS2015 compiler in attachment (over pypy version 5.7.1) I compiled with stock python cd pypy/goalpython ../../rpython/bin/rpython --opt=jit compilation options in files are changed manually to reflect new compiler rpython\translator\platform\windows.py in _get_msvc_env(vsver, x64flag) toolsdir = os.environ['VS140COMNTOOLS'] vcvars = "C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\Common7\\Tools\\vsvars32.bat"#os.path.join(toolsdir, 'vsvars32.bat') otherwise - I followed http://doc.pypy.org/en/latest/windows.html few caveats: VS2015 now loads ucrtbase.dll in place of msvcrt.dll changed _PyVerify_fd implementation as it won't compile, fixed few other compilation issues. the compilation works, but all network calls end with [Errno 10035] A non-blocking socket operation could not be completed immediately ( but _PyVerify_fd works correctly ) did not figure out so far why it happens. while changes are not production ready - they could be a hint what might be needed to change in pypy to compile with new ms compilers ( ucrtbase vs msvcrt etc, new _PyVerify_fd etc ) attachment is a zip file with changed extension ( gmail checks for py files .... ) and contains only changed files (pypy 5.7.1) Now I will try older compilers to see if I could catch my initial crash with python project, but in a few weeks time. Still think, that it is ok to share changes - someone could try and see what is needed for future support of vs2015/vs2017 Best regards Sergey
Hi, btw seems I got why error 10035 appears in VS 2015 build network calls, in pypy code there is check for EWOULDBLOCK and required steps are made to deal with situation. but in VS 2015 EWOULDBLOCK is defined as #define EWOULDBLOCK 140 while 10035 error is defined as #define WSAEWOULDBLOCK 10035L so all code logic checking real EWOULDBLOCK is misused by pypy compiled with VS 2015 now will try to play with that new definition, possibly code will start to make socket calls in VS 2015. Regards Sergey On Tue, Apr 4, 2017 at 8:43 AM, Sergey Kurdakov <sergey.forum@gmail.com> wrote:
Hi, with additionally changed pypy\rpython\rlib\_rsocket_rffi.py file (see attachment) pypy now compiles with vs2015 and work on simple tests ( see previous explanations after this mail text ) ( one more note - I had to change expat to compile to dll - static lib won't pass test when compiled with python.exe and while pypy would pass this step it would later report memory errors ( I even tried setting allocations from higher addresses (as using MEM_TOP_DOWN in VirtualAlloc via registry setting HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Memory Management\AllocationPreference to 0x100000 ) ) even with jit disabled and memory block set PYPY_GC_MAX_DELTA=200MB --jit loop_longevity=300 ) now I have an opportunity to track my project problems. Regards Sergey On Wed, Apr 5, 2017 at 3:58 PM, Sergey Kurdakov <sergey.forum@gmail.com> wrote:
Hi Sergey, On 5 April 2017 at 16:56, Sergey Kurdakov <sergey.forum@gmail.com> wrote:
with additionally changed pypy\rpython\rlib\_rsocket_rffi.py file (see attachment)
Cool! But it would be more useful for us if you could open a pull request on the bitbucket project page, https://bitbucket.org/pypy/pypy . Failing that, I could attempt to turn your attached files into a pull request. A bientôt, Armin.
Hi Armin, ok, I will prepare pull request, though there are other things to change. somehow openssl crashes on callback openssl is compiled by default with __cdecl, but export win32 functions in pypy are defined as __stdcall so callback to openssl is sent as stdcall but it expects __cdecl ( maybe I could just change that callback declaration to __cdecl) but maybe I just miss something, but I will look into issue, and when I'm fine with it I will make pull request then. Regards Sergey On Wed, Apr 5, 2017 at 10:08 PM, Armin Rigo <armin.rigo@gmail.com> wrote:
On 05/04/17 22:54, Sergey Kurdakov wrote:
How are you coping with third party packages? Were these directions at all useful http://pypy.readthedocs.io/en/latest/windows.html#nonabrided-method-building... or are you using the cpython build scripts with your compiler? Matti
Hi Matti,
How are you coping with third party packages?
I built them myself,
Were these directions at all useful
they were useful to the extent, for example openssl won't build with these instructions ( nasm step was missing ) also opensssl will use /MT library option by default after ms/do_ms.bat call, but all libs are recommended to be built with /MD so I changed flag in generated mak file.
or are you using the cpython build scripts with your compiler
no just build them as recommended by each library (and t setting /MD etc for consistensy) Regards Sergey On Wed, Apr 5, 2017 at 11:04 PM, Matti Picus <matti.picus@gmail.com> wrote:
Hi Armin, concerning pull request. there are few issues: the new PyVerify_fd code will not work with vs 2013 as that compiler has a bug: it reports console fd s as positive numbers ( earlier ms compilers and vs 2015 are correct to return negative values ) so both versions ( previous and new one ) of the code ( now lacking in my changes - only new code ) had to be included will add it to my changes. the other new C code also would be ifdef ed for the same reasons. So I will include older code to pypy code - but ifdef ed. there is another issue with current changes cfunc = get_on_lib(ctypes.cdll.ucrtbase, funcname) in addition to cfunc = get_on_lib(ctypes.cdll.msvcrt, funcname) in /rpython/rtyper/lltypesystem/ll2ctypes.py in previews versions of compiler there was no ucrtbase, so some checks will be needed ( so far I have no idea how to make it better ) as for my last error I observed. also to Matti interestingly, debug build of pypy does not exhibit the problem with openssl and the only difference between builds I could observe so far - is that debug allocates more stack /rpython/translator/platform/windows.py stack_size = 8 << 20 # 8 Mb self.link_flags = self.link_flags + ('/STACK:%d' % stack_size,) ( so that function call fail which I reported in this thread might be not calling convention fail but lack of enough stack memory might be a culprit ) , thus I will try to play with this in release build as I do this research is spare time. I hope that I will appear with code in a week time. Best regards Sergey
Hi Armin and Matti, now my project works. So concerning current stock installation of pypy seems that the only problem was a lack of sufficient stack space on windows, in debug pypy is configured to increase stack, but in release it is not. And as stack space is insufficient - some code might crash on windows (on linux stack is by default is bigger - so no problem occurs ). while I'm preparing my changes, is it possible, that current code is changed that pypy on windows is compiled in release build with increased stack, possibly even some more, than currently in debug mode? another point. while compiling pypy lack of memory is an issue even on machines with more memory that required for compilation. Basically pypy will crash when achieves 2 Gb memory limit even with setting pypy to use extended memory I suppose it starts to interfere with system memory addresses which are located exactly at 2 Gb starting point, but VirtualAlloc allows explicitly to allocate memory from higher addresses and avoid system memory clashes when memory is continuously allocated. Here - one change would be required ( which I would not attempt now as have insufficient knowledge of pypy internals ) to use not a stock malloc, but a redefined (wrapper) malloc which will call to windows VirtualAlloc with appropriate MEM_TOP_DOWN flag. The flag could be passed to pypy in command line to indicate, that memory should be allocated from top. Then on machines which have enough memory ( I have 8 Gb - but pypy will still report memory error when compiling itself ) there will not be memory problems. Best regards Sergey
Hi Matti,
Are you using 64-bit windows? Then using the editbin /largeaddressaware trick as mentioned on the windows build >page you should be able to access all your memory from the 32-bit python executable used for translation.
yes I'm on windows 64 ( win 10/64 ) and I did, but to no avail ( btw pypy builds with largeaddressaware flag too - so it is there either way ) so the memory error appears in spite of having /largeaddressaware trick, Regards Sergey On Sat, Apr 8, 2017 at 9:53 PM, Matti Picus <matti.picus@gmail.com> wrote:
Hi Matti, after double checking pypy-c.exe headers with dumpbin /headers pypy-c.exe (actually while objects are compiled with /LARGEADDRESSAWARE flag in windows.py - final exe does not contain required flag so possibly here - either this flag should be removed or set correctly such that produced exe has needed headers as might be expected from setting /LARGEADDRESSAWARE flag in code - will look into this in my merge request) and running editbin /LARGEADDRESSAWARE pypy-c.exe I achieved compilation of pypy by pypy but I also set BCDEDIT /Set IncreaseUserVa 3072 so my intent before pull request is a) to check that /LARGEADDRESSAWARE flag in windows.py start to work b) will check several times to be sure if BCDEDIT /Set IncreaseUserVa 3072 is really needed to work. when I'm done will submit code and if it actually needed - also docs to compile pypy on windows 64 could be changed such that it it does not require additional efforts to investigate. Regards Sergey On Sat, Apr 8, 2017 at 10:36 PM, Sergey Kurdakov <sergey.forum@gmail.com> wrote:
Hi Sergey, On 10 April 2017 at 19:45, Sergey Kurdakov <sergey.forum@gmail.com> wrote:
BCDEDIT /Set IncreaseUserVa 3072
According to the documentation here: https://msdn.microsoft.com/en-us/library/windows/hardware/bb613473(v=vs.85).... this line has no effect on Windows 64 (even if the running program is 32-bit), but is required on a 32-bit Windows in order to configure the system in a way that after reboot(!) it will let all(!) its applications use more than 2GB. A bientôt, Armin.
participants (4)
-
Armin Rigo -
Carl Friedrich Bolz -
Matti Picus -
Sergey Kurdakov