Armin, thanks for merging in the patch. I have now cloned the latest version, and tried to compile under Cygwin. I ran into a couple more issues, attached is a follow-up patch. In parallel I had a longer conversation with Matti about potentially setting up a build slave for Pypy under Cygwin. Clearly some kind of testing is needed. Currently the Cygwin Pypy Python standalone version build with --opt=jit fails on os.fork() calls. The Pypy Python version build with --opt=2 handles os.fork() apparently just fine, on the other hand it cannot be used to build Pypy and simply exits without error message after platcheck_28. So the question is, do you have a regular build for Cygwin? I have not fully checked into whether I could set up a build slave, but it's clear already that because of the long time it takes to run a build (6.5 hours), it would have to be a weekly, not nightly, setup. I am willing to pursue that if it would help the project. --Uwe ________________________________ From: Armin Rigo To: Uwe F. Mayer Cc: "pypy-dev@python.org" Sent: Sunday, June 3, 2012 2:57 AM Subject: Re: [pypy-dev] Patches for Pypy under cygwin Hi Uwe, On Tue, May 1, 2012 at 1:05 AM, Uwe F. Mayer wrote:

I edited / hacked the sources and managed to compile PyPy under Cygwin based on Python 2.6.7 and gcc-4.5.3. If you find this of interest, the patches are available at http://www.tux.org/~mayer/cygwin/pypy/.

Thanks!  I finally took the time to review and merge the patch. I didn't include any change to lib-python: they are changes that look like they should also be done to CPython..?  I may be mistaken, but it looks like fixes for tests that would also fail on CPython+cygwin.  If that's the case, then it should be reported and fixed in CPython. A bientôt, Armin.

The pypy-c executable is not a dll, and hence does not need to be rebased if I understand it correctly. However, just to follow up on this hunch, I rebased it by first running rebaseall -v on the complete Cygwin installation (which does not touch pypy-c even if put into /usr/bin/) and then manually rebasing pypy-c to the next available base address. Unfortunately this does not help. As to the question about runtime libraries, pypy links against the following libraries: $ cygcheck ./pypy-c D:\pypy\pypy-work-no-python-patch\pypy\pypy\translator\goal\pypy-c   C:\cygwin\bin\cygbz2-1.dll     C:\cygwin\bin\cygwin1.dll       C:\windows\system32\KERNEL32.dll         C:\windows\system32\ntdll.dll     C:\cygwin\bin\cyggcc_s-1.dll   C:\cygwin\bin\cygcrypt-0.dll   C:\cygwin\bin\cygcrypto-1.0.0.dll     C:\cygwin\bin\cygz.dll   C:\cygwin\bin\cygncurses-10.dll   C:\cygwin\bin\cygexpat-1.dll   C:\cygwin\bin\cygintl-8.dll     C:\cygwin\bin\cygiconv-2.dll   C:\cygwin\bin\cygssl-1.0.0.dll   C:\cygwin\bin\cygffi-4.dll Just for good measure, here's a typical error message from an interactive pypy session: $ pypy-c Python 2.7.2 (b0005ab7c34f+, Jun 06 2012, 17:33:28) [PyPy 1.9.1-dev0 with GCC 4.5.3] on cygwin Type "help", "copyright", "credits" or "license" for more information. And now for something completely different: ``'that's definitely a case of "uh????"'''

...

...

...

import os os.fork()       3 [main] pypy-c 11788 fixup_mmaps_after_fork: ReadProcessMemory failed for MAP_PRIVATE address 0xB0010000, Win32 error 998     153 [main] pypy-c 11788 D:\pypy\pypy-work-no-python-patch\pypy\pypy\translator\goal\pypy-c: *** fatal error in forked process - recreate_mmaps_after_fork_failed     499 [main] pypy-c 11788 open_stackdumpfile: Dumping stack trace to pypy-c.stackdump       2 [main] pypy-c 2572 fork: child -1 - forked process 11788 died unexpectedly, retry 0, exit code 256, errno 11                                     Traceback (most recent call last):                                                                         File "<console>", line 1, in <module>                              OSError: [Errno 11] Resource temporarily unavailable  >>>> 

I noticed that the address 0xB0010000 in the error message above seems to be the same regardless of rebasing pypy-c or not, and I am seeing this address on two different Windows 2003 64bit servers. For what it's worth, the expected output, from a python session, would be: $ python Python 2.6.7 (r267:88850, Feb  2 2012, 23:50:20)  [GCC 4.5.3] on cygwin Type "help", "copyright", "credits" or "license" for more information.

...

...

import os os.fork() 5552 0  

Finally, today unexpectedly the os.fork() call works on a Windows XP Professional 32bit box, where it previously failed (that's the Cygwin pypy 1.8.1 version I had posted on my web page). --Uwe ________________________________ From: Amaury Forgeot d'Arc To: Uwe F. Mayer Cc: Armin Rigo ; Matti Picus ; "pypy-dev@python.org" Sent: Thursday, June 7, 2012 10:46 AM Subject: Re: [pypy-dev] Patches for Pypy under cygwin 2012/6/7 Uwe F. Mayer Currently the Cygwin Pypy Python standalone version build with --opt=jit fails on os.fork() calls. fork() on Windows... I'm surprised it works at all! You should probably read this page: http://cygwin.com/cygwin-ug-net/highlights.html#ov-hi-process specially the "DLL base address collisions", did you run rebaseall? Is pypy linked with some non-cygwin load-time DLLs? -- Amaury Forgeot d'Arc

As requested, I ran the code with strace. More precisely, I ran from the pypy/pypy/translator/goal/ directory the command: strace sh -c 'pypy-c /d/pypy/test_os_fork.py' I ran this twice, once with a version of pypy built with "--opt=2", and once with with a version of pypy built with "--opt=jit". The "--opt=2" version succeeds, and the "--opt=jit" version fails, as reported earlier. Attached you'll find the test_os_fork.py script, and the two log files cut down to the relevant part. The log is a couple of MB, I cut it down to the relevant section.  I don't know the answers the other two questions. The third question may touch the core of the issue if the JIT marks the (parent) memory as exclusive access before the fork() call, which would be consistent with the observed error. From what I can tell the fork() call never gets off the ground, having no read access to duplicate the process image. --Uwe ________________________________ From: Amaury Forgeot d'Arc To: Uwe F. Mayer Cc: Armin Rigo ; Matti Picus ; "pypy-dev@python.org" Sent: Thursday, June 7, 2012 2:24 PM Subject: Re: [pypy-dev] Patches for Pypy under cygwin 2012/6/7 Uwe F. Mayer 3 [main] pypy-c 11788 fixup_mmaps_after_fork: ReadProcessMemory failed for MAP_PRIVATE address 0xB0010000, Win32 error 998 Aha, looks interesting. Searching for this error message, I found that cygwin unblocks the parent process before the child calls "fixup_mmaps_after_fork": http://www.mail-archive.com/cygwin@cygwin.com/msg05547.html Could you try again with strace? Does the parent really unmap memory? Or is it because of the special access rights that the JIT uses to mark the memory as executable? -- Amaury Forgeot d'Arc

Just to be sure that we are following up on the right path, I added the hack listed below making the JIT memory allocation shared. Then the os.fork() call works just fine for a simple test program (also included below). Obviously that's not an actual solution as the parent and the child will need separate JIT memory allocations, but it pinpoints the issue. I will issue a report to the Cygwin folks, however a fix may or may not be slow in the coming. How hard would it be for Pypy to use malloc() instead of mmap() for Cygwin?. -- Uwe *** pypy/pypy/rlib/rmmap.py     Wed May 30 08:11:31 2012 --- pypy/pypy/rlib/rmmap.py     Sat Jun  9 16:23:45 2012 *************** *** 14,19 **** --- 14,20 ----   _MS_WINDOWS = os.name == "nt"   _LINUX = "linux" in sys.platform   _64BIT = "64bit" in platform.architecture()[0] + _CYGWIN = "cygwin" == sys.platform      class RValueError(Exception):       def __init__(self, message): *************** *** 692,698 ****           so the memory has the executable bit set and gets allocated           internally in case of a sandboxed process.           """ !         flags = MAP_PRIVATE | MAP_ANONYMOUS           prot = PROT_EXEC | PROT_READ | PROT_WRITE           hintp = rffi.cast(PTR, hint.pos)           res = c_mmap_safe(hintp, map_size, prot, flags, -1, 0) --- 693,703 ----           so the memory has the executable bit set and gets allocated           internally in case of a sandboxed process.           """ !         if _CYGWIN: !             # XXX Hack: JIT memory should be private but Cygwin's fork() fails !             flags = MAP_SHARED | MAP_ANONYMOUS !         else: !             flags = MAP_PRIVATE | MAP_ANONYMOUS           prot = PROT_EXEC | PROT_READ | PROT_WRITE           hintp = rffi.cast(PTR, hint.pos)           res = c_mmap_safe(hintp, map_size, prot, flags, -1, 0) PS Here's a little test program. import sys, os from test import pystone # run some code that uses the JIT print "top-level process:", pystone.main() print "top-level process:", pystone.main() pid = os.fork() if pid == 0:     print "I'm the child, my PID is", os.getpid()     print "child process:", pystone.main()     print "child process:", pystone.main()     sys.exit() else:     print "I'm the parent, my PID is", os.getpid(), " the child PID is", pid     print "parent process:", pystone.main()     print "parent process:", pystone.main()     sys.exit() ________________________________ From: Armin Rigo To: Uwe F. Mayer Cc: Amaury Forgeot d'Arc ; Matti Picus ; "pypy-dev@python.org" Sent: Thursday, June 7, 2012 10:27 PM Subject: Re: [pypy-dev] Patches for Pypy under cygwin Hi Uwe, On Thu, Jun 7, 2012 at 9:21 PM, Uwe F. Mayer wrote:

      3 [main] pypy-c 11788 fixup_mmaps_after_fork: ReadProcessMemory failed for MAP_PRIVATE address 0xB0010000, Win32 error 998

Yes, that's one of the mmap addresses used by the JIT, if available. More to the point, googling for "fork cygwin mmap" shows that there are open bugs with cygwin's fork() in the presence of MAP_PRIVATE mmaps.  That's obviously why it fails. I don't really know what we can reasonably do about that, short of asking people to use some specific versions of cygwin where it seems to work.  I can imagine what we could unreasonably do as work-around, but that requires a lot of work and gross hacks that don't seem appropriate to put in the pypy source just to support cygwin...  I would recommend that you try to see it fixed on the cygwin side. A bientôt, Armin.

Hi, On Sun, Jun 10, 2012 at 11:12 AM, Amaury Forgeot d'Arc wrote:

I think memory allocated for JIT must be marked with PROT_EXEC (it contains machine code), so malloc() is not an option.

Exactly. On Linux it would work, because you can use mprotect() to make even malloc()ed memory executable, but I doubt this non-Posix extension works on cygwin. What might work instead would be using the Windows API instead of the POSIX one. In pypy/rlib/rmmap.py, look for "def alloc" and "def free": there are the two versions. A bientôt, Armin.

I have made a final patch (as of now) and linked it from: http://www.tux.org/~mayer/cygwin/pypy/index.html See the bottom third of the page, the patch against the trunk of the mercurial repository is http://www.tux.org/~mayer/cygwin/pypy/pypy-pypy-ac392fb76904.patch With this pypy builds under Cygwin, and os.fork() works. It would be good if this could get merged into the trunk. --Uwe ----- Original Message ----- From: Uwe F. Mayer To: Armin Rigo ; Amaury Forgeot d'Arc Cc: Matti Picus ; "pypy-dev@python.org" Sent: Monday, June 11, 2012 10:27 AM Subject: Re: [pypy-dev] Patches for Pypy under cygwin

----- Original Message ----- From: Armin Rigo Sent: Monday, June 11, 2012 4:59 AM

...

...

On Sun, Jun 10, 2012 at 11:12 AM, Amaury Forgeot d'Arc  wrote: I think memory allocated for JIT must be marked with PROT_EXEC (it contains machine code), so malloc() is not an option.  Exactly.  On Linux it would work, because you can use mprotect() to make even malloc()ed memory executable, but I doubt this non-Posix extension works on cygwin.  What might work instead would be using the Windows API instead of the POSIX one.  In pypy/rlib/rmmap.py, look for "def alloc" and "def free": there are the two versions.

Cygwin implements the POSIX API on top of the Windows API, and some things are tricky to implement, and memory handling and permissions tend to be part of those. To see if it may just work, I modified rmmap.pyto use malloc() and free(), and indeed pypy seems to work just fine. Following up on this email I also added an mprotect() call, which does exist under Cygwin. However, POSIX says that the behavior of mprotect() is unspecified if it is applied to a region of memory that was not obtained via mmap(2), and indeed this fails. Here's the patch that works, using malloc() and free(), and not using mprotect(). *** pypy/pypy/rlib/rmmap.pyMon Jun 11 10:18:55 2012 --- pypy/pypy/rlib/rmmap.pyMon Jun 11 10:16:12 2012 *************** *** 14,19 **** --- 14,20 ----   _MS_WINDOWS = os.name == "nt"   _LINUX = "linux" in sys.platform   _64BIT = "64bit" in platform.architecture()[0] + _CYGWIN = "cygwin" == sys.platform      class RValueError(Exception):       def __init__(self, message): *************** *** 115,120 **** --- 116,126 ----      PTR = rffi.CCHARP    + if _CYGWIN: +     c_malloc, _ = external('malloc', [size_t], PTR) +     #c_mprotect, _ = external('mprotect', [PTR, size_t, rffi.INT], rffi.INT) +     c_free, _ = external('free', [PTR], lltype.Void) +    c_memmove, _ = external('memmove', [PTR, PTR, size_t], lltype.Void)      if _POSIX: *************** *** 692,714 ****           so the memory has the executable bit set and gets allocated           internally in case of a sandboxed process.           """ !         flags = MAP_PRIVATE | MAP_ANONYMOUS !         prot = PROT_EXEC | PROT_READ | PROT_WRITE !         hintp = rffi.cast(PTR, hint.pos) !         res = c_mmap_safe(hintp, map_size, prot, flags, -1, 0) !         if res == rffi.cast(PTR, -1): !             # some systems (some versions of OS/X?) complain if they !             # are passed a non-zero address.  Try again. !             hintp = rffi.cast(PTR, 0) !             res = c_mmap_safe(hintp, map_size, prot, flags, -1, 0) !             if res == rffi.cast(PTR, -1):                   raise MemoryError           else: !             hint.pos += map_size           return res       alloc._annenforceargs_ = (int,)    !     free = c_munmap_safe      elif _MS_WINDOWS:       def mmap(fileno, length, tagname="", access=_ACCESS_DEFAULT, offset=0): --- 698,739 ----           so the memory has the executable bit set and gets allocated           internally in case of a sandboxed process.           """ !         if _CYGWIN: !             # XXX: JIT memory should be using mmap MAP_PRIVATE with !             #      PROT_EXEC but Cygwin's fork() fails !             res = c_malloc(map_size) !             if res == rffi.cast(PTR, 0):                   raise MemoryError +             # XXX POSIX says that the behavior of mprotect() is unspecified +             # if it is applied to a region of memory that was not obtained +             # via mmap(2), and indeed it does not work here. +             # prot = PROT_EXEC | PROT_READ | PROT_WRITE +             # res2 = c_mprotect(res, map_size, prot) +             # if res2 == -1: +             #     errno = rposix.get_errno() +             #     raise OSError(errno, os.strerror(errno))           else: !             flags = MAP_PRIVATE | MAP_ANONYMOUS !             prot = PROT_EXEC | PROT_READ | PROT_WRITE !             hintp = rffi.cast(PTR, hint.pos) !             res = c_mmap_safe(hintp, map_size, prot, flags, -1, 0) !             if res == rffi.cast(PTR, -1): !                 # some systems (some versions of OS/X?) complain if they !                 # are passed a non-zero address.  Try again. !                 hintp = rffi.cast(PTR, 0) !                 res = c_mmap_safe(hintp, map_size, prot, flags, -1, 0) !                 if res == rffi.cast(PTR, -1): !                     raise MemoryError !             else: !                 hint.pos += map_size           return res +        alloc._annenforceargs_ = (int,)    !     if _CYGWIN: !         free = c_free !     else: !         free = c_munmap_safe      elif _MS_WINDOWS:       def mmap(fileno, length, tagname="", access=_ACCESS_DEFAULT, offset=0):

Hi Ronny, On Wed, Jun 13, 2012 at 9:54 AM, Ronny Pfannschmidt wrote:

We would appreciate a unified diff or something that can easily be feed to hg import

I don't think I would reject such a patch from these grounds; at least it *is* a patch, easily usable with "patch", and not, say, entire files that have been modified from some unknown revision, as we got too. I have checked in a slightly edited version of http://www.tux.org/~mayer/cygwin/pypy/pypy-pypy-ac392fb76904.patch (hg revision c7dff5469611). Please tell me if I missed something. A bientôt, Armin.

From: Ronny Pfannschmidt I think we should encourage contributors to use tools that help the core developers/reviewers whenever possible.

Ronny: I certainly don't mind generating diffs in a format that works for the dev team. It would be nice if that format or desired tools were described somewhere easily found on pypy.org (maybe it's there, but I didn't see it). I can foresee that I may stay involved with pypy, so if I understand correctly, you'd prefer the output of "diff -cru" instead of "diff -cr"? --Uwe On 06/13/2012 01:20 PM, Armin Rigo wrote:

I have checked in a slightly edited version of

http://www.tux.org/~mayer/cygwin/pypy/pypy-pypy-ac392fb76904.patch (hg revision c7dff5469611).  Please tell me if I missed something.

Armin: You did not check in the change below (you may have left that one out on purpose, I don't know). If I recall correctly, having write_barrier_failing_case_ptr uninitialized caused an error when compiling with --gc=hybrid --opt=jit. I doubt this is Cygwin specific. On any case, without the patch below "python translate.py -Ojit" succeeds straight out of the repository under Cygwin with pypy source revision: 1613ac9e0cbd, tag: tip, date: Wed Jun 13 16:16:58 2012 +0200. Great! --Uwe *** pypy/pypy/rpython/memory/gctransform/framework.py   Sun Jun  3 11:01:06 2012 --- ../pypy-work/pypy/pypy/rpython/memory/gctransform/framework.py      Tue Jun  *************** *** 448,453 **** --- 448,454 ----           self.write_barrier_ptr = None           self.write_barrier_from_array_ptr = None +         self.write_barrier_failing_case_ptr = None           if GCClass.needs_write_barrier:               self.write_barrier_ptr = getfn(GCClass.write_barrier.im_func,                                              [s_gc,

Hi Uwe, On Thu, Jun 14, 2012 at 1:11 AM, Uwe F. Mayer wrote:

You did not check in the change below (you may have left that one out on purpose, I don't know). If I recall correctly, having write_barrier_failing_case_ptr uninitialized caused an error when compiling with --gc=hybrid --opt=jit. I doubt this is Cygwin specific.

Ah, I see now. But the error is deeper; your fix merely causes the JIT to call a NULL function pointer at run-time, as far as I can tell. We indeed don't have any test for hybrid+JIT, which means that over time it just broke. I "fixed" it by definitely requiring --gc=minimark with the JIT. A bientôt, Armin.

The workaround I implement is based on full-hour differences between localtime and gmtime, and uses hardcoded GMT+/-# strings. It obviously will not work correctly for timezones that have half-hour offsets to GMT, but it seemed to do the trick for what I needed. ----- Original Message ----- From: Alexander Belopolsky To: Uwe F. Mayer Cc: "pypy-dev@python.org" Sent: Wednesday, June 13, 2012 5:22 PM Subject: Re: [pypy-dev] Patches for Pypy under cygwin On Mon, Apr 30, 2012 at 7:05 PM, Uwe F. Mayer wrote:

In particular the work-around for the tm structure to get by without the tm_gmtoff and tm_zone fields may be of interest, as it allows compilation of the rctime module on systems that don't have it, such as some versions of SunOS.

Interesting and timely.  I am about to add limited tm_gmtoff/tm_zone support to CPython.  See http://bugs.python.org/issue1667546.