[issue11277] Crash with mmap and sparse files on Mac OS X

[Steffen Daode Nurpmeso]

Steffen Daode Nurpmeso <sdaoden at googlemail.com> added the comment:

Ok, this patch could be used.
*Unless* the code is not protected by the GIL.

- Gestalt usage is a bit more complicated according to

    http://www.cocoadev.com/index.pl?DeterminingOSVersion

  unless Python only supports OS X 10.4 and later.
  (And platform.py correctly states that in _mac_ver_gestalt(),
  but see below.)

- Due to usage of Gestalt, '-framework CoreServices' must be
  linked against mmapmodule.c.
  The Python configuration stuff is interesting for me, i managed
  compilation by adding the line

    mmap mmapmodule.c -framework CoreServices

  to Modules/Setup, but i guess it's only OS X which is happy
  about that.

platform.py: _mac_ver_xml() should be dropped entirely according
to one of Ned Deily's links ("never officially supported"), and
_mac_ver_gestalt() obviously never executed because i guess it
would fail due to "versioninfo".  Unless i missed something.

By the way: where do you get the info from?  "sys1", "sys2",
"sys3"?  Cannot find it anywhere, only the long names, e.g.
gestaltSystemVersionXy.

Note that i've mailed Apple.  I did not pay 99$ or even 249$, so
i don't know if there will be a response.
--
Ciao, Steffen
sdaoden(*)(gmail.com)
() ascii ribbon campaign - against html e-mail
/\ www.asciiribbon.org - against proprietary attachments

----------
Added file: http://bugs.python.org/file22281/11277.apple-fix-2.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11277>
_______________________________________
-------------- next part --------------
diff --git a/Doc/library/mmap.rst b/Doc/library/mmap.rst
--- a/Doc/library/mmap.rst
+++ b/Doc/library/mmap.rst
@@ -88,7 +88,8 @@
 
    To ensure validity of the created memory mapping the file specified
    by the descriptor *fileno* is internally automatically synchronized
-   with physical backing store on Mac OS X and OpenVMS.
+   with physical backing store on operating systems where this is
+   necessary, e.g. OpenVMS, and some buggy versions of Mac OS X.
 
    This example shows a simple way of using :class:`mmap`::
 
diff --git a/Modules/mmapmodule.c b/Modules/mmapmodule.c
--- a/Modules/mmapmodule.c
+++ b/Modules/mmapmodule.c
@@ -25,6 +25,8 @@
 #define UNIX
 # ifdef __APPLE__
 #  include <fcntl.h>
+
+#  include <CoreServices/CoreServices.h>
 # endif
 #endif
 
@@ -65,6 +67,39 @@
 #define my_getpagesize getpagesize
 #endif
 
+# ifdef __APPLE__
+static void
+apple_osx_needs_fullsync(long *use_fullsync)
+{
+    /* Issue #11277: mmap(2) bug with >32 bit sparse files.
+     * Apple fixed the bug before announcement of OS X "Lion", but since we
+     * need to handle buggy versions, perform a once-only check to see if the
+     * running kernel requires the expensive sync.
+     * >0: F_FULLSYNC is required, <0: kernel has mmap(2) bug fixed */
+    SInt32 ver;
+
+    *use_fullsync = 1;
+    if (Gestalt(gestaltSystemVersion, &ver) != noErr && ver >= 0x1040) {
+        if (Gestalt(gestaltSystemVersionMajor, &ver) != noErr)
+            goto jleave;
+        if (ver > 10) {
+            *use_fullsync = -1;
+            goto jleave;
+        }
+
+        if (Gestalt(gestaltSystemVersionMinor, &ver) != noErr)
+            goto jleave;
+        if (ver >= 7) {
+            *use_fullsync = -1;
+            goto jleave;
+        }
+    }
+
+jleave:
+    return;
+}
+# endif /* __APPLE__ */
+
 #endif /* UNIX */
 
 #include <string.h>
@@ -1128,8 +1163,14 @@
 #ifdef __APPLE__
     /* Issue #11277: fsync(2) is not enough on OS X - a special, OS X specific
        fcntl(2) is necessary to force DISKSYNC and get around mmap(2) bug */
-    if (fd != -1)
-        (void)fcntl(fd, F_FULLFSYNC);
+    if (fd != -1) {
+        /* (GIL protected) */
+        static long use_fullsync /*= 0*/;
+        if (!use_fullsync)
+            apple_osx_needs_fullsync(&use_fullsync);
+        if (use_fullsync > 0)
+            (void)fcntl(fd, F_FULLFSYNC);
+    }
 #endif
 #ifdef HAVE_FSTAT
 #  ifdef __VMS