Mailman 3 January 2013 - Python-checkins

cpython (3.2): Issue #13590: OS X Xcode 4 - improve support for universal extension modules
by ned.deily Jan. 31, 2013

Jan. 31, 2013

http://hg.python.org/cpython/rev/502b139b3b19 changeset: 81844:502b139b3b19 branch: 3.2 parent: 81837:48c5c632d212 user: Ned Deily <nad(a)acm.org> date: Thu Jan 31 01:28:23 2013 -0800 summary: Issue #13590: OS X Xcode 4 - improve support for universal extension modules In particular, fix extension module build failures when trying to use 32-bit-only installer Pythons on systems with Xcode 4 (currently OS X 10.8, 10.7, and optionally 10.6). * Backport 3.3.0 fixes to 3.2 branch (for release in 3.2.4) * Since Xcode 4 removes ppc support, extension module builds now check for ppc compiler support and by default remove ppc and ppc64 archs when they are not available. * Extension module builds now revert to using system installed headers and libs (/usr and /System/Library) if the SDK used to build the interpreter is not installed or has moved. * Try to avoid building extension modules with deprecated and problematic Apple llvm-gcc compiler. If original compiler is not available, use clang instead by default. files: Lib/_osx_support.py | 488 ++++++++++++++ Lib/distutils/sysconfig.py | 88 +- Lib/distutils/tests/test_sysconfig.py | 22 +- Lib/distutils/tests/test_util.py | 9 + Lib/distutils/unixccompiler.py | 70 +- Lib/distutils/util.py | 92 +-- Lib/sysconfig.py | 151 +---- Lib/test/test__osx_support.py | 279 ++++++++ Lib/test/test_sysconfig.py | 17 +- Misc/NEWS | 15 + 10 files changed, 862 insertions(+), 369 deletions(-) diff --git a/Lib/_osx_support.py b/Lib/_osx_support.py new file mode 100644 --- /dev/null +++ b/Lib/_osx_support.py @@ -0,0 +1,488 @@ +"""Shared OS X support functions.""" + +import os +import re +import sys + +__all__ = [ + 'compiler_fixup', + 'customize_config_vars', + 'customize_compiler', + 'get_platform_osx', +] + +# configuration variables that may contain universal build flags, +# like "-arch" or "-isdkroot", that may need customization for +# the user environment +_UNIVERSAL_CONFIG_VARS = ('CFLAGS', 'LDFLAGS', 'CPPFLAGS', 'BASECFLAGS', + 'BLDSHARED', 'LDSHARED', 'CC', 'CXX', + 'PY_CFLAGS', 'PY_LDFLAGS', 'PY_CPPFLAGS', + 'PY_CORE_CFLAGS') + +# configuration variables that may contain compiler calls +_COMPILER_CONFIG_VARS = ('BLDSHARED', 'LDSHARED', 'CC', 'CXX') + +# prefix added to original configuration variable names +_INITPRE = '_OSX_SUPPORT_INITIAL_' + + +def _find_executable(executable, path=None): + """Tries to find 'executable' in the directories listed in 'path'. + + A string listing directories separated by 'os.pathsep'; defaults to + os.environ['PATH']. Returns the complete filename or None if not found. + """ + if path is None: + path = os.environ['PATH'] + + paths = path.split(os.pathsep) + base, ext = os.path.splitext(executable) + + if (sys.platform == 'win32' or os.name == 'os2') and (ext != '.exe'): + executable = executable + '.exe' + + if not os.path.isfile(executable): + for p in paths: + f = os.path.join(p, executable) + if os.path.isfile(f): + # the file exists, we have a shot at spawn working + return f + return None + else: + return executable + + +def _read_output(commandstring): + """Output from succesful command execution or None""" + # Similar to os.popen(commandstring, "r").read(), + # but without actually using os.popen because that + # function is not usable during python bootstrap. + # tempfile is also not available then. + import contextlib + try: + import tempfile + fp = tempfile.NamedTemporaryFile() + except ImportError: + fp = open("/tmp/_osx_support.%s"%( + os.getpid(),), "w+b") + + with contextlib.closing(fp) as fp: + cmd = "%s 2>/dev/null >'%s'" % (commandstring, fp.name) + return fp.read().decode('utf-8').strip() if not os.system(cmd) else None + + +def _find_build_tool(toolname): + """Find a build tool on current path or using xcrun""" + return (_find_executable(toolname) + or _read_output("/usr/bin/xcrun -find %s" % (toolname,)) + or '' + ) + +_SYSTEM_VERSION = None + +def _get_system_version(): + """Return the OS X system version as a string""" + # Reading this plist is a documented way to get the system + # version (see the documentation for the Gestalt Manager) + # We avoid using platform.mac_ver to avoid possible bootstrap issues during + # the build of Python itself (distutils is used to build standard library + # extensions). + + global _SYSTEM_VERSION + + if _SYSTEM_VERSION is None: + _SYSTEM_VERSION = '' + try: + f = open('/System/Library/CoreServices/SystemVersion.plist') + except IOError: + # We're on a plain darwin box, fall back to the default + # behaviour. + pass + else: + try: + m = re.search(r'<key>ProductUserVisibleVersion</key>\s*' + r'<string>(.*?)</string>', f.read()) + finally: + f.close() + if m is not None: + _SYSTEM_VERSION = '.'.join(m.group(1).split('.')[:2]) + # else: fall back to the default behaviour + + return _SYSTEM_VERSION + +def _remove_original_values(_config_vars): + """Remove original unmodified values for testing""" + # This is needed for higher-level cross-platform tests of get_platform. + for k in list(_config_vars): + if k.startswith(_INITPRE): + del _config_vars[k] + +def _save_modified_value(_config_vars, cv, newvalue): + """Save modified and original unmodified value of configuration var""" + + oldvalue = _config_vars.get(cv, '') + if (oldvalue != newvalue) and (_INITPRE + cv not in _config_vars): + _config_vars[_INITPRE + cv] = oldvalue + _config_vars[cv] = newvalue + +def _supports_universal_builds(): + """Returns True if universal builds are supported on this system""" + # As an approximation, we assume that if we are running on 10.4 or above, + # then we are running with an Xcode environment that supports universal + # builds, in particular -isysroot and -arch arguments to the compiler. This + # is in support of allowing 10.4 universal builds to run on 10.3.x systems. + + osx_version = _get_system_version() + if osx_version: + try: + osx_version = tuple(int(i) for i in osx_version.split('.')) + except ValueError: + osx_version = '' + return bool(osx_version >= (10, 4)) if osx_version else False + + +def _find_appropriate_compiler(_config_vars): + """Find appropriate C compiler for extension module builds""" + + # Issue #13590: + # The OSX location for the compiler varies between OSX + # (or rather Xcode) releases. With older releases (up-to 10.5) + # the compiler is in /usr/bin, with newer releases the compiler + # can only be found inside Xcode.app if the "Command Line Tools" + # are not installed. + # + # Futhermore, the compiler that can be used varies between + # Xcode releases. Upto Xcode 4 it was possible to use 'gcc-4.2' + # as the compiler, after that 'clang' should be used because + # gcc-4.2 is either not present, or a copy of 'llvm-gcc' that + # miscompiles Python. + + # skip checks if the compiler was overriden with a CC env variable + if 'CC' in os.environ: + return _config_vars + + # The CC config var might contain additional arguments. + # Ignore them while searching. + cc = oldcc = _config_vars['CC'].split()[0] + if not _find_executable(cc): + # Compiler is not found on the shell search PATH. + # Now search for clang, first on PATH (if the Command LIne + # Tools have been installed in / or if the user has provided + # another location via CC). If not found, try using xcrun + # to find an uninstalled clang (within a selected Xcode). + + # NOTE: Cannot use subprocess here because of bootstrap + # issues when building Python itself (and os.popen is + # implemented on top of subprocess and is therefore not + # usable as well) + + cc = _find_build_tool('clang') + + elif os.path.basename(cc).startswith('gcc'): + # Compiler is GCC, check if it is LLVM-GCC + data = _read_output("'%s' --version" + % (cc.replace("'", "'\"'\"'"),)) + if 'llvm-gcc' in data: + # Found LLVM-GCC, fall back to clang + cc = _find_build_tool('clang') + + if not cc: + raise SystemError( + "Cannot locate working compiler") + + if cc != oldcc: + # Found a replacement compiler. + # Modify config vars using new compiler, if not already explictly + # overriden by an env variable, preserving additional arguments. + for cv in _COMPILER_CONFIG_VARS: + if cv in _config_vars and cv not in os.environ: + cv_split = _config_vars[cv].split() + cv_split[0] = cc if cv != 'CXX' else cc + '++' + _save_modified_value(_config_vars, cv, ' '.join(cv_split)) + + return _config_vars + + +def _remove_universal_flags(_config_vars): + """Remove all universal build arguments from config vars""" + + for cv in _UNIVERSAL_CONFIG_VARS: + # Do not alter a config var explicitly overriden by env var + if cv in _config_vars and cv not in os.environ: + flags = _config_vars[cv] + flags = re.sub('-arch\s+\w+\s', ' ', flags, re.ASCII) + flags = re.sub('-isysroot [^ \t]*', ' ', flags) + _save_modified_value(_config_vars, cv, flags) + + return _config_vars + + +def _remove_unsupported_archs(_config_vars): + """Remove any unsupported archs from config vars""" + # Different Xcode releases support different sets for '-arch' + # flags. In particular, Xcode 4.x no longer supports the + # PPC architectures. + # + # This code automatically removes '-arch ppc' and '-arch ppc64' + # when these are not supported. That makes it possible to + # build extensions on OSX 10.7 and later with the prebuilt + # 32-bit installer on the python.org website. + + # skip checks if the compiler was overriden with a CC env variable + if 'CC' in os.environ: + return _config_vars + + if re.search('-arch\s+ppc', _config_vars['CFLAGS']) is not None: + # NOTE: Cannot use subprocess here because of bootstrap + # issues when building Python itself + status = os.system("'%s' -arch ppc -x c /dev/null 2>/dev/null"%( + _config_vars['CC'].replace("'", "'\"'\"'"),)) + # The Apple compiler drivers return status 255 if no PPC + if (status >> 8) == 255: + # Compiler doesn't support PPC, remove the related + # '-arch' flags if not explicitly overridden by an + # environment variable + for cv in _UNIVERSAL_CONFIG_VARS: + if cv in _config_vars and cv not in os.environ: + flags = _config_vars[cv] + flags = re.sub('-arch\s+ppc\w*\s', ' ', flags) + _save_modified_value(_config_vars, cv, flags) + + return _config_vars + + +def _override_all_archs(_config_vars): + """Allow override of all archs with ARCHFLAGS env var""" + # NOTE: This name was introduced by Apple in OSX 10.5 and + # is used by several scripting languages distributed with + # that OS release. + if 'ARCHFLAGS' in os.environ: + arch = os.environ['ARCHFLAGS'] + for cv in _UNIVERSAL_CONFIG_VARS: + if cv in _config_vars and '-arch' in _config_vars[cv]: + flags = _config_vars[cv] + flags = re.sub('-arch\s+\w+\s', ' ', flags) + flags = flags + ' ' + arch + _save_modified_value(_config_vars, cv, flags) + + return _config_vars + + +def _check_for_unavailable_sdk(_config_vars): + """Remove references to any SDKs not available""" + # If we're on OSX 10.5 or later and the user tries to + # compile an extension using an SDK that is not present + # on the current machine it is better to not use an SDK + # than to fail. This is particularly important with + # the standalong Command Line Tools alternative to a + # full-blown Xcode install since the CLT packages do not + # provide SDKs. If the SDK is not present, it is assumed + # that the header files and dev libs have been installed + # to /usr and /System/Library by either a standalone CLT + # package or the CLT component within Xcode. + cflags = _config_vars.get('CFLAGS', '') + m = re.search(r'-isysroot\s+(\S+)', cflags) + if m is not None: + sdk = m.group(1) + if not os.path.exists(sdk): + for cv in _UNIVERSAL_CONFIG_VARS: + # Do not alter a config var explicitly overriden by env var + if cv in _config_vars and cv not in os.environ: + flags = _config_vars[cv] + flags = re.sub(r'-isysroot\s+\S+(?:\s|$)', ' ', flags) + _save_modified_value(_config_vars, cv, flags) + + return _config_vars + + +def compiler_fixup(compiler_so, cc_args): + """ + This function will strip '-isysroot PATH' and '-arch ARCH' from the + compile flags if the user has specified one them in extra_compile_flags. + + This is needed because '-arch ARCH' adds another architecture to the + build, without a way to remove an architecture. Furthermore GCC will + barf if multiple '-isysroot' arguments are present. + """ + stripArch = stripSysroot = False + + compiler_so = list(compiler_so) + + if not _supports_universal_builds(): + # OSX before 10.4.0, these don't support -arch and -isysroot at + # all. + stripArch = stripSysroot = True + else: + stripArch = '-arch' in cc_args + stripSysroot = '-isysroot' in cc_args + + if stripArch or 'ARCHFLAGS' in os.environ: + while True: + try: + index = compiler_so.index('-arch') + # Strip this argument and the next one: + del compiler_so[index:index+2] + except ValueError: + break + + if 'ARCHFLAGS' in os.environ and not stripArch: + # User specified different -arch flags in the environ, + # see also distutils.sysconfig + compiler_so = compiler_so + os.environ['ARCHFLAGS'].split() + + if stripSysroot: + while True: + try: + index = compiler_so.index('-isysroot') + # Strip this argument and the next one: + del compiler_so[index:index+2] + except ValueError: + break + + # Check if the SDK that is used during compilation actually exists, + # the universal build requires the usage of a universal SDK and not all + # users have that installed by default. + sysroot = None + if '-isysroot' in cc_args: + idx = cc_args.index('-isysroot') + sysroot = cc_args[idx+1] + elif '-isysroot' in compiler_so: + idx = compiler_so.index('-isysroot') + sysroot = compiler_so[idx+1] + + if sysroot and not os.path.isdir(sysroot): + from distutils import log + log.warn("Compiling with an SDK that doesn't seem to exist: %s", + sysroot) + log.warn("Please check your Xcode installation") + + return compiler_so + + +def customize_config_vars(_config_vars): + """Customize Python build configuration variables. + + Called internally from sysconfig with a mutable mapping + containing name/value pairs parsed from the configured + makefile used to build this interpreter. Returns + the mapping updated as needed to reflect the environment + in which the interpreter is running; in the case of + a Python from a binary installer, the installed + environment may be very different from the build + environment, i.e. different OS levels, different + built tools, different available CPU architectures. + + This customization is performed whenever + distutils.sysconfig.get_config_vars() is first + called. It may be used in environments where no + compilers are present, i.e. when installing pure + Python dists. Customization of compiler paths + and detection of unavailable archs is deferred + until the first extention module build is + requested (in distutils.sysconfig.customize_compiler). + + Currently called from distutils.sysconfig + """ + + if not _supports_universal_builds(): + # On Mac OS X before 10.4, check if -arch and -isysroot + # are in CFLAGS or LDFLAGS and remove them if they are. + # This is needed when building extensions on a 10.3 system + # using a universal build of python. + _remove_universal_flags(_config_vars) + + # Allow user to override all archs with ARCHFLAGS env var + _override_all_archs(_config_vars) + + # Remove references to sdks that are not found + _check_for_unavailable_sdk(_config_vars) + + return _config_vars + + +def customize_compiler(_config_vars): + """Customize compiler path and configuration variables. + + This customization is performed when the first + extension module build is requested + in distutils.sysconfig.customize_compiler). + """ + + # Find a compiler to use for extension module builds + _find_appropriate_compiler(_config_vars) + + # Remove ppc arch flags if not supported here + _remove_unsupported_archs(_config_vars) + + # Allow user to override all archs with ARCHFLAGS env var + _override_all_archs(_config_vars) + + return _config_vars + + +def get_platform_osx(_config_vars, osname, release, machine): + """Filter values for get_platform()""" + # called from get_platform() in sysconfig and distutils.util + # + # For our purposes, we'll assume that the system version from + # distutils' perspective is what MACOSX_DEPLOYMENT_TARGET is set + # to. This makes the compatibility story a bit more sane because the + # machine is going to compile and link as if it were + # MACOSX_DEPLOYMENT_TARGET. + + macver = _config_vars.get('MACOSX_DEPLOYMENT_TARGET', '') + macrelease = _get_system_version() or macver + macver = macver or macrelease + + if macver: + release = macver + osname = "macosx" + + # Use the original CFLAGS value, if available, so that we + # return the same machine type for the platform string. + # Otherwise, distutils may consider this a cross-compiling + # case and disallow installs. + cflags = _config_vars.get(_INITPRE+'CFLAGS', + _config_vars.get('CFLAGS', '')) + if ((macrelease + '.') >= '10.4.' and + '-arch' in cflags.strip()): + # The universal build will build fat binaries, but not on + # systems before 10.4 + + machine = 'fat' + + archs = re.findall('-arch\s+(\S+)', cflags) + archs = tuple(sorted(set(archs))) + + if len(archs) == 1: + machine = archs[0] + elif archs == ('i386', 'ppc'): + machine = 'fat' + elif archs == ('i386', 'x86_64'): + machine = 'intel' + elif archs == ('i386', 'ppc', 'x86_64'): + machine = 'fat3' + elif archs == ('ppc64', 'x86_64'): + machine = 'fat64' + elif archs == ('i386', 'ppc', 'ppc64', 'x86_64'): + machine = 'universal' + else: + raise ValueError( + "Don't know machine value for archs=%r" % (archs,)) + + elif machine == 'i386': + # On OSX the machine type returned by uname is always the + # 32-bit variant, even if the executable architecture is + # the 64-bit variant + if sys.maxsize >= 2**32: + machine = 'x86_64' + + elif machine in ('PowerPC', 'Power_Macintosh'): + # Pick a sane name for the PPC architecture. + # See 'i386' case + if sys.maxsize >= 2**32: + machine = 'ppc64' + else: + machine = 'ppc' + + return (osname, release, machine) diff --git a/Lib/distutils/sysconfig.py b/Lib/distutils/sysconfig.py --- a/Lib/distutils/sysconfig.py +++ b/Lib/distutils/sysconfig.py @@ -146,7 +146,7 @@ "I don't know where Python installs its library " "on platform '%s'" % os.name) -_USE_CLANG = None + def customize_compiler(compiler): """Do any platform-specific customization of a CCompiler instance. @@ -155,42 +155,28 @@ varies across Unices and is stored in Python's Makefile. """ if compiler.compiler_type == "unix": + if sys.platform == "darwin": + # Perform first-time customization of compiler-related + # config vars on OS X now that we know we need a compiler. + # This is primarily to support Pythons from binary + # installers. The kind and paths to build tools on + # the user system may vary significantly from the system + # that Python itself was built on. Also the user OS + # version and build tools may not support the same set + # of CPU architectures for universal builds. + global _config_vars + if not _config_vars.get('CUSTOMIZED_OSX_COMPILER', ''): + import _osx_support + _osx_support.customize_compiler(_config_vars) + _config_vars['CUSTOMIZED_OSX_COMPILER'] = 'True' + (cc, cxx, opt, cflags, ccshared, ldshared, so_ext, ar, ar_flags) = \ get_config_vars('CC', 'CXX', 'OPT', 'CFLAGS', 'CCSHARED', 'LDSHARED', 'SO', 'AR', 'ARFLAGS') newcc = None if 'CC' in os.environ: - newcc = os.environ['CC'] - elif sys.platform == 'darwin' and cc == 'gcc-4.2': - # Issue #13590: - # Since Apple removed gcc-4.2 in Xcode 4.2, we can no - # longer assume it is available for extension module builds. - # If Python was built with gcc-4.2, check first to see if - # it is available on this system; if not, try to use clang - # instead unless the caller explicitly set CC. - global _USE_CLANG - if _USE_CLANG is None: - from distutils import log - from subprocess import Popen, PIPE - p = Popen("! type gcc-4.2 && type clang && exit 2", - shell=True, stdout=PIPE, stderr=PIPE) - p.wait() - if p.returncode == 2: - _USE_CLANG = True - log.warn("gcc-4.2 not found, using clang instead") - else: - _USE_CLANG = False - if _USE_CLANG: - newcc = 'clang' - if newcc: - # On OS X, if CC is overridden, use that as the default - # command for LDSHARED as well - if (sys.platform == 'darwin' - and 'LDSHARED' not in os.environ - and ldshared.startswith(cc)): - ldshared = newcc + ldshared[len(cc):] - cc = newcc + cc = os.environ['CC'] if 'CXX' in os.environ: cxx = os.environ['CXX'] if 'LDSHARED' in os.environ: @@ -543,43 +529,11 @@ srcdir = os.path.join(base, _config_vars['srcdir']) _config_vars['srcdir'] = os.path.normpath(srcdir) + # OS X platforms require special customization to handle + # multi-architecture, multi-os-version installers if sys.platform == 'darwin': - kernel_version = os.uname()[2] # Kernel version (8.4.3) - major_version = int(kernel_version.split('.')[0]) - - if major_version < 8: - # On Mac OS X before 10.4, check if -arch and -isysroot - # are in CFLAGS or LDFLAGS and remove them if they are. - # This is needed when building extensions on a 10.3 system - # using a universal build of python. - for key in ('LDFLAGS', 'BASECFLAGS', - # a number of derived variables. These need to be - # patched up as well. - 'CFLAGS', 'PY_CFLAGS', 'BLDSHARED'): - flags = _config_vars[key] - flags = re.sub('-arch\s+\w+\s', ' ', flags, re.ASCII) - flags = re.sub('-isysroot [^ \t]*', ' ', flags) - _config_vars[key] = flags - - else: - - # Allow the user to override the architecture flags using - # an environment variable. - # NOTE: This name was introduced by Apple in OSX 10.5 and - # is used by several scripting languages distributed with - # that OS release. - - if 'ARCHFLAGS' in os.environ: - arch = os.environ['ARCHFLAGS'] - for key in ('LDFLAGS', 'BASECFLAGS', - # a number of derived variables. These need to be - # patched up as well. - 'CFLAGS', 'PY_CFLAGS', 'BLDSHARED'): - - flags = _config_vars[key] - flags = re.sub('-arch\s+\w+\s', ' ', flags) - flags = flags + ' ' + arch - _config_vars[key] = flags + import _osx_support + _osx_support.customize_config_vars(_config_vars) if args: vals = [] diff --git a/Lib/distutils/tests/test_sysconfig.py b/Lib/distutils/tests/test_sysconfig.py --- a/Lib/distutils/tests/test_sysconfig.py +++ b/Lib/distutils/tests/test_sysconfig.py @@ -102,7 +102,27 @@ import sysconfig as global_sysconfig self.assertEqual(global_sysconfig.get_config_var('CFLAGS'), sysconfig.get_config_var('CFLAGS')) self.assertEqual(global_sysconfig.get_config_var('LDFLAGS'), sysconfig.get_config_var('LDFLAGS')) - self.assertEqual(global_sysconfig.get_config_var('LDSHARED'),sysconfig.get_config_var('LDSHARED')) + + @unittest.skipIf(sysconfig.get_config_var('CUSTOMIZED_OSX_COMPILER'),'compiler flags customized') + def test_sysconfig_compiler_vars(self): + # On OS X, binary installers support extension module building on + # various levels of the operating system with differing Xcode + # configurations. This requires customization of some of the + # compiler configuration directives to suit the environment on + # the installed machine. Some of these customizations may require + # running external programs and, so, are deferred until needed by + # the first extension module build. With Python 3.3, only + # the Distutils version of sysconfig is used for extension module + # builds, which happens earlier in the Distutils tests. This may + # cause the following tests to fail since no tests have caused + # the global version of sysconfig to call the customization yet. + # The solution for now is to simply skip this test in this case. + # The longer-term solution is to only have one version of sysconfig. + + import sysconfig as global_sysconfig + if sysconfig.get_config_var('CUSTOMIZED_OSX_COMPILER'): + return + self.assertEqual(global_sysconfig.get_config_var('LDSHARED'), sysconfig.get_config_var('LDSHARED')) self.assertEqual(global_sysconfig.get_config_var('CC'), sysconfig.get_config_var('CC')) diff --git a/Lib/distutils/tests/test_util.py b/Lib/distutils/tests/test_util.py --- a/Lib/distutils/tests/test_util.py +++ b/Lib/distutils/tests/test_util.py @@ -13,6 +13,7 @@ from distutils.sysconfig import get_config_vars from distutils import sysconfig from distutils.tests import support +import _osx_support class UtilTestCase(support.EnvironGuard, unittest.TestCase): @@ -92,6 +93,7 @@ ('Darwin Kernel Version 8.11.1: ' 'Wed Oct 10 18:23:28 PDT 2007; ' 'root:xnu-792.25.20~1/RELEASE_I386'), 'i386')) + _osx_support._remove_original_values(get_config_vars()) get_config_vars()['MACOSX_DEPLOYMENT_TARGET'] = '10.3' get_config_vars()['CFLAGS'] = ('-fno-strict-aliasing -DNDEBUG -g ' @@ -105,6 +107,7 @@ sys.maxsize = cursize # macbook with fat binaries (fat, universal or fat64) + _osx_support._remove_original_values(get_config_vars()) get_config_vars()['MACOSX_DEPLOYMENT_TARGET'] = '10.4' get_config_vars()['CFLAGS'] = ('-arch ppc -arch i386 -isysroot ' '/Developer/SDKs/MacOSX10.4u.sdk ' @@ -113,10 +116,12 @@ self.assertEqual(get_platform(), 'macosx-10.4-fat') + _osx_support._remove_original_values(get_config_vars()) os.environ['MACOSX_DEPLOYMENT_TARGET'] = '10.1' self.assertEqual(get_platform(), 'macosx-10.4-fat') + _osx_support._remove_original_values(get_config_vars()) get_config_vars()['CFLAGS'] = ('-arch x86_64 -arch i386 -isysroot ' '/Developer/SDKs/MacOSX10.4u.sdk ' '-fno-strict-aliasing -fno-common ' @@ -124,18 +129,21 @@ self.assertEqual(get_platform(), 'macosx-10.4-intel') + _osx_support._remove_original_values(get_config_vars()) get_config_vars()['CFLAGS'] = ('-arch x86_64 -arch ppc -arch i386 -isysroot ' '/Developer/SDKs/MacOSX10.4u.sdk ' '-fno-strict-aliasing -fno-common ' '-dynamic -DNDEBUG -g -O3') self.assertEqual(get_platform(), 'macosx-10.4-fat3') + _osx_support._remove_original_values(get_config_vars()) get_config_vars()['CFLAGS'] = ('-arch ppc64 -arch x86_64 -arch ppc -arch i386 -isysroot ' '/Developer/SDKs/MacOSX10.4u.sdk ' '-fno-strict-aliasing -fno-common ' '-dynamic -DNDEBUG -g -O3') self.assertEqual(get_platform(), 'macosx-10.4-universal') + _osx_support._remove_original_values(get_config_vars()) get_config_vars()['CFLAGS'] = ('-arch x86_64 -arch ppc64 -isysroot ' '/Developer/SDKs/MacOSX10.4u.sdk ' '-fno-strict-aliasing -fno-common ' @@ -144,6 +152,7 @@ self.assertEqual(get_platform(), 'macosx-10.4-fat64') for arch in ('ppc', 'i386', 'x86_64', 'ppc64'): + _osx_support._remove_original_values(get_config_vars()) get_config_vars()['CFLAGS'] = ('-arch %s -isysroot ' '/Developer/SDKs/MacOSX10.4u.sdk ' '-fno-strict-aliasing -fno-common ' diff --git a/Lib/distutils/unixccompiler.py b/Lib/distutils/unixccompiler.py --- a/Lib/distutils/unixccompiler.py +++ b/Lib/distutils/unixccompiler.py @@ -23,6 +23,9 @@ DistutilsExecError, CompileError, LibError, LinkError from distutils import log +if sys.platform == 'darwin': + import _osx_support + # XXX Things not currently handled: # * optimization/debug/warning flags; we just use whatever's in Python's # Makefile and live with it. Is this adequate? If not, we might @@ -38,68 +41,6 @@ # should just happily stuff them into the preprocessor/compiler/linker # options and carry on. -def _darwin_compiler_fixup(compiler_so, cc_args): - """ - This function will strip '-isysroot PATH' and '-arch ARCH' from the - compile flags if the user has specified one them in extra_compile_flags. - - This is needed because '-arch ARCH' adds another architecture to the - build, without a way to remove an architecture. Furthermore GCC will - barf if multiple '-isysroot' arguments are present. - """ - stripArch = stripSysroot = False - - compiler_so = list(compiler_so) - kernel_version = os.uname()[2] # 8.4.3 - major_version = int(kernel_version.split('.')[0]) - - if major_version < 8: - # OSX before 10.4.0, these don't support -arch and -isysroot at - # all. - stripArch = stripSysroot = True - else: - stripArch = '-arch' in cc_args - stripSysroot = '-isysroot' in cc_args - - if stripArch or 'ARCHFLAGS' in os.environ: - while True: - try: - index = compiler_so.index('-arch') - # Strip this argument and the next one: - del compiler_so[index:index+2] - except ValueError: - break - - if 'ARCHFLAGS' in os.environ and not stripArch: - # User specified different -arch flags in the environ, - # see also distutils.sysconfig - compiler_so = compiler_so + os.environ['ARCHFLAGS'].split() - - if stripSysroot: - try: - index = compiler_so.index('-isysroot') - # Strip this argument and the next one: - del compiler_so[index:index+2] - except ValueError: - pass - - # Check if the SDK that is used during compilation actually exists, - # the universal build requires the usage of a universal SDK and not all - # users have that installed by default. - sysroot = None - if '-isysroot' in cc_args: - idx = cc_args.index('-isysroot') - sysroot = cc_args[idx+1] - elif '-isysroot' in compiler_so: - idx = compiler_so.index('-isysroot') - sysroot = compiler_so[idx+1] - - if sysroot and not os.path.isdir(sysroot): - log.warn("Compiling with an SDK that doesn't seem to exist: %s", - sysroot) - log.warn("Please check your Xcode installation") - - return compiler_so class UnixCCompiler(CCompiler): @@ -168,7 +109,8 @@ def _compile(self, obj, src, ext, cc_args, extra_postargs, pp_opts): compiler_so = self.compiler_so if sys.platform == 'darwin': - compiler_so = _darwin_compiler_fixup(compiler_so, cc_args + extra_postargs) + compiler_so = _osx_support.compiler_fixup(compiler_so, + cc_args + extra_postargs) try: self.spawn(compiler_so + cc_args + [src, '-o', obj] + extra_postargs) @@ -247,7 +189,7 @@ linker[i] = self.compiler_cxx[i] if sys.platform == 'darwin': - linker = _darwin_compiler_fixup(linker, ld_args) + linker = _osx_support.compiler_fixup(linker, ld_args) self.spawn(linker + ld_args) except DistutilsExecError as msg: diff --git a/Lib/distutils/util.py b/Lib/distutils/util.py --- a/Lib/distutils/util.py +++ b/Lib/distutils/util.py @@ -94,94 +94,10 @@ if m: release = m.group() elif osname[:6] == "darwin": - # - # For our purposes, we'll assume that the system version from - # distutils' perspective is what MACOSX_DEPLOYMENT_TARGET is set - # to. This makes the compatibility story a bit more sane because the - # machine is going to compile and link as if it were - # MACOSX_DEPLOYMENT_TARGET. - from distutils.sysconfig import get_config_vars - cfgvars = get_config_vars() - - macver = cfgvars.get('MACOSX_DEPLOYMENT_TARGET') - - if 1: - # Always calculate the release of the running machine, - # needed to determine if we can build fat binaries or not. - - macrelease = macver - # Get the system version. Reading this plist is a documented - # way to get the system version (see the documentation for - # the Gestalt Manager) - try: - f = open('/System/Library/CoreServices/SystemVersion.plist') - except IOError: - # We're on a plain darwin box, fall back to the default - # behaviour. - pass - else: - try: - m = re.search( - r'<key>ProductUserVisibleVersion</key>\s*' + - r'<string>(.*?)</string>', f.read()) - if m is not None: - macrelease = '.'.join(m.group(1).split('.')[:2]) - # else: fall back to the default behaviour - finally: - f.close() - - if not macver: - macver = macrelease - - if macver: - from distutils.sysconfig import get_config_vars - release = macver - osname = "macosx" - - if (macrelease + '.') >= '10.4.' and \ - '-arch' in get_config_vars().get('CFLAGS', '').strip(): - # The universal build will build fat binaries, but not on - # systems before 10.4 - # - # Try to detect 4-way universal builds, those have machine-type - # 'universal' instead of 'fat'. - - machine = 'fat' - cflags = get_config_vars().get('CFLAGS') - - archs = re.findall('-arch\s+(\S+)', cflags) - archs = tuple(sorted(set(archs))) - - if len(archs) == 1: - machine = archs[0] - elif archs == ('i386', 'ppc'): - machine = 'fat' - elif archs == ('i386', 'x86_64'): - machine = 'intel' - elif archs == ('i386', 'ppc', 'x86_64'): - machine = 'fat3' - elif archs == ('ppc64', 'x86_64'): - machine = 'fat64' - elif archs == ('i386', 'ppc', 'ppc64', 'x86_64'): - machine = 'universal' - else: - raise ValueError( - "Don't know machine value for archs=%r"%(archs,)) - - elif machine == 'i386': - # On OSX the machine type returned by uname is always the - # 32-bit variant, even if the executable architecture is - # the 64-bit variant - if sys.maxsize >= 2**32: - machine = 'x86_64' - - elif machine in ('PowerPC', 'Power_Macintosh'): - # Pick a sane name for the PPC architecture. - machine = 'ppc' - - # See 'i386' case - if sys.maxsize >= 2**32: - machine = 'ppc64' + import _osx_support, distutils.sysconfig + osname, release, machine = _osx_support.get_platform_osx( + distutils.sysconfig.get_config_vars(), + osname, release, machine) return "%s-%s-%s" % (osname, release, machine) diff --git a/Lib/sysconfig.py b/Lib/sysconfig.py --- a/Lib/sysconfig.py +++ b/Lib/sysconfig.py @@ -501,64 +501,11 @@ srcdir = os.path.join(base, _CONFIG_VARS['srcdir']) _CONFIG_VARS['srcdir'] = os.path.normpath(srcdir) + # OS X platforms require special customization to handle + # multi-architecture, multi-os-version installers if sys.platform == 'darwin': - kernel_version = os.uname()[2] # Kernel version (8.4.3) - major_version = int(kernel_version.split('.')[0]) - - if major_version < 8: - # On Mac OS X before 10.4, check if -arch and -isysroot - # are in CFLAGS or LDFLAGS and remove them if they are. - # This is needed when building extensions on a 10.3 system - # using a universal build of python. - for key in ('LDFLAGS', 'BASECFLAGS', - # a number of derived variables. These need to be - # patched up as well. - 'CFLAGS', 'PY_CFLAGS', 'BLDSHARED'): - flags = _CONFIG_VARS[key] - flags = re.sub('-arch\s+\w+\s', ' ', flags) - flags = re.sub('-isysroot [^ \t]*', ' ', flags) - _CONFIG_VARS[key] = flags - else: - # Allow the user to override the architecture flags using - # an environment variable. - # NOTE: This name was introduced by Apple in OSX 10.5 and - # is used by several scripting languages distributed with - # that OS release. - if 'ARCHFLAGS' in os.environ: - arch = os.environ['ARCHFLAGS'] - for key in ('LDFLAGS', 'BASECFLAGS', - # a number of derived variables. These need to be - # patched up as well. - 'CFLAGS', 'PY_CFLAGS', 'BLDSHARED'): - - flags = _CONFIG_VARS[key] - flags = re.sub('-arch\s+\w+\s', ' ', flags) - flags = flags + ' ' + arch - _CONFIG_VARS[key] = flags - - # If we're on OSX 10.5 or later and the user tries to - # compiles an extension using an SDK that is not present - # on the current machine it is better to not use an SDK - # than to fail. - # - # The major usecase for this is users using a Python.org - # binary installer on OSX 10.6: that installer uses - # the 10.4u SDK, but that SDK is not installed by default - # when you install Xcode. - # - CFLAGS = _CONFIG_VARS.get('CFLAGS', '') - m = re.search('-isysroot\s+(\S+)', CFLAGS) - if m is not None: - sdk = m.group(1) - if not os.path.exists(sdk): - for key in ('LDFLAGS', 'BASECFLAGS', - # a number of derived variables. These need to be - # patched up as well. - 'CFLAGS', 'PY_CFLAGS', 'BLDSHARED'): - - flags = _CONFIG_VARS[key] - flags = re.sub('-isysroot\s+\S+(\s|$)', ' ', flags) - _CONFIG_VARS[key] = flags + import _osx_support + _osx_support.customize_config_vars(_CONFIG_VARS) if args: vals = [] @@ -656,92 +603,10 @@ if m: release = m.group() elif osname[:6] == "darwin": - # - # For our purposes, we'll assume that the system version from - # distutils' perspective is what MACOSX_DEPLOYMENT_TARGET is set - # to. This makes the compatibility story a bit more sane because the - # machine is going to compile and link as if it were - # MACOSX_DEPLOYMENT_TARGET. - # - cfgvars = get_config_vars() - macver = cfgvars.get('MACOSX_DEPLOYMENT_TARGET') - - if 1: - # Always calculate the release of the running machine, - # needed to determine if we can build fat binaries or not. - - macrelease = macver - # Get the system version. Reading this plist is a documented - # way to get the system version (see the documentation for - # the Gestalt Manager) - try: - f = open('/System/Library/CoreServices/SystemVersion.plist') - except IOError: - # We're on a plain darwin box, fall back to the default - # behaviour. - pass - else: - try: - m = re.search( - r'<key>ProductUserVisibleVersion</key>\s*' + - r'<string>(.*?)</string>', f.read()) - if m is not None: - macrelease = '.'.join(m.group(1).split('.')[:2]) - # else: fall back to the default behaviour - finally: - f.close() - - if not macver: - macver = macrelease - - if macver: - release = macver - osname = "macosx" - - if (macrelease + '.') >= '10.4.' and \ - '-arch' in get_config_vars().get('CFLAGS', '').strip(): - # The universal build will build fat binaries, but not on - # systems before 10.4 - # - # Try to detect 4-way universal builds, those have machine-type - # 'universal' instead of 'fat'. - - machine = 'fat' - cflags = get_config_vars().get('CFLAGS') - - archs = re.findall('-arch\s+(\S+)', cflags) - archs = tuple(sorted(set(archs))) - - if len(archs) == 1: - machine = archs[0] - elif archs == ('i386', 'ppc'): - machine = 'fat' - elif archs == ('i386', 'x86_64'): - machine = 'intel' - elif archs == ('i386', 'ppc', 'x86_64'): - machine = 'fat3' - elif archs == ('ppc64', 'x86_64'): - machine = 'fat64' - elif archs == ('i386', 'ppc', 'ppc64', 'x86_64'): - machine = 'universal' - else: - raise ValueError( - "Don't know machine value for archs=%r"%(archs,)) - - elif machine == 'i386': - # On OSX the machine type returned by uname is always the - # 32-bit variant, even if the executable architecture is - # the 64-bit variant - if sys.maxsize >= 2**32: - machine = 'x86_64' - - elif machine in ('PowerPC', 'Power_Macintosh'): - # Pick a sane name for the PPC architecture. - # See 'i386' case - if sys.maxsize >= 2**32: - machine = 'ppc64' - else: - machine = 'ppc' + import _osx_support + osname, release, machine = _osx_support.get_platform_osx( + get_config_vars(), + osname, release, machine) return "%s-%s-%s" % (osname, release, machine) diff --git a/Lib/test/test__osx_support.py b/Lib/test/test__osx_support.py new file mode 100644 --- /dev/null +++ b/Lib/test/test__osx_support.py @@ -0,0 +1,279 @@ +""" +Test suite for _osx_support: shared OS X support functions. +""" + +import os +import platform +import shutil +import stat +import sys +import unittest + +import test.support + +import _osx_support + +(a)unittest.skipUnless(sys.platform.startswith("darwin"), "requires OS X") +class Test_OSXSupport(unittest.TestCase): + + def setUp(self): + self.maxDiff = None + self.prog_name = 'bogus_program_xxxx' + self.temp_path_dir = os.path.abspath(os.getcwd()) + self.env = test.support.EnvironmentVarGuard() + self.addCleanup(self.env.__exit__) + for cv in ('CFLAGS', 'LDFLAGS', 'CPPFLAGS', + 'BASECFLAGS', 'BLDSHARED', 'LDSHARED', 'CC', + 'CXX', 'PY_CFLAGS', 'PY_LDFLAGS', 'PY_CPPFLAGS', + 'PY_CORE_CFLAGS'): + if cv in self.env: + self.env.unset(cv) + + def add_expected_saved_initial_values(self, config_vars, expected_vars): + # Ensure that the initial values for all modified config vars + # are also saved with modified keys. + expected_vars.update(('_OSX_SUPPORT_INITIAL_'+ k, + config_vars[k]) for k in config_vars + if config_vars[k] != expected_vars[k]) + + def test__find_executable(self): + if self.env['PATH']: + self.env['PATH'] = self.env['PATH'] + ':' + self.env['PATH'] = self.env['PATH'] + os.path.abspath(self.temp_path_dir) + test.support.unlink(self.prog_name) + self.assertIsNone(_osx_support._find_executable(self.prog_name)) + self.addCleanup(test.support.unlink, self.prog_name) + with open(self.prog_name, 'w') as f: + f.write("#!/bin/sh\n/bin/echo OK\n") + os.chmod(self.prog_name, stat.S_IRWXU) + self.assertEqual(self.prog_name, + _osx_support._find_executable(self.prog_name)) + + def test__read_output(self): + if self.env['PATH']: + self.env['PATH'] = self.env['PATH'] + ':' + self.env['PATH'] = self.env['PATH'] + os.path.abspath(self.temp_path_dir) + test.support.unlink(self.prog_name) + self.addCleanup(test.support.unlink, self.prog_name) + with open(self.prog_name, 'w') as f: + f.write("#!/bin/sh\n/bin/echo ExpectedOutput\n") + os.chmod(self.prog_name, stat.S_IRWXU) + self.assertEqual('ExpectedOutput', + _osx_support._read_output(self.prog_name)) + + def test__find_build_tool(self): + out = _osx_support._find_build_tool('cc') + self.assertTrue(os.path.isfile(out), + 'cc not found - check xcode-select') + + def test__get_system_version(self): + self.assertTrue(platform.mac_ver()[0].startswith( + _osx_support._get_system_version())) + + def test__remove_original_values(self): + config_vars = { + 'CC': 'gcc-test -pthreads', + } + expected_vars = { + 'CC': 'clang -pthreads', + } + cv = 'CC' + newvalue = 'clang -pthreads' + _osx_support._save_modified_value(config_vars, cv, newvalue) + self.assertNotEqual(expected_vars, config_vars) + _osx_support._remove_original_values(config_vars) + self.assertEqual(expected_vars, config_vars) + + def test__save_modified_value(self): + config_vars = { + 'CC': 'gcc-test -pthreads', + } + expected_vars = { + 'CC': 'clang -pthreads', + } + self.add_expected_saved_initial_values(config_vars, expected_vars) + cv = 'CC' + newvalue = 'clang -pthreads' + _osx_support._save_modified_value(config_vars, cv, newvalue) + self.assertEqual(expected_vars, config_vars) + + def test__save_modified_value_unchanged(self): + config_vars = { + 'CC': 'gcc-test -pthreads', + } + expected_vars = config_vars.copy() + cv = 'CC' + newvalue = 'gcc-test -pthreads' + _osx_support._save_modified_value(config_vars, cv, newvalue) + self.assertEqual(expected_vars, config_vars) + + def test__supports_universal_builds(self): + import platform + self.assertEqual(platform.mac_ver()[0].split('.') >= ['10', '4'], + _osx_support._supports_universal_builds()) + + def test__find_appropriate_compiler(self): + compilers = ( + ('gcc-test', 'i686-apple-darwin11-llvm-gcc-4.2'), + ('clang', 'clang version 3.1'), + ) + config_vars = { + 'CC': 'gcc-test -pthreads', + 'CXX': 'cc++-test', + 'CFLAGS': '-fno-strict-aliasing -g -O3 -arch ppc -arch i386 ', + 'LDFLAGS': '-arch ppc -arch i386 -g', + 'CPPFLAGS': '-I. -isysroot /Developer/SDKs/MacOSX10.4u.sdk', + 'BLDSHARED': 'gcc-test -bundle -arch ppc -arch i386 -g', + 'LDSHARED': 'gcc-test -bundle -arch ppc -arch i386 ' + '-isysroot /Developer/SDKs/MacOSX10.4u.sdk -g', + } + expected_vars = { + 'CC': 'clang -pthreads', + 'CXX': 'clang++', + 'CFLAGS': '-fno-strict-aliasing -g -O3 -arch ppc -arch i386 ', + 'LDFLAGS': '-arch ppc -arch i386 -g', + 'CPPFLAGS': '-I. -isysroot /Developer/SDKs/MacOSX10.4u.sdk', + 'BLDSHARED': 'clang -bundle -arch ppc -arch i386 -g', + 'LDSHARED': 'clang -bundle -arch ppc -arch i386 ' + '-isysroot /Developer/SDKs/MacOSX10.4u.sdk -g', + } + self.add_expected_saved_initial_values(config_vars, expected_vars) + + suffix = (':' + self.env['PATH']) if self.env['PATH'] else '' + self.env['PATH'] = os.path.abspath(self.temp_path_dir) + suffix + for c_name, c_output in compilers: + test.support.unlink(c_name) + self.addCleanup(test.support.unlink, c_name) + with open(c_name, 'w') as f: + f.write("#!/bin/sh\n/bin/echo " + c_output) + os.chmod(c_name, stat.S_IRWXU) + self.assertEqual(expected_vars, + _osx_support._find_appropriate_compiler( + config_vars)) + + def test__remove_universal_flags(self): + config_vars = { + 'CFLAGS': '-fno-strict-aliasing -g -O3 -arch ppc -arch i386 ', + 'LDFLAGS': '-arch ppc -arch i386 -g', + 'CPPFLAGS': '-I. -isysroot /Developer/SDKs/MacOSX10.4u.sdk', + 'BLDSHARED': 'gcc-4.0 -bundle -arch ppc -arch i386 -g', + 'LDSHARED': 'gcc-4.0 -bundle -arch ppc -arch i386 ' + '-isysroot /Developer/SDKs/MacOSX10.4u.sdk -g', + } + expected_vars = { + 'CFLAGS': '-fno-strict-aliasing -g -O3 ', + 'LDFLAGS': ' -g', + 'CPPFLAGS': '-I. ', + 'BLDSHARED': 'gcc-4.0 -bundle -g', + 'LDSHARED': 'gcc-4.0 -bundle -g', + } + self.add_expected_saved_initial_values(config_vars, expected_vars) + + self.assertEqual(expected_vars, + _osx_support._remove_universal_flags( + config_vars)) + + def test__remove_unsupported_archs(self): + config_vars = { + 'CC': 'clang', + 'CFLAGS': '-fno-strict-aliasing -g -O3 -arch ppc -arch i386 ', + 'LDFLAGS': '-arch ppc -arch i386 -g', + 'CPPFLAGS': '-I. -isysroot /Developer/SDKs/MacOSX10.4u.sdk', + 'BLDSHARED': 'gcc-4.0 -bundle -arch ppc -arch i386 -g', + 'LDSHARED': 'gcc-4.0 -bundle -arch ppc -arch i386 ' + '-isysroot /Developer/SDKs/MacOSX10.4u.sdk -g', + } + expected_vars = { + 'CC': 'clang', + 'CFLAGS': '-fno-strict-aliasing -g -O3 -arch i386 ', + 'LDFLAGS': ' -arch i386 -g', + 'CPPFLAGS': '-I. -isysroot /Developer/SDKs/MacOSX10.4u.sdk', + 'BLDSHARED': 'gcc-4.0 -bundle -arch i386 -g', + 'LDSHARED': 'gcc-4.0 -bundle -arch i386 ' + '-isysroot /Developer/SDKs/MacOSX10.4u.sdk -g', + } + self.add_expected_saved_initial_values(config_vars, expected_vars) + + suffix = (':' + self.env['PATH']) if self.env['PATH'] else '' + self.env['PATH'] = os.path.abspath(self.temp_path_dir) + suffix + c_name = 'clang' + test.support.unlink(c_name) + self.addCleanup(test.support.unlink, c_name) + # exit status 255 means no PPC support in this compiler chain + with open(c_name, 'w') as f: + f.write("#!/bin/sh\nexit 255") + os.chmod(c_name, stat.S_IRWXU) + self.assertEqual(expected_vars, + _osx_support._remove_unsupported_archs( + config_vars)) + + def test__override_all_archs(self): + self.env['ARCHFLAGS'] = '-arch x86_64' + config_vars = { + 'CC': 'clang', + 'CFLAGS': '-fno-strict-aliasing -g -O3 -arch ppc -arch i386 ', + 'LDFLAGS': '-arch ppc -arch i386 -g', + 'CPPFLAGS': '-I. -isysroot /Developer/SDKs/MacOSX10.4u.sdk', + 'BLDSHARED': 'gcc-4.0 -bundle -arch ppc -arch i386 -g', + 'LDSHARED': 'gcc-4.0 -bundle -arch ppc -arch i386 ' + '-isysroot /Developer/SDKs/MacOSX10.4u.sdk -g', + } + expected_vars = { + 'CC': 'clang', + 'CFLAGS': '-fno-strict-aliasing -g -O3 -arch x86_64', + 'LDFLAGS': ' -g -arch x86_64', + 'CPPFLAGS': '-I. -isysroot /Developer/SDKs/MacOSX10.4u.sdk', + 'BLDSHARED': 'gcc-4.0 -bundle -g -arch x86_64', + 'LDSHARED': 'gcc-4.0 -bundle -isysroot ' + '/Developer/SDKs/MacOSX10.4u.sdk -g -arch x86_64', + } + self.add_expected_saved_initial_values(config_vars, expected_vars) + + self.assertEqual(expected_vars, + _osx_support._override_all_archs( + config_vars)) + + def test__check_for_unavailable_sdk(self): + config_vars = { + 'CC': 'clang', + 'CFLAGS': '-fno-strict-aliasing -g -O3 -arch ppc -arch i386 ' + '-isysroot /Developer/SDKs/MacOSX10.1.sdk', + 'LDFLAGS': '-arch ppc -arch i386 -g', + 'CPPFLAGS': '-I. -isysroot /Developer/SDKs/MacOSX10.1.sdk', + 'BLDSHARED': 'gcc-4.0 -bundle -arch ppc -arch i386 -g', + 'LDSHARED': 'gcc-4.0 -bundle -arch ppc -arch i386 ' + '-isysroot /Developer/SDKs/MacOSX10.1.sdk -g', + } + expected_vars = { + 'CC': 'clang', + 'CFLAGS': '-fno-strict-aliasing -g -O3 -arch ppc -arch i386 ' + ' ', + 'LDFLAGS': '-arch ppc -arch i386 -g', + 'CPPFLAGS': '-I. ', + 'BLDSHARED': 'gcc-4.0 -bundle -arch ppc -arch i386 -g', + 'LDSHARED': 'gcc-4.0 -bundle -arch ppc -arch i386 ' + ' -g', + } + self.add_expected_saved_initial_values(config_vars, expected_vars) + + self.assertEqual(expected_vars, + _osx_support._check_for_unavailable_sdk( + config_vars)) + + def test_get_platform_osx(self): + # Note, get_platform_osx is currently tested more extensively + # indirectly by test_sysconfig and test_distutils + config_vars = { + 'CFLAGS': '-fno-strict-aliasing -g -O3 -arch ppc -arch i386 ' + '-isysroot /Developer/SDKs/MacOSX10.1.sdk', + 'MACOSX_DEPLOYMENT_TARGET': '10.6', + } + result = _osx_support.get_platform_osx(config_vars, ' ', ' ', ' ') + self.assertEqual(('macosx', '10.6', 'fat'), result) + +def test_main(): + if sys.platform == 'darwin': + test.support.run_unittest(Test_OSXSupport) + +if __name__ == "__main__": + test_main() diff --git a/Lib/test/test_sysconfig.py b/Lib/test/test_sysconfig.py --- a/Lib/test/test_sysconfig.py +++ b/Lib/test/test_sysconfig.py @@ -15,6 +15,7 @@ get_path, get_path_names, _INSTALL_SCHEMES, _get_default_scheme, _expand_vars, get_scheme_names, get_config_var, _main) +import _osx_support class TestSysConfig(unittest.TestCase): @@ -135,8 +136,7 @@ ('Darwin Kernel Version 8.11.1: ' 'Wed Oct 10 18:23:28 PDT 2007; ' 'root:xnu-792.25.20~1/RELEASE_I386'), 'PowerPC')) - - + _osx_support._remove_original_values(get_config_vars()) get_config_vars()['MACOSX_DEPLOYMENT_TARGET'] = '10.3' get_config_vars()['CFLAGS'] = ('-fno-strict-aliasing -DNDEBUG -g ' @@ -151,12 +151,11 @@ finally: sys.maxsize = maxint - self._set_uname(('Darwin', 'macziade', '8.11.1', ('Darwin Kernel Version 8.11.1: ' 'Wed Oct 10 18:23:28 PDT 2007; ' 'root:xnu-792.25.20~1/RELEASE_I386'), 'i386')) - get_config_vars()['MACOSX_DEPLOYMENT_TARGET'] = '10.3' + _osx_support._remove_original_values(get_config_vars()) get_config_vars()['MACOSX_DEPLOYMENT_TARGET'] = '10.3' get_config_vars()['CFLAGS'] = ('-fno-strict-aliasing -DNDEBUG -g ' @@ -171,6 +170,7 @@ sys.maxsize = maxint # macbook with fat binaries (fat, universal or fat64) + _osx_support._remove_original_values(get_config_vars()) get_config_vars()['MACOSX_DEPLOYMENT_TARGET'] = '10.4' get_config_vars()['CFLAGS'] = ('-arch ppc -arch i386 -isysroot ' '/Developer/SDKs/MacOSX10.4u.sdk ' @@ -179,6 +179,7 @@ self.assertEqual(get_platform(), 'macosx-10.4-fat') + _osx_support._remove_original_values(get_config_vars()) get_config_vars()['CFLAGS'] = ('-arch x86_64 -arch i386 -isysroot ' '/Developer/SDKs/MacOSX10.4u.sdk ' '-fno-strict-aliasing -fno-common ' @@ -186,18 +187,21 @@ self.assertEqual(get_platform(), 'macosx-10.4-intel') + _osx_support._remove_original_values(get_config_vars()) get_config_vars()['CFLAGS'] = ('-arch x86_64 -arch ppc -arch i386 -isysroot ' '/Developer/SDKs/MacOSX10.4u.sdk ' '-fno-strict-aliasing -fno-common ' '-dynamic -DNDEBUG -g -O3') self.assertEqual(get_platform(), 'macosx-10.4-fat3') + _osx_support._remove_original_values(get_config_vars()) get_config_vars()['CFLAGS'] = ('-arch ppc64 -arch x86_64 -arch ppc -arch i386 -isysroot ' '/Developer/SDKs/MacOSX10.4u.sdk ' '-fno-strict-aliasing -fno-common ' '-dynamic -DNDEBUG -g -O3') self.assertEqual(get_platform(), 'macosx-10.4-universal') + _osx_support._remove_original_values(get_config_vars()) get_config_vars()['CFLAGS'] = ('-arch x86_64 -arch ppc64 -isysroot ' '/Developer/SDKs/MacOSX10.4u.sdk ' '-fno-strict-aliasing -fno-common ' @@ -206,12 +210,13 @@ self.assertEqual(get_platform(), 'macosx-10.4-fat64') for arch in ('ppc', 'i386', 'x86_64', 'ppc64'): + _osx_support._remove_original_values(get_config_vars()) get_config_vars()['CFLAGS'] = ('-arch %s -isysroot ' '/Developer/SDKs/MacOSX10.4u.sdk ' '-fno-strict-aliasing -fno-common ' - '-dynamic -DNDEBUG -g -O3'%(arch,)) + '-dynamic -DNDEBUG -g -O3' % arch) - self.assertEqual(get_platform(), 'macosx-10.4-%s'%(arch,)) + self.assertEqual(get_platform(), 'macosx-10.4-%s' % arch) # linux debian sarge os.name = 'posix' diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -792,6 +792,21 @@ - Issue #8033: sqlite3: Fix 64-bit integer handling in user functions on 32-bit architectures. Initial patch by Philippe Devalkeneer. +- Issue #13590: OS X Xcode 4 - improve support for universal extension modules + In particular, fix extension module build failures when trying to use + 32-bit-only installer Pythons on systems with Xcode 4 (currently + OS X 10.8, 10.7, and optionally 10.6). + * Backport 3.3.0 fixes to 3.2 branch (for release in 3.2.4) + * Since Xcode 4 removes ppc support, extension module builds now + check for ppc compiler support and by default remove ppc and + ppc64 archs when they are not available. + * Extension module builds now revert to using system installed + headers and libs (/usr and /System/Library) if the SDK used + to build the interpreter is not installed or has moved. + * Try to avoid building extension modules with deprecated + and problematic Apple llvm-gcc compiler. If original compiler + is not available, use clang instead by default. + Extension Modules ----------------- -- Repository URL: http://hg.python.org/cpython

1 0

cpython (2.7): Issue #13590: OS X Xcode 4 - improve support for universal extension modules
by ned.deily Jan. 31, 2013

Jan. 31, 2013

http://hg.python.org/cpython/rev/1d061f83a6bf changeset: 81843:1d061f83a6bf branch: 2.7 parent: 81836:7e4c5914ba76 user: Ned Deily <nad(a)acm.org> date: Thu Jan 31 01:24:55 2013 -0800 summary: Issue #13590: OS X Xcode 4 - improve support for universal extension modules In particular, fix extension module build failures when trying to use 32-bit-only installer Pythons on systems with Xcode 4 (currently OS X 10.8, 10.7, and optionally 10.6). * Backport 3.3.0 fixes to 2.7 branch (for release in 2.7.4) * Since Xcode 4 removes ppc support, extension module builds now check for ppc compiler support and by default remove ppc and ppc64 archs when they are not available. * Extension module builds now revert to using system installed headers and libs (/usr and /System/Library) if the SDK used to build the interpreter is not installed or has moved. * Try to avoid building extension modules with deprecated and problematic Apple llvm-gcc compiler. If original compiler is not available, use clang instead by default. files: Lib/_osx_support.py | 488 ++++++++++++++ Lib/distutils/sysconfig.py | 111 +-- Lib/distutils/tests/test_sysconfig.py | 29 + Lib/distutils/unixccompiler.py | 70 +- Lib/distutils/util.py | 92 +-- Lib/sysconfig.py | 150 +---- Lib/test/test__osx_support.py | 279 ++++++++ Lib/test/test_sysconfig.py | 9 + Misc/NEWS | 15 + 9 files changed, 859 insertions(+), 384 deletions(-) diff --git a/Lib/_osx_support.py b/Lib/_osx_support.py new file mode 100644 --- /dev/null +++ b/Lib/_osx_support.py @@ -0,0 +1,488 @@ +"""Shared OS X support functions.""" + +import os +import re +import sys + +__all__ = [ + 'compiler_fixup', + 'customize_config_vars', + 'customize_compiler', + 'get_platform_osx', +] + +# configuration variables that may contain universal build flags, +# like "-arch" or "-isdkroot", that may need customization for +# the user environment +_UNIVERSAL_CONFIG_VARS = ('CFLAGS', 'LDFLAGS', 'CPPFLAGS', 'BASECFLAGS', + 'BLDSHARED', 'LDSHARED', 'CC', 'CXX', + 'PY_CFLAGS', 'PY_LDFLAGS', 'PY_CPPFLAGS', + 'PY_CORE_CFLAGS') + +# configuration variables that may contain compiler calls +_COMPILER_CONFIG_VARS = ('BLDSHARED', 'LDSHARED', 'CC', 'CXX') + +# prefix added to original configuration variable names +_INITPRE = '_OSX_SUPPORT_INITIAL_' + + +def _find_executable(executable, path=None): + """Tries to find 'executable' in the directories listed in 'path'. + + A string listing directories separated by 'os.pathsep'; defaults to + os.environ['PATH']. Returns the complete filename or None if not found. + """ + if path is None: + path = os.environ['PATH'] + + paths = path.split(os.pathsep) + base, ext = os.path.splitext(executable) + + if (sys.platform == 'win32' or os.name == 'os2') and (ext != '.exe'): + executable = executable + '.exe' + + if not os.path.isfile(executable): + for p in paths: + f = os.path.join(p, executable) + if os.path.isfile(f): + # the file exists, we have a shot at spawn working + return f + return None + else: + return executable + + +def _read_output(commandstring): + """Output from succesful command execution or None""" + # Similar to os.popen(commandstring, "r").read(), + # but without actually using os.popen because that + # function is not usable during python bootstrap. + # tempfile is also not available then. + import contextlib + try: + import tempfile + fp = tempfile.NamedTemporaryFile() + except ImportError: + fp = open("/tmp/_osx_support.%s"%( + os.getpid(),), "w+b") + + with contextlib.closing(fp) as fp: + cmd = "%s 2>/dev/null >'%s'" % (commandstring, fp.name) + return fp.read().decode('utf-8').strip() if not os.system(cmd) else None + + +def _find_build_tool(toolname): + """Find a build tool on current path or using xcrun""" + return (_find_executable(toolname) + or _read_output("/usr/bin/xcrun -find %s" % (toolname,)) + or '' + ) + +_SYSTEM_VERSION = None + +def _get_system_version(): + """Return the OS X system version as a string""" + # Reading this plist is a documented way to get the system + # version (see the documentation for the Gestalt Manager) + # We avoid using platform.mac_ver to avoid possible bootstrap issues during + # the build of Python itself (distutils is used to build standard library + # extensions). + + global _SYSTEM_VERSION + + if _SYSTEM_VERSION is None: + _SYSTEM_VERSION = '' + try: + f = open('/System/Library/CoreServices/SystemVersion.plist') + except IOError: + # We're on a plain darwin box, fall back to the default + # behaviour. + pass + else: + try: + m = re.search(r'<key>ProductUserVisibleVersion</key>\s*' + r'<string>(.*?)</string>', f.read()) + finally: + f.close() + if m is not None: + _SYSTEM_VERSION = '.'.join(m.group(1).split('.')[:2]) + # else: fall back to the default behaviour + + return _SYSTEM_VERSION + +def _remove_original_values(_config_vars): + """Remove original unmodified values for testing""" + # This is needed for higher-level cross-platform tests of get_platform. + for k in list(_config_vars): + if k.startswith(_INITPRE): + del _config_vars[k] + +def _save_modified_value(_config_vars, cv, newvalue): + """Save modified and original unmodified value of configuration var""" + + oldvalue = _config_vars.get(cv, '') + if (oldvalue != newvalue) and (_INITPRE + cv not in _config_vars): + _config_vars[_INITPRE + cv] = oldvalue + _config_vars[cv] = newvalue + +def _supports_universal_builds(): + """Returns True if universal builds are supported on this system""" + # As an approximation, we assume that if we are running on 10.4 or above, + # then we are running with an Xcode environment that supports universal + # builds, in particular -isysroot and -arch arguments to the compiler. This + # is in support of allowing 10.4 universal builds to run on 10.3.x systems. + + osx_version = _get_system_version() + if osx_version: + try: + osx_version = tuple(int(i) for i in osx_version.split('.')) + except ValueError: + osx_version = '' + return bool(osx_version >= (10, 4)) if osx_version else False + + +def _find_appropriate_compiler(_config_vars): + """Find appropriate C compiler for extension module builds""" + + # Issue #13590: + # The OSX location for the compiler varies between OSX + # (or rather Xcode) releases. With older releases (up-to 10.5) + # the compiler is in /usr/bin, with newer releases the compiler + # can only be found inside Xcode.app if the "Command Line Tools" + # are not installed. + # + # Futhermore, the compiler that can be used varies between + # Xcode releases. Upto Xcode 4 it was possible to use 'gcc-4.2' + # as the compiler, after that 'clang' should be used because + # gcc-4.2 is either not present, or a copy of 'llvm-gcc' that + # miscompiles Python. + + # skip checks if the compiler was overriden with a CC env variable + if 'CC' in os.environ: + return _config_vars + + # The CC config var might contain additional arguments. + # Ignore them while searching. + cc = oldcc = _config_vars['CC'].split()[0] + if not _find_executable(cc): + # Compiler is not found on the shell search PATH. + # Now search for clang, first on PATH (if the Command LIne + # Tools have been installed in / or if the user has provided + # another location via CC). If not found, try using xcrun + # to find an uninstalled clang (within a selected Xcode). + + # NOTE: Cannot use subprocess here because of bootstrap + # issues when building Python itself (and os.popen is + # implemented on top of subprocess and is therefore not + # usable as well) + + cc = _find_build_tool('clang') + + elif os.path.basename(cc).startswith('gcc'): + # Compiler is GCC, check if it is LLVM-GCC + data = _read_output("'%s' --version" + % (cc.replace("'", "'\"'\"'"),)) + if 'llvm-gcc' in data: + # Found LLVM-GCC, fall back to clang + cc = _find_build_tool('clang') + + if not cc: + raise SystemError( + "Cannot locate working compiler") + + if cc != oldcc: + # Found a replacement compiler. + # Modify config vars using new compiler, if not already explictly + # overriden by an env variable, preserving additional arguments. + for cv in _COMPILER_CONFIG_VARS: + if cv in _config_vars and cv not in os.environ: + cv_split = _config_vars[cv].split() + cv_split[0] = cc if cv != 'CXX' else cc + '++' + _save_modified_value(_config_vars, cv, ' '.join(cv_split)) + + return _config_vars + + +def _remove_universal_flags(_config_vars): + """Remove all universal build arguments from config vars""" + + for cv in _UNIVERSAL_CONFIG_VARS: + # Do not alter a config var explicitly overriden by env var + if cv in _config_vars and cv not in os.environ: + flags = _config_vars[cv] + flags = re.sub('-arch\s+\w+\s', ' ', flags) + flags = re.sub('-isysroot [^ \t]*', ' ', flags) + _save_modified_value(_config_vars, cv, flags) + + return _config_vars + + +def _remove_unsupported_archs(_config_vars): + """Remove any unsupported archs from config vars""" + # Different Xcode releases support different sets for '-arch' + # flags. In particular, Xcode 4.x no longer supports the + # PPC architectures. + # + # This code automatically removes '-arch ppc' and '-arch ppc64' + # when these are not supported. That makes it possible to + # build extensions on OSX 10.7 and later with the prebuilt + # 32-bit installer on the python.org website. + + # skip checks if the compiler was overriden with a CC env variable + if 'CC' in os.environ: + return _config_vars + + if re.search('-arch\s+ppc', _config_vars['CFLAGS']) is not None: + # NOTE: Cannot use subprocess here because of bootstrap + # issues when building Python itself + status = os.system("'%s' -arch ppc -x c /dev/null 2>/dev/null"%( + _config_vars['CC'].replace("'", "'\"'\"'"),)) + # The Apple compiler drivers return status 255 if no PPC + if (status >> 8) == 255: + # Compiler doesn't support PPC, remove the related + # '-arch' flags if not explicitly overridden by an + # environment variable + for cv in _UNIVERSAL_CONFIG_VARS: + if cv in _config_vars and cv not in os.environ: + flags = _config_vars[cv] + flags = re.sub('-arch\s+ppc\w*\s', ' ', flags) + _save_modified_value(_config_vars, cv, flags) + + return _config_vars + + +def _override_all_archs(_config_vars): + """Allow override of all archs with ARCHFLAGS env var""" + # NOTE: This name was introduced by Apple in OSX 10.5 and + # is used by several scripting languages distributed with + # that OS release. + if 'ARCHFLAGS' in os.environ: + arch = os.environ['ARCHFLAGS'] + for cv in _UNIVERSAL_CONFIG_VARS: + if cv in _config_vars and '-arch' in _config_vars[cv]: + flags = _config_vars[cv] + flags = re.sub('-arch\s+\w+\s', ' ', flags) + flags = flags + ' ' + arch + _save_modified_value(_config_vars, cv, flags) + + return _config_vars + + +def _check_for_unavailable_sdk(_config_vars): + """Remove references to any SDKs not available""" + # If we're on OSX 10.5 or later and the user tries to + # compile an extension using an SDK that is not present + # on the current machine it is better to not use an SDK + # than to fail. This is particularly important with + # the standalong Command Line Tools alternative to a + # full-blown Xcode install since the CLT packages do not + # provide SDKs. If the SDK is not present, it is assumed + # that the header files and dev libs have been installed + # to /usr and /System/Library by either a standalone CLT + # package or the CLT component within Xcode. + cflags = _config_vars.get('CFLAGS', '') + m = re.search(r'-isysroot\s+(\S+)', cflags) + if m is not None: + sdk = m.group(1) + if not os.path.exists(sdk): + for cv in _UNIVERSAL_CONFIG_VARS: + # Do not alter a config var explicitly overriden by env var + if cv in _config_vars and cv not in os.environ: + flags = _config_vars[cv] + flags = re.sub(r'-isysroot\s+\S+(?:\s|$)', ' ', flags) + _save_modified_value(_config_vars, cv, flags) + + return _config_vars + + +def compiler_fixup(compiler_so, cc_args): + """ + This function will strip '-isysroot PATH' and '-arch ARCH' from the + compile flags if the user has specified one them in extra_compile_flags. + + This is needed because '-arch ARCH' adds another architecture to the + build, without a way to remove an architecture. Furthermore GCC will + barf if multiple '-isysroot' arguments are present. + """ + stripArch = stripSysroot = False + + compiler_so = list(compiler_so) + + if not _supports_universal_builds(): + # OSX before 10.4.0, these don't support -arch and -isysroot at + # all. + stripArch = stripSysroot = True + else: + stripArch = '-arch' in cc_args + stripSysroot = '-isysroot' in cc_args + + if stripArch or 'ARCHFLAGS' in os.environ: + while True: + try: + index = compiler_so.index('-arch') + # Strip this argument and the next one: + del compiler_so[index:index+2] + except ValueError: + break + + if 'ARCHFLAGS' in os.environ and not stripArch: + # User specified different -arch flags in the environ, + # see also distutils.sysconfig + compiler_so = compiler_so + os.environ['ARCHFLAGS'].split() + + if stripSysroot: + while True: + try: + index = compiler_so.index('-isysroot') + # Strip this argument and the next one: + del compiler_so[index:index+2] + except ValueError: + break + + # Check if the SDK that is used during compilation actually exists, + # the universal build requires the usage of a universal SDK and not all + # users have that installed by default. + sysroot = None + if '-isysroot' in cc_args: + idx = cc_args.index('-isysroot') + sysroot = cc_args[idx+1] + elif '-isysroot' in compiler_so: + idx = compiler_so.index('-isysroot') + sysroot = compiler_so[idx+1] + + if sysroot and not os.path.isdir(sysroot): + from distutils import log + log.warn("Compiling with an SDK that doesn't seem to exist: %s", + sysroot) + log.warn("Please check your Xcode installation") + + return compiler_so + + +def customize_config_vars(_config_vars): + """Customize Python build configuration variables. + + Called internally from sysconfig with a mutable mapping + containing name/value pairs parsed from the configured + makefile used to build this interpreter. Returns + the mapping updated as needed to reflect the environment + in which the interpreter is running; in the case of + a Python from a binary installer, the installed + environment may be very different from the build + environment, i.e. different OS levels, different + built tools, different available CPU architectures. + + This customization is performed whenever + distutils.sysconfig.get_config_vars() is first + called. It may be used in environments where no + compilers are present, i.e. when installing pure + Python dists. Customization of compiler paths + and detection of unavailable archs is deferred + until the first extention module build is + requested (in distutils.sysconfig.customize_compiler). + + Currently called from distutils.sysconfig + """ + + if not _supports_universal_builds(): + # On Mac OS X before 10.4, check if -arch and -isysroot + # are in CFLAGS or LDFLAGS and remove them if they are. + # This is needed when building extensions on a 10.3 system + # using a universal build of python. + _remove_universal_flags(_config_vars) + + # Allow user to override all archs with ARCHFLAGS env var + _override_all_archs(_config_vars) + + # Remove references to sdks that are not found + _check_for_unavailable_sdk(_config_vars) + + return _config_vars + + +def customize_compiler(_config_vars): + """Customize compiler path and configuration variables. + + This customization is performed when the first + extension module build is requested + in distutils.sysconfig.customize_compiler). + """ + + # Find a compiler to use for extension module builds + _find_appropriate_compiler(_config_vars) + + # Remove ppc arch flags if not supported here + _remove_unsupported_archs(_config_vars) + + # Allow user to override all archs with ARCHFLAGS env var + _override_all_archs(_config_vars) + + return _config_vars + + +def get_platform_osx(_config_vars, osname, release, machine): + """Filter values for get_platform()""" + # called from get_platform() in sysconfig and distutils.util + # + # For our purposes, we'll assume that the system version from + # distutils' perspective is what MACOSX_DEPLOYMENT_TARGET is set + # to. This makes the compatibility story a bit more sane because the + # machine is going to compile and link as if it were + # MACOSX_DEPLOYMENT_TARGET. + + macver = _config_vars.get('MACOSX_DEPLOYMENT_TARGET', '') + macrelease = _get_system_version() or macver + macver = macver or macrelease + + if macver: + release = macver + osname = "macosx" + + # Use the original CFLAGS value, if available, so that we + # return the same machine type for the platform string. + # Otherwise, distutils may consider this a cross-compiling + # case and disallow installs. + cflags = _config_vars.get(_INITPRE+'CFLAGS', + _config_vars.get('CFLAGS', '')) + if ((macrelease + '.') >= '10.4.' and + '-arch' in cflags.strip()): + # The universal build will build fat binaries, but not on + # systems before 10.4 + + machine = 'fat' + + archs = re.findall('-arch\s+(\S+)', cflags) + archs = tuple(sorted(set(archs))) + + if len(archs) == 1: + machine = archs[0] + elif archs == ('i386', 'ppc'): + machine = 'fat' + elif archs == ('i386', 'x86_64'): + machine = 'intel' + elif archs == ('i386', 'ppc', 'x86_64'): + machine = 'fat3' + elif archs == ('ppc64', 'x86_64'): + machine = 'fat64' + elif archs == ('i386', 'ppc', 'ppc64', 'x86_64'): + machine = 'universal' + else: + raise ValueError( + "Don't know machine value for archs=%r" % (archs,)) + + elif machine == 'i386': + # On OSX the machine type returned by uname is always the + # 32-bit variant, even if the executable architecture is + # the 64-bit variant + if sys.maxint >= 2**32: + machine = 'x86_64' + + elif machine in ('PowerPC', 'Power_Macintosh'): + # Pick a sane name for the PPC architecture. + # See 'i386' case + if sys.maxint >= 2**32: + machine = 'ppc64' + else: + machine = 'ppc' + + return (osname, release, machine) diff --git a/Lib/distutils/sysconfig.py b/Lib/distutils/sysconfig.py --- a/Lib/distutils/sysconfig.py +++ b/Lib/distutils/sysconfig.py @@ -141,7 +141,7 @@ "I don't know where Python installs its library " "on platform '%s'" % os.name) -_USE_CLANG = None + def customize_compiler(compiler): """Do any platform-specific customization of a CCompiler instance. @@ -150,6 +150,21 @@ varies across Unices and is stored in Python's Makefile. """ if compiler.compiler_type == "unix": + if sys.platform == "darwin": + # Perform first-time customization of compiler-related + # config vars on OS X now that we know we need a compiler. + # This is primarily to support Pythons from binary + # installers. The kind and paths to build tools on + # the user system may vary significantly from the system + # that Python itself was built on. Also the user OS + # version and build tools may not support the same set + # of CPU architectures for universal builds. + global _config_vars + if not _config_vars.get('CUSTOMIZED_OSX_COMPILER', ''): + import _osx_support + _osx_support.customize_compiler(_config_vars) + _config_vars['CUSTOMIZED_OSX_COMPILER'] = 'True' + (cc, cxx, opt, cflags, ccshared, ldshared, so_ext, ar, ar_flags) = \ get_config_vars('CC', 'CXX', 'OPT', 'CFLAGS', 'CCSHARED', 'LDSHARED', 'SO', 'AR', @@ -157,36 +172,7 @@ newcc = None if 'CC' in os.environ: - newcc = os.environ['CC'] - elif sys.platform == 'darwin' and cc == 'gcc-4.2': - # Issue #13590: - # Since Apple removed gcc-4.2 in Xcode 4.2, we can no - # longer assume it is available for extension module builds. - # If Python was built with gcc-4.2, check first to see if - # it is available on this system; if not, try to use clang - # instead unless the caller explicitly set CC. - global _USE_CLANG - if _USE_CLANG is None: - from distutils import log - from subprocess import Popen, PIPE - p = Popen("! type gcc-4.2 && type clang && exit 2", - shell=True, stdout=PIPE, stderr=PIPE) - p.wait() - if p.returncode == 2: - _USE_CLANG = True - log.warn("gcc-4.2 not found, using clang instead") - else: - _USE_CLANG = False - if _USE_CLANG: - newcc = 'clang' - if newcc: - # On OS X, if CC is overridden, use that as the default - # command for LDSHARED as well - if (sys.platform == 'darwin' - and 'LDSHARED' not in os.environ - and ldshared.startswith(cc)): - ldshared = newcc + ldshared[len(cc):] - cc = newcc + cc = os.environ['CC'] if 'CXX' in os.environ: cxx = os.environ['CXX'] if 'LDSHARED' in os.environ: @@ -518,66 +504,11 @@ _config_vars['prefix'] = PREFIX _config_vars['exec_prefix'] = EXEC_PREFIX + # OS X platforms require special customization to handle + # multi-architecture, multi-os-version installers if sys.platform == 'darwin': - kernel_version = os.uname()[2] # Kernel version (8.4.3) - major_version = int(kernel_version.split('.')[0]) - - if major_version < 8: - # On Mac OS X before 10.4, check if -arch and -isysroot - # are in CFLAGS or LDFLAGS and remove them if they are. - # This is needed when building extensions on a 10.3 system - # using a universal build of python. - for key in ('LDFLAGS', 'BASECFLAGS', 'LDSHARED', - # a number of derived variables. These need to be - # patched up as well. - 'CFLAGS', 'PY_CFLAGS', 'BLDSHARED'): - flags = _config_vars[key] - flags = re.sub('-arch\s+\w+\s', ' ', flags) - flags = re.sub('-isysroot [^ \t]*', ' ', flags) - _config_vars[key] = flags - - else: - - # Allow the user to override the architecture flags using - # an environment variable. - # NOTE: This name was introduced by Apple in OSX 10.5 and - # is used by several scripting languages distributed with - # that OS release. - - if 'ARCHFLAGS' in os.environ: - arch = os.environ['ARCHFLAGS'] - for key in ('LDFLAGS', 'BASECFLAGS', 'LDSHARED', - # a number of derived variables. These need to be - # patched up as well. - 'CFLAGS', 'PY_CFLAGS', 'BLDSHARED'): - - flags = _config_vars[key] - flags = re.sub('-arch\s+\w+\s', ' ', flags) - flags = flags + ' ' + arch - _config_vars[key] = flags - - # If we're on OSX 10.5 or later and the user tries to - # compiles an extension using an SDK that is not present - # on the current machine it is better to not use an SDK - # than to fail. - # - # The major usecase for this is users using a Python.org - # binary installer on OSX 10.6: that installer uses - # the 10.4u SDK, but that SDK is not installed by default - # when you install Xcode. - # - m = re.search('-isysroot\s+(\S+)', _config_vars['CFLAGS']) - if m is not None: - sdk = m.group(1) - if not os.path.exists(sdk): - for key in ('LDFLAGS', 'BASECFLAGS', 'LDSHARED', - # a number of derived variables. These need to be - # patched up as well. - 'CFLAGS', 'PY_CFLAGS', 'BLDSHARED'): - - flags = _config_vars[key] - flags = re.sub('-isysroot\s+\S+(\s|$)', ' ', flags) - _config_vars[key] = flags + import _osx_support + _osx_support.customize_config_vars(_config_vars) if args: vals = [] diff --git a/Lib/distutils/tests/test_sysconfig.py b/Lib/distutils/tests/test_sysconfig.py --- a/Lib/distutils/tests/test_sysconfig.py +++ b/Lib/distutils/tests/test_sysconfig.py @@ -72,6 +72,35 @@ 'OTHER': 'foo'}) + def test_sysconfig_module(self): + import sysconfig as global_sysconfig + self.assertEqual(global_sysconfig.get_config_var('CFLAGS'), sysconfig.get_config_var('CFLAGS')) + self.assertEqual(global_sysconfig.get_config_var('LDFLAGS'), sysconfig.get_config_var('LDFLAGS')) + + @unittest.skipIf(sysconfig.get_config_var('CUSTOMIZED_OSX_COMPILER'),'compiler flags customized') + def test_sysconfig_compiler_vars(self): + # On OS X, binary installers support extension module building on + # various levels of the operating system with differing Xcode + # configurations. This requires customization of some of the + # compiler configuration directives to suit the environment on + # the installed machine. Some of these customizations may require + # running external programs and, so, are deferred until needed by + # the first extension module build. With Python 3.3, only + # the Distutils version of sysconfig is used for extension module + # builds, which happens earlier in the Distutils tests. This may + # cause the following tests to fail since no tests have caused + # the global version of sysconfig to call the customization yet. + # The solution for now is to simply skip this test in this case. + # The longer-term solution is to only have one version of sysconfig. + + import sysconfig as global_sysconfig + if sysconfig.get_config_var('CUSTOMIZED_OSX_COMPILER'): + return + self.assertEqual(global_sysconfig.get_config_var('LDSHARED'), sysconfig.get_config_var('LDSHARED')) + self.assertEqual(global_sysconfig.get_config_var('CC'), sysconfig.get_config_var('CC')) + + + def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(SysconfigTestCase)) diff --git a/Lib/distutils/unixccompiler.py b/Lib/distutils/unixccompiler.py --- a/Lib/distutils/unixccompiler.py +++ b/Lib/distutils/unixccompiler.py @@ -26,6 +26,9 @@ DistutilsExecError, CompileError, LibError, LinkError from distutils import log +if sys.platform == 'darwin': + import _osx_support + # XXX Things not currently handled: # * optimization/debug/warning flags; we just use whatever's in Python's # Makefile and live with it. Is this adequate? If not, we might @@ -41,68 +44,6 @@ # should just happily stuff them into the preprocessor/compiler/linker # options and carry on. -def _darwin_compiler_fixup(compiler_so, cc_args): - """ - This function will strip '-isysroot PATH' and '-arch ARCH' from the - compile flags if the user has specified one them in extra_compile_flags. - - This is needed because '-arch ARCH' adds another architecture to the - build, without a way to remove an architecture. Furthermore GCC will - barf if multiple '-isysroot' arguments are present. - """ - stripArch = stripSysroot = 0 - - compiler_so = list(compiler_so) - kernel_version = os.uname()[2] # 8.4.3 - major_version = int(kernel_version.split('.')[0]) - - if major_version < 8: - # OSX before 10.4.0, these don't support -arch and -isysroot at - # all. - stripArch = stripSysroot = True - else: - stripArch = '-arch' in cc_args - stripSysroot = '-isysroot' in cc_args - - if stripArch or 'ARCHFLAGS' in os.environ: - while 1: - try: - index = compiler_so.index('-arch') - # Strip this argument and the next one: - del compiler_so[index:index+2] - except ValueError: - break - - if 'ARCHFLAGS' in os.environ and not stripArch: - # User specified different -arch flags in the environ, - # see also distutils.sysconfig - compiler_so = compiler_so + os.environ['ARCHFLAGS'].split() - - if stripSysroot: - try: - index = compiler_so.index('-isysroot') - # Strip this argument and the next one: - del compiler_so[index:index+2] - except ValueError: - pass - - # Check if the SDK that is used during compilation actually exists, - # the universal build requires the usage of a universal SDK and not all - # users have that installed by default. - sysroot = None - if '-isysroot' in cc_args: - idx = cc_args.index('-isysroot') - sysroot = cc_args[idx+1] - elif '-isysroot' in compiler_so: - idx = compiler_so.index('-isysroot') - sysroot = compiler_so[idx+1] - - if sysroot and not os.path.isdir(sysroot): - log.warn("Compiling with an SDK that doesn't seem to exist: %s", - sysroot) - log.warn("Please check your Xcode installation") - - return compiler_so class UnixCCompiler(CCompiler): @@ -172,7 +113,8 @@ def _compile(self, obj, src, ext, cc_args, extra_postargs, pp_opts): compiler_so = self.compiler_so if sys.platform == 'darwin': - compiler_so = _darwin_compiler_fixup(compiler_so, cc_args + extra_postargs) + compiler_so = _osx_support.compiler_fixup(compiler_so, + cc_args + extra_postargs) try: self.spawn(compiler_so + cc_args + [src, '-o', obj] + extra_postargs) @@ -251,7 +193,7 @@ linker[i] = self.compiler_cxx[i] if sys.platform == 'darwin': - linker = _darwin_compiler_fixup(linker, ld_args) + linker = _osx_support.compiler_fixup(linker, ld_args) self.spawn(linker + ld_args) except DistutilsExecError, msg: diff --git a/Lib/distutils/util.py b/Lib/distutils/util.py --- a/Lib/distutils/util.py +++ b/Lib/distutils/util.py @@ -93,94 +93,10 @@ if m: release = m.group() elif osname[:6] == "darwin": - # - # For our purposes, we'll assume that the system version from - # distutils' perspective is what MACOSX_DEPLOYMENT_TARGET is set - # to. This makes the compatibility story a bit more sane because the - # machine is going to compile and link as if it were - # MACOSX_DEPLOYMENT_TARGET. - from distutils.sysconfig import get_config_vars - cfgvars = get_config_vars() - - macver = cfgvars.get('MACOSX_DEPLOYMENT_TARGET') - - if 1: - # Always calculate the release of the running machine, - # needed to determine if we can build fat binaries or not. - - macrelease = macver - # Get the system version. Reading this plist is a documented - # way to get the system version (see the documentation for - # the Gestalt Manager) - try: - f = open('/System/Library/CoreServices/SystemVersion.plist') - except IOError: - # We're on a plain darwin box, fall back to the default - # behaviour. - pass - else: - try: - m = re.search( - r'<key>ProductUserVisibleVersion</key>\s*' + - r'<string>(.*?)</string>', f.read()) - if m is not None: - macrelease = '.'.join(m.group(1).split('.')[:2]) - # else: fall back to the default behaviour - finally: - f.close() - - if not macver: - macver = macrelease - - if macver: - from distutils.sysconfig import get_config_vars - release = macver - osname = "macosx" - - if (macrelease + '.') >= '10.4.' and \ - '-arch' in get_config_vars().get('CFLAGS', '').strip(): - # The universal build will build fat binaries, but not on - # systems before 10.4 - # - # Try to detect 4-way universal builds, those have machine-type - # 'universal' instead of 'fat'. - - machine = 'fat' - cflags = get_config_vars().get('CFLAGS') - - archs = re.findall('-arch\s+(\S+)', cflags) - archs = tuple(sorted(set(archs))) - - if len(archs) == 1: - machine = archs[0] - elif archs == ('i386', 'ppc'): - machine = 'fat' - elif archs == ('i386', 'x86_64'): - machine = 'intel' - elif archs == ('i386', 'ppc', 'x86_64'): - machine = 'fat3' - elif archs == ('ppc64', 'x86_64'): - machine = 'fat64' - elif archs == ('i386', 'ppc', 'ppc64', 'x86_64'): - machine = 'universal' - else: - raise ValueError( - "Don't know machine value for archs=%r"%(archs,)) - - elif machine == 'i386': - # On OSX the machine type returned by uname is always the - # 32-bit variant, even if the executable architecture is - # the 64-bit variant - if sys.maxint >= 2**32: - machine = 'x86_64' - - elif machine in ('PowerPC', 'Power_Macintosh'): - # Pick a sane name for the PPC architecture. - machine = 'ppc' - - # See 'i386' case - if sys.maxint >= 2**32: - machine = 'ppc64' + import _osx_support, distutils.sysconfig + osname, release, machine = _osx_support.get_platform_osx( + distutils.sysconfig.get_config_vars(), + osname, release, machine) return "%s-%s-%s" % (osname, release, machine) diff --git a/Lib/sysconfig.py b/Lib/sysconfig.py --- a/Lib/sysconfig.py +++ b/Lib/sysconfig.py @@ -445,64 +445,11 @@ srcdir = os.path.join(base, _CONFIG_VARS['srcdir']) _CONFIG_VARS['srcdir'] = os.path.normpath(srcdir) + # OS X platforms require special customization to handle + # multi-architecture, multi-os-version installers if sys.platform == 'darwin': - kernel_version = os.uname()[2] # Kernel version (8.4.3) - major_version = int(kernel_version.split('.')[0]) - - if major_version < 8: - # On Mac OS X before 10.4, check if -arch and -isysroot - # are in CFLAGS or LDFLAGS and remove them if they are. - # This is needed when building extensions on a 10.3 system - # using a universal build of python. - for key in ('LDFLAGS', 'BASECFLAGS', - # a number of derived variables. These need to be - # patched up as well. - 'CFLAGS', 'PY_CFLAGS', 'BLDSHARED'): - flags = _CONFIG_VARS[key] - flags = re.sub('-arch\s+\w+\s', ' ', flags) - flags = re.sub('-isysroot [^ \t]*', ' ', flags) - _CONFIG_VARS[key] = flags - else: - # Allow the user to override the architecture flags using - # an environment variable. - # NOTE: This name was introduced by Apple in OSX 10.5 and - # is used by several scripting languages distributed with - # that OS release. - if 'ARCHFLAGS' in os.environ: - arch = os.environ['ARCHFLAGS'] - for key in ('LDFLAGS', 'BASECFLAGS', - # a number of derived variables. These need to be - # patched up as well. - 'CFLAGS', 'PY_CFLAGS', 'BLDSHARED'): - - flags = _CONFIG_VARS[key] - flags = re.sub('-arch\s+\w+\s', ' ', flags) - flags = flags + ' ' + arch - _CONFIG_VARS[key] = flags - - # If we're on OSX 10.5 or later and the user tries to - # compiles an extension using an SDK that is not present - # on the current machine it is better to not use an SDK - # than to fail. - # - # The major usecase for this is users using a Python.org - # binary installer on OSX 10.6: that installer uses - # the 10.4u SDK, but that SDK is not installed by default - # when you install Xcode. - # - CFLAGS = _CONFIG_VARS.get('CFLAGS', '') - m = re.search('-isysroot\s+(\S+)', CFLAGS) - if m is not None: - sdk = m.group(1) - if not os.path.exists(sdk): - for key in ('LDFLAGS', 'BASECFLAGS', - # a number of derived variables. These need to be - # patched up as well. - 'CFLAGS', 'PY_CFLAGS', 'BLDSHARED'): - - flags = _CONFIG_VARS[key] - flags = re.sub('-isysroot\s+\S+(\s|$)', ' ', flags) - _CONFIG_VARS[key] = flags + import _osx_support + _osx_support.customize_config_vars(_CONFIG_VARS) if args: vals = [] @@ -600,91 +547,10 @@ if m: release = m.group() elif osname[:6] == "darwin": - # - # For our purposes, we'll assume that the system version from - # distutils' perspective is what MACOSX_DEPLOYMENT_TARGET is set - # to. This makes the compatibility story a bit more sane because the - # machine is going to compile and link as if it were - # MACOSX_DEPLOYMENT_TARGET. - cfgvars = get_config_vars() - macver = cfgvars.get('MACOSX_DEPLOYMENT_TARGET') - - if 1: - # Always calculate the release of the running machine, - # needed to determine if we can build fat binaries or not. - - macrelease = macver - # Get the system version. Reading this plist is a documented - # way to get the system version (see the documentation for - # the Gestalt Manager) - try: - f = open('/System/Library/CoreServices/SystemVersion.plist') - except IOError: - # We're on a plain darwin box, fall back to the default - # behaviour. - pass - else: - try: - m = re.search( - r'<key>ProductUserVisibleVersion</key>\s*' + - r'<string>(.*?)</string>', f.read()) - if m is not None: - macrelease = '.'.join(m.group(1).split('.')[:2]) - # else: fall back to the default behaviour - finally: - f.close() - - if not macver: - macver = macrelease - - if macver: - release = macver - osname = "macosx" - - if (macrelease + '.') >= '10.4.' and \ - '-arch' in get_config_vars().get('CFLAGS', '').strip(): - # The universal build will build fat binaries, but not on - # systems before 10.4 - # - # Try to detect 4-way universal builds, those have machine-type - # 'universal' instead of 'fat'. - - machine = 'fat' - cflags = get_config_vars().get('CFLAGS') - - archs = re.findall('-arch\s+(\S+)', cflags) - archs = tuple(sorted(set(archs))) - - if len(archs) == 1: - machine = archs[0] - elif archs == ('i386', 'ppc'): - machine = 'fat' - elif archs == ('i386', 'x86_64'): - machine = 'intel' - elif archs == ('i386', 'ppc', 'x86_64'): - machine = 'fat3' - elif archs == ('ppc64', 'x86_64'): - machine = 'fat64' - elif archs == ('i386', 'ppc', 'ppc64', 'x86_64'): - machine = 'universal' - else: - raise ValueError( - "Don't know machine value for archs=%r"%(archs,)) - - elif machine == 'i386': - # On OSX the machine type returned by uname is always the - # 32-bit variant, even if the executable architecture is - # the 64-bit variant - if sys.maxint >= 2**32: - machine = 'x86_64' - - elif machine in ('PowerPC', 'Power_Macintosh'): - # Pick a sane name for the PPC architecture. - # See 'i386' case - if sys.maxint >= 2**32: - machine = 'ppc64' - else: - machine = 'ppc' + import _osx_support + osname, release, machine = _osx_support.get_platform_osx( + get_config_vars(), + osname, release, machine) return "%s-%s-%s" % (osname, release, machine) diff --git a/Lib/test/test__osx_support.py b/Lib/test/test__osx_support.py new file mode 100644 --- /dev/null +++ b/Lib/test/test__osx_support.py @@ -0,0 +1,279 @@ +""" +Test suite for _osx_support: shared OS X support functions. +""" + +import os +import platform +import shutil +import stat +import sys +import unittest + +import test.test_support + +import _osx_support + +(a)unittest.skipUnless(sys.platform.startswith("darwin"), "requires OS X") +class Test_OSXSupport(unittest.TestCase): + + def setUp(self): + self.maxDiff = None + self.prog_name = 'bogus_program_xxxx' + self.temp_path_dir = os.path.abspath(os.getcwd()) + self.env = test.test_support.EnvironmentVarGuard() + self.addCleanup(self.env.__exit__) + for cv in ('CFLAGS', 'LDFLAGS', 'CPPFLAGS', + 'BASECFLAGS', 'BLDSHARED', 'LDSHARED', 'CC', + 'CXX', 'PY_CFLAGS', 'PY_LDFLAGS', 'PY_CPPFLAGS', + 'PY_CORE_CFLAGS'): + if cv in self.env: + self.env.unset(cv) + + def add_expected_saved_initial_values(self, config_vars, expected_vars): + # Ensure that the initial values for all modified config vars + # are also saved with modified keys. + expected_vars.update(('_OSX_SUPPORT_INITIAL_'+ k, + config_vars[k]) for k in config_vars + if config_vars[k] != expected_vars[k]) + + def test__find_executable(self): + if self.env['PATH']: + self.env['PATH'] = self.env['PATH'] + ':' + self.env['PATH'] = self.env['PATH'] + os.path.abspath(self.temp_path_dir) + test.test_support.unlink(self.prog_name) + self.assertIsNone(_osx_support._find_executable(self.prog_name)) + self.addCleanup(test.test_support.unlink, self.prog_name) + with open(self.prog_name, 'w') as f: + f.write("#!/bin/sh\n/bin/echo OK\n") + os.chmod(self.prog_name, stat.S_IRWXU) + self.assertEqual(self.prog_name, + _osx_support._find_executable(self.prog_name)) + + def test__read_output(self): + if self.env['PATH']: + self.env['PATH'] = self.env['PATH'] + ':' + self.env['PATH'] = self.env['PATH'] + os.path.abspath(self.temp_path_dir) + test.test_support.unlink(self.prog_name) + self.addCleanup(test.test_support.unlink, self.prog_name) + with open(self.prog_name, 'w') as f: + f.write("#!/bin/sh\n/bin/echo ExpectedOutput\n") + os.chmod(self.prog_name, stat.S_IRWXU) + self.assertEqual('ExpectedOutput', + _osx_support._read_output(self.prog_name)) + + def test__find_build_tool(self): + out = _osx_support._find_build_tool('cc') + self.assertTrue(os.path.isfile(out), + 'cc not found - check xcode-select') + + def test__get_system_version(self): + self.assertTrue(platform.mac_ver()[0].startswith( + _osx_support._get_system_version())) + + def test__remove_original_values(self): + config_vars = { + 'CC': 'gcc-test -pthreads', + } + expected_vars = { + 'CC': 'clang -pthreads', + } + cv = 'CC' + newvalue = 'clang -pthreads' + _osx_support._save_modified_value(config_vars, cv, newvalue) + self.assertNotEqual(expected_vars, config_vars) + _osx_support._remove_original_values(config_vars) + self.assertEqual(expected_vars, config_vars) + + def test__save_modified_value(self): + config_vars = { + 'CC': 'gcc-test -pthreads', + } + expected_vars = { + 'CC': 'clang -pthreads', + } + self.add_expected_saved_initial_values(config_vars, expected_vars) + cv = 'CC' + newvalue = 'clang -pthreads' + _osx_support._save_modified_value(config_vars, cv, newvalue) + self.assertEqual(expected_vars, config_vars) + + def test__save_modified_value_unchanged(self): + config_vars = { + 'CC': 'gcc-test -pthreads', + } + expected_vars = config_vars.copy() + cv = 'CC' + newvalue = 'gcc-test -pthreads' + _osx_support._save_modified_value(config_vars, cv, newvalue) + self.assertEqual(expected_vars, config_vars) + + def test__supports_universal_builds(self): + import platform + self.assertEqual(platform.mac_ver()[0].split('.') >= ['10', '4'], + _osx_support._supports_universal_builds()) + + def test__find_appropriate_compiler(self): + compilers = ( + ('gcc-test', 'i686-apple-darwin11-llvm-gcc-4.2'), + ('clang', 'clang version 3.1'), + ) + config_vars = { + 'CC': 'gcc-test -pthreads', + 'CXX': 'cc++-test', + 'CFLAGS': '-fno-strict-aliasing -g -O3 -arch ppc -arch i386 ', + 'LDFLAGS': '-arch ppc -arch i386 -g', + 'CPPFLAGS': '-I. -isysroot /Developer/SDKs/MacOSX10.4u.sdk', + 'BLDSHARED': 'gcc-test -bundle -arch ppc -arch i386 -g', + 'LDSHARED': 'gcc-test -bundle -arch ppc -arch i386 ' + '-isysroot /Developer/SDKs/MacOSX10.4u.sdk -g', + } + expected_vars = { + 'CC': 'clang -pthreads', + 'CXX': 'clang++', + 'CFLAGS': '-fno-strict-aliasing -g -O3 -arch ppc -arch i386 ', + 'LDFLAGS': '-arch ppc -arch i386 -g', + 'CPPFLAGS': '-I. -isysroot /Developer/SDKs/MacOSX10.4u.sdk', + 'BLDSHARED': 'clang -bundle -arch ppc -arch i386 -g', + 'LDSHARED': 'clang -bundle -arch ppc -arch i386 ' + '-isysroot /Developer/SDKs/MacOSX10.4u.sdk -g', + } + self.add_expected_saved_initial_values(config_vars, expected_vars) + + suffix = (':' + self.env['PATH']) if self.env['PATH'] else '' + self.env['PATH'] = os.path.abspath(self.temp_path_dir) + suffix + for c_name, c_output in compilers: + test.test_support.unlink(c_name) + self.addCleanup(test.test_support.unlink, c_name) + with open(c_name, 'w') as f: + f.write("#!/bin/sh\n/bin/echo " + c_output) + os.chmod(c_name, stat.S_IRWXU) + self.assertEqual(expected_vars, + _osx_support._find_appropriate_compiler( + config_vars)) + + def test__remove_universal_flags(self): + config_vars = { + 'CFLAGS': '-fno-strict-aliasing -g -O3 -arch ppc -arch i386 ', + 'LDFLAGS': '-arch ppc -arch i386 -g', + 'CPPFLAGS': '-I. -isysroot /Developer/SDKs/MacOSX10.4u.sdk', + 'BLDSHARED': 'gcc-4.0 -bundle -arch ppc -arch i386 -g', + 'LDSHARED': 'gcc-4.0 -bundle -arch ppc -arch i386 ' + '-isysroot /Developer/SDKs/MacOSX10.4u.sdk -g', + } + expected_vars = { + 'CFLAGS': '-fno-strict-aliasing -g -O3 ', + 'LDFLAGS': ' -g', + 'CPPFLAGS': '-I. ', + 'BLDSHARED': 'gcc-4.0 -bundle -g', + 'LDSHARED': 'gcc-4.0 -bundle -g', + } + self.add_expected_saved_initial_values(config_vars, expected_vars) + + self.assertEqual(expected_vars, + _osx_support._remove_universal_flags( + config_vars)) + + def test__remove_unsupported_archs(self): + config_vars = { + 'CC': 'clang', + 'CFLAGS': '-fno-strict-aliasing -g -O3 -arch ppc -arch i386 ', + 'LDFLAGS': '-arch ppc -arch i386 -g', + 'CPPFLAGS': '-I. -isysroot /Developer/SDKs/MacOSX10.4u.sdk', + 'BLDSHARED': 'gcc-4.0 -bundle -arch ppc -arch i386 -g', + 'LDSHARED': 'gcc-4.0 -bundle -arch ppc -arch i386 ' + '-isysroot /Developer/SDKs/MacOSX10.4u.sdk -g', + } + expected_vars = { + 'CC': 'clang', + 'CFLAGS': '-fno-strict-aliasing -g -O3 -arch i386 ', + 'LDFLAGS': ' -arch i386 -g', + 'CPPFLAGS': '-I. -isysroot /Developer/SDKs/MacOSX10.4u.sdk', + 'BLDSHARED': 'gcc-4.0 -bundle -arch i386 -g', + 'LDSHARED': 'gcc-4.0 -bundle -arch i386 ' + '-isysroot /Developer/SDKs/MacOSX10.4u.sdk -g', + } + self.add_expected_saved_initial_values(config_vars, expected_vars) + + suffix = (':' + self.env['PATH']) if self.env['PATH'] else '' + self.env['PATH'] = os.path.abspath(self.temp_path_dir) + suffix + c_name = 'clang' + test.test_support.unlink(c_name) + self.addCleanup(test.test_support.unlink, c_name) + # exit status 255 means no PPC support in this compiler chain + with open(c_name, 'w') as f: + f.write("#!/bin/sh\nexit 255") + os.chmod(c_name, stat.S_IRWXU) + self.assertEqual(expected_vars, + _osx_support._remove_unsupported_archs( + config_vars)) + + def test__override_all_archs(self): + self.env['ARCHFLAGS'] = '-arch x86_64' + config_vars = { + 'CC': 'clang', + 'CFLAGS': '-fno-strict-aliasing -g -O3 -arch ppc -arch i386 ', + 'LDFLAGS': '-arch ppc -arch i386 -g', + 'CPPFLAGS': '-I. -isysroot /Developer/SDKs/MacOSX10.4u.sdk', + 'BLDSHARED': 'gcc-4.0 -bundle -arch ppc -arch i386 -g', + 'LDSHARED': 'gcc-4.0 -bundle -arch ppc -arch i386 ' + '-isysroot /Developer/SDKs/MacOSX10.4u.sdk -g', + } + expected_vars = { + 'CC': 'clang', + 'CFLAGS': '-fno-strict-aliasing -g -O3 -arch x86_64', + 'LDFLAGS': ' -g -arch x86_64', + 'CPPFLAGS': '-I. -isysroot /Developer/SDKs/MacOSX10.4u.sdk', + 'BLDSHARED': 'gcc-4.0 -bundle -g -arch x86_64', + 'LDSHARED': 'gcc-4.0 -bundle -isysroot ' + '/Developer/SDKs/MacOSX10.4u.sdk -g -arch x86_64', + } + self.add_expected_saved_initial_values(config_vars, expected_vars) + + self.assertEqual(expected_vars, + _osx_support._override_all_archs( + config_vars)) + + def test__check_for_unavailable_sdk(self): + config_vars = { + 'CC': 'clang', + 'CFLAGS': '-fno-strict-aliasing -g -O3 -arch ppc -arch i386 ' + '-isysroot /Developer/SDKs/MacOSX10.1.sdk', + 'LDFLAGS': '-arch ppc -arch i386 -g', + 'CPPFLAGS': '-I. -isysroot /Developer/SDKs/MacOSX10.1.sdk', + 'BLDSHARED': 'gcc-4.0 -bundle -arch ppc -arch i386 -g', + 'LDSHARED': 'gcc-4.0 -bundle -arch ppc -arch i386 ' + '-isysroot /Developer/SDKs/MacOSX10.1.sdk -g', + } + expected_vars = { + 'CC': 'clang', + 'CFLAGS': '-fno-strict-aliasing -g -O3 -arch ppc -arch i386 ' + ' ', + 'LDFLAGS': '-arch ppc -arch i386 -g', + 'CPPFLAGS': '-I. ', + 'BLDSHARED': 'gcc-4.0 -bundle -arch ppc -arch i386 -g', + 'LDSHARED': 'gcc-4.0 -bundle -arch ppc -arch i386 ' + ' -g', + } + self.add_expected_saved_initial_values(config_vars, expected_vars) + + self.assertEqual(expected_vars, + _osx_support._check_for_unavailable_sdk( + config_vars)) + + def test_get_platform_osx(self): + # Note, get_platform_osx is currently tested more extensively + # indirectly by test_sysconfig and test_distutils + config_vars = { + 'CFLAGS': '-fno-strict-aliasing -g -O3 -arch ppc -arch i386 ' + '-isysroot /Developer/SDKs/MacOSX10.1.sdk', + 'MACOSX_DEPLOYMENT_TARGET': '10.6', + } + result = _osx_support.get_platform_osx(config_vars, ' ', ' ', ' ') + self.assertEqual(('macosx', '10.6', 'fat'), result) + +def test_main(): + if sys.platform == 'darwin': + test.test_support.run_unittest(Test_OSXSupport) + +if __name__ == "__main__": + test_main() diff --git a/Lib/test/test_sysconfig.py b/Lib/test/test_sysconfig.py --- a/Lib/test/test_sysconfig.py +++ b/Lib/test/test_sysconfig.py @@ -14,6 +14,7 @@ get_path, get_path_names, _INSTALL_SCHEMES, _get_default_scheme, _expand_vars, get_scheme_names, get_config_var) +import _osx_support class TestSysConfig(unittest.TestCase): @@ -137,6 +138,7 @@ ('Darwin Kernel Version 8.11.1: ' 'Wed Oct 10 18:23:28 PDT 2007; ' 'root:xnu-792.25.20~1/RELEASE_I386'), 'PowerPC')) + _osx_support._remove_original_values(get_config_vars()) get_config_vars()['MACOSX_DEPLOYMENT_TARGET'] = '10.3' get_config_vars()['CFLAGS'] = ('-fno-strict-aliasing -DNDEBUG -g ' @@ -156,6 +158,7 @@ ('Darwin Kernel Version 8.11.1: ' 'Wed Oct 10 18:23:28 PDT 2007; ' 'root:xnu-792.25.20~1/RELEASE_I386'), 'i386')) + _osx_support._remove_original_values(get_config_vars()) get_config_vars()['MACOSX_DEPLOYMENT_TARGET'] = '10.3' get_config_vars()['CFLAGS'] = ('-fno-strict-aliasing -DNDEBUG -g ' @@ -171,6 +174,7 @@ sys.maxint = maxint # macbook with fat binaries (fat, universal or fat64) + _osx_support._remove_original_values(get_config_vars()) get_config_vars()['MACOSX_DEPLOYMENT_TARGET'] = '10.4' get_config_vars()['CFLAGS'] = ('-arch ppc -arch i386 -isysroot ' '/Developer/SDKs/MacOSX10.4u.sdk ' @@ -179,6 +183,7 @@ self.assertEqual(get_platform(), 'macosx-10.4-fat') + _osx_support._remove_original_values(get_config_vars()) get_config_vars()['CFLAGS'] = ('-arch x86_64 -arch i386 -isysroot ' '/Developer/SDKs/MacOSX10.4u.sdk ' '-fno-strict-aliasing -fno-common ' @@ -186,18 +191,21 @@ self.assertEqual(get_platform(), 'macosx-10.4-intel') + _osx_support._remove_original_values(get_config_vars()) get_config_vars()['CFLAGS'] = ('-arch x86_64 -arch ppc -arch i386 -isysroot ' '/Developer/SDKs/MacOSX10.4u.sdk ' '-fno-strict-aliasing -fno-common ' '-dynamic -DNDEBUG -g -O3') self.assertEqual(get_platform(), 'macosx-10.4-fat3') + _osx_support._remove_original_values(get_config_vars()) get_config_vars()['CFLAGS'] = ('-arch ppc64 -arch x86_64 -arch ppc -arch i386 -isysroot ' '/Developer/SDKs/MacOSX10.4u.sdk ' '-fno-strict-aliasing -fno-common ' '-dynamic -DNDEBUG -g -O3') self.assertEqual(get_platform(), 'macosx-10.4-universal') + _osx_support._remove_original_values(get_config_vars()) get_config_vars()['CFLAGS'] = ('-arch x86_64 -arch ppc64 -isysroot ' '/Developer/SDKs/MacOSX10.4u.sdk ' '-fno-strict-aliasing -fno-common ' @@ -206,6 +214,7 @@ self.assertEqual(get_platform(), 'macosx-10.4-fat64') for arch in ('ppc', 'i386', 'x86_64', 'ppc64'): + _osx_support._remove_original_values(get_config_vars()) get_config_vars()['CFLAGS'] = ('-arch %s -isysroot ' '/Developer/SDKs/MacOSX10.4u.sdk ' '-fno-strict-aliasing -fno-common ' diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -679,6 +679,21 @@ - Issue #16012: Fix a regression in pyexpat. The parser's UseForeignDTD() method doesn't require an argument again. +- Issue #13590: OS X Xcode 4 - improve support for universal extension modules + In particular, fix extension module build failures when trying to use + 32-bit-only installer Pythons on systems with Xcode 4 (currently + OS X 10.8, 10.7, and optionally 10.6). + * Backport 3.3.0 fixes to 2.7 branch (for release in 2.7.4) + * Since Xcode 4 removes ppc support, extension module builds now + check for ppc compiler support and by default remove ppc and + ppc64 archs when they are not available. + * Extension module builds now revert to using system installed + headers and libs (/usr and /System/Library) if the SDK used + to build the interpreter is not installed or has moved. + * Try to avoid building extension modules with deprecated + and problematic Apple llvm-gcc compiler. If original compiler + is not available, use clang instead by default. + Tests ----- -- Repository URL: http://hg.python.org/cpython

1 0

Daily reference leaks (58e72cb89848): sum=6
by solipsis＠pitrou.net Jan. 31, 2013

Jan. 31, 2013

results for 58e72cb89848 on branch "default" -------------------------------------------- test_dbm leaked [0, 2, 0] references, sum=2 test_dbm leaked [0, 2, 2] memory blocks, sum=4 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogE1TgxI', '-x']

1 0

cpython: Updated NEWS with fix for #17028.
by vinay.sajip Jan. 30, 2013

Jan. 30, 2013

http://hg.python.org/cpython/rev/58e72cb89848 changeset: 81842:58e72cb89848 user: Vinay Sajip <vinay_sajip(a)yahoo.co.uk> date: Wed Jan 30 22:26:49 2013 +0000 summary: Updated NEWS with fix for #17028. files: Misc/NEWS | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -891,6 +891,9 @@ Tools/Demos ----------- +- Issue #17028: Allowed Python arguments to be supplied to the Windows + launcher. + - Issue #15539: Fix a number of bugs in Tools/scripts/pindent.py. Now pindent.py works with a "with" statement. pindent.py no longer produces improper indentation. pindent.py now works with continued lines broken after -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.3 -> default): Merged documentation update from 3.3.
by vinay.sajip Jan. 30, 2013

Jan. 30, 2013

http://hg.python.org/cpython/rev/ae6c77a883c6 changeset: 81841:ae6c77a883c6 parent: 81839:e81cad0c722a parent: 81840:91435b4dc8d4 user: Vinay Sajip <vinay_sajip(a)yahoo.co.uk> date: Wed Jan 30 13:44:34 2013 +0000 summary: Merged documentation update from 3.3. files: Doc/library/venv.rst | 213 +++++++++++++++++++++++++++++++ 1 files changed, 213 insertions(+), 0 deletions(-) diff --git a/Doc/library/venv.rst b/Doc/library/venv.rst --- a/Doc/library/venv.rst +++ b/Doc/library/venv.rst @@ -177,3 +177,216 @@ Create an :class:`EnvBuilder` with the given keyword arguments, and call its :meth:`~EnvBuilder.create` method with the *env_dir* argument. + +An example of extending ``EnvBuilder`` +-------------------------------------- + +The following script shows how to extend :class:`EnvBuilder` by implementing a +subclass which installs Distribute and pip into a created venv:: + + import os + import os.path + from subprocess import Popen, PIPE + import sys + from threading import Thread + from urllib.parse import urlparse + from urllib.request import urlretrieve + import venv + + class DistributeEnvBuilder(venv.EnvBuilder): + """ + This builder installs Distribute and pip so that you can pip or + easy_install other packages into the created environment. + + :param nodist: If True, Distribute is not installed into the created + environment. + :param nopip: If True, pip is not installed into the created + environment. + :param progress: If Distribute or pip are installed, the progress of the + installation can be monitored by passing a progress + callable. If specified, it is called with two + arguments: a string indicating some progress, and a + context indicating where the string is coming from. + The context argument can have one of three values: + 'main', indicating that it is called from virtualize() + itself, and 'stdout' and 'stderr', which are obtained + by reading lines from the output streams of a subprocess + which is used to install the app. + + If a callable is not specified, default progress + information is output to sys.stderr. + """ + + def __init__(self, *args, **kwargs): + self.nodist = kwargs.pop('nodist', False) + self.nopip = kwargs.pop('nopip', False) + self.progress = kwargs.pop('progress', None) + self.verbose = kwargs.pop('verbose', False) + super().__init__(*args, **kwargs) + + def post_setup(self, context): + """ + Set up any packages which need to be pre-installed into the + environment being created. + + :param context: The information for the environment creation request + being processed. + """ + if not self.nodist: + self.install_distribute(context) + if not self.nopip: + self.install_pip(context) + + def reader(self, stream, context): + """ + Read lines from a subprocess' output stream and either pass to a progress + callable (if specified) or write progress information to sys.stderr. + """ + progress = self.progress + while True: + s = stream.readline() + if not s: + break + if progress is not None: + progress(s, context) + else: + if not self.verbose: + sys.stderr.write('.') + else: + sys.stderr.write(s.decode('utf-8')) + sys.stderr.flush() + + def install_script(self, context, name, url): + _, _, path, _, _, _ = urlparse(url) + fn = os.path.split(path)[-1] + binpath = context.bin_path + distpath = os.path.join(binpath, fn) + # Download script into the env's binaries folder + urlretrieve(url, distpath) + progress = self.progress + if progress is not None: + progress('Installing %s' %name, 'main') + else: + sys.stderr.write('Installing %s ' % name) + sys.stderr.flush() + # Install in the env + args = [context.env_exe, fn] + p = Popen(args, stdout=PIPE, stderr=PIPE, cwd=binpath) + t1 = Thread(target=self.reader, args=(p.stdout, 'stdout')) + t1.start() + t2 = Thread(target=self.reader, args=(p.stderr, 'stderr')) + t2.start() + p.wait() + t1.join() + t2.join() + if progress is not None: + progress('done.', 'main') + else: + sys.stderr.write('done.\n') + # Clean up - no longer needed + os.unlink(distpath) + + def install_distribute(self, context): + """ + Install Distribute in the environment. + + :param context: The information for the environment creation request + being processed. + """ + url = 'http://python-distribute.org/distribute_setup.py' + self.install_script(context, 'distribute', url) + # clear up the distribute archive which gets downloaded + pred = lambda o: o.startswith('distribute-') and o.endswith('.tar.gz') + files = filter(pred, os.listdir(context.bin_path)) + for f in files: + f = os.path.join(context.bin_path, f) + os.unlink(f) + + def install_pip(self, context): + """ + Install pip in the environment. + + :param context: The information for the environment creation request + being processed. + """ + url = 'https://raw.github.com/pypa/pip/master/contrib/get-pip.py' + self.install_script(context, 'pip', url) + + def main(args=None): + compatible = True + if sys.version_info < (3, 3): + compatible = False + elif not hasattr(sys, 'base_prefix'): + compatible = False + if not compatible: + raise ValueError('This script is only for use with ' + 'Python 3.3 or later') + else: + import argparse + + parser = argparse.ArgumentParser(prog=__name__, + description='Creates virtual Python ' + 'environments in one or ' + 'more target ' + 'directories.') + parser.add_argument('dirs', metavar='ENV_DIR', nargs='+', + help='A directory to create the environment in.') + parser.add_argument('--no-distribute', default=False, + action='store_true', dest='nodist', + help="Don't install Distribute in the virtual " + "environment.") + parser.add_argument('--no-pip', default=False, + action='store_true', dest='nopip', + help="Don't install pip in the virtual " + "environment.") + parser.add_argument('--system-site-packages', default=False, + action='store_true', dest='system_site', + help='Give the virtual environment access to the ' + 'system site-packages dir.') + if os.name == 'nt': + use_symlinks = False + else: + use_symlinks = True + parser.add_argument('--symlinks', default=use_symlinks, + action='store_true', dest='symlinks', + help='Try to use symlinks rather than copies, ' + 'when symlinks are not the default for ' + 'the platform.') + parser.add_argument('--clear', default=False, action='store_true', + dest='clear', help='Delete the contents of the ' + 'environment directory if it ' + 'already exists, before ' + 'environment creation.') + parser.add_argument('--upgrade', default=False, action='store_true', + dest='upgrade', help='Upgrade the environment ' + 'directory to use this version ' + 'of Python, assuming Python ' + 'has been upgraded in-place.') + parser.add_argument('--verbose', default=False, action='store_true', + dest='verbose', help='Display the output ' + 'from the scripts which ' + 'install Distribute and pip.') + options = parser.parse_args(args) + if options.upgrade and options.clear: + raise ValueError('you cannot supply --upgrade and --clear together.') + builder = DistributeEnvBuilder(system_site_packages=options.system_site, + clear=options.clear, + symlinks=options.symlinks, + upgrade=options.upgrade, + nodist=options.nodist, + nopip=options.nopip, + verbose=options.verbose) + for d in options.dirs: + builder.create(d) + + if __name__ == '__main__': + rc = 1 + try: + main() + rc = 0 + except Exception as e: + print('Error: %s' % e, file=sys.stderr) + sys.exit(rc) + +This script is also available for download `online +<https://gist.github.com/4673395>`_. -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.3): Updated venv documentation with an example.
by vinay.sajip Jan. 30, 2013

Jan. 30, 2013

http://hg.python.org/cpython/rev/91435b4dc8d4 changeset: 81840:91435b4dc8d4 branch: 3.3 parent: 81838:f39b69494393 user: Vinay Sajip <vinay_sajip(a)yahoo.co.uk> date: Wed Jan 30 13:44:00 2013 +0000 summary: Updated venv documentation with an example. files: Doc/library/venv.rst | 213 +++++++++++++++++++++++++++++++ 1 files changed, 213 insertions(+), 0 deletions(-) diff --git a/Doc/library/venv.rst b/Doc/library/venv.rst --- a/Doc/library/venv.rst +++ b/Doc/library/venv.rst @@ -178,3 +178,216 @@ Create an :class:`EnvBuilder` with the given keyword arguments, and call its :meth:`~EnvBuilder.create` method with the *env_dir* argument. + +An example of extending ``EnvBuilder`` +-------------------------------------- + +The following script shows how to extend :class:`EnvBuilder` by implementing a +subclass which installs Distribute and pip into a created venv:: + + import os + import os.path + from subprocess import Popen, PIPE + import sys + from threading import Thread + from urllib.parse import urlparse + from urllib.request import urlretrieve + import venv + + class DistributeEnvBuilder(venv.EnvBuilder): + """ + This builder installs Distribute and pip so that you can pip or + easy_install other packages into the created environment. + + :param nodist: If True, Distribute is not installed into the created + environment. + :param nopip: If True, pip is not installed into the created + environment. + :param progress: If Distribute or pip are installed, the progress of the + installation can be monitored by passing a progress + callable. If specified, it is called with two + arguments: a string indicating some progress, and a + context indicating where the string is coming from. + The context argument can have one of three values: + 'main', indicating that it is called from virtualize() + itself, and 'stdout' and 'stderr', which are obtained + by reading lines from the output streams of a subprocess + which is used to install the app. + + If a callable is not specified, default progress + information is output to sys.stderr. + """ + + def __init__(self, *args, **kwargs): + self.nodist = kwargs.pop('nodist', False) + self.nopip = kwargs.pop('nopip', False) + self.progress = kwargs.pop('progress', None) + self.verbose = kwargs.pop('verbose', False) + super().__init__(*args, **kwargs) + + def post_setup(self, context): + """ + Set up any packages which need to be pre-installed into the + environment being created. + + :param context: The information for the environment creation request + being processed. + """ + if not self.nodist: + self.install_distribute(context) + if not self.nopip: + self.install_pip(context) + + def reader(self, stream, context): + """ + Read lines from a subprocess' output stream and either pass to a progress + callable (if specified) or write progress information to sys.stderr. + """ + progress = self.progress + while True: + s = stream.readline() + if not s: + break + if progress is not None: + progress(s, context) + else: + if not self.verbose: + sys.stderr.write('.') + else: + sys.stderr.write(s.decode('utf-8')) + sys.stderr.flush() + + def install_script(self, context, name, url): + _, _, path, _, _, _ = urlparse(url) + fn = os.path.split(path)[-1] + binpath = context.bin_path + distpath = os.path.join(binpath, fn) + # Download script into the env's binaries folder + urlretrieve(url, distpath) + progress = self.progress + if progress is not None: + progress('Installing %s' %name, 'main') + else: + sys.stderr.write('Installing %s ' % name) + sys.stderr.flush() + # Install in the env + args = [context.env_exe, fn] + p = Popen(args, stdout=PIPE, stderr=PIPE, cwd=binpath) + t1 = Thread(target=self.reader, args=(p.stdout, 'stdout')) + t1.start() + t2 = Thread(target=self.reader, args=(p.stderr, 'stderr')) + t2.start() + p.wait() + t1.join() + t2.join() + if progress is not None: + progress('done.', 'main') + else: + sys.stderr.write('done.\n') + # Clean up - no longer needed + os.unlink(distpath) + + def install_distribute(self, context): + """ + Install Distribute in the environment. + + :param context: The information for the environment creation request + being processed. + """ + url = 'http://python-distribute.org/distribute_setup.py' + self.install_script(context, 'distribute', url) + # clear up the distribute archive which gets downloaded + pred = lambda o: o.startswith('distribute-') and o.endswith('.tar.gz') + files = filter(pred, os.listdir(context.bin_path)) + for f in files: + f = os.path.join(context.bin_path, f) + os.unlink(f) + + def install_pip(self, context): + """ + Install pip in the environment. + + :param context: The information for the environment creation request + being processed. + """ + url = 'https://raw.github.com/pypa/pip/master/contrib/get-pip.py' + self.install_script(context, 'pip', url) + + def main(args=None): + compatible = True + if sys.version_info < (3, 3): + compatible = False + elif not hasattr(sys, 'base_prefix'): + compatible = False + if not compatible: + raise ValueError('This script is only for use with ' + 'Python 3.3 or later') + else: + import argparse + + parser = argparse.ArgumentParser(prog=__name__, + description='Creates virtual Python ' + 'environments in one or ' + 'more target ' + 'directories.') + parser.add_argument('dirs', metavar='ENV_DIR', nargs='+', + help='A directory to create the environment in.') + parser.add_argument('--no-distribute', default=False, + action='store_true', dest='nodist', + help="Don't install Distribute in the virtual " + "environment.") + parser.add_argument('--no-pip', default=False, + action='store_true', dest='nopip', + help="Don't install pip in the virtual " + "environment.") + parser.add_argument('--system-site-packages', default=False, + action='store_true', dest='system_site', + help='Give the virtual environment access to the ' + 'system site-packages dir.') + if os.name == 'nt': + use_symlinks = False + else: + use_symlinks = True + parser.add_argument('--symlinks', default=use_symlinks, + action='store_true', dest='symlinks', + help='Try to use symlinks rather than copies, ' + 'when symlinks are not the default for ' + 'the platform.') + parser.add_argument('--clear', default=False, action='store_true', + dest='clear', help='Delete the contents of the ' + 'environment directory if it ' + 'already exists, before ' + 'environment creation.') + parser.add_argument('--upgrade', default=False, action='store_true', + dest='upgrade', help='Upgrade the environment ' + 'directory to use this version ' + 'of Python, assuming Python ' + 'has been upgraded in-place.') + parser.add_argument('--verbose', default=False, action='store_true', + dest='verbose', help='Display the output ' + 'from the scripts which ' + 'install Distribute and pip.') + options = parser.parse_args(args) + if options.upgrade and options.clear: + raise ValueError('you cannot supply --upgrade and --clear together.') + builder = DistributeEnvBuilder(system_site_packages=options.system_site, + clear=options.clear, + symlinks=options.symlinks, + upgrade=options.upgrade, + nodist=options.nodist, + nopip=options.nopip, + verbose=options.verbose) + for d in options.dirs: + builder.create(d) + + if __name__ == '__main__': + rc = 1 + try: + main() + rc = 0 + except Exception as e: + print('Error: %s' % e, file=sys.stderr) + sys.exit(rc) + +This script is also available for download `online +<https://gist.github.com/4673395>`_. -- Repository URL: http://hg.python.org/cpython

1 0

peps: PEP 433: more info on apache/php and cloexec
by victor.stinner Jan. 30, 2013

Jan. 30, 2013

http://hg.python.org/peps/rev/ffd254c46e55 changeset: 4703:ffd254c46e55 user: Victor Stinner <victor.stinner(a)gmail.com> date: Wed Jan 30 13:22:54 2013 +0100 summary: PEP 433: more info on apache/php and cloexec files: pep-0433.txt | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-) diff --git a/pep-0433.txt b/pep-0433.txt --- a/pep-0433.txt +++ b/pep-0433.txt @@ -119,6 +119,13 @@ * `Hijacking Apache https by mod_php <http://www.securityfocus.com/archive/1/348368>`_ (Dec 2003) + * Apache: `Apr should set FD_CLOEXEC if APR_FOPEN_NOCLEANUP is not set + <https://issues.apache.org/bugzilla/show_bug.cgi?id=46425>`_ + (fixed in 2009) + * PHP: `system() (and similar) don't cleanup opened handles of Apache + <https://bugs.php.net/bug.php?id=38915>`_ (not fixed in january + 2013) + Atomicity --------- -- Repository URL: http://hg.python.org/peps

1 0

peps: PEP 433: more references
by victor.stinner Jan. 30, 2013

Jan. 30, 2013

http://hg.python.org/peps/rev/651cc14404e8 changeset: 4702:651cc14404e8 user: Victor Stinner <victor.stinner(a)gmail.com> date: Wed Jan 30 12:54:34 2013 +0100 summary: PEP 433: more references files: pep-0433.txt | 22 +++++++++++++++++++--- 1 files changed, 19 insertions(+), 3 deletions(-) diff --git a/pep-0433.txt b/pep-0433.txt --- a/pep-0433.txt +++ b/pep-0433.txt @@ -632,6 +632,7 @@ * ``fcntl()``: ``F_DUPFD_CLOEXEC`` flag, available on Linux 2.6.24, OpenBSD 5.0, FreeBSD 9.1, NetBSD 6.0. This flag is part of POSIX.1-2008. + * ``fcntl()``: ``F_DUP2FD_CLOEXEC`` flag, available on FreeBSD 9.1. * ``recvmsg()``: ``MSG_CMSG_CLOEXEC``, available on Linux 2.6.23, NetBSD 6.0. @@ -644,6 +645,9 @@ socket type, ``socket()`` or ``socketpair()`` fail and ``errno`` is set to ``EINVAL``. +On Windows XPS3, ``WSASocket()`` with with ``WSAEPROTOTYPE`` when +``WSA_FLAG_NO_HANDLE_INHERIT`` flag is used. + New functions: * ``dup3()``: available on Linux 2.6.27 (and glibc 2.9) @@ -666,6 +670,8 @@ <https://bitbucket.org/pvl/gaeseries-tornado/src/c2671cea1842/tornado/win32_…>`_: emulate fcntl(fd, F_SETFD, FD_CLOEXEC) using ``SetHandleInformation(fd, HANDLE_FLAG_INHERIT, 1)`` + * `LKML: [PATCH] nextfd(2) + <https://lkml.org/lkml/2012/4/1/71>`_ Python issues: @@ -689,14 +695,24 @@ * `#17070: PEP 433: Use the new cloexec to improve security and avoid bugs <http://bugs.python.org/issue17070>`_ -Ruby: +Other languages: - * `Set FD_CLOEXEC for all fds (except 0, 1, 2) + * Perl sets the close-on-exec flag on newly created file decriptor if + their number is greater than ``$SYSTEM_FD_MAX`` (``$^F``). + See `$SYSTEM_FD_MAX documentation + <http://perldoc.perl.org/perlvar.html#%24SYSTEM_FD_MAX>`_. Perl does + this since the creation of Perl (it was already present in Perl 1). + * Ruby: `Set FD_CLOEXEC for all fds (except 0, 1, 2) <http://bugs.ruby-lang.org/issues/5041>`_ - * `O_CLOEXEC flag missing for Kernel::open + * Ruby: `O_CLOEXEC flag missing for Kernel::open <http://bugs.ruby-lang.org/issues/1291>`_: the `commit was reverted later <http://bugs.ruby-lang.org/projects/ruby-trunk/repository/revisions/31643>`_ + * OCaml: `PR#5256: Processes opened using Unix.open_process* inherit + all opened file descriptors (including sockets) + <http://caml.inria.fr/mantis/view.php?id=5256>`_. OCaml has a + ``Unix.set_close_on_exec`` function. + Footnotes ========= -- Repository URL: http://hg.python.org/peps

1 0

peps: PEP 433: inherance => inheritance
by victor.stinner Jan. 30, 2013

Jan. 30, 2013

http://hg.python.org/peps/rev/fdf10a400fd9 changeset: 4701:fdf10a400fd9 user: Victor Stinner <victor.stinner(a)gmail.com> date: Wed Jan 30 09:51:02 2013 +0100 summary: PEP 433: inherance => inheritance files: pep-0433.txt | 22 +++++++++++----------- 1 files changed, 11 insertions(+), 11 deletions(-) diff --git a/pep-0433.txt b/pep-0433.txt --- a/pep-0433.txt +++ b/pep-0433.txt @@ -38,7 +38,7 @@ close-on-exec flag is cleared and if ``CreateProcess()`` is called with the *bInheritHandles* parameter set to ``TRUE`` (when ``subprocess.Popen`` is created with ``close_fds=False`` for example). -Windows does now have "close-on-exec" flag but an inherance flag which +Windows does now have "close-on-exec" flag but an inheritance flag which is just the opposite value. For example, setting close-on-exec flag means clearing the ``HANDLE_FLAG_INHERIT`` flag of an handle. @@ -235,7 +235,7 @@ * It is not more possible to know if the close-on-exec flag will be set or not on a newly created file descriptor just by reading the source code. - * If the inherance of a file descriptor matters, the *cloexec* + * If the inheritance of a file descriptor matters, the *cloexec* parameter must now be specified explicitly, or the library or the application will not work depending on the default value of the *cloexec* parameter. @@ -244,12 +244,12 @@ Alternatives ============ -Enable inherance by default and no configurable default +Inheritance enabled by default, default no configurable ------------------------------------------------------- Add a new optional parameter *cloexec* on functions creating file descriptors. The default value of the *cloexec* parameter is ``False``, -and this default cannot be changed. No file descriptor inherance by +and this default cannot be changed. No file descriptor inheritance by default is also the default on POSIX and on Windows. This alternative is the most convervative option. @@ -259,8 +259,8 @@ module used by an application to fix all these issues. -Enable inherance by default, default can only be set to True ------------------------------------------------------------- +Inheritance enabled by default, default can only be set to True +--------------------------------------------------------------- This alternative is based on the proposal: the only difference is that ``sys.setdefaultcloexec()`` does not take any argument, it can only be @@ -286,7 +286,7 @@ * There are far more programs that are bitten by FD inheritance upon exec (see `Inherited file descriptors issues`_ and `Security`_) - than programs relying on it (see `Applications using inherance of + than programs relying on it (see `Applications using inheritance of file descriptors`_). Drawbacks of setting close-on-exec flag by default: @@ -302,7 +302,7 @@ * Extra system calls, if any, may slow down Python: see `Performances`_. -Backward compatibility: only a few programs rely on inherance of file +Backward compatibility: only a few programs rely on inheritance of file descriptors, and they only pass a few file descriptors, usually just one. These programs will fail immediatly with ``EBADF`` error, and it will be simple to fix them: add ``cloexec=False`` parameter or use @@ -366,11 +366,11 @@ -Applications using inherance of file descriptors -================================================ +Applications using inheritance of file descriptors +================================================== Most developers don't know that file descriptors are inherited by -default. Most programs do not rely on inherance of file descriptors. +default. Most programs do not rely on inheritance of file descriptors. For example, ``subprocess.Popen`` was changed in Python 3.2 to close all file descriptors greater than 2 in the child process by default. No user complained about this behavior change yet. -- Repository URL: http://hg.python.org/peps

1 0

peps: PEP 433: fix typo, sys.setdefaultcloexec() has one mandatory parameter
by victor.stinner Jan. 30, 2013

Jan. 30, 2013

http://hg.python.org/peps/rev/c8017823a4dc changeset: 4700:c8017823a4dc user: Victor Stinner <victor.stinner(a)gmail.com> date: Wed Jan 30 09:40:36 2013 +0100 summary: PEP 433: fix typo, sys.setdefaultcloexec() has one mandatory parameter files: pep-0433.txt | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/pep-0433.txt b/pep-0433.txt --- a/pep-0433.txt +++ b/pep-0433.txt @@ -20,7 +20,7 @@ * ``os.get_cloexec(fd)`` * ``os.set_cloexec(fd, cloexec=True)`` * ``sys.getdefaultcloexec()`` - * ``sys.setdefaultcloexec(cloexec=True)`` + * ``sys.setdefaultcloexec(cloexec)`` Rationale @@ -39,7 +39,7 @@ the *bInheritHandles* parameter set to ``TRUE`` (when ``subprocess.Popen`` is created with ``close_fds=False`` for example). Windows does now have "close-on-exec" flag but an inherance flag which -is just the opposite value. For example, setting close-on-exec flag set +is just the opposite value. For example, setting close-on-exec flag means clearing the ``HANDLE_FLAG_INHERIT`` flag of an handle. @@ -192,7 +192,7 @@ platforms. * ``sys.getdefaultcloexec() -> bool``: get the current default value of the *cloexec* parameter - * ``sys.setdefaultcloexec(cloexec: bool=True)``: set the default value + * ``sys.setdefaultcloexec(cloexec: bool)``: set the default value of the *cloexec* parameter Add a new optional *cloexec* parameter to: -- Repository URL: http://hg.python.org/peps

1 0