Mailman 3 September 2013 - Python-Dev

Re: [Python-Dev] [Python-checkins] cpython: Close #19030: improvements to inspect and Enum.
by Nick Coghlan Sept. 25, 2013

Sept. 25, 2013

Hmm, I think I just noticed a subtle bug in the new version that I missed previously - I believe it may now misclassify a function in an instance dict as a method. Would need to try it out to be sure. Cheers, Nick.

1 0

Installing python27-devel on Centos5.8 running python2.4
by alok bhan Sept. 25, 2013

Sept. 25, 2013

Hi All I need help in installing python development packages (python27-devel) on Centos5.8. Upgrading hardware is not an option as of now. I've installed python2.7 in the same machine by $ wget --no-check-certificate http://www.python.org/ftp/python/2.7.3/Python-2.7.3.tgz $ ./configure --prefix=/usr/local/Python2.7 $ make altinstall Now I need help in installing python-devel package for 2.7. $ sudo yum info python2.7-devel gives no info. Whereas the package is there for python26-devel. Tried many suggestions after googling but still no luck. Any help or pointer for exact steps will be great. Thanks Alok Bhan

2 1

astgen.py vs. ASDL for ASTs
by Eli Bendersky Sept. 25, 2013

Sept. 25, 2013

Hello, Some history (as best as I could collect it) followed by a question: Before Python 2.5, the ast stdlib module was auto generated by a script named astgen.py from a textual AST definition in Tools/compiler/ast.txt. Since 2.5 (http://www.python.org/dev/peps/pep-0339/) ASTs are part of the normal compilation flow by the Python compiler itself, and the ast module uses the same ASTs generated from Parser/Python.asdl My question is, when the switch was made in 2.5 - why didn't the existing AST-generating code was used and the path moved to ASDL instead? What advantages does ASDL have over the previous approach? One reason I could think of is that ASDL nodes are typed and that's maybe better for the generated C code to handle. [My interest here is personal. One of my projects (pycparser) uses a astgen.py-like approach, and in a new project I'm considering the options again and remembered ASDL. ADSL's documentation it extremely scarce online - seems like CPython is one of its only somewhat-visible users these days] Thanks in advance, Eli

3 5

Enum Eccentricities
by Ethan Furman Sept. 23, 2013

Sept. 23, 2013

http://bugs.python.org/issue19011 So, as anyone who has worked with Enum is probably aware by now, Enum's are a strange duck -- you might even call them platypuses. For example, Enum members are instances of the class, but they are defined inside the class structure, and new ones cannot be created afterwards; Enum classes also support iteration and containment checks. What I'm looking for feedback on is the question is #19011: should an Enum member be considered a class level attribute? On the one hand, they are defined inside the class, and they are accessed via dot notation (EnumClass.member). On the other hand, inside the class they look like 3 and 5 and 'up' and 'east', and they don't live in the class dictionary. Also, if we change Enum so that members do act more like class attributes, then things like Color.red.blue.green.blue will result in Color.blue, and that seems stranger to me than having class instances be available on the class without be full-fledged class-attributes. Thoughts? Opinions? Pearls of wisdom? -- ~Ethan~

8 15

sys.intern should work on bytes
by Jesus Cea Sept. 23, 2013

Sept. 23, 2013

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 """ sys.intern(b'12121212') Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: must be str, not bytes """ I wonder why. - -- Jesús Cea Avión _/_/ _/_/_/ _/_/_/ jcea(a)jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQCVAwUBUjwvPJlgi5GaxT1NAQIINQP/ZmEyPBSapa52yZRhQf8QSVSBm5tXpWrC k9MbcvsK/5K6ArRogkulk1GSu1EJPPvuHMAXX8EenNFBXPvDRm0mOxrHkcYw5IKx Ml2ORENm+EEM/ziUJMRtY4aqD5Jp6pXSSl9UmP5OQMDJfuabSrVqs7X2409OOhUj BXeg9HvURxo= =78M+ -----END PGP SIGNATURE-----

7 15

Use an empty def as a lambda
by Ben Gift Sept. 22, 2013

Sept. 22, 2013

I think the lambda keyword is difficult to understand for many people. It would be more pythonic to use an empty def call instead. For instance this: words.sort(key = lambda x: x[2]) could look like this: words.sort(key = def (x): x[2]) It's obvious and explicit that we're creating an unnamed, anonymous function this way.

12 14

Summary of Python tracker Issues
by Python tracker Sept. 20, 2013

Sept. 20, 2013

ACTIVITY SUMMARY (2013-09-13 - 2013-09-20) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4228 (+16) closed 26602 (+31) total 30830 (+47) Open issues with patches: 1949 Issues opened (37) ================== #12085: subprocess.Popen.__del__ raises AttributeError if __init__ was http://bugs.python.org/issue12085 reopened by serhiy.storchaka #14927: add "Do not supply 'int' argument" to random.shuffle docstring http://bugs.python.org/issue14927 reopened by orsenthil #19011: Enum should have a __getattr__ that makes all the instances av http://bugs.python.org/issue19011 opened by lambacck #19012: urllib2: bad proxy configuration throws "getaddrinfo" error http://bugs.python.org/issue19012 opened by SjoerdOptLand #19014: Allow memoryview.cast() for empty views http://bugs.python.org/issue19014 opened by serhiy.storchaka #19016: autospecced namedtuples should be truthy by default http://bugs.python.org/issue19016 opened by skrisman #19017: selectors: towards uniform EBADF handling http://bugs.python.org/issue19017 opened by neologix #19019: Investigate using Apple clang for building OS X installers http://bugs.python.org/issue19019 opened by ned.deily #19020: Regression: Windows-tkinter-idle, unicode, and 0xxx filename http://bugs.python.org/issue19020 opened by terry.reedy #19021: AttributeError in Popen.__del__ http://bugs.python.org/issue19021 opened by serhiy.storchaka #19022: Improve handling of type.__abstractmethods__ descriptor http://bugs.python.org/issue19022 opened by ncoghlan #19023: ctypes docs: Unimplemented and undocumented features http://bugs.python.org/issue19023 opened by syeberman #19024: Document asterisk (*), splat or star operator http://bugs.python.org/issue19024 opened by techtonik #19025: Deleting attribute of Enum gives misleading error message http://bugs.python.org/issue19025 opened by vajrasky #19027: undefined symbol: _PyParser_Grammar http://bugs.python.org/issue19027 opened by arigo #19028: tkinter.tkapp.merge() fails on non-strings http://bugs.python.org/issue19028 opened by serhiy.storchaka #19030: inspect.getmembers and inspect.classify_class_attrs mishandle http://bugs.python.org/issue19030 opened by ethan.furman #19031: Make help() enum aware http://bugs.python.org/issue19031 opened by ethan.furman #19034: More useful repr for Tcl_Obj http://bugs.python.org/issue19034 opened by serhiy.storchaka #19035: tokenize.generate_tokens treat '\f' symbol as the end of file http://bugs.python.org/issue19035 opened by Alexey.Umnov #19036: setlocale fails due to locale.h being wrapped up in LANGINFO c http://bugs.python.org/issue19036 opened by alanh #19040: Problems with overriding Enum.__new__ http://bugs.python.org/issue19040 opened by Drekin #19042: Idle: add option to autosave 'Untitled' edit window http://bugs.python.org/issue19042 opened by terry.reedy #19043: Remove detailed listing of all versions from LICENSE, Doc/lice http://bugs.python.org/issue19043 opened by georg.brandl #19044: getaddrinfo raises near-useless exception http://bugs.python.org/issue19044 opened by Nikratio #19045: Make on Solaris 11 x64 with OracleStudio12.3 failed http://bugs.python.org/issue19045 opened by podlipnik #19046: SystemError: ..\Objects\weakrefobject.c:903: bad argument to i http://bugs.python.org/issue19046 opened by mherrmann.at #19047: Assorted weakref docs improvements http://bugs.python.org/issue19047 opened by ncoghlan #19048: itertools.tee doesn't have a __sizeof__ method http://bugs.python.org/issue19048 opened by pitrou #19049: itertools.tee uses int for indices http://bugs.python.org/issue19049 opened by pitrou #19050: crash while writing to a closed file descriptor http://bugs.python.org/issue19050 opened by damiro #19051: Unify buffered readers http://bugs.python.org/issue19051 opened by serhiy.storchaka #19053: read1() from zipfile returns empty data http://bugs.python.org/issue19053 opened by serhiy.storchaka #19054: Descriptors howto http://bugs.python.org/issue19054 opened by marco.buttu #19055: Regular expressions: * does not match as many repetitions as p http://bugs.python.org/issue19055 opened by Jason.Stumpf #19056: Windows 7, script exec not working without explicit cal of pyt http://bugs.python.org/issue19056 opened by tsunwell #19057: Sometimes urllib2 raises URLError when trying POST with httpS http://bugs.python.org/issue19057 opened by mrDoctorWho0.. Most recent 15 issues with no replies (15) ========================================== #19056: Windows 7, script exec not working without explicit cal of pyt http://bugs.python.org/issue19056 #19054: Descriptors howto http://bugs.python.org/issue19054 #19040: Problems with overriding Enum.__new__ http://bugs.python.org/issue19040 #19034: More useful repr for Tcl_Obj http://bugs.python.org/issue19034 #19031: Make help() enum aware http://bugs.python.org/issue19031 #19028: tkinter.tkapp.merge() fails on non-strings http://bugs.python.org/issue19028 #19027: undefined symbol: _PyParser_Grammar http://bugs.python.org/issue19027 #19022: Improve handling of type.__abstractmethods__ descriptor http://bugs.python.org/issue19022 #19010: Make XMLPullParser in ElementTree inherit from XMLParser http://bugs.python.org/issue19010 #19005: PyIter_Next crashes if passed a non-iterator http://bugs.python.org/issue19005 #19003: email.generator.BytesGenerator corrupts data by changing line http://bugs.python.org/issue19003 #18982: Add tests for CLI of the calendar module http://bugs.python.org/issue18982 #18979: Use argparse in the uu module http://bugs.python.org/issue18979 #18977: The -t option has no effect in for uu command-line http://bugs.python.org/issue18977 #18976: distutils/command/build_ext passes wrong linker flags http://bugs.python.org/issue18976 Most recent 15 issues waiting for review (15) ============================================= #19054: Descriptors howto http://bugs.python.org/issue19054 #19053: read1() from zipfile returns empty data http://bugs.python.org/issue19053 #19051: Unify buffered readers http://bugs.python.org/issue19051 #19049: itertools.tee uses int for indices http://bugs.python.org/issue19049 #19048: itertools.tee doesn't have a __sizeof__ method http://bugs.python.org/issue19048 #19043: Remove detailed listing of all versions from LICENSE, Doc/lice http://bugs.python.org/issue19043 #19036: setlocale fails due to locale.h being wrapped up in LANGINFO c http://bugs.python.org/issue19036 #19034: More useful repr for Tcl_Obj http://bugs.python.org/issue19034 #19030: inspect.getmembers and inspect.classify_class_attrs mishandle http://bugs.python.org/issue19030 #19028: tkinter.tkapp.merge() fails on non-strings http://bugs.python.org/issue19028 #19025: Deleting attribute of Enum gives misleading error message http://bugs.python.org/issue19025 #19023: ctypes docs: Unimplemented and undocumented features http://bugs.python.org/issue19023 #19021: AttributeError in Popen.__del__ http://bugs.python.org/issue19021 #19020: Regression: Windows-tkinter-idle, unicode, and 0xxx filename http://bugs.python.org/issue19020 #19016: autospecced namedtuples should be truthy by default http://bugs.python.org/issue19016 Top 10 most discussed issues (10) ================================= #19048: itertools.tee doesn't have a __sizeof__ method http://bugs.python.org/issue19048 28 msgs #18986: Add a case-insensitive case-preserving dict http://bugs.python.org/issue18986 20 msgs #18967: Find a less conflict prone approach to Misc/NEWS http://bugs.python.org/issue18967 18 msgs #11798: Test cases not garbage collected after run http://bugs.python.org/issue11798 15 msgs #19030: inspect.getmembers and inspect.classify_class_attrs mishandle http://bugs.python.org/issue19030 14 msgs #16042: smtplib: unlimited readline() from connection http://bugs.python.org/issue16042 10 msgs #19036: setlocale fails due to locale.h being wrapped up in LANGINFO c http://bugs.python.org/issue19036 10 msgs #19049: itertools.tee uses int for indices http://bugs.python.org/issue19049 8 msgs #14927: add "Do not supply 'int' argument" to random.shuffle docstring http://bugs.python.org/issue14927 7 msgs #18990: Return root element from ElementTree.XMLPullParser.close() to http://bugs.python.org/issue18990 7 msgs Issues closed (30) ================== #13758: compile() should not encode 'filename' (at least on Windows) http://bugs.python.org/issue13758 closed by haypo #14984: netrc module allows read of non-secured .netrc file http://bugs.python.org/issue14984 closed by r.david.murray #16201: socket.gethostbyname incorrectly parses ip http://bugs.python.org/issue16201 closed by neologix #17734: Failure when running test_builtin after test_genexps http://bugs.python.org/issue17734 closed by serhiy.storchaka #17764: Support http.server passing bind address via commend line argu http://bugs.python.org/issue17764 closed by orsenthil #18230: test_builtin fails/hangs when run after test_getopt http://bugs.python.org/issue18230 closed by serhiy.storchaka #18526: Add resource management/guarding to unittest http://bugs.python.org/issue18526 closed by zach.ware #18824: Adding LogRecord attribute "traceback" http://bugs.python.org/issue18824 closed by vinay.sajip #18856: Added test coverage for calendar print functions http://bugs.python.org/issue18856 closed by ezio.melotti #18929: inspect.classify_class_attrs ignores metaclass http://bugs.python.org/issue18929 closed by python-dev #18937: add unittest assertion for logging http://bugs.python.org/issue18937 closed by pitrou #18945: Name collision handling in tempfile is not covered by tests http://bugs.python.org/issue18945 closed by python-dev #18951: In unittest.TestCase.assertRegex change "re" and "regex" to "r http://bugs.python.org/issue18951 closed by ezio.melotti #18955: Confusing documentation in Lib/importlib/util.py http://bugs.python.org/issue18955 closed by brett.cannon #18981: Typo in the ctypes tests http://bugs.python.org/issue18981 closed by ezio.melotti #18989: reuse of enum names in class creation inconsistent http://bugs.python.org/issue18989 closed by python-dev #18995: Enum does not work with reversed http://bugs.python.org/issue18995 closed by ethan.furman #18998: iter() not working in ElementTree http://bugs.python.org/issue18998 closed by eli.bendersky #19013: unittest's own test suite is not CLI-friendly http://bugs.python.org/issue19013 closed by pitrou #19015: Too long command returns 32512 http://bugs.python.org/issue19015 closed by Sworddragon #19018: Heapq.merge suppreses IndexError from user generator http://bugs.python.org/issue19018 closed by rhettinger #19026: OrderedDict should not accept dict as parameter http://bugs.python.org/issue19026 closed by rhettinger #19029: tix.py uses StringType http://bugs.python.org/issue19029 closed by serhiy.storchaka #19032: __reduce_ex__ on lock object http://bugs.python.org/issue19032 closed by cool-RR #19033: Python 3 won't go on PC-BSD 9.1 http://bugs.python.org/issue19033 closed by ned.deily #19037: mailbox module modifies maildir file times after moving from t http://bugs.python.org/issue19037 closed by r.david.murray #19038: Fix sort order in Misc/ACKS. http://bugs.python.org/issue19038 closed by georg.brandl #19039: sphinx search, result sorting http://bugs.python.org/issue19039 closed by georg.brandl #19041: requested (DLL) module could not be found http://bugs.python.org/issue19041 closed by r.david.murray #19052: Python's CGIHTTPServer does not handle Expect: 100-continue gr http://bugs.python.org/issue19052 closed by Joseph.Warren

1 0

PEP 447: add type.__locallookup__
by Ronald Oussoren Sept. 19, 2013

Sept. 19, 2013

Hi, This PEP proposed to add a __locallookup__ slot to type objects, which is used by _PyType_Lookup and super_getattro instead of peeking in the tp_dict of classes. The PEP text explains why this is needed. Differences with the previous version: * Better explanation of why this is a useful addition * type.__locallookup__ is no longer optional. * I've added benchmarking results using pybench. (using the patch attached to issue 18181) Ronald PEP: 447 Title: Add __locallookup__ method to metaclass Version: $Revision$ Last-Modified: $Date$ Author: Ronald Oussoren <ronaldoussoren(a)mac.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 12-Jun-2013 Post-History: 2-Jul-2013, 15-Jul-2013, 29-Jul-2013 Abstract ======== Currently ``object.__getattribute__`` and ``super.__getattribute__`` peek in the ``__dict__`` of classes on the MRO for a class when looking for an attribute. This PEP adds an optional ``__locallookup__`` method to a metaclass that can be used to override this behavior. Rationale ========= It is currently not possible to influence how the `super class`_ looks up attributes (that is, ``super.__getattribute__`` unconditionally peeks in the class ``__dict__``), and that can be problematic for dynamic classes that can grow new methods on demand. The ``__locallookup__`` method makes it possible to dynamicly add attributes even when looking them up using the `super class`_. The new method affects ``object.__getattribute__`` (and `PyObject_GenericGetAttr`_) as well for consistency. Background ---------- The current behavior of ``super.__getattribute__`` causes problems for classes that are dynamic proxies for other (non-Python) classes or types, an example of which is `PyObjC`_. PyObjC creates a Python class for every class in the Objective-C runtime, and looks up methods in the Objective-C runtime when they are used. This works fine for normal access, but doesn't work for access with ``super`` objects. Because of this PyObjC currently includes a custom ``super`` that must be used with its classes. The API in this PEP makes it possible to remove the custom ``super`` and simplifies the implementation because the custom lookup behavior can be added in a central location. The superclass attribute lookup hook ==================================== Both ``super.__getattribute__`` and ``object.__getattribute__`` (or `PyObject_GenericGetAttr`_ in C code) walk an object's MRO and peek in the class' ``__dict__`` to look up attributes. A way to affect this lookup is using a method on the meta class for the type, that by default looks up the name in the class ``__dict__``. In Python code -------------- A meta type can define a method ``__locallookup__`` that is called during attribute resolution by both ``super.__getattribute__`` and ``object.__getattribute``:: class MetaType(type): def __locallookup__(cls, name): try: return cls.__dict__[name] except KeyError: raise AttributeError(name) from None The ``__locallookup__`` method has as its arguments a class and the name of the attribute that is looked up. It should return the value of the attribute without invoking descriptors, or raise `AttributeError`_ when the name cannot be found. The `type`_ class provides a default implementation for ``__locallookup__``, that looks up the name in the class dictionary. Example usage ............. The code below implements a silly metaclass that redirects attribute lookup to uppercase versions of names:: class UpperCaseAccess (type): def __locallookup__(cls, name): return cls.__dict__[name.upper()] class SillyObject (metaclass=UpperCaseAccess): def m(self): return 42 def M(self): return "fourtytwo" obj = SillyObject() assert obj.m() == "fortytwo" In C code --------- A new slot ``tp_locallookup`` is added to the ``PyTypeObject`` struct, this slot corresponds to the ``__locallookup__`` method on `type`_. The slot has the following prototype:: PyObject* (*locallookupfunc)(PyTypeObject* cls, PyObject* name); This method should lookup *name* in the namespace of *cls*, without looking at superclasses, and should not invoke descriptors. The method returns ``NULL`` without setting an exception when the *name* cannot be found, and returns a new reference otherwise (not a borrowed reference). Use of this hook by the interpreter ----------------------------------- The new method is required for metatypes and as such is defined on `type_`. Both ``super.__getattribute__`` and ``object.__getattribute__``/`PyObject_GenericGetAttr`_ (through ``_PyType_Lookup``) use the this ``__locallookup__`` method when walking the MRO. Other changes to the implementation ----------------------------------- The change for `PyObject_GenericGetAttr`_ will be done by changing the private function ``_PyType_Lookup``. This currently returns a borrowed reference, but must return a new reference when the ``__locallookup__`` method is present. Because of this ``_PyType_Lookup`` will be renamed to ``_PyType_LookupName``, this will cause compile-time errors for all out-of-tree users of this private API. The attribute lookup cache in ``Objects/typeobject.c`` is disabled for classes that have a metaclass that overrides ``__locallookup__``, because using the cache might not be valid for such classes. Performance impact ------------------ The pybench output below compares an implementation of this PEP with the regular source tree, both based on changeset a5681f50bae2, run on an idle machine an Core i7 processor running Centos 6.4. Even though the machine was idle there were clear differences between runs, I've seen difference in "minimum time" vary from -0.1% to +1.5%, with simular (but slightly smaller) differences in the "average time" difference. .. :: ------------------------------------------------------------------------------- PYBENCH 2.1 ------------------------------------------------------------------------------- * using CPython 3.4.0a0 (default, Jul 29 2013, 13:01:34) [GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] * disabled garbage collection * system check interval set to maximum: 2147483647 * using timer: time.perf_counter * timer: resolution=1e-09, implementation=clock_gettime(CLOCK_MONOTONIC) ------------------------------------------------------------------------------- Benchmark: pep447.pybench ------------------------------------------------------------------------------- Rounds: 10 Warp: 10 Timer: time.perf_counter Machine Details: Platform ID: Linux-2.6.32-358.114.1.openstack.el6.x86_64-x86_64-with-centos-6.4-Final Processor: x86_64 Python: Implementation: CPython Executable: /tmp/default-pep447/bin/python3 Version: 3.4.0a0 Compiler: GCC 4.4.7 20120313 (Red Hat 4.4.7-3) Bits: 64bit Build: Jul 29 2013 14:09:12 (#default) Unicode: UCS4 ------------------------------------------------------------------------------- Comparing with: default.pybench ------------------------------------------------------------------------------- Rounds: 10 Warp: 10 Timer: time.perf_counter Machine Details: Platform ID: Linux-2.6.32-358.114.1.openstack.el6.x86_64-x86_64-with-centos-6.4-Final Processor: x86_64 Python: Implementation: CPython Executable: /tmp/default/bin/python3 Version: 3.4.0a0 Compiler: GCC 4.4.7 20120313 (Red Hat 4.4.7-3) Bits: 64bit Build: Jul 29 2013 13:01:34 (#default) Unicode: UCS4 Test minimum run-time average run-time this other diff this other diff ------------------------------------------------------------------------------- BuiltinFunctionCalls: 45ms 44ms +1.3% 45ms 44ms +1.3% BuiltinMethodLookup: 26ms 27ms -2.4% 27ms 27ms -2.2% CompareFloats: 33ms 34ms -0.7% 33ms 34ms -1.1% CompareFloatsIntegers: 66ms 67ms -0.9% 66ms 67ms -0.8% CompareIntegers: 51ms 50ms +0.9% 51ms 50ms +0.8% CompareInternedStrings: 34ms 33ms +0.4% 34ms 34ms -0.4% CompareLongs: 29ms 29ms -0.1% 29ms 29ms -0.0% CompareStrings: 43ms 44ms -1.8% 44ms 44ms -1.8% ComplexPythonFunctionCalls: 44ms 42ms +3.9% 44ms 42ms +4.1% ConcatStrings: 33ms 33ms -0.4% 33ms 33ms -1.0% CreateInstances: 47ms 48ms -2.9% 47ms 49ms -3.4% CreateNewInstances: 35ms 36ms -2.5% 36ms 36ms -2.5% CreateStringsWithConcat: 69ms 70ms -0.7% 69ms 70ms -0.9% DictCreation: 52ms 50ms +3.1% 52ms 50ms +3.0% DictWithFloatKeys: 40ms 44ms -10.1% 43ms 45ms -5.8% DictWithIntegerKeys: 32ms 36ms -11.2% 35ms 37ms -4.6% DictWithStringKeys: 29ms 34ms -15.7% 35ms 40ms -11.0% ForLoops: 30ms 29ms +2.2% 30ms 29ms +2.2% IfThenElse: 38ms 41ms -6.7% 38ms 41ms -6.9% ListSlicing: 36ms 36ms -0.7% 36ms 37ms -1.3% NestedForLoops: 43ms 45ms -3.1% 43ms 45ms -3.2% NestedListComprehensions: 39ms 40ms -1.7% 39ms 40ms -2.1% NormalClassAttribute: 86ms 82ms +5.1% 86ms 82ms +5.0% NormalInstanceAttribute: 42ms 42ms +0.3% 42ms 42ms +0.0% PythonFunctionCalls: 39ms 38ms +3.5% 39ms 38ms +2.8% PythonMethodCalls: 51ms 49ms +3.0% 51ms 50ms +2.8% Recursion: 67ms 68ms -1.4% 67ms 68ms -1.4% SecondImport: 41ms 36ms +12.5% 41ms 36ms +12.6% SecondPackageImport: 45ms 40ms +13.1% 45ms 40ms +13.2% SecondSubmoduleImport: 92ms 95ms -2.4% 95ms 98ms -3.6% SimpleComplexArithmetic: 28ms 28ms -0.1% 28ms 28ms -0.2% SimpleDictManipulation: 57ms 57ms -1.0% 57ms 58ms -1.0% SimpleFloatArithmetic: 29ms 28ms +4.7% 29ms 28ms +4.9% SimpleIntFloatArithmetic: 37ms 41ms -8.5% 37ms 41ms -8.7% SimpleIntegerArithmetic: 37ms 41ms -9.4% 37ms 42ms -10.2% SimpleListComprehensions: 33ms 33ms -1.9% 33ms 34ms -2.9% SimpleListManipulation: 28ms 30ms -4.3% 29ms 30ms -4.1% SimpleLongArithmetic: 26ms 26ms +0.5% 26ms 26ms +0.5% SmallLists: 40ms 40ms +0.1% 40ms 40ms +0.1% SmallTuples: 46ms 47ms -2.4% 46ms 48ms -3.0% SpecialClassAttribute: 126ms 120ms +4.7% 126ms 121ms +4.4% SpecialInstanceAttribute: 42ms 42ms +0.6% 42ms 42ms +0.8% StringMappings: 94ms 91ms +3.9% 94ms 91ms +3.8% StringPredicates: 48ms 49ms -1.7% 48ms 49ms -2.1% StringSlicing: 45ms 45ms +1.4% 46ms 45ms +1.5% TryExcept: 23ms 22ms +4.9% 23ms 22ms +4.8% TryFinally: 32ms 32ms -0.1% 32ms 32ms +0.1% TryRaiseExcept: 17ms 17ms +0.9% 17ms 17ms +0.5% TupleSlicing: 49ms 48ms +1.1% 49ms 49ms +1.0% WithFinally: 48ms 47ms +2.3% 48ms 47ms +2.4% WithRaiseExcept: 45ms 44ms +0.8% 45ms 45ms +0.5% ------------------------------------------------------------------------------- Totals: 2284ms 2287ms -0.1% 2306ms 2308ms -0.1% (this=pep447.pybench, other=default.pybench) Alternative proposals --------------------- ``__getattribute_super__`` .......................... An earlier version of this PEP used the following static method on classes:: def __getattribute_super__(cls, name, object, owner): pass This method performed name lookup as well as invoking descriptors and was necessarily limited to working only with ``super.__getattribute__``. Reuse ``tp_getattro`` ..................... It would be nice to avoid adding a new slot, thus keeping the API simpler and easier to understand. A comment on `Issue 18181`_ asked about reusing the ``tp_getattro`` slot, that is super could call the ``tp_getattro`` slot of all methods along the MRO. That won't work because ``tp_getattro`` will look in the instance ``__dict__`` before it tries to resolve attributes using classes in the MRO. This would mean that using ``tp_getattro`` instead of peeking the class dictionaries changes the semantics of the `super class`_. References ========== * `Issue 18181`_ contains a prototype implementation Copyright ========= This document has been placed in the public domain. .. _`Issue 18181`: http://bugs.python.org/issue18181 .. _`super class`: http://docs.python.org/3/library/functions.html#super .. _`NotImplemented`: http://docs.python.org/3/library/constants.html#NotImplemented .. _`PyObject_GenericGetAttr`: http://docs.python.org/3/c-api/object.html#PyObject_GenericGetAttr .. _`type`: http://docs.python.org/3/library/functions.html#type .. _`AttributeError`: http://docs.python.org/3/library/exceptions.html#AttributeError .. _`PyObjC`: http://pyobjc.sourceforge.net/ .. _`classmethod`: http://docs.python.org/3/library/functions.html#classmethod

14 44

PEP 453 Round 4 - Explicit bootstrapping of pip in Python installations
by Donald Stufft Sept. 19, 2013

Sept. 19, 2013

We've updated PEP453 based on some of the early feedback we've gotten from -dev and Martin. Major changes: * Removal of the option to fetch pip from PyPI in order not to modify the trust model of the Python installers * Consequently rename the model from ``getpip`` to ``extractpip`` Also available online at: http://www.python.org/dev/peps/pep-0453/ Abstract ======== This PEP proposes that the `pip`_ package manager be made available by default when installing CPython and when creating virtual environments using the standard library's ``venv`` module via the ``pyvenv`` command line utility). To clearly demarcate development responsibilities, and to avoid inadvertently downgrading ``pip`` when updating CPython, the proposed mechanism to achieve this is to include an explicit `pip`_ bootstrapping mechanism in the standard library that is invoked automatically by the CPython installers provided on python.org. The PEP also strongly recommends that CPython redistributors and other Python implementations ensure that ``pip`` is available by default, or at the very least, explicitly document the fact that it is not included. Proposal ======== This PEP proposes the inclusion of an ``extractpip`` bootstrapping module in Python 3.4, as well as in the next maintenance releases of Python 3.3 and 2.7. This PEP does *not* propose making pip (or any dependencies) directly available as part of the standard library. Instead, pip will be a bundled application provided along with CPython for the convenience of Python users, but subject to its own development life cycle and able to be upgraded independently of the core interpreter and standard library. Rationale ========= Currently, on systems without a platform package manager and repository, installing a third-party Python package into a freshly installed Python requires first identifying an appropriate package manager and then installing it. Even on systems that *do* have a platform package manager, it is unlikely to include every package that is available on the Python Package Index, and even when a desired third-party package is available, the correct name in the platform package manager may not be clear. This means that, to work effectively with the Python Package Index ecosystem, users must know which package manager to install, where to get it, and how to install it. The effect of this is that third-party Python projects are currently required to choose from a variety of undesirable alternatives: * Assume the user already has a suitable cross-platform package manager installed. * Duplicate the instructions and tell their users how to install the package manager. * Completely forgo the use of dependencies to ease installation concerns for their users. All of these available options have significant drawbacks. If a project simply assumes a user already has the tooling then beginning users may get a confusing error message when the installation command doesn't work. Some operating systems may ease this pain by providing a global hook that looks for commands that don't exist and suggest an OS package they can install to make the command work, but that only works on systems with platform package managers (such as major Linux distributions). No such assistance is available for Windows and Mac OS X users. The challenges of dealing with this problem are a regular feature of feedback the core Python developers receive from professional educators and others introducing new users to Python. If a project chooses to duplicate the installation instructions and tell their users how to install the package manager before telling them how to install their own project then whenever these instructions need updates they need updating by every project that has duplicated them. This is particular problematic when there are multiple competing installation tools available, and different projects recommend different tools. This specific problem can be partially alleviated by strongly promoting ``pip`` as the default installer and recommending that other projects reference `pip's own bootstrapping instructions <http://www.pip-installer.org/en/latest/installing.html>`__ rather than duplicating them. However the user experience created by this approach still isn't good (especially on Windows, where downloading and running the ``get-pip.py`` bootstrap script with the default OS configuration is significantly more painful than downloading and running a binary executable or installer). The situation becomes even more complicated when multiple Python versions are involved (for example, parallel installations of Python 2 and Python 3), since that makes it harder to create and maintain good platform specific ``pip`` installers independently of the CPython installers. The projects that have decided to forgo dependencies altogether are forced to either duplicate the efforts of other projects by inventing their own solutions to problems or are required to simply include the other projects in their own source trees. Both of these options present their own problems either in duplicating maintenance work across the ecosystem or potentially leaving users vulnerable to security issues because the included code or duplicated efforts are not automatically updated when upstream releases a new version. By providing a cross-platform package manager by default it will be easier for users trying to install these third-party packages as well as easier for the people distributing them as they should now be able to safely assume that most users will have the appropriate installation tools available. This is expected to become more important in the future as the Wheel_ package format (deliberately) does not have a built in "installer" in the form of ``setup.py`` so users wishing to install from a wheel file will want an installer even in the simplest cases. Reducing the burden of actually installing a third-party package should also decrease the pressure to add every useful module to the standard library. This will allow additions to the standard library to focus more on why Python should have a particular tool out of the box, and why it is reasonable for that package to adopt the standard library's 18-24 month feature release cycle, instead of using the general difficulty of installing third-party packages as justification for inclusion. Providing a standard installation system also helps with bootstrapping alternate build and installer systems, such as ``setuptools``, ``zc.buildout`` and the ``hashdist``/``conda`` combination that is aimed specifically at the scientific community. So long as ``pip install <tool>`` works, then a standard Python-specific installer provides a reasonably secure, cross platform mechanism to get access to these utilities. Why pip? -------- ``pip`` has been chosen as the preferred default installer, as it addresses several design and user experience issues with its predecessor ``easy_install`` (these issues can't readily be fixed in ``easy_install`` itself due to backwards compatibility concerns). ``pip`` is also well suited to working within the bounds of a single Python runtime installation (including associated virtual environments), which is a desirable feature for a tool bundled with CPython. Other tools like ``zc.buildout`` and ``conda`` are more ambitious in their aims (and hence substantially better than ``pip`` at handling external binary dependencies), so it makes sense for the Python ecosystem to treat them more like platform package managers to inter operate with rather than as the default cross-platform installation tool. This relationship is similar to that between ``pip`` and platform package management systems like ``apt`` and ``yum`` (which are also designed to handle arbitrary binary dependencies). Explicit bootstrapping mechanism ================================ An additional module called ``extractpip`` will be added to the standard library whose purpose is to install pip and any of its dependencies into the appropriate location (most commonly site-packages). It will expose a callable named ``bootstrap()`` as well as offer direct execution via ``python -m extractpip``. The bootstrap will *not* contact PyPI, but instead rely on a private copy of pip stored inside the standard library. Accordingly, only options related to the installation location will be supported (``--user``, ``--root``, etc). It is considered desirable that users be strongly encouraged to use the latest available version of ``pip``, in order to take advantage of the ongoing efforts to improve the security of the PyPI based ecosystem, as well as benefiting from the efforts to improve the speed, reliability and flexibility of that ecosystem. In order to satisfy this goal of providing the most recent version of ``pip`` by default, the private copy of ``pip`` will be updated in CPython maintenance releases, which should align well with the 6-month cycle used for new ``pip`` releases. Security considerations ----------------------- The design in this PEP has been deliberately chosen to avoid making any significant changes to the trust model of the CPython installers for end users that do not subsequently make use of ``pip``. The installers will contain all the components of a fully functioning version of Python, including the ``pip`` installer. The installation process will *not* require network access, and will *not* rely on trusting the security of the network connection established between ``pip`` and the Python package index. Only users that choose to use ``pip`` directly will need to pay attention to any PyPI related security considerations. Implementation strategy ----------------------- To ensure there is no need for network access when installing Python or creating virtual environments, the ``extractpip`` module will, as an implementation detail, include a complete private copy of pip and its dependencies which will be used to extract pip and install it into the target environment. It is important to stress that this private copy of pip is *only* an implementation detail and it should *not* be relied on or assumed to exist beyond the public capabilities exposed through the ``extractpip`` module (and indirectly through ``venv``). There is not yet a reference ``extractpip`` implementation. The existing ``get-pip.py`` bootstrap script demonstrates an earlier variation of the general concept, but the standard library version would take advantage of the improved distribution capabilities offered by the CPython installers to include private copies of ``pip`` and ``setuptools`` as wheel files (rather than as embedded base64 encoded data), and would not try to contact PyPI (instead installing directly from the private wheel files. Rather than including separate code to handle the bootstrapping, the ``extractpip`` module will manipulate sys.path appropriately to allow the wheel files to be used to install themselves, either into the current Python installation or into a virtual environment (as determined by the options passed to the bootstrap command). Proposed CLI ------------ The proposed CLI is based on a subset of the existing ``pip install`` options:: Usage: python -m extractpip [options] General Options: -h, --help Show help. -v, --verbose Give more output. Option is additive, and can be used up to 3 times. -V, --version Show the pip version that would be extracted and exit. -q, --quiet Give less output. Installation Options: -U, --upgrade Upgrade pip and dependencies, even if already installed --user Install using the user scheme. --root <dir> Install everything relative to this alternate root directory. In most cases, end users won't need to use this CLI directly, as ``pip`` should have been installed automatically when installing Python or when creating a virtual environment. Users that want to retrieve the latest version from PyPI, or otherwise needing more flexibility, should invoke the extracted ``pip`` appropriately. Proposed module API ------------------- The proposed ``extractpip`` module API consists of the following two functions:: def version(): """ Returns a string specifying the bundled version of pip. """ def bootstrap(root=None, upgrade=False, user=False, verbosity=0): """ Bootstrap pip into the current Python installation (or the given root directory). """ Invocation from the CPython installers -------------------------------------- The CPython Windows and Mac OS X installers will each gain a new option: * Install pip (the default Python package management utility)? This option will be checked by default. If the option is checked, then the installer will invoke the following command with the just installed Python:: python -m extractpip --upgrade This ensures that, by default, installing or updating CPython will ensure that the installed version of pip is at least as recent as the one included with that version of CPython. Installing from source ---------------------- While the prebuilt binary installers will be updated to run ``python -m extractpip`` by default, no such change will be made to the ``make install`` and ``make altinstall`` commands of the source distribution. ``extractpip`` itself (including the private copy of ``pip`` and its dependencies) will still be installed normally (as it is a regular part of the standard library), only the implicit installation of pip and its dependencies will be skipped. Keeping the pip bootstrapping as a separate step for ``make``-based installations should minimize the changes CPython redistributors need to make to their build processes. Avoiding the layer of indirection through ``make`` for the ``extractpip`` invocation avoids any challenges associated with determining where to install the extracted ``pip``. Changes to virtual environments ------------------------------- Python 3.3 included a standard library approach to virtual Python environments through the ``venv`` module. Since its release it has become clear that very few users have been willing to use this feature directly, in part due to the lack of an installer present by default inside of the virtual environment. They have instead opted to continue using the ``virtualenv`` package which *does* include pip installed by default. To make the ``venv`` more useful to users it will be modified to issue the pip bootstrap by default inside of the new environment while creating it. This will allow people the same convenience inside of the virtual environment as this PEP provides outside of it as well as bringing the ``venv`` module closer to feature parity with the external ``virtualenv`` package, making it a more suitable replacement. To handle cases where a user does not wish to have pip bootstrapped into their virtual environment a ``--without-pip`` option will be added. The ``venv.EnvBuilder`` and ``venv.create`` APIs will be updated to accept one new parameter: ``with_pip`` (defaulting to ``False``). The new default for the module API is chosen for backwards compatibility with the current behaviour (as it is assumed that most invocation of the ``venv`` module happens through third part tools that likely will not want ``pip`` installed without explicitly requesting it), while the default for the command line interface is chosen to try to ensure ``pip`` is available in most virtual environments without additional action on the part of the end user. This particular change will be made only for Python 3.4 and later versions. The third-party ``virtualenv`` project will still be needed to obtain a consistent cross-version experience in Python 3.3 and 2.7. Documentation ------------- The "Installing Python Modules" section of the standard library documentation will be updated to recommend the use of the bootstrapped `pip` installer. It will give a brief description of the most common commands and options, but delegate to the externally maintained ``pip`` documentation for the full details. The existing content of the module installation guide will be retained, but under a new "Invoking distutils directly" subsection. Bundling CA certificates with CPython ------------------------------------- The ``extractpip`` implementation will include the ``pip`` CA bundle along with the rest of ``pip``. This means CPython effectively includes a CA bundle that is used solely by ``pip`` after it has been extracted. This is considered preferable to relying solely on the system certificate stores, as it ensures that ``pip`` will behave the same across all supported versions of Python, even those prior to Python 3.4 that cannot access the system certificate store on Windows. Automatic installation of setuptools ------------------------------------ ``pip`` currently depends on ``setuptools`` to handle metadata generation during the build process, along with some other features. While work is ongoing to reduce or eliminate this dependency, it is not clear if that work will be complete for pip 1.5 (which is the version likely to be current when Python 3.4.0 is released). This PEP proposes that, if pip still requires it as a dependency, ``extractpip`` will include a private copy of ``setuptools`` (in addition to the private copy of ``extractpip``). ``python -m extractpip`` will then install the private copy in addition to installing ``pip`` itself. However, this behavior is officially considered an implementation detail. Other projects which explicitly require ``setuptools`` must still provide an appropriate dependency declaration, rather than assuming ``setuptools`` will always be installed alongside ``pip``. Once pip is able to run ``pip install --upgrade pip`` without needing ``setuptools`` installed first, then the private copy of ``setuptools`` will be removed from ``extractpip`` in subsequent CPython releases. Updating the private copy of pip -------------------------------- In order to keep up with evolutions in packaging as well as providing users with as recent version a possible the ``extractpip`` module will be regularly updated to the latest versions of everything it bootstraps. After each new ``pip`` release, and again during the preparation for any release of Python (including feature releases), a script, provided as part of this PEP, will be run to ensure the private copies stored in the CPython source repository have been updated to the latest versions. Updating the extractpip module API and CLI ------------------------------------------ Like ``venv`` and ``pyvenv``, the ``extractpip`` module API and CLI will be governed by the normal rules for the standard library: no new features are permitted in maintenance releases. However, the embedded components may be updated as noted above, so the extracted ``pip`` may offer additional functionality in maintenance releases. Feature addition in maintenance releases ======================================== Adding a new module to the standard library in Python 2.7 and 3.3 maintenance releases breaks the usual policy of "no new features in maintenance releases". It is being proposed in this case as the current bootstrapping issues for the third-party Python package ecosystem greatly affects the experience of new users, especially on Python 2 where many Python 3 standard library improvements are available as backports on PyPI, but are not included in the Python 2 standard library. By updating Python 2.7, 3.3 and 3.4 to easily bootstrap the PyPI ecosystem, this change should aid the vast majority of current Python users, rather than only those with the freedom to adopt Python 3.4 as soon as it is released. Open Question: Uninstallation ============================= No changes are currently proposed to the uninstallation process. The bootstrapped pip will be installed the same way as any other pip installed packages, and will be handled in the same way as any other post-install additions to the Python environment. At least on Windows, that means the bootstrapped files will be left behind after uninstallation, since those files won't be associated with the Python MSI installer. .. note:: Perhaps the installer should be updated to clobber everything in site-packages and the Scripts directory when uninstalled (treating them as "data directories" from Python's point of view), but I would prefer not to make this PEP conditional on that change. Open Question: Script Execution on Windows ========================================== .. note:: Perhaps this question should be separated out from the PEP? While the Windows installer was updated in Python 3.3 to optionally make ``python`` available on the PATH, no such change was made to include the Scripts directory. Independently of this PEP, a proposal has also been made to rename the ``Tools\Scripts`` subdirectory to ``bin`` in order to improve consistency with the typical script installation directory names on \*nix systems. Accordingly, in addition to adding the option to extract and install ``pip`` during installation, this PEP proposes that the Windows installer (and ``sysconfig``) in Python 3.4 and later be updated to: - install scripts to PythonXY\bin rather than PythonXY\Tools\Scripts - add PythonXY\bin to the Windows PATH (in addition to PythonXY) when the PATH modification option is enabled during installation For Python 2.7 and 3.3, it is proposed that the only change be the one to bootstrap ``pip`` by default. This means that, for Python 3.3, the most reliable way to invoke pip on Windows (without tinkering manually with PATH) will actually be ``py -m pip`` (or ``py -3 -m pip`` to select the Python 3 version if both Python 2 and 3 are installed) rather than simply calling ``pip``. For Python 2.7 and 3.2, the most reliable mechanism will be to install the standalone Python launcher for Windows and then use ``py -m pip`` as noted above. Adding the scripts directory to the system PATH would mean that ``pip`` works reliably in the "only one Python installation on the system PATH" case, with ``py -m pip``, ``pipX``, or ``pipX.Y`` needed only to select a non-default version in the parallel installation case (and outside a virtual environment). This change should also make the ``pyvenv`` command substantially easier to invoke on Windows, along with all scripts installed by ``pip``, ``easy_install`` and similar tools. While the script invocations on recent versions of Python will run through the Python launcher for Windows, this shouldn't cause any issues, as long as the Python files in the Scripts directory correctly specify a Python version in their shebang line or have an adjacent Windows executable (as ``easy_install`` and ``pip`` do). Recommendations for Downstream Distributors =========================================== A common source of Python installations are through downstream distributors such as the various Linux Distributions [#ubuntu]_ [#debian]_ [#fedora]_, OSX package managers [#homebrew]_, or Python-specific tools [#conda]_. In order to provide a consistent, user-friendly experience to all users of Python regardless of how they attained Python this PEP recommends and asks that downstream distributors: * Ensure that whenever Python is installed pip is also installed. * This may take the form of separate packages with dependencies on each other so that installing the Python package installs the pip package and installing the pip package installs the Python package. * Do not remove the bundled copy of pip. * This is required for installation of pip into a virtual environment by the ``venv`` module. * This is similar to the existing ``virtualenv`` package for which many downstream distributors have already made exception to the common "debundling" policy. * This does mean that if ``pip`` needs to be updated due to a security issue, so does the bundled version in the ``extractpip`` bootstrap module * However, altering the bundled version of pip to remove the embedded CA certificate bundle and rely on the system CA bundle instead is a reasonable change. * Migrate build systems to utilize `pip`_ and `Wheel`_ instead of directly using ``setup.py``. * This will ensure that downstream packages can more easily utilize the new metadata formats which may not have a ``setup.py``. * Ensure that all features of this PEP continue to work with any modifications made to the redistributed version of Python. * Checking the version of pip that will be bootstrapped using ``python -m extractpip --version`` or ``extractpip.version()``. * Installation of the bundled version of pip into a global or virtual python environment using ``python -m extractpip`` or ``extractpip.bootstrap()``. * ``pip install --upgrade pip`` in a global installation should not affect any already created virtual environments (but is permitted to affect future virtual environments, even though it will not do so in the normal case). * ``pip install --upgrade pip`` in a virtual environment should not affect the global installation. In the event that a Python redistributor chooses *not* to follow these recommendations, we request that they explicitly document this fact and provide their users with suitable guidance on translating upstream ``pip`` based installation instructions into something appropriate for the platform. Other Python implementations are also encouraged to follow these guidelines where applicable. Policies & Governance ===================== The maintainers of the bootstrapped software and the CPython core team will work together in order to address the needs of both. The bootstrapped software will still remain external to CPython and this PEP does not include CPython subsuming the development responsibilities or design decisions of the bootstrapped software. This PEP aims to decrease the burden on end users wanting to use third-party packages and the decisions inside it are pragmatic ones that represent the trust that the Python community has already placed in the Python Packaging Authority as the authors and maintainers of ``pip``, ``setuptools``, PyPI, ``virtualenv`` and other related projects. Backwards Compatibility ----------------------- The public API and CLI of the ``extractpip`` module itself will fall under the typical backwards compatibility policy of Python for its standard library. The externally developed software that this PEP bundles does not. Most importantly, this means that the bootstrapped version of pip may gain new features in CPython maintenance releases, and pip continues to operate on its own 6 month release cycle rather than CPython's 18-24 month cycle. Security Releases ----------------- Any security update that affects the ``extractpip`` module will be shared prior to release with the Python Security Response Team (security(a)python.org). The PSRT will then decide if the reported issue warrants a security release of CPython with an updated private copy of ``pip``. Appendix: Rejected Proposals ============================ Automatically contacting PyPI when bootstrapping pip ---------------------------------------------------- Earlier versions of this PEP called the bootstrapping module ``getpip`` and defaulted to downloading and installing ``pip`` from PyPI, with the private copy used only as a fallback option or when explicitly requested. This resulted in several complex edge cases, along with difficulties in defining a clean API and CLI for the bootstrap module. It also significantly altered the default trust model for the binary installers published on python.org, as end users would need to explicitly *opt-out* of trusting the security of the PyPI ecosystem (rather than opting in to it by explicitly invoking ``pip`` following installation). As a result, the PEP was simplified to the current design, where the bootstrapping *always* uses the private copy of ``pip``. Contacting PyPI is now always an explicit separate step, with direct access to the full pip interface. Implicit bootstrap ------------------ `PEP439`_, the predecessor for this PEP, proposes its own solution. Its solution involves shipping a fake ``pip`` command that when executed would implicitly bootstrap and install pip if it does not already exist. This has been rejected because it is too "magical". It hides from the end user when exactly the pip command will be installed or that it is being installed at all. It also does not provide any recommendations or considerations towards downstream packagers who wish to manage the globally installed pip through the mechanisms typical for their system. The implicit bootstrap mechanism also ran into possible permissions issues, if a user inadvertently attempted to bootstrap pip without write access to the appropriate installation directories. Including pip directly in the standard library ---------------------------------------------- Similar to this PEP is the proposal of just including pip in the standard library. This would ensure that Python always includes pip and fixes all of the end user facing problems with not having pip present by default. This has been rejected because we've learned, through the inclusion and history of ``distutils`` in the standard library, that losing the ability to update the packaging tools independently can leave the tooling in a state of constant limbo. Making it unable to ever reasonably evolve in a time frame that actually affects users as any new features will not be available to the general population for *years*. Allowing the packaging tools to progress separately from the Python release and adoption schedules allows the improvements to be used by *all* members of the Python community and not just those able to live on the bleeding edge of Python releases. There have also been issues in the past with the "dual maintenance" problem if a project continues to be maintained externally while *also* having a fork maintained in the standard library. Since external maintenance of ``pip`` will always be needed to support earlier Python versions, the proposed bootstrapping mechanism will becoming the explicit responsibility of the CPython core developers (assisted by the pip developers), while pip issues reported to the CPython tracker will be migrated to the pip issue tracker. There will no doubt still be some user confusion over which tracker to use, but hopefully less than has been seen historically when including complete public copies of third-party projects in the standard library. Finally, the approach described in this PEP avoids some technical issues related to handle CPython maintenance updates when pip has been independently updated to a more recent version. The proposed pip-based bootstrapping mechanism handles that automatically, since pip and the system installer never get into a fight about who owns the pip installation (it is always managed through pip, either directly, or indirectly via the getpip bootstrap module). Defaulting to --user installation --------------------------------- Some consideration was given to bootstrapping pip into the per-user site-packages directory by default. However, this behavior would be surprising (as it differs from the default behavior of pip itself) and is also not currently considered reliable (there are some edge cases which are not handled correctly when pip is installed into the user site-packages directory rather than the system site-packages). .. _Wheel: http://www.python.org/dev/peps/pep-0427/ .. _pip: http://www.pip-installer.org .. _setuptools: https://pypi.python.org/pypi/setuptools .. _PEP439: http://www.python.org/dev/peps/pep-0439/ References ========== .. [#ubuntu] `Ubuntu <http://www.ubuntu.com/>` .. [#debian] `Debian <http://www.debian.org>` .. [#fedora] `Fedora <https://fedoraproject.org/>` .. [#homebrew] `Homebrew <http://brew.sh/>` .. [#conda] `Conda <http://www.continuum.io/blog/conda>` ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

5 14

Compiler for the Mac OS X version of Python 3.4
by Raymond Hettinger Sept. 19, 2013

Sept. 19, 2013

I was exercising the alpha two release of 3.4 and noticed that it is still being built under GCC 4.2.1. Is there any reason we have to use an old compiler? I would like to see it built under the latest version of Clang (like the other tools on the Mac) or under GCC 4.8.1. I've better using the latter for many months. It works great and produces *much* better code (in terms of readability for those of us who look at assembly output and in terms of speed (approx 20-25% better for cpu bound code)). I'm wondering whether is some issue that forces us to use GCC 4.2.1 or whether that is just an artifact of continuing to do what we've always been doing. Raymond

14 22