Mailman 3 June 2013 - Python-Dev

[engineering.redhat.com #216557] [Fwd: Validating SSL By Default (aka Including a Cert Bundle in CPython)]
by Red Hat Security Response Team 04 Jun '13

04 Jun '13

Hello David, Donald, David, thank you for sharing these intentions with us. On Mon Jun 03 15:56:09 2013, dmalcolm(a)redhat.com wrote: > > As of right now, as far as I can tell, Python does not validate HTTPS certificates by default. As far as I can tell this > is because there is no guaranteed certificates available. > > So I would like to propose that CPython adopt the Mozilla SSL certificate list and include it in core, and switch over > the API's so that they verify HTTPS by default. Donald, we would only welcome this enhancement / proposal. To mention some examples - urllib2 and httplib modules: http://docs.python.org/2/library/urllib2.html http://docs.python.org/2/library/httplib.html are documented upstream not to perform SSL certificate verification by default (and due this fact there has been couple of CVE identifiers assigned in the past for applications that incorrectly assumed certificates would be validated when using these modules). So any enhancement, which can upstream done in this area, would be only welcome. > This is what most people are going to expect when using a https url > (Especially after learning that Python 2.x doesn't verify TLS, but Python 3.x "does"). > > Ideally this would take the shape of attempting to locate the system certificate store if possible, and if that doesn't > work falling back to the bundled certificates. That way the various Linux distros can easily have their copies of Python > depend soley on their built in certs, but Windows, OSX, Source compiles etc will all still have a fallback value. AFAWCT that proposal looks reasonable. Thank you && Regards, Jan. -- Jan iankko Lieskovsky / Red Hat Security Response Team > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > >

1 0

Validating SSL By Default (aka Including a Cert Bundle in CPython)
by Donald Stufft 04 Jun '13

04 Jun '13

As of right now, as far as I can tell, Python does not validate HTTPS certificates by default. As far as I can tell this is because there is no guaranteed certificates available. So I would like to propose that CPython adopt the Mozilla SSL certificate list and include it in core, and switch over the API's so that they verify HTTPS by default. This is what most people are going to expect when using a https url (Especially after learning that Python 2.x doesn't verify TLS, but Python 3.x "does"). Ideally this would take the shape of attempting to locate the system certificate store if possible, and if that doesn't work falling back to the bundled certificates. That way the various Linux distros can easily have their copies of Python depend soley on their built in certs, but Windows, OSX, Source compiles etc will all still have a fallback value. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

16 48

Putting the Mac Build in the Apple App Store
by Raymond Hettinger 04 Jun '13

04 Jun '13

Does anyone know what we would need to do to get Python in the Apple application store as a free App? The default security settings on OS X 10.8 block the installation of the DMG (or any software downloaded outside the app store). A number of my students are having difficulty getting around it will help. If we were in the app store, installation and upgrade would be a piece of cake. Raymond

5 4

PEP 442: Safe object finalization
by Antoine Pitrou 04 Jun '13

04 Jun '13

Hello, I would like to submit the following PEP for discussion and evaluation. Regards Antoine. PEP: 442 Title: Safe object finalization Version: $Revision$ Last-Modified: $Date$ Author: Antoine Pitrou <solipsis(a)pitrou.net> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 2013-05-18 Python-Version: 3.4 Post-History: Resolution: TBD Abstract ======== This PEP proposes to deal with the current limitations of object finalization. The goal is to be able to define and run finalizers for any object, regardless of their position in the object graph. This PEP doesn't call for any change in Python code. Objects with existing finalizers will benefit automatically. Definitions =========== Reference A directional link from an object to another. The target of the reference is kept alive by the reference, as long as the source is itself alive and the reference isn't cleared. Weak reference A directional link from an object to another, which doesn't keep alive its target. This PEP focusses on non-weak references. Reference cycle A cyclic subgraph of directional links between objects, which keeps those objects from being collected in a pure reference-counting scheme. Cyclic isolate (CI) A reference cycle in which no object is referenced from outside the cycle *and* whose objects are still in a usable, non-broken state: they can access each other from their respective finalizers. Cyclic garbage collector (GC) A device able to detect cyclic isolates and turn them into cyclic trash. Objects in cyclic trash are eventually disposed of by the natural effect of the references being cleared and their reference counts dropping to zero. Cyclic trash (CT) A reference cycle, or former reference cycle, in which no object is referenced from outside the cycle *and* whose objects have started being cleared by the GC. Objects in cyclic trash are potential zombies; if they are accessed by Python code, the symptoms can vary from weird AttributeErrors to crashes. Zombie / broken object An object part of cyclic trash. The term stresses that the object is not safe: its outgoing references may have been cleared, or one of the objects it references may be zombie. Therefore, it should not be accessed by arbitrary code (such as finalizers). Finalizer A function or method called when an object is intended to be disposed of. The finalizer can access the object and release any resource held by the object (for example mutexes or file descriptors). An example is a ``__del__`` method. Resurrection The process by which a finalizer creates a new reference to an object in a CI. This can happen as a quirky but supported side-effect of ``__del__`` methods. Impact ====== While this PEP discusses CPython-specific implementation details, the change in finalization semantics is expected to affect the Python ecosystem as a whole. In particular, this PEP obsoletes the current guideline that "objects with a ``__del__`` method should not be part of a reference cycle". Benefits ======== The primary benefits of this PEP regard objects with finalizers, such as objects with a ``__del__`` method and generators with a ``finally`` block. Those objects can now be reclaimed when they are part of a reference cycle. The PEP also paves the way for further benefits: * The module shutdown procedure may not need to set global variables to None anymore. This could solve a well-known class of irritating issues. The PEP doesn't change the semantics of: * Weak references caught in reference cycles. * C extension types with a custom ``tp_dealloc`` function. Description =========== Reference-counted disposal -------------------------- In normal reference-counted disposal, an object's finalizer is called just before the object is deallocated. If the finalizer resurrects the object, deallocation is aborted. *However*, if the object was already finalized, then the finalizer isn't called. This prevents us from finalizing zombies (see below). Disposal of cyclic isolates --------------------------- Cyclic isolates are first detected by the garbage collector, and then disposed of. The detection phase doesn't change and won't be described here. Disposal of a CI traditionally works in the following order: 1. Weakrefs to CI objects are cleared, and their callbacks called. At this point, the objects are still safe to use. 2. The CI becomes a CT as the GC systematically breaks all known references inside it (using the ``tp_clear`` function). 3. Nothing. All CT objects should have been disposed of in step 2 (as a side-effect of clearing references); this collection is finished. This PEP proposes to turn CI disposal into the following sequence (new steps are in bold): 1. Weakrefs to CI objects are cleared, and their callbacks called. At this point, the objects are still safe to use. 2. **The finalizers of all CI objects are called.** 3. **The CI is traversed again to determine if it is still isolated. If it is determined that at least one object in CI is now reachable from outside the CI, this collection is aborted and the whole CI is resurrected. Otherwise, proceed.** 4. The CI becomes a CT as the GC systematically breaks all known references inside it (using the ``tp_clear`` function). 5. Nothing. All CT objects should have been disposed of in step 4 (as a side-effect of clearing references); this collection is finished. C-level changes =============== Type objects get a new ``tp_finalize`` slot to which ``__del__`` methods are bound. Generators are also modified to use this slot, rather than ``tp_del``. At the C level, a ``tp_finalize`` function is a normal function which will be called with a regular, alive object as its only argument. It should not attempt to revive or collect the object. For compatibility, ``tp_del`` is kept in the type structure. Handling of objects with a non-NULL ``tp_del`` is unchanged: when part of a CI, they are not finalized and end up in ``gc.garbage``. However, a non-NULL ``tp_del`` is not encountered anymore in the CPython source tree (except for testing purposes). On the internal side, a bit is reserved in the GC header for GC-managed objects to signal that they were finalized. This helps avoid finalizing an object twice (and, especially, finalizing a CT object after it was broken by the GC). Discussion ========== Predictability -------------- Following this scheme, an object's finalizer is always called exactly once. The only exception is if an object is resurrected: the finalizer will be called again later. For CI objects, the order in which finalizers are called (step 2 above) is undefined. Safety ------ It is important to explain why the proposed change is safe. There are two aspects to be discussed: * Can a finalizer access zombie objects (including the object being finalized)? * What happens if a finalizer mutates the object graph so as to impact the CI? Let's discuss the first issue. We will divide possible cases in two categories: * If the object being finalized is part of the CI: by construction, no objects in CI are zombies yet, since CI finalizers are called before any reference breaking is done. Therefore, the finalizer cannot access zombie objects, which don't exist. * If the object being finalized is not part of the CI/CT: by definition, objects in the CI/CT don't have any references pointing to them from outside the CI/CT. Therefore, the finalizer cannot reach any zombie object (that is, even if the object being finalized was itself referenced from a zombie object). Now for the second issue. There are three potential cases: * The finalizer clears an existing reference to a CI object. The CI object may be disposed of before the GC tries to break it, which is fine (the GC simply has to be aware of this possibility). * The finalizer creates a new reference to a CI object. This can only happen from a CI object's finalizer (see above why). Therefore, the new reference will be detected by the GC after all CI finalizers are called (step 3 above), and collection will be aborted without any objects being broken. * The finalizer clears or creates a reference to a non-CI object. By construction, this is not a problem. Implementation ============== An implementation is available in branch ``finalize`` of the repository at http://hg.python.org/features/finalize/. Validation ========== Besides running the normal Python test suite, the implementation adds test cases for various finalization possibilities including reference cycles, object resurrection and legacy ``tp_del`` slots. The implementation has also been checked to not produce any regressions on the following test suites: * `Tulip <http://code.google.com/p/tulip/>`_, which makes an extensive use of generators * `Tornado <http://www.tornadoweb.org>`_ * `SQLAlchemy <http://www.sqlalchemy.org/>`_ * `Django <https://www.djangoproject.com/>`_ * `zope.interface <http://pypi.python.org/pypi/zope.interface>`_ References ========== Notes about reference cycle collection and weak reference callbacks: http://hg.python.org/cpython/file/4e687d53b645/Modules/gc_weakref.txt Generator memory leak: http://bugs.python.org/issue17468 Allow objects to decide if they can be collected by GC: http://bugs.python.org/issue9141 Module shutdown procedure based on GC http://bugs.python.org/issue812369 Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

8 21

Problem building Python 2.7.5 with separate sysroot
by Paul Smith 02 Jun '13

02 Jun '13

Hi all. I'm trying to build Python 2.7.5 on a GNU/Linux (Linux Mint 14) system, but using a different sysroot (that is, a separate <d>/usr/include, <d>/usr/lib, etc., not the real one for my system). I have shell script wrappers around GCC and its various tools that invoke it with the right paths to force this to happen, and when I call Python's configure I send along "CC=sysroot-gcc", etc. for all the various tools. Note that it's not really a cross-compilation because the target is also a GNU/Linux system on the same hardware architecture. The majority of Python builds just fine like this. However, I'm having serious problems building modules such as fcntl, etc. Looking at the output from the makefile, I can see that somehow, someone is forcibly adding "-I/usr/include/x86_64-linux-gnu" to the link line: building 'termios' extension sysroot-gcc -pthread -fPIC -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I/usr/include/x86_64-linux-gnu -I/common/sysroot/tools/usr/include -I/common/sysroot/tools/usr/include-fixed -I/usr/local/include -I/home/workspaces/psmith/python/obj/src/Python-2.7.5/Include -I/home/workspaces/psmith/python/obj/bld/python -c /home/workspaces/psmith/python/obj/src/Python-2.7.5/Modules/termios.c -o build/temp.linux-x86_64-2.7/home/workspaces/psmith/python/obj/src/Python-2.7.5/Modules/termios.o This fails miserably because the headers in /usr/include/x86_64-linux-gnu do not play at all nicely with my other sysroot headers. Ditto for other extensions like fcntl, etc. I've searched high and low in the Python source, generated makefiles, config.log, etc. and I cannot find where this -I flag is coming from anywhere. I found the --oldincludedir flag to configure and set it to point into my sysroot as well, but that didn't help: the /usr/include still appears when building these extensions. Can anyone tell me where Python is getting these -I flags and what I need to do to tell it to NOT use those flags when building extensions? I'd also like to remove the -I/usr/local/include, although this is not actually causing me problems right now.

3 4