Mailman 3 January 2012 - Python-Dev

Status of the fix for the hash collision vulnerability
by Victor Stinner Jan. 18, 2012

Jan. 18, 2012

Many people proposed their own idea to fix the vulnerability, but only 3 wrote a patch: - Glenn Linderman proposes to fix the vulnerability by adding a new "safe" dict type (only accepting string keys). His proof-of-concept (SafeDict.py) uses a secret of 64 random bits and uses it to compute the hash of a key. - Marc Andre Lemburg proposes to fix the vulnerability directly in dict (for any key type). The patch raises an exception if a lookup causes more than 1000 collisions. - I propose to fix … [View More]the vulnerability only in the Unicode hash (not for other types). My patch adds a random secret initialized at startup (it can be disabled or fixed using an environment variable). -- I consider that Glenn's proposition is not applicable in practice because all applications and all libraries have to be patched to use the new "safe" dict type. Some people are concerned by possible regression introduced by Marc's proposition: his patch may raise an exception for legitimate data. My proposition tries to be "just enough" secure with a low (runtime performance) overhead. My patch becomes huge (and so backporting is more complex), whereas Marc's patch is very simple and so trivial to backport. -- It is still unclear to me if the fix should be enabled by default for Python < 3.3. Because the overhead (of my patch) is low, I would prefer to enable the fix by default, to protect everyone with a simple Python upgrade. I prefer to explain how to disable explicitly the randomized hash (PYTHONHASHSEED=0) (or how to fix application bugs) to people having troubles with randomized hash, instead of leaving the hole open by default. -- We might change hash() for types other than str, but it looks like web servers are only concerned by dict with string keys. We may use Paul's hash function if mine is not enough secure. My patch doesn't fix the DoS, it just make the attack more complex. The attacker cannot pregenerate data for an attack: (s)he has first to compute the hash secret, and then compute hash collisions using the secret. The hash secret is a least 64 bits long (128 bits on a 64 bit system). So I hope that computing collisions requires a lot of CPU time (is slow) to make the attack ineffective with today computers. -- I plan to write a nice patch for Python 3.3, then write a simpler patch for 3.1 and 3.2 (duplicate os.urandom code to keep it unchanged, maybe don't create a new random.c file, maybe don't touch the test suite while the patch breaks many tests), and finally write patches for Python 2.6 and 2.7. Details about my patch: - I tested it on Linux (32 and 64 bits) and Windows (Seven 64 bits) - a new PYTHONSEED environment variable allow to control the randomized hash: PYTHONSEED=0 disables completly the randomized hash (restore the previous behaviour), PYTHONSEED=value uses a fixed seed for processes sharing data and needind same hash values (multiprocessing users?) - no overhead on hash(str) - no startup overhead on Linux - startup overhead is 10% on Windows (see the issue, I propose another solution with a startup overhead of 1%) The patch is not done, some tests are still failing because of the randomized hash. -- FYI, PHP released a version 5.3.9 adding "max_input_vars directive to prevent attacks based on hash collisions (CVE-2011-4885)". Victor [View Less]

26 66

Re: [Python-Dev] [Python-checkins] cpython: Refactored logging rotating handlers for improved flexibility.
by Éric Araujo Jan. 17, 2012

Jan. 17, 2012

Hi, > changeset: 57295c4d81ac > user: Vinay Sajip <vinay_sajip(a)yahoo.co.uk> > date: Wed Jan 04 12:02:26 2012 +0000 > summary: > Refactored logging rotating handlers for improved flexibility. > diff --git a/Doc/howto/logging-cookbook.rst b/Doc/howto/logging-cookbook.rst > --- a/Doc/howto/logging-cookbook.rst > +++ b/Doc/howto/logging-cookbook.rst > [snip] > +These are not “true” .gz files, as they are bare compressed data, with no > +“… [View More]

1 0

Backwards incompatible sys.stdout.write() behavior in Python 3 (Was: [Python-ideas] Pythonic buffering in Py3 print())
by anatoly techtonik Jan. 17, 2012

Jan. 17, 2012

Posting to python-dev as it is no more relates to the idea of improving print(). sys.stdout.write() in Python 3 causes backwards incompatible behavior that breaks recipe for unbuffered character reading from stdin on Linux - http://code.activestate.com/recipes/134892/ At first I though that the problem is in the new print() function, but it appeared that the culprit is sys.stdout.write() Attached is a test script which is a stripped down version of the recipe above. If executed with Python … [View More]

6 7

Python as a Metro-style App
by "Martin v. Löwis" Jan. 17, 2012

Jan. 17, 2012

I just tried porting Python as a Metro (Windows 8) App, and failed. Metro Apps use a variant of the Windows API called WinRT that still allows to write native applications in C++, but restricts various APIs to a subset of the full Win32 functionality. For example, everything related to subprocess creation would not work; none of the byte-oriented file API seems to be present, and a number of file operation functions are absent as well (such as MoveFile). Regardless, porting Python ought to be … [View More]

13 27

Script(s) for building Python on Windows
by Jason R. Coombs Jan. 17, 2012

Jan. 17, 2012

The current scripts for building Python lack some things to be desired. The first thing I notice when I try to build Python on Windows is the scripts expect to be run inside of a Visual Studio environment, the environment of which is only defined inside of a cmd.exe context. This means the scripts can't be executed from within Powershell (my preferred shell on Windows). One must first shell out to cmd.exe, which disables any Powershell-specific features the developer might have installed (… [View More]aliases, functions, etc). The second thing I notice is the scripts assume Visual Studio 2008. And while I recognize that Python is specifically built against Visual Studio 2008 for the official releases and that Visual Studio 2008 may be the only officially-supported build environment, later releases, such as Visual Studio 2010 are also adequate for testing purposes. I've been developing Python against Visual Studio 2010 for quite a while and it seems to be more than adequate. And while it's not the responsibility of the scripts to accommodate such environments, if the scripts could allow for such environments, that would be nice. Furthermore, having scripts that codify the process to upgrade will facilitate the migration should someone make the decision to officially upgrade to Visual Studio 2010. The third thing that I notice is that the command-line argument handling by the batch scripts is clumsy (compared to argparse, for example). This clumsiness is not a criticism of the authors, who have done well with the tools they had. However, batch programming is probably one of the least powerful ways to automate builds these days. So to ease my experience, I've developed my own library of functions and commands to facilitate building Python that aren't subject to the above limitations. Of course, I built these in Python, so they do require Python to build Python (not a huge burden, but worth mentioning). All of these modules are open-source and part of the jaraco.develop package <http://pypi.python.org/pypi/jaraco.develop> . The first of these modules is jaraco.develop.vstudio <https://bitbucket.org/jaraco/jaraco.develop/src/b7263c9d9c93/jaraco/develop /vstudio.py> . It exposes a class for locating Visual Studio in the usual locations, loading the environment for that instance of Visual Studio, and upgrading a project or solution file to that version. This class in particular enables running Visual Studio commands (including msbuild) from within a Visual Studio environment without actually requiring a cmd.exe context with that environment. Another module is jaraco.develop.python <https://bitbucket.org/jaraco/jaraco.develop/src/b7263c9d9c93/jaraco/develop /python.py> , which includes build_python, a function (and command) to build Python using whatever version of Visual Studio can be found (9 or 10 required). It has no environmental requirements except that Visual Studio be installed. Simply run build-python (part of jaraco.develop's console scripts) and it will build PCbuild.sln from the current directory to whatever targets are specified (or all of them if none are specified). The builder currently makes some assumptions (such as always building the 64-bit Release targets), but those could easily be customized using argparse parameters. This package and these modules have been tested and run on Python 2.7+. These tools solve the three shortcomings I mentioned above and make the development process so much smoother, IMO. If these modules were built into the repository, building Python could be as simple as "hg clone; cd cpython/pcbuild; ./build.py" (assuming only Visual Studio and Python available). I'd like to propose migrating this functionality (mainly these two modules) into the CPython heads for Python 2.7, 3.1, 3.2, and default as PCbuild/build.py (or similar). This functionality doesn't necessarily need to supersede the existing scripts (env, build_env, build), though it certainly could (and would as far as my usage is concerned). If there are no objections, I'll work to extract the aforementioned functionality from the jaraco.develop modules and into a portable script and put together a proof-of-concept in the default branch. The build script should not interfere with any build bots or other existing build processes, but should enable another more powerful technique for producing builds. I look forward to your comments and feedback. Regards, Jason [View Less]

3 7

Sphinx version for Python 2.x docs
by Chris Withers Jan. 17, 2012

Jan. 17, 2012

Hi All, Any chance the version of sphinx used to generate the docs on docs.python.org could be updated? I'd love to take advantage of the "new format" intersphinx mapping: http://sphinx.pocoo.org/ext/intersphinx.html#confval-intersphinx_mapping ...but since it looks like docs.python.org uses a version of sphinx that's too old for that, I can't like to: :ref:`Foo <python:logrecord-attributes>` ...and have to link to: `LogRecord attributes <http://docs.python.org/library/… [View More]

7 12

Re: [Python-Dev] [Python-checkins] cpython: provide a common method to check for RETR_DATA validity, first checking the
by Éric Araujo Jan. 17, 2012

Jan. 17, 2012

Hi Giampaolo, > changeset: 53a5a5b8859d > user: Giampaolo Rodola' <g.rodola(a)gmail.com> > date: Mon Jan 09 17:10:10 2012 +0100 > summary: > provide a common method to check for RETR_DATA validity, first > checking the expected len and then the actual data content; this > way we get a failure on len mismatch rather than content mismatch > (which is very long and unreadable) My trick is to convert long strings to lists (with data.split(… [View More]

1 0

Re: [Python-Dev] [Python-checkins] peps: Bring the Python 3.3 feature list up to date.
by Brett Cannon Jan. 17, 2012

Jan. 17, 2012

Is the change to the pyc format big enough news to go into the release PEP? Or should that just be a "What's New" topic? On Fri, Jan 13, 2012 at 15:18, georg.brandl <python-checkins(a)python.org>wrote: > http://hg.python.org/peps/rev/ea3ffa3611e5 > changeset: 4012:ea3ffa3611e5 > user: Georg Brandl <georg(a)python.org> > date: Fri Jan 13 21:18:11 2012 +0100 > summary: > Bring the Python 3.3 feature list up to date. > > files: > pep-0398.… [View More]

3 2

Re: [Python-Dev] Python-Dev Digest, Vol 102, Issue 35
by python Jan. 16, 2012

Jan. 16, 2012

jbk python-dev-request(a)python.xn--org:-7z8fi32v >Send Python-Dev mailing list submissions to > python-dev(a)python.org > >To subscribe or unsubscribe via the World Wide Web, visit > http://mail.python.org/mailman/listinfo/python-dev >or, via email, send a message with subject or body 'help' to > python-dev-request(a)python.org > >You can reach the person managing the list at > python-dev-owner(a)python.org > >When replying, please edit your Subject line … [View More]so it is more specific >than "Re: Contents of Python-Dev digest..." > > >Today's Topics: > > 1. Re: Status of the fix for the hash collision vulnerability > (Gregory P. Smith) > 2. Re: Status of the fix for the hash collision vulnerability > (Barry Warsaw) > 3. Re: Sphinx version for Python 2.x docs (?ric Araujo) > 4. Re: Status of the fix for the hash collision vulnerability > (martin(a)v.loewis.de) > 5. Re: Status of the fix for the hash collision vulnerability > (Guido van Rossum) > 6. Re: [Python-checkins] cpython: add test, which was missing > from d64ac9ab4cd0 (Nick Coghlan) > 7. Re: Status of the fix for the hash collision vulnerability > (Terry Reedy) > 8. Re: Status of the fix for the hash collision vulnerability > (Jack Diederich) > 9. Re: cpython: Implement PEP 380 - 'yield from' (closes #11682) > (Nick Coghlan) > 10. Re: Status of the fix for the hash collision vulnerability > (Nick Coghlan) > > >---------------------------------------------------------------------- > >Message: 1 >Date: Fri, 13 Jan 2012 19:06:00 -0800 >From: "Gregory P. Smith" <greg(a)krypto.org> >Cc: python-dev(a)python.org >Subject: Re: [Python-Dev] Status of the fix for the hash collision > vulnerability >Message-ID: > <CAGE7PNKkHW-_WqiuQC9bhqxnoU77f+eprs_q3nqmycstM3JZag(a)mail.gmail.com> >Content-Type: text/plain; charset="iso-8859-1" > >On Fri, Jan 13, 2012 at 5:58 PM, Gregory P. Smith <greg(a)krypto.org> wrote: > >> >> On Fri, Jan 13, 2012 at 5:38 PM, Guido van Rossum <guido(a)python.org>wrote: >> >>> On Fri, Jan 13, 2012 at 5:17 PM, Antoine Pitrou <solipsis(a)pitrou.net>wrote: >>> >>>> On Thu, 12 Jan 2012 18:57:42 -0800 >>>> Guido van Rossum <guido(a)python.org> wrote: >>>> > Hm... I started out as a big fan of the randomized hash, but thinking >>>> more >>>> > about it, I actually believe that the chances of some legitimate app >>>> having >>>> > >1000 collisions are way smaller than the chances that somebody's code >>>> will >>>> > break due to the variable hashing. >>>> >>>> Breaking due to variable hashing is deterministic: you notice it as >>>> soon as you upgrade (and then you use PYTHONHASHSEED to disable >>>> variable hashing). That seems better than unpredictable breaking when >>>> some legitimate collision chain happens. >>> >>> >>> Fair enough. But I'm now uncomfortable with turning this on for bugfix >>> releases. I'm fine with making this the default in 3.3, just not in 3.2, >>> 3.1 or 2.x -- it will break too much code and organizations will have to >>> roll back the release or do extensive testing before installing a bugfix >>> release -- exactly what we *don't* want for those. >>> >>> FWIW, I don't believe in the SafeDict solution -- you never know which >>> dicts you have to change. >>> >>> >> Agreed. >> >> Of the three options Victor listed only one is good. >> >> I don't like *SafeDict*. *-1*. It puts the onerous on the coder to >> always get everything right with regards to data that came from outside the >> process never ending up hashed in a non-safe dict or set *anywhere*. >> "Safe" needs to be the default option for all hash tables. >> >> I don't like the "*too many hash collisions*" exception. *-1*. It >> provides non-deterministic application behavior for data driven >> applications with no way for them to predict when it'll happen or where and >> prepare for it. It may work in practice for many applications but is simply >> odd behavior. >> >> I do like *randomly seeding the hash*. *+1*. This is easy. It can easily >> be back ported to any Python version. >> >> It is perfectly okay to break existing users who had anything depending on >> ordering of internal hash tables. Their code was already broken. We *will*provide a flag and/or environment variable that can be set to turn the >> feature off at their own peril which they can use in their test harnesses >> that are stupid enough to use doctests with order dependencies. >> > >What an implementation looks like: > > http://pastebin.com/9ydETTag > >some stuff to be filled in, but this is all that is really required. add >logic to allow a particular seed to be specified or forced to 0 from the >command line or environment. add the logic to grab random bytes. add the >autoconf glue to disable it. done. > >-gps > > >> This approach worked fine for Perl 9 years ago. >> https://rt.perl.org/rt3//Public/Bug/Display.html?id=22371 >> >> -gps >> > [View Less]

2 1

Dinsdale is no more
by Łukasz Langa Jan. 15, 2012

Jan. 15, 2012

Gentlemen, www.python.org is down at the moment. -- Best regards, Łukasz Langa Senior Systems Architecture Engineer IT Infrastructure Department Grupa Allegro Sp. z o.o.

2 1