From g.brandl at gmx.net  Mon Jul  1 08:33:38 2013
From: g.brandl at gmx.net (Georg Brandl)
Date: Mon, 01 Jul 2013 08:33:38 +0200
Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug"
In-Reply-To: <CAMpsgwYST2OSOyKjowQaNjUwDpESHzstkc7LqqOYLRYh=6uabw@mail.gmail.com>
References: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>
	<CAPZV6o9vvjDVOzTzB0CgCECW9gEgmnSBa9_-XxJ-FO4guuDzNg@mail.gmail.com>
	<CAPOVWOTj8TBGsDvoium4cBXjerY5gkrPA3jOvVf9d8zB=9aFag@mail.gmail.com>
	<51CA4DB8.8060802@trueblade.com>
	<CAMpsgwYST2OSOyKjowQaNjUwDpESHzstkc7LqqOYLRYh=6uabw@mail.gmail.com>
Message-ID: <kqr7qn$3h3$1@ger.gmane.org>

Am 26.06.2013 16:24, schrieb Victor Stinner:
> 2013/6/26 Eric V. Smith <eric at trueblade.com>:
>> I think that's exactly what's happening.
>>
>> From the bug report:
>>
>>         find $(srcdir) '(' -name '*.fdc' -o -name '*~' \
>>                            -o -name '[@,#]*' -o -name '*.old' \
>>                            -o -name '*.orig' -o -name '*.rej' \
>>                            -o -name '*.bak' ')' \
>>                            -exec rm -f {} ';'
>>
>> Will find files beginning with '@' inside subdirectories of $(srcdir)/.hg.
> 
> In my opinion, make distclean should only remove files generated by
> configure and a build. It should not remove random files.
> 
> *~, .orig, .rej, .back should be kept. They are not generated by
> configure nor make.

I think you want "make clean" then.

Georg



From rdmurray at bitdance.com  Mon Jul  1 13:19:12 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Mon, 01 Jul 2013 07:19:12 -0400
Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug"
In-Reply-To: <kqr7qn$3h3$1@ger.gmane.org>
References: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>
	<CAPZV6o9vvjDVOzTzB0CgCECW9gEgmnSBa9_-XxJ-FO4guuDzNg@mail.gmail.com>
	<CAPOVWOTj8TBGsDvoium4cBXjerY5gkrPA3jOvVf9d8zB=9aFag@mail.gmail.com>
	<51CA4DB8.8060802@trueblade.com>
	<CAMpsgwYST2OSOyKjowQaNjUwDpESHzstkc7LqqOYLRYh=6uabw@mail.gmail.com>
	<kqr7qn$3h3$1@ger.gmane.org>
Message-ID: <20130701111913.5CA932505F1@webabinitio.net>

On Mon, 01 Jul 2013 08:33:38 +0200, Georg Brandl <g.brandl at gmx.net> wrote:
> Am 26.06.2013 16:24, schrieb Victor Stinner:
> > 2013/6/26 Eric V. Smith <eric at trueblade.com>:
> >> I think that's exactly what's happening.
> >>
> >> From the bug report:
> >>
> >>         find $(srcdir) '(' -name '*.fdc' -o -name '*~' \
> >>                            -o -name '[@,#]*' -o -name '*.old' \
> >>                            -o -name '*.orig' -o -name '*.rej' \
> >>                            -o -name '*.bak' ')' \
> >>                            -exec rm -f {} ';'
> >>
> >> Will find files beginning with '@' inside subdirectories of $(srcdir)/.hg.
> > 
> > In my opinion, make distclean should only remove files generated by
> > configure and a build. It should not remove random files.
> > 
> > *~, .orig, .rej, .back should be kept. They are not generated by
> > configure nor make.
> 
> I think you want "make clean" then.

That's a good point.  If the find were dropped, the target would have to
be renamed.  "make configureclean", maybe.  But I think it is easier and
less confusing just to leave things as they will be after Eric applies
the fix proposed in http://bugs.python.org/issue18312.

--David

From greg at krypto.org  Mon Jul  1 22:54:08 2013
From: greg at krypto.org (Gregory P. Smith)
Date: Mon, 1 Jul 2013 13:54:08 -0700
Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug"
In-Reply-To: <20130701111913.5CA932505F1@webabinitio.net>
References: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>
	<CAPZV6o9vvjDVOzTzB0CgCECW9gEgmnSBa9_-XxJ-FO4guuDzNg@mail.gmail.com>
	<CAPOVWOTj8TBGsDvoium4cBXjerY5gkrPA3jOvVf9d8zB=9aFag@mail.gmail.com>
	<51CA4DB8.8060802@trueblade.com>
	<CAMpsgwYST2OSOyKjowQaNjUwDpESHzstkc7LqqOYLRYh=6uabw@mail.gmail.com>
	<kqr7qn$3h3$1@ger.gmane.org>
	<20130701111913.5CA932505F1@webabinitio.net>
Message-ID: <CAGE7PNJ0spOCMY8p34HvMSjMRiLx76XwwfkXSmwGgngTNx6Vww@mail.gmail.com>

If we disallowed builds *from in source tree* requiring all output to go
into a separate build output directory instead (like any sane person does*)
we wouldn't need a crazy find in the source tree to mess things up. ;)

this can be done today:

$ mkdir foo && cd foo && ../my-hg/2.7/configure --srcdir=../my-hg/2.7 &&
make -j12

I think all we'd need to do is disallow the cwd when configuring or
building from being within srcdir.

-gps

* note: the author is normally too lazy to be sane because it involves
slightly more typing.


On Mon, Jul 1, 2013 at 4:19 AM, R. David Murray <rdmurray at bitdance.com>wrote:

> On Mon, 01 Jul 2013 08:33:38 +0200, Georg Brandl <g.brandl at gmx.net> wrote:
> > Am 26.06.2013 16:24, schrieb Victor Stinner:
> > > 2013/6/26 Eric V. Smith <eric at trueblade.com>:
> > >> I think that's exactly what's happening.
> > >>
> > >> From the bug report:
> > >>
> > >>         find $(srcdir) '(' -name '*.fdc' -o -name '*~' \
> > >>                            -o -name '[@,#]*' -o -name '*.old' \
> > >>                            -o -name '*.orig' -o -name '*.rej' \
> > >>                            -o -name '*.bak' ')' \
> > >>                            -exec rm -f {} ';'
> > >>
> > >> Will find files beginning with '@' inside subdirectories of
> $(srcdir)/.hg.
> > >
> > > In my opinion, make distclean should only remove files generated by
> > > configure and a build. It should not remove random files.
> > >
> > > *~, .orig, .rej, .back should be kept. They are not generated by
> > > configure nor make.
> >
> > I think you want "make clean" then.
>
> That's a good point.  If the find were dropped, the target would have to
> be renamed.  "make configureclean", maybe.  But I think it is easier and
> less confusing just to leave things as they will be after Eric applies
> the fix proposed in http://bugs.python.org/issue18312.
>
> --David
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130701/b1a8b232/attachment.html>

From victor.stinner at gmail.com  Mon Jul  1 23:05:56 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 1 Jul 2013 23:05:56 +0200
Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug"
In-Reply-To: <kqr7qn$3h3$1@ger.gmane.org>
References: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>
	<CAPZV6o9vvjDVOzTzB0CgCECW9gEgmnSBa9_-XxJ-FO4guuDzNg@mail.gmail.com>
	<CAPOVWOTj8TBGsDvoium4cBXjerY5gkrPA3jOvVf9d8zB=9aFag@mail.gmail.com>
	<51CA4DB8.8060802@trueblade.com>
	<CAMpsgwYST2OSOyKjowQaNjUwDpESHzstkc7LqqOYLRYh=6uabw@mail.gmail.com>
	<kqr7qn$3h3$1@ger.gmane.org>
Message-ID: <CAMpsgwaBq3YJfZjM5Q+zM7UruWX4fYvv9Hhu2ubhLYHntmZxbg@mail.gmail.com>

Hi Georg,

2013/7/1 Georg Brandl <g.brandl at gmx.net>:
> Am 26.06.2013 16:24, schrieb Victor Stinner:
>> 2013/6/26 Eric V. Smith <eric at trueblade.com>:
>>> I think that's exactly what's happening.
>>>
>>> From the bug report:
>>>
>>>         find $(srcdir) '(' -name '*.fdc' -o -name '*~' \
>>>                            -o -name '[@,#]*' -o -name '*.old' \
>>>                            -o -name '*.orig' -o -name '*.rej' \
>>>                            -o -name '*.bak' ')' \
>>>                            -exec rm -f {} ';'
>>>
>>> Will find files beginning with '@' inside subdirectories of $(srcdir)/.hg.
>>
>> In my opinion, make distclean should only remove files generated by
>> configure and a build. It should not remove random files.
>>
>> *~, .orig, .rej, .back should be kept. They are not generated by
>> configure nor make.
>
> I think you want "make clean" then.

I don't understand why you are suggesting me to use "make clean". I
would like to start a fresh build, so remove configure and Makefile,
but I also want to keep my local changes and local files not tracked
by Mercurial. I need this when the build does not work because a new
file was added or a build script was modified. "make clean" does not
change anything for this use case.

For example, I don't understand why "make distclean" removes the
"tags" file. Generating this file takes 20 to 30 seconds on my slow
laptop, and it is not generated by Python build system, but by the
external ctags program.

Don't you think that we need two different "distclean" commands? One
GNU-style "distclean" which only removes configure and Makefile, and
another "distclean" which is the GNU "distclean" + the extra find
removing temporary files.

Victor

From rdmurray at bitdance.com  Mon Jul  1 23:13:28 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Mon, 01 Jul 2013 17:13:28 -0400
Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug"
In-Reply-To: <CAMpsgwaBq3YJfZjM5Q+zM7UruWX4fYvv9Hhu2ubhLYHntmZxbg@mail.gmail.com>
References: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>
	<CAPZV6o9vvjDVOzTzB0CgCECW9gEgmnSBa9_-XxJ-FO4guuDzNg@mail.gmail.com>
	<CAPOVWOTj8TBGsDvoium4cBXjerY5gkrPA3jOvVf9d8zB=9aFag@mail.gmail.com>
	<51CA4DB8.8060802@trueblade.com>
	<CAMpsgwYST2OSOyKjowQaNjUwDpESHzstkc7LqqOYLRYh=6uabw@mail.gmail.com>
	<kqr7qn$3h3$1@ger.gmane.org>
	<CAMpsgwaBq3YJfZjM5Q+zM7UruWX4fYvv9Hhu2ubhLYHntmZxbg@mail.gmail.com>
Message-ID: <20130701211328.ECF1D2500E3@webabinitio.net>

On Mon, 01 Jul 2013 23:05:56 +0200, Victor Stinner <victor.stinner at gmail.com> wrote:
> Hi Georg,
> 
> 2013/7/1 Georg Brandl <g.brandl at gmx.net>:
> > Am 26.06.2013 16:24, schrieb Victor Stinner:
> >> 2013/6/26 Eric V. Smith <eric at trueblade.com>:
> >>> I think that's exactly what's happening.
> >>>
> >>> From the bug report:
> >>>
> >>>         find $(srcdir) '(' -name '*.fdc' -o -name '*~' \
> >>>                            -o -name '[@,#]*' -o -name '*.old' \
> >>>                            -o -name '*.orig' -o -name '*.rej' \
> >>>                            -o -name '*.bak' ')' \
> >>>                            -exec rm -f {} ';'
> >>>
> >>> Will find files beginning with '@' inside subdirectories of $(srcdir)/.hg.
> >>
> >> In my opinion, make distclean should only remove files generated by
> >> configure and a build. It should not remove random files.
> >>
> >> *~, .orig, .rej, .back should be kept. They are not generated by
> >> configure nor make.
> >
> > I think you want "make clean" then.
> 
> I don't understand why you are suggesting me to use "make clean". I
> would like to start a fresh build, so remove configure and Makefile,
> but I also want to keep my local changes and local files not tracked
> by Mercurial. I need this when the build does not work because a new
> file was added or a build script was modified. "make clean" does not
> change anything for this use case.
> 
> For example, I don't understand why "make distclean" removes the
> "tags" file. Generating this file takes 20 to 30 seconds on my slow
> laptop, and it is not generated by Python build system, but by the
> external ctags program.
> 
> Don't you think that we need two different "distclean" commands? One
> GNU-style "distclean" which only removes configure and Makefile, and
> another "distclean" which is the GNU "distclean" + the extra find
> removing temporary files.

The command that does not remove the extra files is *not* a 'distclean'
command.  'buildclean' or 'configclean', but not 'distclean'.

distclean still needs to be fixed, so please open a new issue for
adding buildclean or whatever you want to call it, as Eric requested
in the existing issue.

--David

From eric at trueblade.com  Mon Jul  1 23:19:55 2013
From: eric at trueblade.com (Eric V. Smith)
Date: Mon, 01 Jul 2013 17:19:55 -0400
Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug"
In-Reply-To: <20130701211328.ECF1D2500E3@webabinitio.net>
References: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>
	<CAPZV6o9vvjDVOzTzB0CgCECW9gEgmnSBa9_-XxJ-FO4guuDzNg@mail.gmail.com>
	<CAPOVWOTj8TBGsDvoium4cBXjerY5gkrPA3jOvVf9d8zB=9aFag@mail.gmail.com>
	<51CA4DB8.8060802@trueblade.com>
	<CAMpsgwYST2OSOyKjowQaNjUwDpESHzstkc7LqqOYLRYh=6uabw@mail.gmail.com>
	<kqr7qn$3h3$1@ger.gmane.org>
	<CAMpsgwaBq3YJfZjM5Q+zM7UruWX4fYvv9Hhu2ubhLYHntmZxbg@mail.gmail.com>
	<20130701211328.ECF1D2500E3@webabinitio.net>
Message-ID: <51D1F27B.6070001@trueblade.com>

On 7/1/2013 5:13 PM, R. David Murray wrote:
> distclean still needs to be fixed, so please open a new issue for
> adding buildclean or whatever you want to call it, as Eric requested
> in the existing issue.

I finally got around to testing it today, so I'm going to check in my
change to 18312 sometime shortly.

-- 
Eric.

From g.brandl at gmx.net  Tue Jul  2 00:06:37 2013
From: g.brandl at gmx.net (Georg Brandl)
Date: Tue, 02 Jul 2013 00:06:37 +0200
Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug"
In-Reply-To: <CAMpsgwaBq3YJfZjM5Q+zM7UruWX4fYvv9Hhu2ubhLYHntmZxbg@mail.gmail.com>
References: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>
	<CAPZV6o9vvjDVOzTzB0CgCECW9gEgmnSBa9_-XxJ-FO4guuDzNg@mail.gmail.com>
	<CAPOVWOTj8TBGsDvoium4cBXjerY5gkrPA3jOvVf9d8zB=9aFag@mail.gmail.com>
	<51CA4DB8.8060802@trueblade.com>
	<CAMpsgwYST2OSOyKjowQaNjUwDpESHzstkc7LqqOYLRYh=6uabw@mail.gmail.com>
	<kqr7qn$3h3$1@ger.gmane.org>
	<CAMpsgwaBq3YJfZjM5Q+zM7UruWX4fYvv9Hhu2ubhLYHntmZxbg@mail.gmail.com>
Message-ID: <kqsug2$4vl$1@ger.gmane.org>

Am 01.07.2013 23:05, schrieb Victor Stinner:
> Hi Georg,
> 
> 2013/7/1 Georg Brandl <g.brandl at gmx.net>:
>> Am 26.06.2013 16:24, schrieb Victor Stinner:
>>> 2013/6/26 Eric V. Smith <eric at trueblade.com>:
>>>> I think that's exactly what's happening.
>>>>
>>>> From the bug report:
>>>>
>>>>         find $(srcdir) '(' -name '*.fdc' -o -name '*~' \
>>>>                            -o -name '[@,#]*' -o -name '*.old' \
>>>>                            -o -name '*.orig' -o -name '*.rej' \
>>>>                            -o -name '*.bak' ')' \
>>>>                            -exec rm -f {} ';'
>>>>
>>>> Will find files beginning with '@' inside subdirectories of $(srcdir)/.hg.
>>>
>>> In my opinion, make distclean should only remove files generated by
>>> configure and a build. It should not remove random files.
>>>
>>> *~, .orig, .rej, .back should be kept. They are not generated by
>>> configure nor make.
>>
>> I think you want "make clean" then.
> 
> I don't understand why you are suggesting me to use "make clean". I
> would like to start a fresh build, so remove configure and Makefile,
> but I also want to keep my local changes and local files not tracked
> by Mercurial. I need this when the build does not work because a new
> file was added or a build script was modified. "make clean" does not
> change anything for this use case.

Right, I had wrongly remembered that "clean" also removed the Makefile.

Note that to add to the confusion, there's an additional target
named "clobber" (which is called by distclean).  Both "clobber" and
"distclean" remove Python generated and potentially user generated files.

IMO things could be rearranged without too much effort so that "distclean"
only removes Python-build-generated files, whereas "clobber" removes also
user generated files.  But please don't introduce yet another target.

Georg


From agriff at tin.it  Tue Jul  2 11:27:02 2013
From: agriff at tin.it (Andrea Griffini)
Date: Tue, 2 Jul 2013 11:27:02 +0200
Subject: [Python-Dev] Why are captured parameters also listed in co_varnames?
Message-ID: <CALy7ODvNuK0G=aWH_dDt90yOi6uJKo9Ks7wO6jJ-+x1d1-j=nw@mail.gmail.com>

I'm trying to understand how CPython implements closure variable capture
and there is one minor point I can't understand.

When a local is captured it gets allocated in co_cellvars and is accessed
with (LOAD|STORE)_DEREF, and this is clear.
However when a local is coming from a parameter it gets ALSO allocated in
co_varnames even if the local slot apparently is not accesible because
*_FAST opcodes are not generated.

Is there a technical reason for this? It happens in CPython 2, 3 and even
in PyPy...

Andrea Griffini
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130702/d86dd488/attachment.html>

From amauryfa at gmail.com  Tue Jul  2 11:45:27 2013
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Tue, 2 Jul 2013 11:45:27 +0200
Subject: [Python-Dev] Why are captured parameters also listed in
	co_varnames?
In-Reply-To: <CALy7ODvNuK0G=aWH_dDt90yOi6uJKo9Ks7wO6jJ-+x1d1-j=nw@mail.gmail.com>
References: <CALy7ODvNuK0G=aWH_dDt90yOi6uJKo9Ks7wO6jJ-+x1d1-j=nw@mail.gmail.com>
Message-ID: <CAGmFidYhuGcLjFLALAzF1ajmhUYOW4EPmc5DYPaafUsJE9coww@mail.gmail.com>

2013/7/2 Andrea Griffini <agriff at tin.it>

> I'm trying to understand how CPython implements closure variable capture
> and there is one minor point I can't understand.
>
> When a local is captured it gets allocated in co_cellvars and is accessed
> with (LOAD|STORE)_DEREF, and this is clear.
> However when a local is coming from a parameter it gets ALSO allocated in
> co_varnames even if the local slot apparently is not accesible because
> *_FAST opcodes are not generated.
>
> Is there a technical reason for this? It happens in CPython 2, 3 and even
> in PyPy...
>
>
co_varnames is also used in error messages, for example in the following
code:

>>> def f():
...     def g():
...         x
...     print x
...     x = 1
...     return g
...
>>> f()
UnboundLocalError: local variable 'x' referenced before assignment

This is also needed when x is a parameter of f(), for inspect.signature of
course,
but also because in python3 you can "del x".

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130702/8f93fe71/attachment.html>

From ronaldoussoren at mac.com  Tue Jul  2 13:51:13 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Tue, 2 Jul 2013 13:51:13 +0200
Subject: [Python-Dev] Hooking into super() attribute resolution
Message-ID: <DCEB2B77-B9DD-405A-A084-EFB761B947BB@mac.com>

Hi,

Below is a very preliminary draft PEP for adding a special method that can be used to hook into the attribute resolution process of the super object.

The primary usecase for using this special method are classes that perform custom logic in their __getattribute__ method, where the default behavior of super (peekin the the class __dict__) is not appropriate.  The primary reason I wrote this proposal is PyObjC: it dynamicly looks up methods in its __getattribute__ and caches the result in the class __dict__, because of this super() will often not work correctly and therefore I'm currently shipping a custom subclass of super() that basicly contains an in-line implementation of the hook that would be used by PyObjC.

I have a partial implementation of the hook system in issue 18181  and a PyObjC patch that uses it. The implementation currently does not contain tests, and I'm sure that I'll find edge cases that I haven't thought about yet when I add tests. 

Ronald




PEP: TODO
Title: Hooking into super attribute resolution
Version: $Revision$
Last-Modified: $Date$
Author: Ronald Oussoren <ronaldoussoren at mac.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 12-Jun-2013
Post-History: 2-Jul-2013


Abstract
========

In current python releases the attribute resolution of the `super class`_
peeks in the ``__dict__`` attribute of classes on the MRO to look
for attributes. This PEP introduces a hook that classes can use
to override that behavior for specific classes.


Rationale
=========

Peeking in the class ``__dict__`` works for regular classes, but can
cause problems when a class dynamicly looks up attributes in a
``__getattribute__`` method.

The new hook makes it possible to introduce the same customization for
attribute lookup through the `super class`_.


The superclass attribute lookup hook
====================================

In C code
---------

A new slot ``tp_getattro_super`` is added to the ``PyTypeObject`` struct. The
``tp_getattro`` slot for super will call this slot when it is not ``NULL``,
otherwise it will peek in the class ``tp_dict``.

The slot has the following prototype::

    PyObject* (*getattrosuperfunc)(PyTypeObject* tp, PyObject* self, PyObject* name);

The function should perform attribute lookup for *name*, but only looking in
type *tp* (which will be one of the types on the MRO for *self*) and without looking
in the instance *__dict__*.

The function returns ``NULL`` when the attribute cannot be found, and raises and
exception. Exception other than ``AttributeError`` will cause failure of super's
attribute resolution.


In Python code
--------------

A Python class can contain a definition for a method ``__getattribute_super__`` with
the following prototype::

   def __getattribute_super__(self, cls, name): pass

The method should perform attribute lookup for *name* on instance *self* while only
looking at *cls* (it should not look in super classes or the instance *__dict__*


Alternative proposals
---------------------

Reuse ``tp_getattro``
.....................

It would be nice to avoid adding a new slot, thus keeping the API simpler and easier
to understand.  A comment on `Issue 18181`_ asked about reusing the ``tp_getattro`` slot,
that is super could call the ``tp_getattro`` slot of all methods along the MRO.

AFAIK that won't work because ``tp_getattro`` will look in the instance ``__dict__`` before
it tries to resolve attributes using classes in the MRO. This would mean that using
``tp_getattro`` instead of peeking the class dictionaries changes the semantics of
the `super class`_.


Open Issues
===========

* The names of the new slot and magic method are far from settled.

* I'm not too happy with the prototype for the new hook.

* Should ``__getattribute_super__`` be a class method instead?


References
==========

* `Issue 18181`_ contains a prototype implementation

  The prototype uses different names than this proposal.


Copyright
=========

This document has been placed in the public domain.

.. _`Issue 18181`: http://bugs.python.org/issue18181

.. _`super class`: http://docs.python.org/3/library/functions.html?highlight=super#super

From eric at trueblade.com  Tue Jul  2 15:52:56 2013
From: eric at trueblade.com (Eric V. Smith)
Date: Tue, 02 Jul 2013 09:52:56 -0400
Subject: [Python-Dev] Issue 18312 "fix" broke buildbots
Message-ID: <51D2DB38.8090106@trueblade.com>

I'm not sure how my change broke the buildbots, but apparently it did. I
need to run to a meeting now, but I can roll this back in the next few
hours.

If someone else wants to roll it back before I get to it, feel free.

Sorry about the problem. I tested it locally, I'm not sure how the
buildbots are affected.

Eric.

From rdmurray at bitdance.com  Tue Jul  2 16:33:35 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Tue, 02 Jul 2013 10:33:35 -0400
Subject: [Python-Dev] Issue 18312 "fix" broke buildbots
In-Reply-To: <51D2DB38.8090106@trueblade.com>
References: <51D2DB38.8090106@trueblade.com>
Message-ID: <20130702143335.ACD702500E3@webabinitio.net>

On Tue, 02 Jul 2013 09:52:56 -0400, "Eric V. Smith" <eric at trueblade.com> wrote:
> I'm not sure how my change broke the buildbots, but apparently it did. I
> need to run to a meeting now, but I can roll this back in the next few
> hours.
> 
> If someone else wants to roll it back before I get to it, feel free.
> 
> Sorry about the problem. I tested it locally, I'm not sure how the
> buildbots are affected.

I'm no longer sure it was your patch.  Anyone who wants to look
at the buildbots please do:

http://buildbot.python.org/all/builders/x86%20Tiger%202.7/builds/2030
http://buildbot.python.org/all/builders/x86%20Tiger%203.3/builds/742
http://buildbot.python.org/all/builders/x86%20Tiger%203.x/builds/6522
http://buildbot.python.org/all/builders/AMD64%20FreeBSD%209.0%20dtrace%2Bclang%203.3/builds/699
http://buildbot.python.org/all/builders/x86%20Windows7%202.7/builds/2125
http://buildbot.python.org/all/builders/bolen-dmg-3.x/builds/278
http://buildbot.python.org/all/builders/x86%20XP-4%203.3/builds/832

I haven't looked at all of these.  The blamelist cites Richard Oudkerk,
but at least one of the logs I looked at was about @test files, in
a number of different tests.

I don't really have time to look at it firther right now either, I've
got a client with an issue that needs solved asap...

--David

From eric at trueblade.com  Tue Jul  2 16:57:11 2013
From: eric at trueblade.com (Eric V. Smith)
Date: Tue, 02 Jul 2013 10:57:11 -0400
Subject: [Python-Dev] Issue 18312 "fix" broke buildbots
In-Reply-To: <20130702143335.ACD702500E3@webabinitio.net>
References: <51D2DB38.8090106@trueblade.com>
	<20130702143335.ACD702500E3@webabinitio.net>
Message-ID: <51D2EA47.4030004@trueblade.com>

On 07/02/2013 10:33 AM, R. David Murray wrote:
> On Tue, 02 Jul 2013 09:52:56 -0400, "Eric V. Smith" <eric at trueblade.com> wrote:
>> I'm not sure how my change broke the buildbots, but apparently it did. I
>> need to run to a meeting now, but I can roll this back in the next few
>> hours.
>>
>> If someone else wants to roll it back before I get to it, feel free.
>>
>> Sorry about the problem. I tested it locally, I'm not sure how the
>> buildbots are affected.
> 
> I'm no longer sure it was your patch.  Anyone who wants to look
> at the buildbots please do:
> 
> http://buildbot.python.org/all/builders/x86%20Tiger%202.7/builds/2030
> http://buildbot.python.org/all/builders/x86%20Tiger%203.3/builds/742
> http://buildbot.python.org/all/builders/x86%20Tiger%203.x/builds/6522
> http://buildbot.python.org/all/builders/AMD64%20FreeBSD%209.0%20dtrace%2Bclang%203.3/builds/699
> http://buildbot.python.org/all/builders/x86%20Windows7%202.7/builds/2125
> http://buildbot.python.org/all/builders/bolen-dmg-3.x/builds/278
> http://buildbot.python.org/all/builders/x86%20XP-4%203.3/builds/832
> 
> I haven't looked at all of these.  The blamelist cites Richard Oudkerk,
> but at least one of the logs I looked at was about @test files, in
> a number of different tests.
> 
> I don't really have time to look at it firther right now either, I've
> got a client with an issue that needs solved asap...

I've looked at a few of these, and they don't seem related to my change.
I'll review it some more this afternoon.

Eric.



From pythoniks at gmail.com  Tue Jul  2 20:31:48 2013
From: pythoniks at gmail.com (Pascal Chambon)
Date: Tue, 02 Jul 2013 20:31:48 +0200
Subject: [Python-Dev] Solving the import-deadlock case
Message-ID: <51D31C94.3010203@gmail.com>

Hello everyone,

I'd like to bring your attention to this issue, since it touches the 
fundamentals of python's import workflow:
http://bugs.python.org/issue17716

/I've tried to post it on the python-import ML for weeks, but it must 
still be blocked somewhere in a moderation queue, so here I come ^^/

TLDR version: because of the way import current works, if importing a 
package "temporarily" fails whereas importing one of its children 
succeeded, we reach an unusable state, all subsequent attempts at 
importing that package will fail if a "from...import" is used somewhere. 
Typically, it makes a web worker broken, even though the typical 
behaviour of such process woudl be to retry loading, again and again, 
the failing view.

I agree that a module loading should be, as much as possible, "side 
effects free", and thus shouldn't have temporary errors. But well, in 
practice, module loading is typically the time where process-wide 
initialization are done (modifying sys.path, os.environ, instantiating 
connection or thread pools, registering atexit handler, starting 
maintenance threads...), so that case has chances to happen at a moment 
or another, especially if accesses to filesystem or network (SQL...) are 
done at module loading, due to the lack of initialization system at 
upper levels.

That's why I propose modifying the behaviour of module import, so that 
submodules are deleted as well when a parent module import fails. True, 
it means they will be reloaded as well when importing the parent will 
start again, but anyway we already have a "double execution" problem 
with the reloading of the parent module, so it shouldn't make a big 
difference.
The only other solution I'd see would be to SYSTEMATICALLY perform name 
(re)binding when processing a from...import statement, to recover from 
the previously failed initialization. Dunno if it's a good idea.

On a (separate but related) topic, to be safer on module reimports or 
reloadings, it could be interesting to add some "idempotency" to common 
initialization tasks ; for example the "atexit" registration system, 
wouldn't it be worth adding a boolean flag to explicitely skip 
registration if a callable with same fully qualified name is already 
registered.

Do you have opinions on these subjects ?

thanks,
regards,
Pascal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130702/1ecdb18e/attachment.html>

From rdmurray at bitdance.com  Tue Jul  2 21:41:46 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Tue, 02 Jul 2013 15:41:46 -0400
Subject: [Python-Dev] Solving the import-deadlock case
In-Reply-To: <51D31C94.3010203@gmail.com>
References: <51D31C94.3010203@gmail.com>
Message-ID: <20130702194146.780342500CB@webabinitio.net>

On Tue, 02 Jul 2013 20:31:48 +0200, Pascal Chambon <pythoniks at gmail.com> wrote:
> I agree that a module loading should be, as much as possible, "side 
> effects free", and thus shouldn't have temporary errors. But well, in 
> practice, module loading is typically the time where process-wide 
> initialization are done (modifying sys.path, os.environ, instantiating 
> connection or thread pools, registering atexit handler, starting 
> maintenance threads...), so that case has chances to happen at a moment 
> or another, especially if accesses to filesystem or network (SQL...) are 
> done at module loading, due to the lack of initialization system at 
> upper levels.

There may well be a bug that could be/should be fixed here, but...it
seems to me that other than the sys.path modifications, doing any of that
at module import time has a strong code smell.

--David

From ncoghlan at gmail.com  Tue Jul  2 23:24:46 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 3 Jul 2013 07:24:46 +1000
Subject: [Python-Dev] Solving the import-deadlock case
In-Reply-To: <20130702194146.780342500CB@webabinitio.net>
References: <51D31C94.3010203@gmail.com>
	<20130702194146.780342500CB@webabinitio.net>
Message-ID: <CADiSq7eypavz5AnSss9LLsPZUrjJ6LAMNObevt-tFFY89SO=Dw@mail.gmail.com>

On 3 Jul 2013 05:44, "R. David Murray" <rdmurray at bitdance.com> wrote:
>
> On Tue, 02 Jul 2013 20:31:48 +0200, Pascal Chambon <pythoniks at gmail.com>
wrote:
> > I agree that a module loading should be, as much as possible, "side
> > effects free", and thus shouldn't have temporary errors. But well, in
> > practice, module loading is typically the time where process-wide
> > initialization are done (modifying sys.path, os.environ, instantiating
> > connection or thread pools, registering atexit handler, starting
> > maintenance threads...), so that case has chances to happen at a moment
> > or another, especially if accesses to filesystem or network (SQL...) are
> > done at module loading, due to the lack of initialization system at
> > upper levels.
>
> There may well be a bug that could be/should be fixed here, but...it
> seems to me that other than the sys.path modifications, doing any of that
> at module import time has a strong code smell.

Unfortunately it's one of those "Your code is dubious, but so many people
do it anyway we should handle it better than we do" cases.

We could also be a lot more emphatic about "import side effects are what
marks the boundary between a library and a framework. To stay on the
library side of that fence provide a 'start' or 'configure' function
instead of doing things implicitly on import".

Heck, even *defining* library, framework and application would be a good
thing.

OTOH, it's hard to find motivation to work on improving the handling of
things you think people shouldn't be doing in the first place (that's one
of the reasons circular import handling has never been made more
consistent).

(That's not to dismiss the work Pascal's already done - just pointing out
why it may sometimes feel like it's difficult to get interest and feedback
on things like this).

Cheers,
Nick.

>
> --David
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130703/b221c80e/attachment.html>

From ncoghlan at gmail.com  Tue Jul  2 23:32:53 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 3 Jul 2013 07:32:53 +1000
Subject: [Python-Dev] Solving the import-deadlock case
In-Reply-To: <51D31C94.3010203@gmail.com>
References: <51D31C94.3010203@gmail.com>
Message-ID: <CADiSq7erK0ZoYNzpXftk13jr1EU27P_PYLHPX=-y-DA6pTH2Yw@mail.gmail.com>

On 3 Jul 2013 04:34, "Pascal Chambon" <pythoniks at gmail.com> wrote:
>
> Hello everyone,
>
> I'd like to bring your attention to this issue, since it touches the
fundamentals of python's import workflow:
> http://bugs.python.org/issue17716
>
> I've tried to post it on the python-import ML for weeks, but it must
still be blocked somewhere in a moderation queue, so here I come ^^
>
> TLDR version: because of the way import current works, if importing a
package "temporarily" fails whereas importing one of its children
succeeded, we reach an unusable state, all subsequent attempts at importing
that package will fail if a "from...import" is used somewhere. Typically,
it makes a web worker broken, even though the typical behaviour of such
process woudl be to retry loading, again and again, the failing view.
>
> I agree that a module loading should be, as much as possible, "side
effects free", and thus shouldn't have temporary errors. But well, in
practice, module loading is typically the time where process-wide
initialization are done (modifying sys.path, os.environ, instantiating
connection or thread pools, registering atexit handler, starting
maintenance threads...), so that case has chances to happen at a moment or
another, especially if accesses to filesystem or network (SQL...) are done
at module loading, due to the lack of initialization system at upper
levels.
>
> That's why I propose modifying the behaviour of module import, so that
submodules are deleted as well when a parent module import fails. True, it
means they will be reloaded as well when importing the parent will start
again, but anyway we already have a "double execution" problem with the
reloading of the parent module, so it shouldn't make a big difference.
> The only other solution I'd see would be to SYSTEMATICALLY perform name
(re)binding when processing a from...import statement, to recover from the
previously failed initialization. Dunno if it's a good idea.
>
> On a (separate but related) topic, to be safer on module reimports or
reloadings, it could be interesting to add some "idempotency" to common
initialization tasks ; for example the "atexit" registration system,
wouldn't it be worth adding a boolean flag to explicitely skip registration
if a callable with same fully qualified name is already registered.
>
> Do you have opinions on these subjects ?

Back on topic...

As I stated on the issue, I think purging the whole subtree when a package
implicitly imports child modules is the least bad of the available options,
and better than leaving the child modules in place in violation of the "all
parent packages can be assumed to be in sys.modules" invariant (which is
what we do now).

Cheers,
Nick.
>
> thanks,
> regards,
> Pascal
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130703/c88841fd/attachment.html>

From guido at python.org  Wed Jul  3 05:49:23 2013
From: guido at python.org (Guido van Rossum)
Date: Tue, 2 Jul 2013 20:49:23 -0700
Subject: [Python-Dev] Hooking into super() attribute resolution
In-Reply-To: <DCEB2B77-B9DD-405A-A084-EFB761B947BB@mac.com>
References: <DCEB2B77-B9DD-405A-A084-EFB761B947BB@mac.com>
Message-ID: <CAP7+vJLZqeZYH_G6iE7GvTaBLk-UPxYQfEVpPfLpJG58ebZLHg@mail.gmail.com>

No time to read the PEP in detail but the motivation sound reasonable.

--Guido van Rossum (sent from Android phone)
On Jul 2, 2013 4:53 AM, "Ronald Oussoren" <ronaldoussoren at mac.com> wrote:

> Hi,
>
> Below is a very preliminary draft PEP for adding a special method that can
> be used to hook into the attribute resolution process of the super object.
>
> The primary usecase for using this special method are classes that perform
> custom logic in their __getattribute__ method, where the default behavior
> of super (peekin the the class __dict__) is not appropriate.  The primary
> reason I wrote this proposal is PyObjC: it dynamicly looks up methods in
> its __getattribute__ and caches the result in the class __dict__, because
> of this super() will often not work correctly and therefore I'm currently
> shipping a custom subclass of super() that basicly contains an in-line
> implementation of the hook that would be used by PyObjC.
>
> I have a partial implementation of the hook system in issue 18181  and a
> PyObjC patch that uses it. The implementation currently does not contain
> tests, and I'm sure that I'll find edge cases that I haven't thought about
> yet when I add tests.
>
> Ronald
>
>
>
>
> PEP: TODO
> Title: Hooking into super attribute resolution
> Version: $Revision$
> Last-Modified: $Date$
> Author: Ronald Oussoren <ronaldoussoren at mac.com>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 12-Jun-2013
> Post-History: 2-Jul-2013
>
>
> Abstract
> ========
>
> In current python releases the attribute resolution of the `super class`_
> peeks in the ``__dict__`` attribute of classes on the MRO to look
> for attributes. This PEP introduces a hook that classes can use
> to override that behavior for specific classes.
>
>
> Rationale
> =========
>
> Peeking in the class ``__dict__`` works for regular classes, but can
> cause problems when a class dynamicly looks up attributes in a
> ``__getattribute__`` method.
>
> The new hook makes it possible to introduce the same customization for
> attribute lookup through the `super class`_.
>
>
> The superclass attribute lookup hook
> ====================================
>
> In C code
> ---------
>
> A new slot ``tp_getattro_super`` is added to the ``PyTypeObject`` struct.
> The
> ``tp_getattro`` slot for super will call this slot when it is not ``NULL``,
> otherwise it will peek in the class ``tp_dict``.
>
> The slot has the following prototype::
>
>     PyObject* (*getattrosuperfunc)(PyTypeObject* tp, PyObject* self,
> PyObject* name);
>
> The function should perform attribute lookup for *name*, but only looking
> in
> type *tp* (which will be one of the types on the MRO for *self*) and
> without looking
> in the instance *__dict__*.
>
> The function returns ``NULL`` when the attribute cannot be found, and
> raises and
> exception. Exception other than ``AttributeError`` will cause failure of
> super's
> attribute resolution.
>
>
> In Python code
> --------------
>
> A Python class can contain a definition for a method
> ``__getattribute_super__`` with
> the following prototype::
>
>    def __getattribute_super__(self, cls, name): pass
>
> The method should perform attribute lookup for *name* on instance *self*
> while only
> looking at *cls* (it should not look in super classes or the instance
> *__dict__*
>
>
> Alternative proposals
> ---------------------
>
> Reuse ``tp_getattro``
> .....................
>
> It would be nice to avoid adding a new slot, thus keeping the API simpler
> and easier
> to understand.  A comment on `Issue 18181`_ asked about reusing the
> ``tp_getattro`` slot,
> that is super could call the ``tp_getattro`` slot of all methods along the
> MRO.
>
> AFAIK that won't work because ``tp_getattro`` will look in the instance
> ``__dict__`` before
> it tries to resolve attributes using classes in the MRO. This would mean
> that using
> ``tp_getattro`` instead of peeking the class dictionaries changes the
> semantics of
> the `super class`_.
>
>
> Open Issues
> ===========
>
> * The names of the new slot and magic method are far from settled.
>
> * I'm not too happy with the prototype for the new hook.
>
> * Should ``__getattribute_super__`` be a class method instead?
>
>
> References
> ==========
>
> * `Issue 18181`_ contains a prototype implementation
>
>   The prototype uses different names than this proposal.
>
>
> Copyright
> =========
>
> This document has been placed in the public domain.
>
> .. _`Issue 18181`: http://bugs.python.org/issue18181
>
> .. _`super class`:
> http://docs.python.org/3/library/functions.html?highlight=super#super
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130702/e9981371/attachment-0001.html>

From db3l.net at gmail.com  Wed Jul  3 06:34:14 2013
From: db3l.net at gmail.com (David Bolen)
Date: Wed, 03 Jul 2013 00:34:14 -0400
Subject: [Python-Dev] Issue 18312 "fix" broke buildbots
References: <51D2DB38.8090106@trueblade.com>
	<20130702143335.ACD702500E3@webabinitio.net>
	<51D2EA47.4030004@trueblade.com>
Message-ID: <m2d2r06z3t.fsf@valheru.db3l.homeip.net>

"Eric V. Smith" <eric at trueblade.com> writes:

>> http://buildbot.python.org/all/builders/x86%20Tiger%202.7/builds/2030
>> http://buildbot.python.org/all/builders/x86%20Tiger%203.3/builds/742
>> http://buildbot.python.org/all/builders/x86%20Tiger%203.x/builds/6522
>> http://buildbot.python.org/all/builders/bolen-dmg-3.x/builds/278
(...)
> I've looked at a few of these, and they don't seem related to my change.
> I'll review it some more this afternoon.

The above set (which are all on the x86 Tiger buildbot) appear due to some
sort of filesystem corruption on the buildbot machine, so it appears that
the timing of the failures and particular changesets is coincidental.

I've repaired the filesystem and the disk passes a surface scan, so
hopefully it'll be cleaned up.  I've also re-queued the most recent builds
in each branch.

-- David


From larry at hastings.org  Wed Jul  3 10:36:24 2013
From: larry at hastings.org (Larry Hastings)
Date: Wed, 03 Jul 2013 10:36:24 +0200
Subject: [Python-Dev] Expose stack effects to Python?
Message-ID: <51D3E288.6000904@hastings.org>



I wrote my own assembler for Python bytecode called "Maynard".  I had to 
statically compute the stack effects for each bytecode instruction by 
hand; what I did was copied and pasted opcode_stack_effect() (which is 
static) out of Python/compile.c and into my own driver program, then I 
probed it with test values to produce a table.  I then coded up a 
function using that table, but hand-calculating the value sometimes as 
there are some opcodes whose stack effect varies based on the oparg.

It sure would be nice if this information was simply available to the 
Python interpreter; theoretically it can change between point releases.  
Would anybody mind if I added it somewhere?  I'd probably just expose 
opcode_stack_effect to Python, then add all this other junk to the dis 
module and make it available there.


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130703/e6edf6c6/attachment.html>

From victor.stinner at gmail.com  Wed Jul  3 12:06:08 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 3 Jul 2013 12:06:08 +0200
Subject: [Python-Dev] Expose stack effects to Python?
In-Reply-To: <51D3E288.6000904@hastings.org>
References: <51D3E288.6000904@hastings.org>
Message-ID: <CAMpsgwYSV+mnpaEubbTf2fa1876d1au0q5Dpnp7iayKgTe+GnA@mail.gmail.com>

Hi,

For my registervm project (fork of CPython using register-based
bytecode, instead of stack-based bytecode), I implemented a
Instruction.use_stack() method which just checks if the stack is
"used": read the stack, exchange values in the stack (like "ROT"
instruction), push or pop a value.

Instruction.use_stack():
http://hg.python.org/sandbox/registervm/file/ff24dfecc27d/Lib/registervm.py#l546

The method uses a dummy heuristic, just because it was quick to
implement it, and it's enough for my use case.

To answer your question: yes, I would like a opcode_stack_effect() function.

registervm has its own disassembler which creates objects, rather than
just generating a text representation of the bytecode. I'm using
objects because I rewrite most instructions and I need more
information and functions:

* disassembler (raw bytes => list of instructions)
* assembler (list of instructions => raw bytes)
* format an instruction (human readable assembler)
* is_terminal(): last instruction of a block
* is_cond_jump(): the instruction is a conditional jump? hesitate to
move this disassembler to CPython directly. I'm not sure that it would
be useful, its API is maybe too specific to my registervm project.
* is_reg_used(), is_reg_replaced(), is_reg_modified(), etc.: checks on registers
* etc.

Would it be useful to have such high-level API in Python?

Victor

2013/7/3 Larry Hastings <larry at hastings.org>:
>
>
> I wrote my own assembler for Python bytecode called "Maynard".  I had to
> statically compute the stack effects for each bytecode instruction by hand;
> what I did was copied and pasted opcode_stack_effect() (which is static) out
> of Python/compile.c and into my own driver program, then I probed it with
> test values to produce a table.  I then coded up a function using that
> table, but hand-calculating the value sometimes as there are some opcodes
> whose stack effect varies based on the oparg.
>
> It sure would be nice if this information was simply available to the Python
> interpreter; theoretically it can change between point releases.  Would
> anybody mind if I added it somewhere?  I'd probably just expose
> opcode_stack_effect to Python, then add all this other junk to the dis
> module and make it available there.
>
>
> /arry
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
>

From ncoghlan at gmail.com  Wed Jul  3 14:16:51 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 3 Jul 2013 22:16:51 +1000
Subject: [Python-Dev] Expose stack effects to Python?
In-Reply-To: <CAMpsgwYSV+mnpaEubbTf2fa1876d1au0q5Dpnp7iayKgTe+GnA@mail.gmail.com>
References: <51D3E288.6000904@hastings.org>
	<CAMpsgwYSV+mnpaEubbTf2fa1876d1au0q5Dpnp7iayKgTe+GnA@mail.gmail.com>
Message-ID: <CADiSq7embEBFEoYcdNU3XfmeTgocsbXdtqhdfo1++d_nLk_MUw@mail.gmail.com>

On 3 July 2013 20:06, Victor Stinner <victor.stinner at gmail.com> wrote:

> Hi,
>
> For my registervm project (fork of CPython using register-based
> bytecode, instead of stack-based bytecode), I implemented a
> Instruction.use_stack() method which just checks if the stack is
> "used": read the stack, exchange values in the stack (like "ROT"
> instruction), push or pop a value.
>
> Instruction.use_stack():
>
> http://hg.python.org/sandbox/registervm/file/ff24dfecc27d/Lib/registervm.py#l546
>
> The method uses a dummy heuristic, just because it was quick to
> implement it, and it's enough for my use case.
>
> To answer your question: yes, I would like a opcode_stack_effect()
> function.
>
> registervm has its own disassembler which creates objects, rather than
> just generating a text representation of the bytecode. I'm using
> objects because I rewrite most instructions and I need more
> information and functions:
>
> * disassembler (raw bytes => list of instructions)
> * assembler (list of instructions => raw bytes)
> * format an instruction (human readable assembler)
>
* is_terminal(): last instruction of a block
> * is_cond_jump(): the instruction is a conditional jump? hesitate to
> move this disassembler to CPython directly. I'm not sure that it would
> be useful, its API is maybe too specific to my registervm project.
> * is_reg_used(), is_reg_replaced(), is_reg_modified(), etc.: checks on
> registers
> * etc.
>
> Would it be useful to have such high-level API in Python?
>

I finally committed a longstanding patch to add something like that a while
ago for 3.4: http://docs.python.org/dev/library/dis#bytecode-analysis

It's still fairly low level, but already far more programmatically useful
than the old disassembler text.

I'm still inclined to push higher level stuff out to external libraries -
this patch was mostly about making some of our compiler tests a bit more
maintainable, as well as giving third party libraries better building
blocks without changing the dis module too much.

To get back to Larry's question, though, I think exposing the stack effects
through dis.Instruction would be a good idea (since that will have access
to the oparg to calculate the variable effects).

As far as how to expose the data to Python goes, I suggest adding an
_opcode C module to back opcode.py and eliminate the manual duplication of
the opcode values while you're at it.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130703/9dd7b5fb/attachment.html>

From fijall at gmail.com  Wed Jul  3 14:22:32 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 3 Jul 2013 14:22:32 +0200
Subject: [Python-Dev] Expose stack effects to Python?
In-Reply-To: <CADiSq7embEBFEoYcdNU3XfmeTgocsbXdtqhdfo1++d_nLk_MUw@mail.gmail.com>
References: <51D3E288.6000904@hastings.org>
	<CAMpsgwYSV+mnpaEubbTf2fa1876d1au0q5Dpnp7iayKgTe+GnA@mail.gmail.com>
	<CADiSq7embEBFEoYcdNU3XfmeTgocsbXdtqhdfo1++d_nLk_MUw@mail.gmail.com>
Message-ID: <CAK5idxQ-4FvG0r-rZEcxV-nkuaPEykBZg-pQpPF4nkuUgDEwZA@mail.gmail.com>

I wrote a thing that adds more structure to dis (but is not finished)
https://bitbucket.org/pypy/pypy/src/15b0489c15d8150b22815312dd283aa5bafcdd67/lib_pypy/disassembler.py?at=default

On Wed, Jul 3, 2013 at 2:16 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 3 July 2013 20:06, Victor Stinner <victor.stinner at gmail.com> wrote:
>>
>> Hi,
>>
>> For my registervm project (fork of CPython using register-based
>> bytecode, instead of stack-based bytecode), I implemented a
>> Instruction.use_stack() method which just checks if the stack is
>> "used": read the stack, exchange values in the stack (like "ROT"
>> instruction), push or pop a value.
>>
>> Instruction.use_stack():
>>
>> http://hg.python.org/sandbox/registervm/file/ff24dfecc27d/Lib/registervm.py#l546
>>
>> The method uses a dummy heuristic, just because it was quick to
>> implement it, and it's enough for my use case.
>>
>> To answer your question: yes, I would like a opcode_stack_effect()
>> function.
>>
>> registervm has its own disassembler which creates objects, rather than
>> just generating a text representation of the bytecode. I'm using
>> objects because I rewrite most instructions and I need more
>> information and functions:
>>
>> * disassembler (raw bytes => list of instructions)
>> * assembler (list of instructions => raw bytes)
>> * format an instruction (human readable assembler)
>>
>> * is_terminal(): last instruction of a block
>> * is_cond_jump(): the instruction is a conditional jump? hesitate to
>> move this disassembler to CPython directly. I'm not sure that it would
>> be useful, its API is maybe too specific to my registervm project.
>> * is_reg_used(), is_reg_replaced(), is_reg_modified(), etc.: checks on
>> registers
>> * etc.
>>
>> Would it be useful to have such high-level API in Python?
>
>
> I finally committed a longstanding patch to add something like that a while
> ago for 3.4: http://docs.python.org/dev/library/dis#bytecode-analysis
>
> It's still fairly low level, but already far more programmatically useful
> than the old disassembler text.
>
> I'm still inclined to push higher level stuff out to external libraries -
> this patch was mostly about making some of our compiler tests a bit more
> maintainable, as well as giving third party libraries better building blocks
> without changing the dis module too much.
>
> To get back to Larry's question, though, I think exposing the stack effects
> through dis.Instruction would be a good idea (since that will have access to
> the oparg to calculate the variable effects).
>
> As far as how to expose the data to Python goes, I suggest adding an _opcode
> C module to back opcode.py and eliminate the manual duplication of the
> opcode values while you're at it.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
>

From chambon.pascal at wanadoo.fr  Wed Jul  3 21:55:57 2013
From: chambon.pascal at wanadoo.fr (Pascal Chambon)
Date: Wed, 03 Jul 2013 21:55:57 +0200
Subject: [Python-Dev] Solving the import-deadlock case
In-Reply-To: <CADiSq7erK0ZoYNzpXftk13jr1EU27P_PYLHPX=-y-DA6pTH2Yw@mail.gmail.com>
References: <51D31C94.3010203@gmail.com>
	<CADiSq7erK0ZoYNzpXftk13jr1EU27P_PYLHPX=-y-DA6pTH2Yw@mail.gmail.com>
Message-ID: <51D481CD.8040108@wanadoo.fr>

Thanks for the comments,

in my particular case we're actually on a provisioning /framework/, so 
we chose the easy (lazy?) way, i.e initializing miscellaneous modules at 
loading times (like Django or others do, I think), rather than building 
an proper initialization dispatcher to be called from eg. a wsgi launcher.
It works pretty well actually, except that nasty (but fortunately very 
rare) import deadlock. ^^

Since module loading errors *might* occur for tons of reasons (i.e 
searching the disk for py files already IS a side effect...), and since 
the current behaviour (letting children module survive disconnected from 
their parent) is more harmful than useful, I guess that the cleanup that 
Nick evocated iwould be the path to follow, wouldn't it ?

thanks,
Regards,
Pascal

Le 02/07/2013 23:32, Nick Coghlan a ?crit :
>
>
> On 3 Jul 2013 04:34, "Pascal Chambon" <pythoniks at gmail.com 
> <mailto:pythoniks at gmail.com>> wrote:
> >
> > Hello everyone,
> >
> > I'd like to bring your attention to this issue, since it touches the 
> fundamentals of python's import workflow:
> > http://bugs.python.org/issue17716
> >
> > I've tried to post it on the python-import ML for weeks, but it must 
> still be blocked somewhere in a moderation queue, so here I come ^^
> >
> > TLDR version: because of the way import current works, if importing 
> a package "temporarily" fails whereas importing one of its children 
> succeeded, we reach an unusable state, all subsequent attempts at 
> importing that package will fail if a "from...import" is used 
> somewhere. Typically, it makes a web worker broken, even though the 
> typical behaviour of such process woudl be to retry loading, again and 
> again, the failing view.
> >
> > I agree that a module loading should be, as much as possible, "side 
> effects free", and thus shouldn't have temporary errors. But well, in 
> practice, module loading is typically the time where process-wide 
> initialization are done (modifying sys.path, os.environ, instantiating 
> connection or thread pools, registering atexit handler, starting 
> maintenance threads...), so that case has chances to happen at a 
> moment or another, especially if accesses to filesystem or network 
> (SQL...) are done at module loading, due to the lack of initialization 
> system at upper levels.
> >
> > That's why I propose modifying the behaviour of module import, so 
> that submodules are deleted as well when a parent module import fails. 
> True, it means they will be reloaded as well when importing the parent 
> will start again, but anyway we already have a "double execution" 
> problem with the reloading of the parent module, so it shouldn't make 
> a big difference.
> > The only other solution I'd see would be to SYSTEMATICALLY perform 
> name (re)binding when processing a from...import statement, to recover 
> from the previously failed initialization. Dunno if it's a good idea.
> >
> > On a (separate but related) topic, to be safer on module reimports 
> or reloadings, it could be interesting to add some "idempotency" to 
> common initialization tasks ; for example the "atexit" registration 
> system, wouldn't it be worth adding a boolean flag to explicitely skip 
> registration if a callable with same fully qualified name is already 
> registered.
> >
> > Do you have opinions on these subjects ?
>
> Back on topic...
>
> As I stated on the issue, I think purging the whole subtree when a 
> package implicitly imports child modules is the least bad of the 
> available options, and better than leaving the child modules in place 
> in violation of the "all parent packages can be assumed to be in 
> sys.modules" invariant (which is what we do now).
>
> Cheers,
> Nick.
> >
> > thanks,
> > regards,
> > Pascal
> >
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org <mailto:Python-Dev at python.org>
> > http://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
> >
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/chambon.pascal%40wanadoo.fr
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130703/e6e108e5/attachment.html>

From victor.stinner at gmail.com  Thu Jul  4 13:03:06 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 4 Jul 2013 13:03:06 +0200
Subject: [Python-Dev] PEP 446: Add new parameters to configure the inherance
 of files and for non-blocking sockets
Message-ID: <CAMpsgwYxV3HPY0fodrUBTE8wCHpcJJ-_ESVt04HOno+CcQAF0w@mail.gmail.com>

HTML version:
http://www.python.org/dev/peps/pep-0446/


PEP: 446
Title: Add new parameters to configure the inherance of files and for
non-blocking sockets
Version: $Revision$
Last-Modified: $Date$
Author: Victor Stinner <victor.stinner at gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 3-July-2013
Python-Version: 3.4


Abstract
========

This PEP proposes new portable parameters and functions to configure the
inherance of file descriptors and the non-blocking flag of sockets.


Rationale
=========

Inherance of file descriptors
-----------------------------

The inherance of file descriptors in child processes can be configured
on each file descriptor using a *close-on-exec* flag. By default, the
close-on-exec flag is not set.

On Windows, file descriptors are not inherited if the
``bInheritHandles`` parameter of the ``CreateProcess()`` function is
``FALSE``, even if the close-on-exec flag is not set.

On UNIX, file descriptors with the close-and-exec flag set are closed at
the execution of a new program (ex: when calling ``execv()``). The flag
has no effect on ``fork()``, all file descriptors are inherited by the
child process.

Issues of the inherance of file descriptors
-------------------------------------------

Inherance of file descriptors causes issues. For example, closing a file
descriptor in the parent process does not release the resource (file,
socket, ...), because the file descriptor is still open in the child
process.

Leaking file descriptors is also a major security vulnerability. An
untrusted child process can read sensitive data like passwords and take
control of the parent process though leaked file descriptors. It is for
example a known vulnerability to escape from a chroot.


Non-blocking sockets
--------------------

To handle multiple network clients in a single thread, a multiplexing
function like ``select()`` can be used. For best performances, sockets
must be configured as non-blocking. Operations like ``send()`` and
``recv()`` return an ``EAGAIN`` or ``EWOULDBLOCK`` error if the
operation would block.

By default, newly created sockets are blocking. Setting the non-blocking
mode requires additional system calls.


Setting flags at the creation of the file descriptor
----------------------------------------------------

Windows and recent versions of other operating systems like Linux
support setting the close-on-exec flag directly at the creation of file
descriptors, and close-on-exec and blocking flags at the creation of
sockets.

Setting these flags at the creation is atomic and avoids additional
system calls.


Proposal
========

New cloexec And blocking Parameters
-----------------------------------

Add a new optional *cloexec* on functions creating file descriptors:

* ``io.FileIO``
* ``io.open()``
* ``open()``
* ``os.dup()``
* ``os.dup2()``
* ``os.fdopen()``
* ``os.open()``
* ``os.openpty()``
* ``os.pipe()``
* ``select.devpoll()``
* ``select.epoll()``
* ``select.kqueue()``

Add new optional *cloexec* and *blocking* parameters to functions
creating sockets:

* ``asyncore.dispatcher.create_socket()``
* ``socket.socket()``
* ``socket.socket.accept()``
* ``socket.socket.dup()``
* ``socket.socket.fromfd``
* ``socket.socketpair()``

The default value of *cloexec* is ``False`` and the default value of
*blocking* is ``True``.

The atomicity is not guaranteed. If the platform does not support
setting close-on-exec and blocking flags at the creation of the file
descriptor or socket, the flags are set using additional system calls.

New Functions
-------------

Add new functions the get and set the close-on-exec flag of a file
descriptor:

* ``os.get_cloexec(fd:int) -> bool``
* ``os.set_cloexec(fd:int, cloexec: bool)``


Other Changes
-------------

The ``subprocess.Popen`` class must clear the close-on-exec flag of file
descriptors of the ``pass_fds`` parameter.

The close-on-exec flag must also be set on private file descriptors and
sockets in the Python standard library. For example, on UNIX,
os.urandom() opens ``/dev/urandom`` to read some random bytes and the
file descriptor is closed at function exit. The file descriptor is not
expected to be inherited by child processes.


Rejected Alternatives
=====================

PEP 433
-------

The PEP 433 entitled "Easier suppression of file descriptor inheritance"
is a previous attempt proposing various other alternatives, but no
consensus could be reached.

This PEP has a well defined behaviour (the default value of the new
*cloexec* parameter is not configurable), is more conservative (no
backward compatibility issue), and is much simpler.


Add blocking parameter for file descriptors and Windows overlapped I/O
----------------------------------------------------------------------

Windows supports non-blocking operations on files using an extension of
the Windows API called "Overlapped I/O". Using this extension requires
to modify the Python standard library and applications to pass a
``OVERLAPPED`` structure and an event loop to wait for the completion of
operations.

This PEP only tries to expose portable flags on file descriptors and
sockets. Supporting overlapped I/O requires an abstraction providing a
high-level and portable API for asynchronous operations on files and
sockets. Overlapped I/O are out of the scope of this PEP.

UNIX supports non-blocking files, moreover recent versions of operating
systems support setting the non-blocking flag at the creation of a file
descriptor. It would be possible to add a new optional *blocking*
parameter to Python functions creating file descriptors. On Windows,
creating a file descriptor with ``blocking=False``  would raise a
``NotImplementedError``. This behaviour is not acceptable for the ``os``
module which is designed as a thin wrapper on the C functions of the
operating system. If a platform does not support a function, the
function should not be available on the platform. For example,
the ``os.fork()`` function is not available on Windows.

For all these reasons, this alternative was rejected. The PEP 3156
proposes an abstraction for asynchronous I/O supporting non-blocking
files on Windows.


Links
=====

Python issues:

* `#10115: Support accept4() for atomic setting of flags at socket
  creation <http://bugs.python.org/issue10115>`_
* `#12105: open() does not able to set flags, such as O_CLOEXEC
  <http://bugs.python.org/issue12105>`_
* `#12107: TCP listening sockets created without FD_CLOEXEC flag
  <http://bugs.python.org/issue12107>`_
* `#16850: Add "e" mode to open(): close-and-exec
  (O_CLOEXEC) / O_NOINHERIT <http://bugs.python.org/issue16850>`_
* `#16860: Use O_CLOEXEC in the tempfile module
  <http://bugs.python.org/issue16860>`_
* `#16946: subprocess: _close_open_fd_range_safe() does not set
  close-on-exec flag on Linux < 2.6.23 if O_CLOEXEC is defined
  <http://bugs.python.org/issue16946>`_
* `#17070: Use the new cloexec to improve security and avoid bugs
  <http://bugs.python.org/issue17070>`_

Other links:

* `Secure File Descriptor Handling
  <http://udrepper.livejournal.com/20407.html>`_ (Ulrich Drepper,
  2008)


Copyright
=========

This document has been placed into the public domain.

From victor.stinner at gmail.com  Thu Jul  4 13:19:01 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 4 Jul 2013 13:19:01 +0200
Subject: [Python-Dev] PEP 446: Add new parameters to configure the
 inherance of files and for non-blocking sockets
In-Reply-To: <CAMpsgwYxV3HPY0fodrUBTE8wCHpcJJ-_ESVt04HOno+CcQAF0w@mail.gmail.com>
References: <CAMpsgwYxV3HPY0fodrUBTE8wCHpcJJ-_ESVt04HOno+CcQAF0w@mail.gmail.com>
Message-ID: <CAMpsgwbK2uUSuwh9nuHKrKKUN3gqwD_2OARHxEVYU5SDgw8boA@mail.gmail.com>

2013/7/4 Victor Stinner <victor.stinner at gmail.com>:
> Add a new optional *cloexec* on functions creating file descriptors:
>
> * ``io.FileIO``
> * ``io.open()``
> * ``open()``

The PEP 433 proposes adding an "e" mode to open in alternatives. I
didn't keep this idea because the fopen() function of the GNU libc
library has no mode for the O_NONBLOCK flag. IMO it is not interesting
to mention it in the PEP 466.

Victor

From victor.stinner at gmail.com  Thu Jul  4 13:13:06 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 4 Jul 2013 13:13:06 +0200
Subject: [Python-Dev] PEP 446: Add new parameters to configure the
 inherance of files and for non-blocking sockets
In-Reply-To: <CAMpsgwYxV3HPY0fodrUBTE8wCHpcJJ-_ESVt04HOno+CcQAF0w@mail.gmail.com>
References: <CAMpsgwYxV3HPY0fodrUBTE8wCHpcJJ-_ESVt04HOno+CcQAF0w@mail.gmail.com>
Message-ID: <CAMpsgwZo14i6NTmVkm3ZZkTJeWc71XdaWzH-Up19iGAOW1H7Hg@mail.gmail.com>

> PEP: 446
> Title: Add new parameters to configure the inherance of files and for
> non-blocking sockets
> (...)
> Rejected Alternatives
> =====================
>
> PEP 433
> -------
>
> The PEP 433 entitled "Easier suppression of file descriptor inheritance"
> is a previous attempt proposing various other alternatives, but no
> consensus could be reached.
>
> This PEP has a well defined behaviour (the default value of the new
> *cloexec* parameter is not configurable), is more conservative (no
> backward compatibility issue), and is much simpler.

Even if the PEP 433 was not explicitly rejected, no consensus could be
reached. I didn't want to loose all my work on this PEP and so I'm
proposing something new which should make everbody agrees :-)

The PEP 446 is written to supersed the PEP 433. Even if they are very
close, many drawbacks of the PEP 433 (especially the configurable
default value of the new cloexec parameter) goes away with the PEP
466, which explains why the PEP 466 is much shorter. I preferred to
create a new PEP to keep the historic in the PEP 433. I also chose to
not detail the implementation in the PEP 466, because the PEP 433
already contains a lot of information.

Victor

From chris at simplistix.co.uk  Thu Jul  4 13:21:47 2013
From: chris at simplistix.co.uk (Chris Withers)
Date: Thu, 04 Jul 2013 12:21:47 +0100
Subject: [Python-Dev] lament for the demise of unbound methods
Message-ID: <51D55ACB.4050900@simplistix.co.uk>

Hi All,

In Python 2, I can figure out whether I have a method or a function, 
and, more importantly, for an unbound method, I can figure out what 
class the method belongs to:

 >>> class MyClass(object):
...   def method(self): pass
...
 >>> MyClass.method
<unbound method MyClass.method>
 >>> MyClass.method.im_class
<class '__main__.MyClass'>

There doesn't appear to be any way in Python 3 to do this, which is a 
little surprising and frustrating...

What am I missing here?

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From ronaldoussoren at mac.com  Thu Jul  4 13:52:59 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Thu, 4 Jul 2013 13:52:59 +0200
Subject: [Python-Dev] PEP 446: Add new parameters to configure the
	inherance of files and for non-blocking sockets
In-Reply-To: <CAMpsgwbK2uUSuwh9nuHKrKKUN3gqwD_2OARHxEVYU5SDgw8boA@mail.gmail.com>
References: <CAMpsgwYxV3HPY0fodrUBTE8wCHpcJJ-_ESVt04HOno+CcQAF0w@mail.gmail.com>
	<CAMpsgwbK2uUSuwh9nuHKrKKUN3gqwD_2OARHxEVYU5SDgw8boA@mail.gmail.com>
Message-ID: <C3BBB059-95D0-4E28-89C6-0ED229B819BF@mac.com>


On 4 Jul, 2013, at 13:19, Victor Stinner <victor.stinner at gmail.com> wrote:

> 2013/7/4 Victor Stinner <victor.stinner at gmail.com>:
>> Add a new optional *cloexec* on functions creating file descriptors:
>> 
>> * ``io.FileIO``
>> * ``io.open()``
>> * ``open()``
> 
> The PEP 433 proposes adding an "e" mode to open in alternatives. I
> didn't keep this idea because the fopen() function of the GNU libc
> library has no mode for the O_NONBLOCK flag. IMO it is not interesting
> to mention it in the PEP 466.

I don't understand your reasoning, that is what has GNU libc to do with adding "e" mode to io.open?

BTW. I have no particular fondness for an "e" flag, adding a clo_exec flag would be fine and I'm just curious.

Ronald


From ronaldoussoren at mac.com  Thu Jul  4 13:55:58 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Thu, 4 Jul 2013 13:55:58 +0200
Subject: [Python-Dev] lament for the demise of unbound methods
In-Reply-To: <51D55ACB.4050900@simplistix.co.uk>
References: <51D55ACB.4050900@simplistix.co.uk>
Message-ID: <BCC6A79D-F86C-49D3-9639-9E53C6C75E75@mac.com>


On 4 Jul, 2013, at 13:21, Chris Withers <chris at simplistix.co.uk> wrote:

> Hi All,
> 
> In Python 2, I can figure out whether I have a method or a function, and, more importantly, for an unbound method, I can figure out what class the method belongs to:
> 
> >>> class MyClass(object):
> ...   def method(self): pass
> ...
> >>> MyClass.method
> <unbound method MyClass.method>
> >>> MyClass.method.im_class
> <class '__main__.MyClass'>
> 
> There doesn't appear to be any way in Python 3 to do this, which is a little surprising and frustrating...
> 
> What am I missing here?

You can find the fully qualified name of the method with the qualname attribute:

>>> class A:
...    def method(self): pass
... 
>>> A.method.__qualname__
'A.method'

Ronald
> 
> Chris
> 
> -- 
> Simplistix - Content Management, Batch Processing & Python Consulting
>            - http://www.simplistix.co.uk
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com


From christian at python.org  Thu Jul  4 13:59:23 2013
From: christian at python.org (Christian Heimes)
Date: Thu, 04 Jul 2013 13:59:23 +0200
Subject: [Python-Dev] lament for the demise of unbound methods
In-Reply-To: <51D55ACB.4050900@simplistix.co.uk>
References: <51D55ACB.4050900@simplistix.co.uk>
Message-ID: <51D5639B.4060604@python.org>

Am 04.07.2013 13:21, schrieb Chris Withers:
> There doesn't appear to be any way in Python 3 to do this, which is a
> little surprising and frustrating...
> 
> What am I missing here?

I removed unbound methods almost six years ago:

http://hg.python.org/cpython/rev/48af6375207e

Christian

From victor.stinner at gmail.com  Thu Jul  4 14:04:25 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 4 Jul 2013 14:04:25 +0200
Subject: [Python-Dev] PEP 446: Add new parameters to configure the
 inherance of files and for non-blocking sockets
In-Reply-To: <C3BBB059-95D0-4E28-89C6-0ED229B819BF@mac.com>
References: <CAMpsgwYxV3HPY0fodrUBTE8wCHpcJJ-_ESVt04HOno+CcQAF0w@mail.gmail.com>
	<CAMpsgwbK2uUSuwh9nuHKrKKUN3gqwD_2OARHxEVYU5SDgw8boA@mail.gmail.com>
	<C3BBB059-95D0-4E28-89C6-0ED229B819BF@mac.com>
Message-ID: <CAMpsgwawrfBW07bobQw_u=vK_-RigJTqzD2wWPpsxxS2=+HnDA@mail.gmail.com>

2013/7/4 Ronald Oussoren <ronaldoussoren at mac.com>:
>> The PEP 433 proposes adding an "e" mode to open in alternatives. I
>> didn't keep this idea because the fopen() function of the GNU libc
>> library has no mode for the O_NONBLOCK flag. IMO it is not interesting
>> to mention it in the PEP 466.
>
> I don't understand your reasoning, that is what has GNU libc to do with adding "e" mode to io.open?

The GNU libc supports fopen(filename, "re") to set O_CLOEXEC flag on
the file. I pick the idea into the PEP 433, in alternatives:
open(filename, "re") sets O_CLOEXEC flag on the file.

For the PEP 466, I only proposed the original API of the PEP 433:
open(filename, "r", cloexec=True).

Victor

From chris at simplistix.co.uk  Thu Jul  4 14:13:57 2013
From: chris at simplistix.co.uk (Chris Withers)
Date: Thu, 04 Jul 2013 13:13:57 +0100
Subject: [Python-Dev] lament for the demise of unbound methods
In-Reply-To: <51D5639B.4060604@python.org>
References: <51D55ACB.4050900@simplistix.co.uk> <51D5639B.4060604@python.org>
Message-ID: <51D56705.1000908@simplistix.co.uk>

On 04/07/2013 12:59, Christian Heimes wrote:
> Am 04.07.2013 13:21, schrieb Chris Withers:
>> There doesn't appear to be any way in Python 3 to do this, which is a
>> little surprising and frustrating...
>>
>> What am I missing here?
>
> I removed unbound methods almost six years ago:
>
> http://hg.python.org/cpython/rev/48af6375207e

Not disputing when it happened, more the why...

...the recommended change doesn't work, for obvious reasons:

 >>> MyClass.method.__self__.__class__
Traceback (most recent call last):
   File "<console>", line 1, in <module>
AttributeError: 'function' object has no attribute '__self__'

The loss of the ability to figure out the class from an unbound method 
seems quite an annoying step back from an introspection point of view.

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From chris at simplistix.co.uk  Thu Jul  4 14:16:12 2013
From: chris at simplistix.co.uk (Chris Withers)
Date: Thu, 04 Jul 2013 13:16:12 +0100
Subject: [Python-Dev] lament for the demise of unbound methods
In-Reply-To: <BCC6A79D-F86C-49D3-9639-9E53C6C75E75@mac.com>
References: <51D55ACB.4050900@simplistix.co.uk>
	<BCC6A79D-F86C-49D3-9639-9E53C6C75E75@mac.com>
Message-ID: <51D5678C.9050205@simplistix.co.uk>

On 04/07/2013 12:55, Ronald Oussoren wrote:
>
> You can find the fully qualified name of the method with the qualname attribute:
>
>>>> class A:
> ...    def method(self): pass
> ...
>>>> A.method.__qualname__
> 'A.method'

That doesn't seem helpful as a sensible way to get back to the class object:

 >> globals()[MyClass.method.__qualname__.split('.')[0]]
<class '__main__.MyClass'>

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From brett at python.org  Thu Jul  4 14:34:05 2013
From: brett at python.org (Brett Cannon)
Date: Thu, 4 Jul 2013 08:34:05 -0400
Subject: [Python-Dev] lament for the demise of unbound methods
In-Reply-To: <51D56705.1000908@simplistix.co.uk>
References: <51D55ACB.4050900@simplistix.co.uk> <51D5639B.4060604@python.org>
	<51D56705.1000908@simplistix.co.uk>
Message-ID: <CAP1=2W5--OmzkQDoS58=vXzmWZH0zasF2yNqZhxyo48JPvg8eQ@mail.gmail.com>

On Thu, Jul 4, 2013 at 8:13 AM, Chris Withers <chris at simplistix.co.uk>wrote:

> On 04/07/2013 12:59, Christian Heimes wrote:
>
>> Am 04.07.2013 13:21, schrieb Chris Withers:
>>
>>> There doesn't appear to be any way in Python 3 to do this, which is a
>>> little surprising and frustrating...
>>>
>>> What am I missing here?
>>>
>>
>> I removed unbound methods almost six years ago:
>>
>> http://hg.python.org/cpython/**rev/48af6375207e<http://hg.python.org/cpython/rev/48af6375207e>
>>
>
> Not disputing when it happened, more the why...
>
> ...the recommended change doesn't work, for obvious reasons:
>
> >>> MyClass.method.__self__.__**class__
> Traceback (most recent call last):
>   File "<console>", line 1, in <module>
> AttributeError: 'function' object has no attribute '__self__'
>
> The loss of the ability to figure out the class from an unbound method
> seems quite an annoying step back from an introspection point of view.
>

It's only annoying if you take the perspective that methods are somehow
special compared to functions. With the removal of bound class methods that
makes methods == functions that are an attribute on a class. And when you
take that perspective it makes having anything special about methods seem
wrong. It also makes adding a function to a class post-class creation make
more sense since there is no difference technically.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130704/f3a894ce/attachment.html>

From victor.stinner at gmail.com  Thu Jul  4 15:59:23 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 4 Jul 2013 15:59:23 +0200
Subject: [Python-Dev] lament for the demise of unbound methods
In-Reply-To: <51D5678C.9050205@simplistix.co.uk>
References: <51D55ACB.4050900@simplistix.co.uk>
	<BCC6A79D-F86C-49D3-9639-9E53C6C75E75@mac.com>
	<51D5678C.9050205@simplistix.co.uk>
Message-ID: <CAMpsgwYGNg_kNwzDACujMtsKo+7CB3Dh_EkRvGRqbQo7c67+zg@mail.gmail.com>

2013/7/4 Chris Withers <chris at simplistix.co.uk>:
> That doesn't seem helpful as a sensible way to get back to the class object:
>
>>> globals()[MyClass.method.__qualname__.split('.')[0]]
> <class '__main__.MyClass'>

globals() can only be used if MyClass is in the same module.
Otherwise, you a more complex function:
---------------
import types

def get_function_class(func):
    obj = func
    for name in func.__qualname__.split('.')[:-1]:
        if name == "<locals>":
            raise ValueError("you lose")
        if isinstance(obj, types.FunctionType):
            obj = func.__globals__[name]
        else:
            # get a method of a class, or a class defined in a child
            obj = getattr(obj, name)
    return obj
---------------

Victor

From guido at python.org  Thu Jul  4 18:25:31 2013
From: guido at python.org (Guido van Rossum)
Date: Thu, 4 Jul 2013 09:25:31 -0700
Subject: [Python-Dev] lament for the demise of unbound methods
In-Reply-To: <51D55ACB.4050900@simplistix.co.uk>
References: <51D55ACB.4050900@simplistix.co.uk>
Message-ID: <CAP7+vJLVwzg0y-H3TbbbAbfsL8Bcmx3Y3=57uBA+O0hARhO8OA@mail.gmail.com>

Chris, what do you want to do with the knowledge you are seeking?

--Guido van Rossum (sent from Android phone)
On Jul 4, 2013 4:28 AM, "Chris Withers" <chris at simplistix.co.uk> wrote:

> Hi All,
>
> In Python 2, I can figure out whether I have a method or a function, and,
> more importantly, for an unbound method, I can figure out what class the
> method belongs to:
>
> >>> class MyClass(object):
> ...   def method(self): pass
> ...
> >>> MyClass.method
> <unbound method MyClass.method>
> >>> MyClass.method.im_class
> <class '__main__.MyClass'>
>
> There doesn't appear to be any way in Python 3 to do this, which is a
> little surprising and frustrating...
>
> What am I missing here?
>
> Chris
>
> --
> Simplistix - Content Management, Batch Processing & Python Consulting
>             - http://www.simplistix.co.uk
> ______________________________**_________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/**mailman/listinfo/python-dev<http://mail.python.org/mailman/listinfo/python-dev>
> Unsubscribe: http://mail.python.org/**mailman/options/python-dev/**
> guido%40python.org<http://mail.python.org/mailman/options/python-dev/guido%40python.org>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130704/710c4258/attachment-0001.html>

From chris at simplistix.co.uk  Thu Jul  4 18:42:07 2013
From: chris at simplistix.co.uk (Chris Withers)
Date: Thu, 04 Jul 2013 17:42:07 +0100
Subject: [Python-Dev] lament for the demise of unbound methods
In-Reply-To: <CAP7+vJLVwzg0y-H3TbbbAbfsL8Bcmx3Y3=57uBA+O0hARhO8OA@mail.gmail.com>
References: <51D55ACB.4050900@simplistix.co.uk>
	<CAP7+vJLVwzg0y-H3TbbbAbfsL8Bcmx3Y3=57uBA+O0hARhO8OA@mail.gmail.com>
Message-ID: <51D5A5DF.3000103@simplistix.co.uk>

Hi Guido,

I've bumped into this a couple of times.

First time was when I wanted to know whether what I had was a 
classmethod, staticmethod or normal method here:

https://github.com/Simplistix/testfixtures/blob/master/testfixtures/replace.py#L59

This resulted in having to trawl through __dict__ here:

https://github.com/Simplistix/testfixtures/blob/master/testfixtures/resolve.py#L17

...rather than just using getattr.

I bumped into it again, yesterday, trying to add support for classes to 
this lightweight dependency injection framework I'm developing:

https://github.com/Simplistix/mush/blob/master/tests/test_runner.py#L189

Here's my local copy of that test:

https://gist.github.com/cjw296/db64279c69cdc0c5e112

The workaround I was playing with this morning is a wrapper so that I 
know I have a class method, although what I really want to write at this 
line is:

https://gist.github.com/cjw296/db64279c69cdc0c5e112#file-gistfile1-txt-L40

runner = Runner(T0, C1.meth, C2.meth1, C2.meth2)

...but if I do that, how can the runner know that what it gets for its 
second argument is a class method of C1?
(which is this case means that it should do C1().meth() rather than 
C1.meth())

cheers,

Chris

On 04/07/2013 17:25, Guido van Rossum wrote:
> Chris, what do you want to do with the knowledge you are seeking?
>
> --Guido van Rossum (sent from Android phone)
>
> On Jul 4, 2013 4:28 AM, "Chris Withers" <chris at simplistix.co.uk
> <mailto:chris at simplistix.co.uk>> wrote:
>
>     Hi All,
>
>     In Python 2, I can figure out whether I have a method or a function,
>     and, more importantly, for an unbound method, I can figure out what
>     class the method belongs to:
>
>      >>> class MyClass(object):
>     ...   def method(self): pass
>     ...
>      >>> MyClass.method
>     <unbound method MyClass.method>
>      >>> MyClass.method.im_class
>     <class '__main__.MyClass'>
>
>     There doesn't appear to be any way in Python 3 to do this, which is
>     a little surprising and frustrating...
>
>     What am I missing here?
>
>     Chris
>
>     --
>     Simplistix - Content Management, Batch Processing & Python Consulting
>                  - http://www.simplistix.co.uk
>     _________________________________________________
>     Python-Dev mailing list
>     Python-Dev at python.org <mailto:Python-Dev at python.org>
>     http://mail.python.org/__mailman/listinfo/python-dev
>     <http://mail.python.org/mailman/listinfo/python-dev>
>     Unsubscribe:
>     http://mail.python.org/__mailman/options/python-dev/__guido%40python.org
>     <http://mail.python.org/mailman/options/python-dev/guido%40python.org>
>
>
> ______________________________________________________________________
> This email has been scanned by the Symantec Email Security.cloud service.
> For more information please visit http://www.symanteccloud.com
> ______________________________________________________________________

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From guido at python.org  Thu Jul  4 19:00:46 2013
From: guido at python.org (Guido van Rossum)
Date: Thu, 4 Jul 2013 10:00:46 -0700
Subject: [Python-Dev] lament for the demise of unbound methods
In-Reply-To: <51D5A5DF.3000103@simplistix.co.uk>
References: <51D55ACB.4050900@simplistix.co.uk>
	<CAP7+vJLVwzg0y-H3TbbbAbfsL8Bcmx3Y3=57uBA+O0hARhO8OA@mail.gmail.com>
	<51D5A5DF.3000103@simplistix.co.uk>
Message-ID: <CAP7+vJLeHPkq-UgFreNBHR67UVx7Tih3jZz-VjEdbtBZ2mp3tw@mail.gmail.com>

Thanks for the code pointers. So it's all about monkeypatching. :-) I have
only a little sympathy, as there still seems to be a way to do this, it's
just less convenient. Too bad.

--Guido

On Thu, Jul 4, 2013 at 9:42 AM, Chris Withers <chris at simplistix.co.uk>wrote:

> Hi Guido,
>
> I've bumped into this a couple of times.
>
> First time was when I wanted to know whether what I had was a classmethod,
> staticmethod or normal method here:
>
> https://github.com/Simplistix/**testfixtures/blob/master/**
> testfixtures/replace.py#L59<https://github.com/Simplistix/testfixtures/blob/master/testfixtures/replace.py#L59>
>
> This resulted in having to trawl through __dict__ here:
>
> https://github.com/Simplistix/**testfixtures/blob/master/**
> testfixtures/resolve.py#L17<https://github.com/Simplistix/testfixtures/blob/master/testfixtures/resolve.py#L17>
>
> ...rather than just using getattr.
>
> I bumped into it again, yesterday, trying to add support for classes to
> this lightweight dependency injection framework I'm developing:
>
> https://github.com/Simplistix/**mush/blob/master/tests/test_**
> runner.py#L189<https://github.com/Simplistix/mush/blob/master/tests/test_runner.py#L189>
>
> Here's my local copy of that test:
>
> https://gist.github.com/**cjw296/db64279c69cdc0c5e112<https://gist.github.com/cjw296/db64279c69cdc0c5e112>
>
> The workaround I was playing with this morning is a wrapper so that I know
> I have a class method, although what I really want to write at this line is:
>
> https://gist.github.com/**cjw296/db64279c69cdc0c5e112#**
> file-gistfile1-txt-L40<https://gist.github.com/cjw296/db64279c69cdc0c5e112#file-gistfile1-txt-L40>
>
> runner = Runner(T0, C1.meth, C2.meth1, C2.meth2)
>
> ...but if I do that, how can the runner know that what it gets for its
> second argument is a class method of C1?
> (which is this case means that it should do C1().meth() rather than
> C1.meth())
>
> cheers,
>
> Chris
>
>
> On 04/07/2013 17:25, Guido van Rossum wrote:
>
>> Chris, what do you want to do with the knowledge you are seeking?
>>
>> --Guido van Rossum (sent from Android phone)
>>
>> On Jul 4, 2013 4:28 AM, "Chris Withers" <chris at simplistix.co.uk
>> <mailto:chris at simplistix.co.uk**>> wrote:
>>
>>     Hi All,
>>
>>     In Python 2, I can figure out whether I have a method or a function,
>>     and, more importantly, for an unbound method, I can figure out what
>>     class the method belongs to:
>>
>>      >>> class MyClass(object):
>>     ...   def method(self): pass
>>     ...
>>      >>> MyClass.method
>>     <unbound method MyClass.method>
>>      >>> MyClass.method.im_class
>>     <class '__main__.MyClass'>
>>
>>     There doesn't appear to be any way in Python 3 to do this, which is
>>     a little surprising and frustrating...
>>
>>     What am I missing here?
>>
>>     Chris
>>
>>     --
>>     Simplistix - Content Management, Batch Processing & Python Consulting
>>                  - http://www.simplistix.co.uk
>>     ______________________________**___________________
>>     Python-Dev mailing list
>>     Python-Dev at python.org <mailto:Python-Dev at python.org>
>>     http://mail.python.org/__**mailman/listinfo/python-dev<http://mail.python.org/__mailman/listinfo/python-dev>
>>     <http://mail.python.org/**mailman/listinfo/python-dev<http://mail.python.org/mailman/listinfo/python-dev>
>> >
>>     Unsubscribe:
>>     http://mail.python.org/__**mailman/options/python-dev/__**
>> guido%40python.org<http://mail.python.org/__mailman/options/python-dev/__guido%40python.org>
>>     <http://mail.python.org/**mailman/options/python-dev/**
>> guido%40python.org<http://mail.python.org/mailman/options/python-dev/guido%40python.org>
>> >
>>
>>
>> ______________________________**______________________________**
>> __________
>> This email has been scanned by the Symantec Email Security.cloud service.
>> For more information please visit http://www.symanteccloud.com
>> ______________________________**______________________________**
>> __________
>>
>
> --
> Simplistix - Content Management, Batch Processing & Python Consulting
>             - http://www.simplistix.co.uk
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130704/6af3f21d/attachment.html>

From chris at simplistix.co.uk  Thu Jul  4 19:20:54 2013
From: chris at simplistix.co.uk (Chris Withers)
Date: Thu, 04 Jul 2013 18:20:54 +0100
Subject: [Python-Dev] lament for the demise of unbound methods
In-Reply-To: <CAP7+vJLeHPkq-UgFreNBHR67UVx7Tih3jZz-VjEdbtBZ2mp3tw@mail.gmail.com>
References: <51D55ACB.4050900@simplistix.co.uk>
	<CAP7+vJLVwzg0y-H3TbbbAbfsL8Bcmx3Y3=57uBA+O0hARhO8OA@mail.gmail.com>
	<51D5A5DF.3000103@simplistix.co.uk>
	<CAP7+vJLeHPkq-UgFreNBHR67UVx7Tih3jZz-VjEdbtBZ2mp3tw@mail.gmail.com>
Message-ID: <51D5AEF6.50804@simplistix.co.uk>

On 04/07/2013 18:00, Guido van Rossum wrote:
> Thanks for the code pointers. So it's all about monkeypatching. :-)

Well, that's the testfixtures use case, but for mush it's about figuring 
out whether you need to instantiate a class before calling a callable.

MyClass.a_method is a bit like a functools.partial in the mush case, if 
I can pass that object around and know what to do with it (which I can 
in Python 2) then I only have to pass that around.

In Python3, I either have to pass around the class, the method and a 
flag to indicate that a class and method are being passed, or wrap my 
own unboundmethod equivalent, meaning mush users would have to write 
method(MyClass, 'a_method') under Python 3 when they can just write 
MyClass.a_method under Python 2.

> I
> have only a little sympathy, as there still seems to be a way to do
> this, it's just less convenient. Too bad.

I don't know that Victor's suggestion will actually work in all the 
cases that MyClass.a_method.im_class does :-S

Chris

> On Thu, Jul 4, 2013 at 9:42 AM, Chris Withers <chris at simplistix.co.uk
>
>     Hi Guido,
>
>     I've bumped into this a couple of times.
>
>     First time was when I wanted to know whether what I had was a
>     classmethod, staticmethod or normal method here:
>
>     https://github.com/Simplistix/__testfixtures/blob/master/__testfixtures/replace.py#L59
>     <https://github.com/Simplistix/testfixtures/blob/master/testfixtures/replace.py#L59>
>
>     This resulted in having to trawl through __dict__ here:
>
>     https://github.com/Simplistix/__testfixtures/blob/master/__testfixtures/resolve.py#L17
>     <https://github.com/Simplistix/testfixtures/blob/master/testfixtures/resolve.py#L17>
>
>     ...rather than just using getattr.
>
>     I bumped into it again, yesterday, trying to add support for classes
>     to this lightweight dependency injection framework I'm developing:
>
>     https://github.com/Simplistix/__mush/blob/master/tests/test___runner.py#L189
>     <https://github.com/Simplistix/mush/blob/master/tests/test_runner.py#L189>
>
>     Here's my local copy of that test:
>
>     https://gist.github.com/__cjw296/db64279c69cdc0c5e112
>     <https://gist.github.com/cjw296/db64279c69cdc0c5e112>
>
>     The workaround I was playing with this morning is a wrapper so that
>     I know I have a class method, although what I really want to write
>     at this line is:
>
>     https://gist.github.com/__cjw296/db64279c69cdc0c5e112#__file-gistfile1-txt-L40
>     <https://gist.github.com/cjw296/db64279c69cdc0c5e112#file-gistfile1-txt-L40>
>
>     runner = Runner(T0, C1.meth, C2.meth1, C2.meth2)
>
>     ...but if I do that, how can the runner know that what it gets for
>     its second argument is a class method of C1?
>     (which is this case means that it should do C1().meth() rather than
>     C1.meth())
>
>     cheers,
>
>     Chris
>
>
>     On 04/07/2013 17:25, Guido van Rossum wrote:
>
>         Chris, what do you want to do with the knowledge you are seeking?
>
>         --Guido van Rossum (sent from Android phone)
>
>         On Jul 4, 2013 4:28 AM, "Chris Withers" <chris at simplistix.co.uk
>         <mailto:chris at simplistix.co.uk>
>         <mailto:chris at simplistix.co.uk
>         <mailto:chris at simplistix.co.uk>__>> wrote:
>
>              Hi All,
>
>              In Python 2, I can figure out whether I have a method or a
>         function,
>              and, more importantly, for an unbound method, I can figure
>         out what
>              class the method belongs to:
>
>          >>> class MyClass(object):
>              ...   def method(self): pass
>              ...
>          >>> MyClass.method
>         <unbound method MyClass.method>
>          >>> MyClass.method.im_class
>         <class '__main__.MyClass'>
>
>              There doesn't appear to be any way in Python 3 to do this,
>         which is
>              a little surprising and frustrating...
>
>              What am I missing here?
>
>              Chris
>
>              --
>              Simplistix - Content Management, Batch Processing & Python
>         Consulting
>                           - http://www.simplistix.co.uk
>              ___________________________________________________
>              Python-Dev mailing list
>         Python-Dev at python.org <mailto:Python-Dev at python.org>
>         <mailto:Python-Dev at python.org <mailto:Python-Dev at python.org>>
>         http://mail.python.org/____mailman/listinfo/python-dev
>         <http://mail.python.org/__mailman/listinfo/python-dev>
>         <http://mail.python.org/__mailman/listinfo/python-dev
>         <http://mail.python.org/mailman/listinfo/python-dev>>
>              Unsubscribe:
>         http://mail.python.org/____mailman/options/python-dev/____guido%40python.org
>         <http://mail.python.org/__mailman/options/python-dev/__guido%40python.org>
>         <http://mail.python.org/__mailman/options/python-dev/__guido%40python.org
>         <http://mail.python.org/mailman/options/python-dev/guido%40python.org>>
>
>
>         __________________________________________________________________________
>         This email has been scanned by the Symantec Email Security.cloud
>         service.
>         For more information please visit http://www.symanteccloud.com
>         __________________________________________________________________________
>
>
>     --
>     Simplistix - Content Management, Batch Processing & Python Consulting
>                  - http://www.simplistix.co.uk
>
>
>
>
> --
> --Guido van Rossum (python.org/~guido <http://python.org/~guido>)
> ______________________________________________________________________
> This email has been scanned by the Symantec Email Security.cloud service.
> For more information please visit http://www.symanteccloud.com
> ______________________________________________________________________

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From ericsnowcurrently at gmail.com  Thu Jul  4 20:05:08 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Thu, 4 Jul 2013 12:05:08 -0600
Subject: [Python-Dev] lament for the demise of unbound methods
In-Reply-To: <51D55ACB.4050900@simplistix.co.uk>
References: <51D55ACB.4050900@simplistix.co.uk>
Message-ID: <CALFfu7CTE7J-Y1u+aMX_AUmui0OE2jVJ6yESY8NZ7B9_3BqUfQ@mail.gmail.com>

On Thu, Jul 4, 2013 at 5:21 AM, Chris Withers <chris at simplistix.co.uk>wrote:

> Hi All,
>
> In Python 2, I can figure out whether I have a method or a function, and,
> more importantly, for an unbound method, I can figure out what class the
> method belongs to:
>
> >>> class MyClass(object):
> ...   def method(self): pass
> ...
> >>> MyClass.method
> <unbound method MyClass.method>
> >>> MyClass.method.im_class
> <class '__main__.MyClass'>
>
> There doesn't appear to be any way in Python 3 to do this, which is a
> little surprising and frustrating...
>
> What am I missing here?


You could always monkeypatch builtins.__build_class__ to add an attribute
to every "unbound method" pointing to the class.

-eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130704/719c49dd/attachment.html>

From solipsis at pitrou.net  Thu Jul  4 21:09:06 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 4 Jul 2013 21:09:06 +0200
Subject: [Python-Dev] Oddity in MISC/News on default
Message-ID: <20130704210906.74208200@fsol>


Hello,

In 3.4's Misc/NEWS, there's a huge section entitled "What's New in
Python 3.3.1 release candidate 1". It seems it shouldn't be there,
should it?

Regards

Antoine.



From guido at python.org  Thu Jul  4 21:17:05 2013
From: guido at python.org (Guido van Rossum)
Date: Thu, 4 Jul 2013 12:17:05 -0700
Subject: [Python-Dev] Oddity in MISC/News on default
In-Reply-To: <20130704210906.74208200@fsol>
References: <20130704210906.74208200@fsol>
Message-ID: <CAP7+vJKF=ZHUrtB8bfj8oVNToGi1=KWDef9EXR4i7UjW7MYmYA@mail.gmail.com>

Why not? Presumably those news items were all merged into the default
branch, and Misc/NEWS also has all the news for all 3.3.0 releases. Why
pick on 3.3.1 rc1?

On Thu, Jul 4, 2013 at 12:09 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:

>
> Hello,
>
> In 3.4's Misc/NEWS, there's a huge section entitled "What's New in
> Python 3.3.1 release candidate 1". It seems it shouldn't be there,
> should it?
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130704/90940aee/attachment.html>

From solipsis at pitrou.net  Thu Jul  4 21:19:59 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 04 Jul 2013 21:19:59 +0200
Subject: [Python-Dev] Oddity in MISC/News on default
In-Reply-To: <CAP7+vJKF=ZHUrtB8bfj8oVNToGi1=KWDef9EXR4i7UjW7MYmYA@mail.gmail.com>
References: <20130704210906.74208200@fsol>
	<CAP7+vJKF=ZHUrtB8bfj8oVNToGi1=KWDef9EXR4i7UjW7MYmYA@mail.gmail.com>
Message-ID: <1372965599.2599.2.camel@fsol>

Le jeudi 04 juillet 2013 ? 12:17 -0700, Guido van Rossum a ?crit :
> Why not? Presumably those news items were all merged into the default
> branch, and Misc/NEWS also has all the news for all 3.3.0 releases.
> Why pick on 3.3.1 rc1?

3.3.1rc1 is the only post-3.3.0 release that's mentioned there (e.g.
3.3.2 isn't), which is what makes me thing a mistake was made along the
way (probably when merging).

Regards

Antoine.



From guido at python.org  Thu Jul  4 21:36:20 2013
From: guido at python.org (Guido van Rossum)
Date: Thu, 4 Jul 2013 12:36:20 -0700
Subject: [Python-Dev] Oddity in MISC/News on default
In-Reply-To: <1372965599.2599.2.camel@fsol>
References: <20130704210906.74208200@fsol>
	<CAP7+vJKF=ZHUrtB8bfj8oVNToGi1=KWDef9EXR4i7UjW7MYmYA@mail.gmail.com>
	<1372965599.2599.2.camel@fsol>
Message-ID: <CAP7+vJKR2=OK1cpdX_fEV=ay-pA1GLmc4xSZEjUYXLfLUn+5aw@mail.gmail.com>

Maybe the mistake is that the others aren't mentioned? Or perhaps
everything before 3.4a1 should be dropped? I forget what kind of policy we
have for this -- is it all changes in this branch or only changes unique to
this branch?

On Thu, Jul 4, 2013 at 12:19 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> Le jeudi 04 juillet 2013 ? 12:17 -0700, Guido van Rossum a ?crit :
> > Why not? Presumably those news items were all merged into the default
> > branch, and Misc/NEWS also has all the news for all 3.3.0 releases.
> > Why pick on 3.3.1 rc1?
>
> 3.3.1rc1 is the only post-3.3.0 release that's mentioned there (e.g.
> 3.3.2 isn't), which is what makes me thing a mistake was made along the
> way (probably when merging).
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130704/95965e10/attachment.html>

From benjamin at python.org  Thu Jul  4 21:47:30 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Thu, 4 Jul 2013 12:47:30 -0700
Subject: [Python-Dev] Oddity in MISC/News on default
In-Reply-To: <20130704210906.74208200@fsol>
References: <20130704210906.74208200@fsol>
Message-ID: <CAPZV6o-YwQYBdxbe=iXT_Z80uiXCitC3hcwJwmp5n-8sie8kpA@mail.gmail.com>

Does annotate show it might have been added accidently in a merge?

2013/7/4 Antoine Pitrou <solipsis at pitrou.net>:
>
> Hello,
>
> In 3.4's Misc/NEWS, there's a huge section entitled "What's New in
> Python 3.3.1 release candidate 1". It seems it shouldn't be there,
> should it?
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/benjamin%40python.org



-- 
Regards,
Benjamin

From tseaver at palladion.com  Thu Jul  4 21:47:38 2013
From: tseaver at palladion.com (Tres Seaver)
Date: Thu, 04 Jul 2013 15:47:38 -0400
Subject: [Python-Dev] PEP 446: Add new parameters to configure the
 inherance of files and for non-blocking sockets
In-Reply-To: <CAMpsgwYxV3HPY0fodrUBTE8wCHpcJJ-_ESVt04HOno+CcQAF0w@mail.gmail.com>
References: <CAMpsgwYxV3HPY0fodrUBTE8wCHpcJJ-_ESVt04HOno+CcQAF0w@mail.gmail.com>
Message-ID: <kr4jgk$3uu$1@ger.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 07/04/2013 07:03 AM, Victor Stinner wrote:
> Title: Add new parameters to configure the inherance of files and for 
> non-blocking sockets

Not commenting on either the form or the substance (pun intended), but
the word you want is "inheritance" -- "inherence" is a valid term[1], but
would a good deal stranger notion to apply to a file descriptor. ;)


[1] https://en.wikipedia.org/wiki/Inherence


Platonic'ly,


Tres.
- -- 
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlHV0VoACgkQ+gerLs4ltQ4YCQCgp6mFPxEVVoXAXib/jrChjRxu
QkAAoLJQIfBCQezj61LCAgmVaE1kwNmM
=yiPj
-----END PGP SIGNATURE-----


From benjamin at python.org  Thu Jul  4 21:50:09 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Thu, 4 Jul 2013 12:50:09 -0700
Subject: [Python-Dev] lament for the demise of unbound methods
In-Reply-To: <CALFfu7CTE7J-Y1u+aMX_AUmui0OE2jVJ6yESY8NZ7B9_3BqUfQ@mail.gmail.com>
References: <51D55ACB.4050900@simplistix.co.uk>
	<CALFfu7CTE7J-Y1u+aMX_AUmui0OE2jVJ6yESY8NZ7B9_3BqUfQ@mail.gmail.com>
Message-ID: <CAPZV6o8n-Ti8q5Q_-wwqRXpCiP76GGaHHwmCW7T5MrvOycOepw@mail.gmail.com>

2013/7/4 Eric Snow <ericsnowcurrently at gmail.com>:
>
> On Thu, Jul 4, 2013 at 5:21 AM, Chris Withers <chris at simplistix.co.uk>
> wrote:
>>
>> Hi All,
>>
>> In Python 2, I can figure out whether I have a method or a function, and,
>> more importantly, for an unbound method, I can figure out what class the
>> method belongs to:
>>
>> >>> class MyClass(object):
>> ...   def method(self): pass
>> ...
>> >>> MyClass.method
>> <unbound method MyClass.method>
>> >>> MyClass.method.im_class
>> <class '__main__.MyClass'>
>>
>> There doesn't appear to be any way in Python 3 to do this, which is a
>> little surprising and frustrating...
>>
>> What am I missing here?
>
>
> You could always monkeypatch builtins.__build_class__ to add an attribute to
> every "unbound method" pointing to the class.

I would not reccomend that. __build_class__ is very internal and it's
contract may change between versions.



--
Regards,
Benjamin

From solipsis at pitrou.net  Thu Jul  4 21:51:27 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 4 Jul 2013 21:51:27 +0200
Subject: [Python-Dev] Oddity in MISC/News on default
In-Reply-To: <CAPZV6o-YwQYBdxbe=iXT_Z80uiXCitC3hcwJwmp5n-8sie8kpA@mail.gmail.com>
References: <20130704210906.74208200@fsol>
	<CAPZV6o-YwQYBdxbe=iXT_Z80uiXCitC3hcwJwmp5n-8sie8kpA@mail.gmail.com>
Message-ID: <20130704215127.1a92ffb9@fsol>

On Thu, 4 Jul 2013 12:47:30 -0700
Benjamin Peterson <benjamin at python.org> wrote:

> Does annotate show it might have been added accidently in a merge?

Annotate shows the original changesets on the 3.3 branch, which doesn't
help ;-)

Regards

Antoine.

From chris at simplistix.co.uk  Thu Jul  4 21:59:03 2013
From: chris at simplistix.co.uk (Chris Withers)
Date: Thu, 04 Jul 2013 20:59:03 +0100
Subject: [Python-Dev] lament for the demise of unbound methods
In-Reply-To: <CAPZV6o8n-Ti8q5Q_-wwqRXpCiP76GGaHHwmCW7T5MrvOycOepw@mail.gmail.com>
References: <51D55ACB.4050900@simplistix.co.uk>
	<CALFfu7CTE7J-Y1u+aMX_AUmui0OE2jVJ6yESY8NZ7B9_3BqUfQ@mail.gmail.com>
	<CAPZV6o8n-Ti8q5Q_-wwqRXpCiP76GGaHHwmCW7T5MrvOycOepw@mail.gmail.com>
Message-ID: <51D5D407.1070609@simplistix.co.uk>

On 04/07/2013 20:50, Benjamin Peterson wrote:
> 2013/7/4 Eric Snow <ericsnowcurrently at gmail.com>:
>>
>> You could always monkeypatch builtins.__build_class__ to add an attribute to
>> every "unbound method" pointing to the class.
>
> I would not reccomend that. __build_class__ is very internal and it's
> contract may change between versions.

Right, not to mention the fact that there's no way a library can ensure 
it monkeypatches that before the users of the library have created any 
classes.

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk

From victor.stinner at gmail.com  Thu Jul  4 22:12:46 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 4 Jul 2013 22:12:46 +0200
Subject: [Python-Dev] PEP 446: Add new parameters to configure the
 inherance of files and for non-blocking sockets
In-Reply-To: <kr4jgk$3uu$1@ger.gmane.org>
References: <CAMpsgwYxV3HPY0fodrUBTE8wCHpcJJ-_ESVt04HOno+CcQAF0w@mail.gmail.com>
	<kr4jgk$3uu$1@ger.gmane.org>
Message-ID: <CAMpsgwZKu9_jSWkFRv2xmSXGtsdZB83Yu9ieBwJmVX85-ZxjfA@mail.gmail.com>

2013/7/4 Tres Seaver <tseaver at palladion.com>:
> Not commenting on either the form or the substance (pun intended), but
> the word you want is "inheritance" -- "inherence" is a valid term[1], but
> would a good deal stranger notion to apply to a file descriptor. ;)

Oh... I don't know why I wrote "inherance", it was "inheritance" in the PEP 433.

Thanks, I fixed the typo in the PEP 466.

Victor

From cs at zip.com.au  Fri Jul  5 02:41:02 2013
From: cs at zip.com.au (Cameron Simpson)
Date: Fri, 5 Jul 2013 10:41:02 +1000
Subject: [Python-Dev] PEP 446: Add new parameters to configure the
 inherance of files and for non-blocking sockets
In-Reply-To: <CAMpsgwYxV3HPY0fodrUBTE8wCHpcJJ-_ESVt04HOno+CcQAF0w@mail.gmail.com>
References: <CAMpsgwYxV3HPY0fodrUBTE8wCHpcJJ-_ESVt04HOno+CcQAF0w@mail.gmail.com>
Message-ID: <20130705004102.GA19819@cskk.homeip.net>

First up I broadly like this.

You might want to make clear that the "blocking" parameter refers
only to the file creation calls (eg socket.socket) and not to the
file descriptor itself, and is not to be confused with the UNIX
O_NONBLOCK file descriptor flag (and whatever equivalent flag may
apply on Windows).

This is deducable from your PEP, but I was at first confused, and
initially expected get_blocking/set_blocking functions in New
Functions.

On 04Jul2013 13:03, Victor Stinner <victor.stinner at gmail.com> wrote:
| Other Changes
| -------------
| The ``subprocess.Popen`` class must clear the close-on-exec flag of file
| descriptors of the ``pass_fds`` parameter.

I would expect Popen and friends to need to both clear the flag to
get the descriptors across the fork() call, and _possibly_ to set
the flag again after the fork. Naively, I would expect the the flag
to be as it was before the Popen call, after the call.

This is not addressed.

Cheers,
-- 
Cameron Simpson <cs at zip.com.au>

Time is nature's way of keeping everything from happening at once.

From cs at zip.com.au  Fri Jul  5 07:20:36 2013
From: cs at zip.com.au (Cameron Simpson)
Date: Fri, 5 Jul 2013 15:20:36 +1000
Subject: [Python-Dev] PEP 446: Add new parameters to configure the
 inherance of files and for non-blocking sockets
In-Reply-To: <20130705004102.GA19819@cskk.homeip.net>
References: <20130705004102.GA19819@cskk.homeip.net>
Message-ID: <20130705052036.GA37545@cskk.homeip.net>

On 05Jul2013 10:41, I wrote:
| On 04Jul2013 13:03, Victor Stinner <victor.stinner at gmail.com> wrote:
| | Other Changes
| | -------------
| | The ``subprocess.Popen`` class must clear the close-on-exec flag of file
| | descriptors of the ``pass_fds`` parameter.
| 
| I would expect Popen and friends to need to both clear the flag to
| get the descriptors across the fork() call, and _possibly_ to set
| the flag again after the fork. Naively, I would expect the the flag
| to be as it was before the Popen call, after the call.

This is harder than I'd thought through.

Descriptors go across a fork() anyway, and after an exec() there is nobody to
change their state.

In order for the parent to restore the prior close-on-exec state
(if deemed the right thing to do) it would need to know when the
exec was done in the child.

You'd need something like an extra descriptor attached to a pipe
marked close-on-exec so that the parent could see EOF on the read
end after the exec, since the exec would close the write end.
tractable, but baroque.

So I'd have the PEP take a position on the post-exec() state of the
passed file descriptors, and if state is not going to be restored
it should outright say that passing an fd clears the close-on-exec
flag. Which is it does, but I think the doco should be pretty overt
about:

  - this side effect of clearing close-on-exec on these fds

  - that the file descriptor and its close-on-exec state is common
    between the parent and the child

  - that setting of close-on-exec on the fds in the parent could
    in principle comprimise the exec in the child if done too soon

If you decide state should get restored, the implementation gets a
bit fiddlier to detect the exec() in the child. I can't speak for
what might be required on non-UNIX.

Cheers,
-- 
Cameron Simpson <cs at zip.com.au>

Agree, for Law is costly. -- Very good advice to litigious Persons, founded
upon Reason and Experience; for many Times the Charges of a Suit exceed the
Value of the Thing in Dispute. - Bailey's dictionary, 1736

From victor.stinner at gmail.com  Fri Jul  5 08:24:11 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Fri, 5 Jul 2013 08:24:11 +0200
Subject: [Python-Dev] PEP 446: Add new parameters to configure the
 inherance of files and for non-blocking sockets
In-Reply-To: <20130705004102.GA19819@cskk.homeip.net>
References: <CAMpsgwYxV3HPY0fodrUBTE8wCHpcJJ-_ESVt04HOno+CcQAF0w@mail.gmail.com>
	<20130705004102.GA19819@cskk.homeip.net>
Message-ID: <CAMpsgwbfiGLYb+M=_d7Znbzg=1XX=Zk6esH0n=c9fYxvpEPH3A@mail.gmail.com>

2013/7/5 Cameron Simpson <cs at zip.com.au>:
> You might want to make clear that the "blocking" parameter refers
> only to the file creation calls (eg socket.socket) and not to the
> file descriptor itself, and is not to be confused with the UNIX
> O_NONBLOCK file descriptor flag (and whatever equivalent flag may
> apply on Windows).

The two following codes are the same:

s = socket.socket(..., blocking=False)

and

s = socket.socket(...)
s.setblocking(False)

Both set O_NONBLOCK flag (UNIX) or clear HANDLE_FLAG_INHERIT (Windows)
on the socket (which is a file descriptor).

socket.socket(..., blocking=False) cannot fail with EAGAIN or
EWOULDBLOCK (at least on Linux according to the manual page).

> This is deducable from your PEP, but I was at first confused, and
> initially expected get_blocking/set_blocking functions in New
> Functions.

socket objects already have get_blocking() and set_blocking() methods.
If we supported the blocking flag on any file descriptor, it would
make sense to add such function to the os module. But I explained in
the "Add blocking parameter for file descriptors and Windows
overlapped I/O" section why I only want to set/clear the blocking file
on sockets.

> I would expect Popen and friends to need to both clear the flag to
> get the descriptors across the fork() call, and _possibly_ to set
> the flag again after the fork. Naively, I would expect the the flag
> to be as it was before the Popen call, after the call.

As explained in the "Inheritance of file descriptors" section, the
close-on-exec flag has no effect on fork:

"The flag has no effect on fork(), all file descriptors are inherited
by the child process."

The close-on-exec flag is cleared after the fork and before the
exec(). Pseudo-code for Popen:

pid = os.fork()
if pid:
  return pid
# child process
for fd in pass_fds: os.set_cloexec(fd, False)
os.execv(...)

Victor

From tjreedy at udel.edu  Fri Jul  5 09:10:39 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 05 Jul 2013 03:10:39 -0400
Subject: [Python-Dev] Oddity in MISC/News on default
In-Reply-To: <CAP7+vJKR2=OK1cpdX_fEV=ay-pA1GLmc4xSZEjUYXLfLUn+5aw@mail.gmail.com>
References: <20130704210906.74208200@fsol>
	<CAP7+vJKF=ZHUrtB8bfj8oVNToGi1=KWDef9EXR4i7UjW7MYmYA@mail.gmail.com>
	<1372965599.2599.2.camel@fsol>
	<CAP7+vJKR2=OK1cpdX_fEV=ay-pA1GLmc4xSZEjUYXLfLUn+5aw@mail.gmail.com>
Message-ID: <kr5rhc$18e$1@ger.gmane.org>

On 7/4/2013 3:36 PM, Guido van Rossum wrote:
> Maybe the mistake is that the others aren't mentioned? Or perhaps
> everything before 3.4a1 should be dropped? I forget what kind of policy
> we have for this -- is it all changes in this branch or only changes
> unique to this branch?

It cannot be 'unique' because most bug fixes are not unique to any 
branch. News for 3.x should have all relevant changes to 3.x, and, I 
think, only those.

The question is whether the entries under 3.3.1rc1 in 3.4 News are 
changes unique to 3.3.1rc1 that should not be in News 3.4; changes to 
both 3.3 and 3.4 that are now listed twice, and should only be listed 
once; or changes to 3.4 that are now listed only under 3.3 but should 
instead be listed as 3.4 changes. I am not looking at the moment because 
is it 3am here.

-- 
Terry Jan Reedy


From fuzzyman at voidspace.org.uk  Fri Jul  5 11:23:36 2013
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Fri, 5 Jul 2013 11:23:36 +0200
Subject: [Python-Dev] lament for the demise of unbound methods
In-Reply-To: <CAP7+vJLeHPkq-UgFreNBHR67UVx7Tih3jZz-VjEdbtBZ2mp3tw@mail.gmail.com>
References: <51D55ACB.4050900@simplistix.co.uk>
	<CAP7+vJLVwzg0y-H3TbbbAbfsL8Bcmx3Y3=57uBA+O0hARhO8OA@mail.gmail.com>
	<51D5A5DF.3000103@simplistix.co.uk>
	<CAP7+vJLeHPkq-UgFreNBHR67UVx7Tih3jZz-VjEdbtBZ2mp3tw@mail.gmail.com>
Message-ID: <761E2333-74A7-4F50-9943-E8148189507B@voidspace.org.uk>


On 4 Jul 2013, at 19:00, Guido van Rossum <guido at python.org> wrote:

> Thanks for the code pointers. So it's all about monkeypatching. :-) I have only a little sympathy, as there still seems to be a way to do this, it's just less convenient. Too bad.
> 


I've also lamented the death of bound methods in Python 3 for mock "autospeccing". Autospec introspects objects and provides mock objects with the same attributes - and with the same method signatures. For methods it needs to trim the first argument (because instances are called externally without self of course). Not being able to tell the difference between a module level function and an unbound method caused some pain then. (I worked round it by flowing the information about where the object came from through the code but it did add ugliness).

Michael


> --Guido
> 
> On Thu, Jul 4, 2013 at 9:42 AM, Chris Withers <chris at simplistix.co.uk> wrote:
> Hi Guido,
> 
> I've bumped into this a couple of times.
> 
> First time was when I wanted to know whether what I had was a classmethod, staticmethod or normal method here:
> 
> https://github.com/Simplistix/testfixtures/blob/master/testfixtures/replace.py#L59
> 
> This resulted in having to trawl through __dict__ here:
> 
> https://github.com/Simplistix/testfixtures/blob/master/testfixtures/resolve.py#L17
> 
> ...rather than just using getattr.
> 
> I bumped into it again, yesterday, trying to add support for classes to this lightweight dependency injection framework I'm developing:
> 
> https://github.com/Simplistix/mush/blob/master/tests/test_runner.py#L189
> 
> Here's my local copy of that test:
> 
> https://gist.github.com/cjw296/db64279c69cdc0c5e112
> 
> The workaround I was playing with this morning is a wrapper so that I know I have a class method, although what I really want to write at this line is:
> 
> https://gist.github.com/cjw296/db64279c69cdc0c5e112#file-gistfile1-txt-L40
> 
> runner = Runner(T0, C1.meth, C2.meth1, C2.meth2)
> 
> ...but if I do that, how can the runner know that what it gets for its second argument is a class method of C1?
> (which is this case means that it should do C1().meth() rather than C1.meth())
> 
> cheers,
> 
> Chris
> 
> 
> On 04/07/2013 17:25, Guido van Rossum wrote:
> Chris, what do you want to do with the knowledge you are seeking?
> 
> --Guido van Rossum (sent from Android phone)
> 
> On Jul 4, 2013 4:28 AM, "Chris Withers" <chris at simplistix.co.uk
> <mailto:chris at simplistix.co.uk>> wrote:
> 
>     Hi All,
> 
>     In Python 2, I can figure out whether I have a method or a function,
>     and, more importantly, for an unbound method, I can figure out what
>     class the method belongs to:
> 
>      >>> class MyClass(object):
>     ...   def method(self): pass
>     ...
>      >>> MyClass.method
>     <unbound method MyClass.method>
>      >>> MyClass.method.im_class
>     <class '__main__.MyClass'>
> 
>     There doesn't appear to be any way in Python 3 to do this, which is
>     a little surprising and frustrating...
> 
>     What am I missing here?
> 
>     Chris
> 
>     --
>     Simplistix - Content Management, Batch Processing & Python Consulting
>                  - http://www.simplistix.co.uk
>     _________________________________________________
>     Python-Dev mailing list
>     Python-Dev at python.org <mailto:Python-Dev at python.org>
>     http://mail.python.org/__mailman/listinfo/python-dev
>     <http://mail.python.org/mailman/listinfo/python-dev>
>     Unsubscribe:
>     http://mail.python.org/__mailman/options/python-dev/__guido%40python.org
>     <http://mail.python.org/mailman/options/python-dev/guido%40python.org>
> 
> 
> ______________________________________________________________________
> This email has been scanned by the Symantec Email Security.cloud service.
> For more information please visit http://www.symanteccloud.com
> ______________________________________________________________________
> 
> -- 
> Simplistix - Content Management, Batch Processing & Python Consulting
>             - http://www.simplistix.co.uk
> 
> 
> 
> -- 
> --Guido van Rossum (python.org/~guido) _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk


--
http://www.voidspace.org.uk/


May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing 
http://www.sqlite.org/different.html






From martin at v.loewis.de  Fri Jul  5 12:07:15 2013
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 05 Jul 2013 12:07:15 +0200
Subject: [Python-Dev] lament for the demise of unbound methods
In-Reply-To: <761E2333-74A7-4F50-9943-E8148189507B@voidspace.org.uk>
References: <51D55ACB.4050900@simplistix.co.uk>
	<CAP7+vJLVwzg0y-H3TbbbAbfsL8Bcmx3Y3=57uBA+O0hARhO8OA@mail.gmail.com>
	<51D5A5DF.3000103@simplistix.co.uk>
	<CAP7+vJLeHPkq-UgFreNBHR67UVx7Tih3jZz-VjEdbtBZ2mp3tw@mail.gmail.com>
	<761E2333-74A7-4F50-9943-E8148189507B@voidspace.org.uk>
Message-ID: <51D69AD3.7050003@v.loewis.de>

Am 05.07.13 11:23, schrieb Michael Foord:
> I've also lamented the death of bound methods in Python 3 for mock
> "autospeccing". Autospec introspects objects and provides mock
> objects with the same attributes - and with the same method
> signatures.

I wonder why you need to figure out the signatures in advance.
Can you just wait until the function is actually used, and then
process the parameters as you get them?

Regards,
Martin

From lukasz at langa.pl  Fri Jul  5 12:26:44 2013
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Fri, 5 Jul 2013 12:26:44 +0200
Subject: [Python-Dev] lament for the demise of unbound methods
In-Reply-To: <51D69AD3.7050003@v.loewis.de>
References: <51D55ACB.4050900@simplistix.co.uk>
	<CAP7+vJLVwzg0y-H3TbbbAbfsL8Bcmx3Y3=57uBA+O0hARhO8OA@mail.gmail.com>
	<51D5A5DF.3000103@simplistix.co.uk>
	<CAP7+vJLeHPkq-UgFreNBHR67UVx7Tih3jZz-VjEdbtBZ2mp3tw@mail.gmail.com>
	<761E2333-74A7-4F50-9943-E8148189507B@voidspace.org.uk>
	<51D69AD3.7050003@v.loewis.de>
Message-ID: <076A4BF7-C8CC-4325-985F-BB905B174F01@langa.pl>

On 5 lip 2013, at 12:07, Martin v. L?wis <martin at v.loewis.de> wrote:

> I wonder why you need to figure out the signatures in advance.
> Can you just wait until the function is actually used, and then
> process the parameters as you get them?
> 

My guess is that Michael's design lets mock objects be introspected as well, i.e. they don't appear as magical as they really are to the user code.

-- 
Best regards,
?ukasz Langa

WWW: http://lukasz.langa.pl/
Twitter: @llanga
IRC: ambv on #python-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130705/f3a4cd6a/attachment.html>

From cf.natali at gmail.com  Fri Jul  5 12:26:48 2013
From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=)
Date: Fri, 5 Jul 2013 12:26:48 +0200
Subject: [Python-Dev] PEP 446: Add new parameters to configure the
 inherance of files and for non-blocking sockets
In-Reply-To: <CAMpsgwZo14i6NTmVkm3ZZkTJeWc71XdaWzH-Up19iGAOW1H7Hg@mail.gmail.com>
References: <CAMpsgwYxV3HPY0fodrUBTE8wCHpcJJ-_ESVt04HOno+CcQAF0w@mail.gmail.com>
	<CAMpsgwZo14i6NTmVkm3ZZkTJeWc71XdaWzH-Up19iGAOW1H7Hg@mail.gmail.com>
Message-ID: <CAH_1eM11MdRvEWFc=PTA3a_34aaJmZ_eDkxUY77vJBb1gRAKmQ@mail.gmail.com>

2013/7/4 Victor Stinner <victor.stinner at gmail.com>:
> Even if the PEP 433 was not explicitly rejected, no consensus could be
> reached. I didn't want to loose all my work on this PEP and so I'm
> proposing something new which should make everbody agrees :-)

Thanks Victor, I think this one is perfectly fine!

cf

From martin at v.loewis.de  Fri Jul  5 12:26:06 2013
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 05 Jul 2013 12:26:06 +0200
Subject: [Python-Dev] lament for the demise of unbound methods
In-Reply-To: <51D5A5DF.3000103@simplistix.co.uk>
References: <51D55ACB.4050900@simplistix.co.uk>
	<CAP7+vJLVwzg0y-H3TbbbAbfsL8Bcmx3Y3=57uBA+O0hARhO8OA@mail.gmail.com>
	<51D5A5DF.3000103@simplistix.co.uk>
Message-ID: <51D69F3E.8060103@v.loewis.de>

Am 04.07.13 18:42, schrieb Chris Withers:
> Hi Guido,
> 
> I've bumped into this a couple of times.
> 
> First time was when I wanted to know whether what I had was a
> classmethod, staticmethod or normal method here:
> 
> https://github.com/Simplistix/testfixtures/blob/master/testfixtures/replace.py#L59
> 
> 
> This resulted in having to trawl through __dict__ here:
> 
> https://github.com/Simplistix/testfixtures/blob/master/testfixtures/resolve.py#L17

You could use __getattribute__ instead:

>>> class A:
...   @staticmethod
...   def s():
...     pass
...   @classmethod
...   def c(cl):
...     pass
...   def r(self):
...     pass
...
>>> A.__getattribute__(A,'s')
<staticmethod object at 0x100937828>
>>> A.__getattribute__(A,'c')
<classmethod object at 0x100937860>
>>> A.__getattribute__(A,'r')
<function A.r at 0x100938378>

Regards,
Martin


From cs at zip.com.au  Fri Jul  5 11:15:59 2013
From: cs at zip.com.au (Cameron Simpson)
Date: Fri, 5 Jul 2013 19:15:59 +1000
Subject: [Python-Dev] PEP 446: Add new parameters to configure the
 inherance of files and for non-blocking sockets
In-Reply-To: <CAMpsgwbfiGLYb+M=_d7Znbzg=1XX=Zk6esH0n=c9fYxvpEPH3A@mail.gmail.com>
References: <CAMpsgwbfiGLYb+M=_d7Znbzg=1XX=Zk6esH0n=c9fYxvpEPH3A@mail.gmail.com>
Message-ID: <20130705091559.GA65340@cskk.homeip.net>

On 05Jul2013 08:24, Victor Stinner <victor.stinner at gmail.com> wrote:
| 2013/7/5 Cameron Simpson <cs at zip.com.au>:
| > You might want to make clear that the "blocking" parameter refers
| > only to the file creation calls (eg socket.socket) and not to the
| > file descriptor itself, and is not to be confused with the UNIX
| > O_NONBLOCK file descriptor flag (and whatever equivalent flag may
| > apply on Windows).
| 
| The two following codes are the same:
| 
| s = socket.socket(..., blocking=False)
| and
| s = socket.socket(...)
| s.setblocking(False)
| 
| Both set O_NONBLOCK flag (UNIX)

Oh, how embarassing.

| or clear HANDLE_FLAG_INHERIT (Windows)
| on the socket (which is a file descriptor).
| 
| socket.socket(..., blocking=False) cannot fail with EAGAIN or
| EWOULDBLOCK (at least on Linux according to the manual page).

Fair enough.

| > This is deducable from your PEP, but I was at first confused, and
| > initially expected get_blocking/set_blocking functions in New
| > Functions.
| 
| socket objects already have get_blocking() and set_blocking() methods.

Ah, ok then. As far as it goes. Shouldn't there be a general purpose
os.get_blocking() and os.set_blocking() functions that operate on
any file descriptor, paralleling os.get_cloexec() and os.set_cloexec()?
They're not socket specific operations in principle, merely commonly
used with sockets.

| If we supported the blocking flag on any file descriptor, it would
| make sense to add such function to the os module. But I explained in
| the "Add blocking parameter for file descriptors and Windows
| overlapped I/O" section why I only want to set/clear the blocking file
| on sockets.

But WHY? I think socket file decriptors should be treated little
differently from other file descriptors. Certainly a stream socket
connection is outstanding like other unseekable file descriptors
in many respects. Just "only wanting it on sockets" seems a bit
special purpose.

If you're going address O_NONBLOCK in this proposal in addition to
the close-on-exec flag, it should be general purpose.

Otherwise, I think it should be separated out into a separate
proposal if you're proposing it just for sockets; make this proposal
just close-on-exec and forget the blocking stuff for this specific
PEP.

| > I would expect Popen and friends to need to both clear the flag to
| > get the descriptors across the fork() call, and _possibly_ to set
| > the flag again after the fork. Naively, I would expect the the flag
| > to be as it was before the Popen call, after the call.
| 
| As explained in the "Inheritance of file descriptors" section, the
| close-on-exec flag has no effect on fork:
| 
| "The flag has no effect on fork(), all file descriptors are inherited
| by the child process."

Agreed, see my second post where I explained I'd misthought it.
However, as stated there, I think the side effects should be fairly
overtly stated in the docuemntation.

| The close-on-exec flag is cleared after the fork and before the
| exec(). Pseudo-code for Popen:
| 
| pid = os.fork()
| if pid:
|   return pid
| # child process
| for fd in pass_fds: os.set_cloexec(fd, False)
| os.execv(...)

Fine. No state restore is fine with me. I think it should be as
clear as possible in the doco.

Cheers,
-- 
Cameron Simpson <cs at zip.com.au>

Experience is what you have got after you needed it.

From status at bugs.python.org  Fri Jul  5 18:07:31 2013
From: status at bugs.python.org (Python tracker)
Date: Fri,  5 Jul 2013 18:07:31 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20130705160731.7655E56A3B@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2013-06-28 - 2013-07-05)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    4072 ( +3)
  closed 26066 (+39)
  total  30138 (+42)

Open issues with patches: 1803 


Issues opened (32)
==================

#5308: cannot marshal objects with more than 2**31 elements
http://bugs.python.org/issue5308  reopened by christian.heimes

#13153: IDLE crashes when pasting non-BMP unicode char on Py3
http://bugs.python.org/issue13153  reopened by serhiy.storchaka

#18324: set_payload does not handle binary payloads correctly
http://bugs.python.org/issue18324  opened by r.david.murray

#18325: test_kqueue fails in OpenBSD
http://bugs.python.org/issue18325  opened by Federico.Schwindt

#18326: Not Clear Docs
http://bugs.python.org/issue18326  opened by icedream91

#18327: swapped arguments in compatible_for_assignment()?
http://bugs.python.org/issue18327  opened by christian.heimes

#18329: for line in socket.makefile() speed degradation
http://bugs.python.org/issue18329  opened by mmarkk

#18330: Fix idlelib.PyShell.build_subprocess_arglist  use of __import_
http://bugs.python.org/issue18330  opened by terry.reedy

#18331: Document that runpy.run_path and run_module copy the module gl
http://bugs.python.org/issue18331  opened by Drekin

#18334: type(name, bases, dict) does not call metaclass' __prepare__ a
http://bugs.python.org/issue18334  opened by Nikratio

#18335: Add textwrap.dedent, .indent, as str methods.
http://bugs.python.org/issue18335  opened by terry.reedy

#18336: codecs: Link to readline module (history) instead of fd.readli
http://bugs.python.org/issue18336  opened by guettli

#18338: python --version should send output to STDOUT
http://bugs.python.org/issue18338  opened by jaalto

#18340: float related test has problem with Denormal Flush to Zero com
http://bugs.python.org/issue18340  opened by V.E.O

#18342: Use the repr of a module name for ModuleNotFoundError in ceval
http://bugs.python.org/issue18342  opened by brett.cannon

#18344: _bufferedreader_read_all() may leak reference to data
http://bugs.python.org/issue18344  opened by christian.heimes

#18345: logging: file creation options with FileHandler and friends
http://bugs.python.org/issue18345  opened by pitrou

#18348: Additional code pages for EBCDIC
http://bugs.python.org/issue18348  opened by roskakori

#18349: argparse usage should preserve () in metavars such as range(20
http://bugs.python.org/issue18349  opened by paul.j3

#18351: Incorrect variable name in importlib._bootstrap._get_sourcefil
http://bugs.python.org/issue18351  opened by brett.cannon

#18352: collections.Counter with added attributes are not deepcopied p
http://bugs.python.org/issue18352  opened by Olivier.Gagnon

#18353: PyUnicode_WRITE_CHAR macro definition missing
http://bugs.python.org/issue18353  opened by Wolf.Ihlenfeldt

#18354: http://www.python.org/doc/ has outdated note
http://bugs.python.org/issue18354  opened by dholth

#18355: Merge super() guide into documentation
http://bugs.python.org/issue18355  opened by Russkel

#18356: help(numpy) causes segfault on exit
http://bugs.python.org/issue18356  opened by Leeward

#18357: add tests for dictview set difference operations
http://bugs.python.org/issue18357  opened by frasertweedale

#18360: Won't install.  Keeps telling me DLL is missing.
http://bugs.python.org/issue18360  opened by daleastar

#18361: Move dev-in-a-box to os.cpu_count()
http://bugs.python.org/issue18361  opened by brett.cannon

#18362: Make build_cpython.py from dev-in-a-box work outside of a box
http://bugs.python.org/issue18362  opened by brett.cannon

#18363: Change use of acronym tag in devinabox index.html to abbr
http://bugs.python.org/issue18363  opened by brett.cannon

#18364: Remove _not_found hack from importlib
http://bugs.python.org/issue18364  opened by brett.cannon

#18365: Idle: mock Text class and test thereof
http://bugs.python.org/issue18365  opened by terry.reedy



Most recent 15 issues with no replies (15)
==========================================

#18364: Remove _not_found hack from importlib
http://bugs.python.org/issue18364

#18363: Change use of acronym tag in devinabox index.html to abbr
http://bugs.python.org/issue18363

#18362: Make build_cpython.py from dev-in-a-box work outside of a box
http://bugs.python.org/issue18362

#18361: Move dev-in-a-box to os.cpu_count()
http://bugs.python.org/issue18361

#18355: Merge super() guide into documentation
http://bugs.python.org/issue18355

#18349: argparse usage should preserve () in metavars such as range(20
http://bugs.python.org/issue18349

#18348: Additional code pages for EBCDIC
http://bugs.python.org/issue18348

#18345: logging: file creation options with FileHandler and friends
http://bugs.python.org/issue18345

#18342: Use the repr of a module name for ModuleNotFoundError in ceval
http://bugs.python.org/issue18342

#18336: codecs: Link to readline module (history) instead of fd.readli
http://bugs.python.org/issue18336

#18330: Fix idlelib.PyShell.build_subprocess_arglist  use of __import_
http://bugs.python.org/issue18330

#18325: test_kqueue fails in OpenBSD
http://bugs.python.org/issue18325

#18324: set_payload does not handle binary payloads correctly
http://bugs.python.org/issue18324

#18320: idle, pydoc search python on wrong path at make altinstall wit
http://bugs.python.org/issue18320

#18319: gettext() cannot find translations with plural forms
http://bugs.python.org/issue18319



Most recent 15 issues waiting for review (15)
=============================================

#18357: add tests for dictview set difference operations
http://bugs.python.org/issue18357

#18352: collections.Counter with added attributes are not deepcopied p
http://bugs.python.org/issue18352

#18351: Incorrect variable name in importlib._bootstrap._get_sourcefil
http://bugs.python.org/issue18351

#18349: argparse usage should preserve () in metavars such as range(20
http://bugs.python.org/issue18349

#18344: _bufferedreader_read_all() may leak reference to data
http://bugs.python.org/issue18344

#18338: python --version should send output to STDOUT
http://bugs.python.org/issue18338

#18334: type(name, bases, dict) does not call metaclass' __prepare__ a
http://bugs.python.org/issue18334

#18324: set_payload does not handle binary payloads correctly
http://bugs.python.org/issue18324

#18321: Multivolume support in tarfile module
http://bugs.python.org/issue18321

#18317: gettext: DoS via crafted Plural-Forms
http://bugs.python.org/issue18317

#18309: Make python slightly more relocatable
http://bugs.python.org/issue18309

#18308: checkRecvmsgAddress wrong in test_socket.py (AIX failures)
http://bugs.python.org/issue18308

#18305: [patch] Fast sum() for non-numbers
http://bugs.python.org/issue18305

#18301: In itertools.chain.from_iterable() there is no cls argument
http://bugs.python.org/issue18301

#18294: zlib module is not completly 64-bit safe
http://bugs.python.org/issue18294



Top 10 most discussed issues (10)
=================================

#18329: for line in socket.makefile() speed degradation
http://bugs.python.org/issue18329  18 msgs

#18305: [patch] Fast sum() for non-numbers
http://bugs.python.org/issue18305   7 msgs

#18340: float related test has problem with Denormal Flush to Zero com
http://bugs.python.org/issue18340   7 msgs

#18226: IDLE Unit test for FormatParagrah.py
http://bugs.python.org/issue18226   6 msgs

#18338: python --version should send output to STDOUT
http://bugs.python.org/issue18338   6 msgs

#13582: IDLE and pythonw.exe stderr problem
http://bugs.python.org/issue13582   5 msgs

#18335: Add textwrap.dedent, .indent, as str methods.
http://bugs.python.org/issue18335   5 msgs

#14455: plistlib unable to read json and binary plist files
http://bugs.python.org/issue14455   4 msgs

#17206: Py_XDECREF() expands its argument multiple times
http://bugs.python.org/issue17206   4 msgs

#18334: type(name, bases, dict) does not call metaclass' __prepare__ a
http://bugs.python.org/issue18334   4 msgs



Issues closed (37)
==================

#4199: add shorthand global and nonlocal statements
http://bugs.python.org/issue4199  closed by gregory.p.smith

#7136: Idle File Menu Option Improvement
http://bugs.python.org/issue7136  closed by terry.reedy

#11016: Re-implementation of the stat module in C
http://bugs.python.org/issue11016  closed by christian.heimes

#11185: test_wait4 error on AIX
http://bugs.python.org/issue11185  closed by pitrou

#12520: spurious output in test_warnings
http://bugs.python.org/issue12520  closed by haypo

#12716: Reorganize os docs for files/dirs/fds
http://bugs.python.org/issue12716  closed by akuchling

#14206: multiprocessing.Queue documentation is lacking important detai
http://bugs.python.org/issue14206  closed by sbt

#17097: multiprocessing BaseManager serve_client() does not check EINT
http://bugs.python.org/issue17097  closed by sbt

#17261: multiprocessing.manager BaseManager cannot return proxies from
http://bugs.python.org/issue17261  closed by sbt

#17273: Pool methods can only be used by parent process.
http://bugs.python.org/issue17273  closed by sbt

#17914: add os.cpu_count()
http://bugs.python.org/issue17914  closed by neologix

#18081: test_logging failure in WarningsTest on buildbots
http://bugs.python.org/issue18081  closed by terry.reedy

#18103: Create a GUI test framework for Idle
http://bugs.python.org/issue18103  closed by terry.reedy

#18155: csv.Sniffer.has_header doesn't escape characters used in regex
http://bugs.python.org/issue18155  closed by r.david.murray

#18189: IDLE Improvements: Unit test for Delegator.py
http://bugs.python.org/issue18189  closed by terry.reedy

#18224: pyvenv pydoc.py script causing AttributeErrors on Windows
http://bugs.python.org/issue18224  closed by python-dev

#18237: unittest.assertRaisesRegex(p) example is wrong in docs
http://bugs.python.org/issue18237  closed by terry.reedy

#18240: hmac unnecessarily restricts input to "bytes"
http://bugs.python.org/issue18240  closed by christian.heimes

#18242: IDLE should not be replacing warnings.formatwarning
http://bugs.python.org/issue18242  closed by terry.reedy

#18244: Adopt C3-based linearization for improved ABC support in funct
http://bugs.python.org/issue18244  closed by lukasz.langa

#18288: Idle 2.7: Run Module does not set __file__
http://bugs.python.org/issue18288  closed by terry.reedy

#18302: test_multiprocessing: test.support.import_module() does not ig
http://bugs.python.org/issue18302  closed by brett.cannon

#18312: "make distclean" deletes files under .hg directory
http://bugs.python.org/issue18312  closed by python-dev

#18315: Fix fileinput doc and docstrings (add 'buffer' in 2.7)
http://bugs.python.org/issue18315  closed by terry.reedy

#18322: test_stat nits
http://bugs.python.org/issue18322  closed by pitrou

#18328: Use after free in pystate.c
http://bugs.python.org/issue18328  closed by christian.heimes

#18332: _posix_listdir may leak FD
http://bugs.python.org/issue18332  closed by christian.heimes

#18333: Memory leak in _pickle.c:Unpickler_set_memo()
http://bugs.python.org/issue18333  closed by christian.heimes

#18337: codecs: StremReader readline() breaks on undocumented characte
http://bugs.python.org/issue18337  closed by serhiy.storchaka

#18339: Segfault in Unpickler_set_memo()
http://bugs.python.org/issue18339  closed by christian.heimes

#18341: enhancements zlib.compress/decompress to accept Py_buffer
http://bugs.python.org/issue18341  closed by jcea

#18343: faulthandler_register_py() may use _Py_sighandler_t previous w
http://bugs.python.org/issue18343  closed by haypo

#18346: Backporting of atomic file rename to Python 2.7
http://bugs.python.org/issue18346  closed by r.david.murray

#18347: ElementTree corrupts cAse of closing tags when html method is 
http://bugs.python.org/issue18347  closed by christian.heimes

#18350: Python 3 unittest framework broken?
http://bugs.python.org/issue18350  closed by amaury.forgeotdarc

#18358: update links to Apple Style Guide
http://bugs.python.org/issue18358  closed by ronaldoussoren

#18359: pickle fail to handle some class with __new__
http://bugs.python.org/issue18359  closed by amaury.forgeotdarc

From victor.stinner at gmail.com  Fri Jul  5 19:03:35 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Fri, 5 Jul 2013 19:03:35 +0200
Subject: [Python-Dev] PEP 446: Add new parameters to configure the
 inherance of files and for non-blocking sockets
In-Reply-To: <20130705091559.GA65340@cskk.homeip.net>
References: <CAMpsgwbfiGLYb+M=_d7Znbzg=1XX=Zk6esH0n=c9fYxvpEPH3A@mail.gmail.com>
	<20130705091559.GA65340@cskk.homeip.net>
Message-ID: <CAMpsgwYjVGxyF4PT=g33R+SoU1tRfXT5yFwgUt3KTpnn4cHcXA@mail.gmail.com>

2013/7/5 Cameron Simpson <cs at zip.com.au>:
> | Both set O_NONBLOCK flag (UNIX)
>
> Oh, how embarassing.

You said that the PEP is not cristal clear. Do you have a suggestion
to make it more clear?

Should I mention that the close-on-exec flag is O_CLOEXEC on UNIX, and
HANDLE_FLAG_INHERIT on Windows? (except that HANDLE_FLAG_INHERIT set
means inheritable, whereas O_CLOEXEC set means *not* inheritable)

> | > This is deducable from your PEP, but I was at first confused, and
> | > initially expected get_blocking/set_blocking functions in New
> | > Functions.
> |
> | socket objects already have get_blocking() and set_blocking() methods.
>
> Ah, ok then. As far as it goes. Shouldn't there be a general purpose
> os.get_blocking() and os.set_blocking() functions that operate on
> any file descriptor, paralleling os.get_cloexec() and os.set_cloexec()?

I didn't propose to add these two functions, because I'm not sure that
they are really useful.

We can add os.get_blocking() and os.set_blocking() on UNIX, but these
functions would not be available on Windows. On Windows,
os.set_blocking() only makes sense for sockets, and sockets already
have a set_blocking() method. On UNIX, it makes sense because
set_blocking(fd, True) can be implemented as a single syscall on some
platforms (using ioctl), whereas most developers implement it using
two syscalls (fcntl to get current flags, and fcntl to set the
O_NONBLOCK flag) maybe because it is more portable.

But we cannot add a new "blocking" parameter to all functions creating
file descriptors, like open(), because of Windows. Or do you like the
"raise NotImplementedError on Windows" option?

> But WHY? I think socket file decriptors should be treated little
> differently from other file descriptors.

Because of Windows. On Windows, sockets and files are two different
things. sockets have no "file descriptor", they have a HANDLE. For
example, sockobj.fileno() on Windows returns a huge number, not
something like 3 or 10.

The C type HANDLE is larger than a file descriptor on Windows 64-bit
(sizeof(void*) > sizeof(int)) and so functions must be modified to use
a wider type (to parse their argument), and must support the HANDLE
type (_open_osfhandle() can be used to have a single version of the
code).

> Otherwise, I think it should be separated out into a separate
> proposal if you're proposing it just for sockets; make this proposal
> just close-on-exec and forget the blocking stuff for this specific
> PEP.

The reason for addressing close-on-exec and blocking parameters in the
same PEP is that new versions of operating systems support setting the
two flags at the creation a new file descriptor and a new socket. Read
the article linked at the end of the PEP for the background:
http://udrepper.livejournal.com/20407.html

The Python call socket.socket(..., cloexec=True, blocking=False) only
calls one syscall on Linux >= 2.6.27.

> | > I would expect Popen and friends to need to both clear the flag to
> | > get the descriptors across the fork() call, and _possibly_ to set
> | > the flag again after the fork. Naively, I would expect the the flag
> | > to be as it was before the Popen call, after the call.
> |
> | As explained in the "Inheritance of file descriptors" section, the
> | close-on-exec flag has no effect on fork:
> |
> | "The flag has no effect on fork(), all file descriptors are inherited
> | by the child process."
>
> Agreed, see my second post where I explained I'd misthought it.
> However, as stated there, I think the side effects should be fairly
> overtly stated in the docuemntation.

Sorry, I don't understand. Which "side effect"?

The close-on-exec flag only affects inheritance of file descriptors at
the execv() syscall, not at fork(). And execv() can be called without
fork().

Popen must clear close-on-exec flag on files from its pass_fds
parameter for convinience. It would be surprising to not inherit a
file descriptor passed to Popen in pass_fds, don't you think so?

os.execv() has no pass_fds parameter, and it is a thin wrapper on the
syscall. Popen is a high-level API, that's why it prepares more things
before calling the new program.

> | The close-on-exec flag is cleared after the fork and before the
> | exec(). Pseudo-code for Popen:
> |
> | pid = os.fork()
> | if pid:
> |   return pid
> | # child process
> | for fd in pass_fds: os.set_cloexec(fd, False)
> | os.execv(...)
>
> Fine. No state restore is fine with me. I think it should be as
> clear as possible in the doco.

I don't understand. I already wrote "The flag has no effect on fork(),
all file descriptors are inherited by the child process" in the PEP.
It is not enough? Do you have a suggestion to explain it better?

Victor

From benoit at marmelune.net  Fri Jul  5 09:38:36 2013
From: benoit at marmelune.net (=?UTF-8?B?QmVub8OudCBCcnlvbg==?=)
Date: Fri, 05 Jul 2013 09:38:36 +0200
Subject: [Python-Dev] PEP 423 : naming conventions and recipes related
 to packaging
In-Reply-To: <20120627125055.196efe9f@pitrou.net>
References: <4FEACD9D.8090208@marmelune.net>
	<20120627125055.196efe9f@pitrou.net>
Message-ID: <51D677FC.8050107@marmelune.net>

Hi!

Attached is a an updated proposal for PEP 423.
You can also find it online at https://gist.github.com/benoitbryon/2815051

I am attending at EuroPython 2013 in Florence. Isn't it a great 
opportunity to get feedback and discuss about a PEP? I registered an 
open-space session and a lightning-talk today!

Some notes about the update...

The main point that was discussed in the previous proposal was the 
"top-level namespace relates to code ownership rule". Here is a quote 
from Antoine Pitrou:

Le 27/06/2012 12:50, Antoine Pitrou a ?crit :
> On Wed, 27 Jun 2012 11:08:45 +0200
> Beno?t Bryon<benoit at marmelune.net>  wrote:
>> Hi,
>>
>> Here is an informational PEP proposal:
>> http://hg.python.org/peps/file/52767ab7e140/pep-0423.txt
>>
>> Could you review it for style, consistency and content?
> There is one Zen principle this PEP is missing:
>
> Flat is better than nested.
>
> This PEP seems to promote the practice of having a top-level namespace
> denote ownership. I think it should do the reverse: promote
> meaningful top-level packages (e.g. "sphinx") as standard practice, and
> allow an exception for when a piece of software is part of a larger
> organizational body.

So, the main change in the proposal I'm sending today is the removal of 
this "ownership" rule.
It has been replaced by "Use a single namespace (except special cases)".

Some additional changes have been performed, such as removal of some 
sections about "opportunity" or "promote migrations". I also added a 
"Rationale" section where I pointed out some issues related to naming.

The PEP has been marked as "deferred" because it was inactive and it is 
partly related to PEP 426. I left this deferred state.

I am aware that some links in the PEP are broken... I will fix them 
later. My very first motivation is to get feedback about the "big" 
changes in the PEP. I wanted the update to be sent before 
EuroPython-2013's open-space session. I guess a detailed review would be 
nice anyway, for links, style, grammar...

Also, I wonder whether the PEP could be shortened or not. Sometimes I 
cannot find straightforward words to explain things, so perhaps someone 
with better skills in english language could help. Or maybe some parts, 
such as the "How to rename a project" section, could be moved in other 
documents.

Regards,

Beno?t
-------------- next part --------------
PEP: 423
Title: Naming conventions and recipes related to packaging
Version: $Revision$
Last-Modified: $Date$
Author: Beno?t Bryon <benoit at marmelune.net>
Discussions-To: <distutils-sig at python.org>
Status: Deferred
Type: Informational
Content-Type: text/x-rst
Created: 24-May-2012
Post-History: 5-Jul-2013


Abstract
========

This document deals with:

* names of Python projects,
* names of Python packages or modules being distributed,
* namespace packages.

It provides guidelines and recipes for distribution authors:

* new projects should follow the `guidelines <#overview>`_ below.

* existing projects should be aware of these guidelines and can
  follow `specific recipes for existing projects
  <#how-to-apply-naming-guidelines-on-existing-projects>`_.


PEP Deferral
============

Further consideration of this PEP has been deferred at least until
after PEP 426 (package metadata 2.0) and related updates have been
resolved.


Rationale: issues related to names
==================================

For a long time, there have been no official reference on the "how to
choose names" topic in the Python community. As a consequence, the
Python package index (`PyPI`_) contains many naming patterns.

The fact is that this heterogeneity causes some issues. Some of them
are described below.

.. note:: Examples were taken on July 2013.

Naming things is a hard task, and naming Python projects or packages
is not an exception. The purpose of this PEP is to help project
authors to avoid common traps about naming, and focus on valuable
things.

Clashes
-------

Projects names are unique on `PyPI`_. But names of distributed things
(packages, modules) are not. And there are clashes.

As an example, "pysendfile" and "django-sendfile" projects both
distribute a "sendfile" package. Users cannot use both in an
environment.

Deep nested hierarchies
-----------------------

Deep nested namespaces mean deep nested hierarchies. It obfuscates
valuable project contents.

As an example, with "plone.app.content" you get a deeply nested
directory hierarchy:

.. code:: text

   plone/
   ??? app/
    ?? ??? command/
           ??? ... valuable code is here...

Whereas, with flat packages like "sphinx", you have valuable
code near the top-level directory:

.. code:: text

   sphinx/
   ??? ... valuable code is here...

Unrelated namespaces
--------------------

When project names are made of nested namespaces, and these
namespaces are not strongly related, then there is confusion.

As an example, it is not obvious that "zc.rst2" project is a general
Python project, not tied to "zc" (Zope Corporation), but related to
docutils' reStructuredText. As a consequence, some users discard
"zc.rst2" project, because they think it is specific to "zc".

This issue occurs with branded namespaces, i.e. when top-level
namespace relates to an organization and various projects are put
into this namespace.

This issue also occurs when namespaces are used for categorization.

Inconsistent names
------------------

When project and distributed packages do not share a single name,
users have to think when they use names (install project or import
package).

As an example, which package does "django-pipeline" project
distributes? Is it "djangopipeline", "django_pipeline" or "pipeline"?
The answer is not obvious and the pattern varies depending on
project. Users have to remember or search for the name to use.

There is no obvious bijection between project name and
package/module name. It means that a user have to remember at least
two names (project and package) whereas one could be enough.

This fact is a cause of clashes, because when you register a project
name on `PyPI`_, you have no idea about package name clashes, because
package names are not predictable.


Terminology
===========

Reference of terminology used in this PEP is `packaging terminology
in Python documentation`_.


Relationship with other PEPs
============================

* `PEP 8`_ deals with code style guide, including names of Python
  packages and modules. It covers syntax of package/modules names.

* `PEP 426`_ deals with packaging metadata, and defines name argument
  of the ``packaging.core.setup()`` function.

* `PEP 420`_ deals with namespace packages. It brings support of
  namespace packages to Python core. Before, namespaces packages were
  implemented by external libraries.

* `PEP 3108`_ deals with transition between Python 2.x and Python 3.x
  applied to standard library: some modules to be deleted, some to be
  renamed. It points out that naming conventions matter and is an
  example of transition plan.


Overview
========

Here is a summarized list of recommendations for you to choose names:

* `understand and respect namespace ownership
  <#understand-and-respect-namespace-ownership>`_.

* if your project is related to another project or community, first
  search for conventions in main project's documentation, then:

  * `follow specific project or related community conventions
    <#follow-community-or-related-project-conventions-if-any>`_, if
    any.

  * else (there is no specific convention), `follow a standard naming pattern
    <#use-standard-pattern-for-community-contributions>`_.

* make sure names are unique, i.e. avoid duplicates:

  * `check for name availability`_,
  * `register names with PyPI`_.

  Exception is when you explicitely want to distribute alternatives
  to existing packages or modules.

* `Use a single name`_. It implies `a project distributes a single
  package or module <#multiple-packages-modules-should-be-rare>`_

* `distribute only one package or module at a time
  <#multiple-packages-modules-should-be-rare>`_, unless you are in
  a special case.

* make it easy to discover and remember your project:
 
  * `pick memorable names`_,
  * `pick meaningful names`_,
  * `use packaging metadata`_.

* `avoid deep nesting`_:
 
  * one single level is the recommended way to go.

  * two levels can be used to point out strict relationships: the
    second level is specific to the first one. Main use cases are
    community contributions related to one project, and
    vendor specific projects.

  * you should not need more than two levels. Having more than three
    levels is strongly discouraged.

* `follow PEP 8`_ for syntax of package and module names.

* if, for some reason, your project does not follow the
  recommendations above, `document specific naming policy`_.
  In particular, projects which are receiving community
  contributions should `organize community contributions`_.

* `if still in doubt, ask <#if-in-doubt-ask>`_.


If in doubt, ask
================

If you feel unsure after reading this document, ask `Python
community`_ on IRC or on a mailing list.


Understand and respect namespace ownership
==========================================

On `PyPI`_, all projects are put at index root. There is no owner or
user level. PyPI cannot host two projects with the exact same name,
even if owners are different. One name relies to one project, which
relies to one ownership.

.. note:: A project's ownership can be hold by several users.

The top-level namespace relates to ownership.
As an example, `Django`_ is owned and maintained by the Django
Software Fundation.

Understand the purpose of namespace before you use it.

Do not plug into a namespace you do not own, unless explicitely
authorized. `If in doubt, ask`_.
As an example, do not plug in "django.contrib" namespace because it
is managed by Django's core contributors.

Project owners may define exceptions. See `Organize community
contributions`_ below.
As an example, `flask`_ project explicitely invites contributors to
release projects in "flask.ext" namespace.

Also, whenever possible, try to consider non-Python projects.
As an example, you should not use "apache" as top-level namespace:
"Apache" is the name of another (non Python) project. This is more an
advice than a strict rule, but it could help identify your project on
the internet or prevent some trademark issues.

Private projects may use a namespace
------------------------------------

For internal/customer projects, feel free to use the company or main
project name as the namespace. But keep in mind that, if a project is
general purpose (i.e. not specific to your company or to some main
project), then one level should be enough.

This rule applies to closed-source projects.

As an example, if you are creating a "climbing" project that is
specific to the "Python Sport" company: you may use
"pythonsport.climbing" name, even if it is closed source.

Use a single name
=================

Distribute only one package (or only one module) per project, and use
package (or module) name as project name.

* It avoids possible confusion between project name and distributed
  package or module name.

* It makes the name consistent.

* It is explicit: when one sees project name, he guesses
  package/module name, and vice versa.

* It also limits implicit clashes between package/module names.
  By using a single name, when you register a project name to
  `PyPI`_, you also perform a basic package/module name availability
  verification.

  As an example, `pipeline`_, `python-pipeline`_ and
  `django-pipeline`_ all distribute a package or module called
  "pipeline". So installing two of them leads to errors. This issue
  wouldn't have occurred if these distributions used a single name.

* As a bonus, it allows easier setup of the project: you provide the
  name of the package/module you want to distribute, and you get
  other names.

Example:

* Yes: Package name is "kheops" and project name is "kheops".

* Yes: Package name is "kheops.history", i.e.
  ``import kheops.history`` and project name is "kheops.history",
  i.e. ``pip install kheops.history``.

* No: Package name is "kheops" and project name is "KheopsPyramid".

.. note::

   For historical reasons, `PyPI`_ contains many distributions where
   project and distributed package/module names differ.

Multiple packages/modules should be rare
----------------------------------------

Technically, Python distributions can provide multiple packages
and/or modules. See `setup script reference`_ for details.

Some distributions actually do.
As an example, `setuptools`_ and `distribute`_ are both declaring
"pkg_resources", "easy_install" and "site" modules in addition to
respective "setuptools" and "distribute" packages.

Consider this use case as exceptional. In most cases, you do not need
this feature and distributing a single package or module is enough.

Distinct names should be rare
-----------------------------

A notable exception to the `Use a single name`_ rule is when you
explicitely need distinct names.

As an example, the `Pillow`_ project provides an alternative to the
original `PIL`_ distribution. Both projects distribute a "PIL"
package.

Consider this use case as exceptional. In most cases, you don't need
this feature and naming the project after the distributed
package/module is enough.

Follow PEP 8
============

`PEP 8`_ applies to names of Python packages and modules.

If you `Use a single name`_, `PEP 8`_ also applies to project names.
The exceptions are namespace packages, where dots are required in
project name.


Pick memorable names
====================

One important thing about a project name is that it be memorable.

As an example, `celery`_ is not a meaningful name. At first, it is
not obvious that it deals with message queuing. But it is memorable
because it can be used to feed a `RabbitMQ`_ server.


Pick meaningful names
=====================

Ask yourself "how would I describe in one sentence what this name is
for?", and then "could anyone have guessed that by looking at the
name?".

As an example, `DateUtils`_ is a meaningful name. It is obvious that
it deals with utilities for dates.

When you are using namespaces, try to make each part meaningful.

.. note::

   Sometimes, you cannot find a name that is both memorable and
   meaningful. In such a situation, consider "memorable" feature is
   more important than "meaningful" one. Generally, by choosing a
   memorable name, you tend to make the name unique, and simplicity
   is a key of success. Whereas a meaningful name tend to be
   similar to names of other projects that deal with same concepts,
   and you tend make the name out of keywords/buzzwords.


Use packaging metadata
======================

Consider that **project names are unique identifiers on PyPI**, i.e.
their primary purpose is to identify, not to classify or describe.

**Classifiers and keywords metadata are made for categorization.**
Summary and description metadata are meant to describe the project.

As an example, there is a "`Framework :: Twisted`_" classifier. Names
of projects that have this classifier are quite heterogeneous. They
do not follow a particular pattern to claim relation with Twisted.
But we get the list using the classifier and that is fine.

In order to `Organize community contributions`_, conventions about
names and namespaces matter, but conventions about metadata are
important too.

As an example, we can find Plone portlets in many places:

* plone.portlet.*
* collective.portlet.*
* collective.portlets.*
* collective.*.portlets
* some vendor-related projects such as "quintagroup.portlet.cumulus"
* and even projects where "portlet" pattern doesn't appear in the
  name.

Even if Plone community has conventions, using the name to categorize
distributions is inapropriate. It's impossible to get the full list
of distributions that provide portlets for Plone by filtering on
names. But it would be possible if all these distributions used
"Framework :: Plone" classifier and "portlet" keyword.

When you release a project on `PyPI`_, you obviously want your
project to be visible, and findable. Keep in mind that the name is
not the only mean to make a project discoverable. If you do care
about your project's visibility, take care of package metadata:
including keywords, classifiers, README... And you may also take care
of project's documentation and some stuff not related to packaging.


Avoid deep nesting
==================

`The Zen of Python`_ says:

  Flat is better than nested.

A single level is recommended
-----------------------------

In most cases, one level is enough. So, unless you are in a special
situation mentioned below, your project name should be made of a
single namespace.

Lower levels indicate strict relationship to upper levels
---------------------------------------------------------

In nested namespaces, lower levels point out strict relationship to
higher ones. It means the second level is specific to the first one.

Main use cases are community contributions related to one project
and vendor specific (mostly private) projects.

Two levels is almost always enough
----------------------------------

Don't define everything in deeply nested hierarchies: you will end up
with projects and packages like "pythonsport.common.maps.forest".
This type of name is both verbose and cumbersome (e.g. if you have
many imports from the package).

Furthermore, big hierarchies tend to break down over time as the
boundaries between different packages blur.

The consensus is that two levels of nesting are preferred.

For example, we have ``plone.principalsource`` instead of
``plone.source.principal`` or something like that. The name is
shorter, the package structure is simpler, and there would be very
little to gain from having three levels of nesting here. It would be
impractical to try to put all "core Plone" sources (a source is kind
of vocabulary) into the ``plone.source.*`` namespace, in part because
some sources are part of other packages, and in part because sources
already exist in other places. Had we made a new namespace, it would
be inconsistently used from the start.

Yes: "pythonsport.climbing"

Yes: "pythonsport.forestmap"

No: "pythonsport.maps.forest"

Do not use namespace levels for categorization
----------------------------------------------

`Use packaging metadata`_ instead.

Don't use more than 3 levels
----------------------------

Technically, you have the ability to create deeply nested
hierarchies. However, it is strongly discouraged.


Document specific naming policy
===============================

A project that does not follow this PEP's recommendations should
mention and explain it in documentation.

This rule is the simplest way to make an existing project comply with
this PEP, without a rename.


Follow community or related project conventions, if any
=======================================================

Projects or related communities can have specific naming conventions,
which may differ from those explained in this document. Specific
conventions override this PEP.

This rule exists for backward-compatibility purpose: new projects
should follow this PEP's conventions.

In such a case, `they should declare specific conventions in
documentation <#organize-community-contributions>`_.

So, if your project belongs to another project or to a community,
first look for specific conventions in main project's documentation.

If there is no specific conventions, follow the ones declared in this
document.

As an example, `Plone community`_ releases community contributions in
the "collective" namespace package. It differs from the `standard
namespace for contributions
<#use-standard-pattern-for-community-contributions>`_ proposed here.
But since it is documented, there is no ambiguity and you should
follow this specific convention.


Use standard pattern for community contributions
================================================

When no specific rule is defined, use the
``{MAINPROJECT}contrib.{PROJECT}`` pattern to store community
contributions for any product or framework, where:

* ``{MAINPROJECT}`` is the name of the related project. "pyranha" in
  the example below.

* ``{PROJECT}`` is the name of your project. "giantteeth" in the
  example below.

As an example:

* you are the author of "pyranha" project.

* you didn't defined specific naming conventions for community
  contributions.

* a third-party developer wants to publish a "giantteeth" project
  related to your "pyranha" project in a community namespace. So he
  should publish it as "pyranhacontrib.giantteeth".

It is the simplest way to `Organize community contributions`_.

.. note::

   Why ``{MAINPROJECT}contrib.*`` pattern?

   * ``{MAINPROJECT}c.*`` is not explicit enough. As examples, "zc"
     belongs to "Zope Corporation" whereas "z3c" belongs to "Zope 3
     community".

   * ``{MAINPROJECT}community`` is too long.

   * ``{MAINPROJECT}community`` conflicts with existing namespaces
     such as "iccommunity" or "PyCommunity".

   * ``{MAINPROJECT}.contrib.*`` is inside {MAINPROJECT} namespace,
     i.e. it is owned by ${MAINPROJECT} authors.

   * ``{MAINPROJECT}.contrib.*`` breaks the `Avoid deep nesting`_
     rule.

   * names where ``{MAINPROJECT}`` does not appear are not explicit
     enough, i.e. nobody can guess they are related to
     ``{MAINPROJECT}``. As an example, it is not obvious that
     "collective.*" belongs to Plone community.

   * ``{DIST}contrib.*`` looks like existing ``sphinxcontrib-*``
     packages. But ``sphinxcontrib-*`` is actually about Sphinx
     contrib, so this is not a real conflict... In fact, the
     "contrib" suffix was inspired by "sphinxcontrib".


Organize community contributions
================================

This is the counterpart of the `follow community conventions
<#follow-community-or-related-project-conventions-if-any>`_ and
`standard pattern for contributions
<#use-standard-pattern-for-community-contributions>`_ rules.

Actions:

* Choose a naming convention for community contributions.

* If it is not `the default
  <#use-standard-pattern-for-community-contributions>`_, then
  document it.

  * if you use the `default convention
    <#use-standard-pattern-for-community-contributions>`_, then this
    document should be enough. Don't reapeat it. You may reference
    it.

  * else, tell users about custom conventions in project's
    "contribute" or "create modules" documentation.

* Also recommend the use of additional metadata, such as
  `classifiers and keywords <#use-packaging-metadata>`_.

Example: "pyranha" is your project name and package name.
Tell contributors that:

* pyranha-related distributions should use the "pyranha" keyword

* pyranha-related distributions providing templates should also use
  "templates" keyword.

* community contributions should be released under "pyranhacontrib"
  namespace (i.e. use "pyranhacontrib.*" pattern).


Register names with PyPI
========================

`PyPI`_ is the central place for distributions in Python community.
So, it is also the place where to register project and package names.

See `Registering with the Package Index`_ for details.


Check for name availability
===========================

Make sure it project name has not already been registered on `PyPI`_.

.. note::

   `PyPI`_ is the only official place where to register names.

Also make sure the names of distributed packages or modules have not
already been registered:

* in the `Python Standard Library`_.
* inside projects at `PyPI`_.

.. note::

   The `use a single name`_ rule helps you avoid clashes with package
   names: if a project name is available, then the package name has
   good chances to be available too.


How to rename a project?
========================

Renaming a project is possible, but it should be done with care.
Pay particular attention to README and documentation, so that users
understand what happened.

#. First of all, **do not remove legacy distributions from PyPI**.
   Because some users may be using them.

#. Copy the legacy project, then change names (project and
   package/module). Pay attention to, at least:

   * packaging files,
   * folder name that contains source files,
   * documentation, including README,
   * import statements in code.

#. Assign ``obsoleted_by`` metadata to new distribution in setup.cfg
   file. See `PEP 426 about obsoleted_by`_ and `setup.cfg
   specification`_.

#. Release a new version of the renamed project, then publish it.

#. Edit legacy project:

   * add dependency to new project,
   * drop everything except packaging stuff,
   * add the ``Development Status :: 7 - Inactive`` classifier in
     setup script,
   * publish a new release.

So, users of the legacy package:

* can continue using the legacy distributions at a deprecated
  version,
* can upgrade to last version of legacy distribution, which is
  empty...
* ... and automatically download new distribution as a dependency of
  the legacy one.

Users who discover the legacy project see it is inactive.


How to apply naming guidelines on existing projects?
====================================================

**There is no obligation for existing projects to be renamed**. The
choice is left to project authors and maintainers for obvious
reasons.

However, project authors are invited to:

* at least, state about current naming, i.e., if necessary

 * explain the reason why the project has this name in documentation,
 * `document specific naming policy`_.

* optionally, `rename existing project or distributed
  packages/modules <#how-to-rename-a-project>`_.

Projects that are meant to receive contributions from community
should also `organize community contributions`_.


References
==========

Additional background:

* `Martin Aspeli's article about names`_. Some parts of this document
  are quotes from this article.

* `in development official packaging documentation`_.

* `The Hitchhiker's Guide to Packaging`_, which has an empty
  placeholder for "naming specification".

References and footnotes:

.. _`packaging terminology in Python documentation`:
   http://docs.python.org/dev/packaging/introduction.html#general-python-terminology
.. _`PEP 8`:
   http://www.python.org/dev/peps/pep-0008/#package-and-module-names
.. _`PEP 426`: http://www.python.org/dev/peps/pep-0426/
.. _`PEP 420`: http://www.python.org/dev/peps/pep-0420/
.. _`PEP 3108`: http://www.python.org/dev/peps/pep-3108/
.. _`Python community`: http://www.python.org/community/
.. _`gp.fileupload`: http://pypi.python.org/pypi/gp.fileupload/
.. _`zest.releaser`: http://pypi.python.org/pypi/zest.releaser/
.. _`django`: http://djangoproject.com/
.. _`flask`: http://flask.pocoo.org/
.. _`sphinx`: http://sphinx.pocoo.org
.. _`pypi`: http://pypi.python.org
.. _`collective.recaptcha`:
   http://pypi.python.org/pypi/collective.recaptcha/
.. _`pipeline`: http://pypi.python.org/pypi/pipeline/ 
.. _`python-pipeline`: http://pypi.python.org/pypi/python-pipeline/
.. _`django-pipeline`: http://pypi.python.org/pypi/django-pipeline/
.. _`setup script reference`:
   http://docs.python.org/dev/packaging/setupscript.html
.. _`setuptools`: http://pypi.python.org/pypi/setuptools
.. _`distribute`: http://packages.python.org/distribute/
.. _`Pillow`: http://pypi.python.org/pypi/Pillow/
.. _`PIL`: http://pypi.python.org/pypi/PIL/
.. _`celery`: http://pypi.python.org/pypi/celery/
.. _`RabbitMQ`: http://www.rabbitmq.com
.. _`DateUtils`: http://pypi.python.org/pypi/DateUtils/
.. _`Framework :: Twisted`:
   http://pypi.python.org/pypi?:action=browse&show=all&c=525
.. _`The Zen of Python`: http://www.python.org/dev/peps/pep-0020/
.. _`Plone community`: http://plone.org/community/develop
.. _`Registering with the Package Index`:
   http://docs.python.org/dev/packaging/packageindex.html
.. _`Python Standard Library`:
   http://docs.python.org/library/index.html
.. _`PEP 426 about obsoleted_by`:
   http://www.python.org/dev/peps/pep-0426/#obsoleted-by
.. _`setup.cfg specification`:
   http://docs.python.org/dev/packaging/setupcfg.html
.. _`Martin Aspeli's article about names`:
   http://www.martinaspeli.net/articles/the-naming-of-things-package-names-and-namespaces
.. _`in development official packaging documentation`:
   http://docs.python.org/dev/packaging/
.. _`The Hitchhiker's Guide to Packaging`:
   http://guide.python-distribute.org/specification.html#naming-specification


Copyright
=========

This document has been placed in the public domain.



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End:

From markus at unterwaditzer.net  Fri Jul  5 21:31:46 2013
From: markus at unterwaditzer.net (Markus Unterwaditzer)
Date: Fri, 05 Jul 2013 21:31:46 +0200
Subject: [Python-Dev] PEP 423 : naming conventions and recipes related
	to packaging
In-Reply-To: <51D677FC.8050107@marmelune.net>
References: <4FEACD9D.8090208@marmelune.net>
	<20120627125055.196efe9f@pitrou.net>
	<51D677FC.8050107@marmelune.net>
Message-ID: <0a300e6e-ca3a-44e5-a029-162c37faa44f@email.android.com>

In your first plone example you first use plone.app.content, but then present the directory structure of plone.app.command.

Apart from that, the PEP seems legit to me, contentwise. I think some parts are clumsily formulated, but IMO rewriting these parts wouldn't even decrease the text's length or improve readability.

-- Markus (from phone)

"Beno?t Bryon" <benoit at marmelune.net> wrote:
>Hi!
>
>Attached is a an updated proposal for PEP 423.
>You can also find it online at
>https://gist.github.com/benoitbryon/2815051
>
>I am attending at EuroPython 2013 in Florence. Isn't it a great 
>opportunity to get feedback and discuss about a PEP? I registered an 
>open-space session and a lightning-talk today!
>
>Some notes about the update...
>
>The main point that was discussed in the previous proposal was the 
>"top-level namespace relates to code ownership rule". Here is a quote 
>from Antoine Pitrou:
>
>Le 27/06/2012 12:50, Antoine Pitrou a ?crit :
>> On Wed, 27 Jun 2012 11:08:45 +0200
>> Beno?t Bryon<benoit at marmelune.net>  wrote:
>>> Hi,
>>>
>>> Here is an informational PEP proposal:
>>> http://hg.python.org/peps/file/52767ab7e140/pep-0423.txt
>>>
>>> Could you review it for style, consistency and content?
>> There is one Zen principle this PEP is missing:
>>
>> Flat is better than nested.
>>
>> This PEP seems to promote the practice of having a top-level
>namespace
>> denote ownership. I think it should do the reverse: promote
>> meaningful top-level packages (e.g. "sphinx") as standard practice,
>and
>> allow an exception for when a piece of software is part of a larger
>> organizational body.
>
>So, the main change in the proposal I'm sending today is the removal of
>
>this "ownership" rule.
>It has been replaced by "Use a single namespace (except special
>cases)".
>
>Some additional changes have been performed, such as removal of some 
>sections about "opportunity" or "promote migrations". I also added a 
>"Rationale" section where I pointed out some issues related to naming.
>
>The PEP has been marked as "deferred" because it was inactive and it is
>
>partly related to PEP 426. I left this deferred state.
>
>I am aware that some links in the PEP are broken... I will fix them 
>later. My very first motivation is to get feedback about the "big" 
>changes in the PEP. I wanted the update to be sent before 
>EuroPython-2013's open-space session. I guess a detailed review would
>be 
>nice anyway, for links, style, grammar...
>
>Also, I wonder whether the PEP could be shortened or not. Sometimes I 
>cannot find straightforward words to explain things, so perhaps someone
>
>with better skills in english language could help. Or maybe some parts,
>
>such as the "How to rename a project" section, could be moved in other 
>documents.
>
>Regards,
>
>Beno?t
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Python-Dev mailing list
>Python-Dev at python.org
>http://mail.python.org/mailman/listinfo/python-dev
>Unsubscribe:
>http://mail.python.org/mailman/options/python-dev/markus%40unterwaditzer.net


From cs at zip.com.au  Sat Jul  6 02:43:52 2013
From: cs at zip.com.au (Cameron Simpson)
Date: Sat, 6 Jul 2013 10:43:52 +1000
Subject: [Python-Dev] PEP 446: Add new parameters to configure the
 inherance of files and for non-blocking sockets
In-Reply-To: <CAMpsgwYjVGxyF4PT=g33R+SoU1tRfXT5yFwgUt3KTpnn4cHcXA@mail.gmail.com>
References: <CAMpsgwYjVGxyF4PT=g33R+SoU1tRfXT5yFwgUt3KTpnn4cHcXA@mail.gmail.com>
Message-ID: <20130706004352.GA66371@cskk.homeip.net>

On 05Jul2013 19:03, Victor Stinner <victor.stinner at gmail.com> wrote:
| 2013/7/5 Cameron Simpson <cs at zip.com.au>:
| > | Both set O_NONBLOCK flag (UNIX)
| >
| > Oh, how embarassing.
| 
| You said that the PEP is not cristal clear. Do you have a suggestion
| to make it more clear?
| 
| Should I mention that the close-on-exec flag is O_CLOEXEC on UNIX, and
| HANDLE_FLAG_INHERIT on Windows? (except that HANDLE_FLAG_INHERIT set
| means inheritable, whereas O_CLOEXEC set means *not* inheritable)

It wouldn't hurt.

Otherwise, colour me mostly convinced.

I've read your "Rejected Alternatives" more closely and Ulrich
Drepper's article, though I think the article also supports adding
a blocking (default True) parameter to open() and os.open(). If you
try to change that default on a platform where it doesn't work, an
exception should be raised.

| The close-on-exec flag only affects inheritance of file descriptors at
| the execv() syscall, not at fork(). And execv() can be called without
| fork().

Yes. Please forget I mentioned fork(); it is only relevant if you
were offering some facility to undo the addition of cloexec to a
Popen passed file descriptor. Which you are not.

| Popen must clear close-on-exec flag on files from its pass_fds
| parameter for convinience. It would be surprising to not inherit a
| file descriptor passed to Popen in pass_fds, don't you think so?

Yes yes yes.

| I don't understand. I already wrote "The flag has no effect on fork(),
| all file descriptors are inherited by the child process" in the PEP.
| It is not enough? Do you have a suggestion to explain it better?

My concerns are probably as well bundled with concerns about users
not realising the a file descriptor state after a fork is shared,
and then being confused if the other process did a seek() etc.

I think I'm just concerned about a naive caller using this scenario:

  # gather up some already open fds to pass
  fds = (5,6,7)
  P = Popen(...., pass_fds=fds)
  # now restore the cloexec state from before
  for fd in fds:
      os.set_cloexec(fd)

where the cleanup loop in the parent gets to run before the exec()
in the child.

We could just consider it one of the pitfalls of fork/exec situations
in general an expect people being this fiddly to need to add some
synchornisation.

Please consider your argument won.

Cheers,
-- 
Cameron Simpson <cs at zip.com.au>

From ronaldoussoren at mac.com  Sat Jul  6 09:45:53 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Sat, 6 Jul 2013 09:45:53 +0200
Subject: [Python-Dev] Hooking into super() attribute resolution
In-Reply-To: <DCEB2B77-B9DD-405A-A084-EFB761B947BB@mac.com>
References: <DCEB2B77-B9DD-405A-A084-EFB761B947BB@mac.com>
Message-ID: <46EC256E-9ACE-4C4C-BE3D-325D7DB87B85@mac.com>

I've updated the implementation in issue 18181 <http://bugs.python.org/issue18181> while adding some tests, and have updated the proposal as well. 

The proposal has some open issues at the moment, most important of which is the actual signature for the new special method; in particular I haven't been able to decide if this should be an instance-, class- or static method. It is a static method in the proposal and prototype, but I'm not convinced that that is the right solution.

Ronald




PEP: TODO
Title: Hooking into super attribute resolution
Version: $Revision$
Last-Modified: $Date$
Author: Ronald Oussoren <ronaldoussoren at mac.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 12-Jun-2013
Post-History: 2-Jul-2013, ?


Abstract
========

In current python releases the attribute resolution of the `super class`_
peeks in the ``__dict__`` attribute of classes on the MRO to look
for attributes. This PEP introduces a hook that classes can use
to override that behavior for specific classes.


Rationale
=========

Peeking in the class ``__dict__`` works for regular classes, but can
cause problems when a class dynamicly looks up attributes in a
``__getattribute__`` method.

The new hook makes it possible to introduce the same customization for
attribute lookup through the `super class`_.


The superclass attribute lookup hook
====================================

In C code
---------

A new slot ``tp_getattro_super`` is added to the ``PyTypeObject`` struct. The
``tp_getattro`` slot for super will call this slot when it is not ``NULL``,
and will raise an exception when it is not set (which shouldn't happen because
the method is implemented for :class:`object`).

The slot has the following prototype::

    PyObject* (*getattrosuperfunc)(PyTypeObject* cls, PyObject* name,
        PyObject* object, PyObject* owner);

The function should perform attribute lookup on *object* for *name*, but only
looking in type *tp* (which will be one of the types on the MRO for *self*)
and without looking in the instance *__dict__*.

The function returns ``NULL`` when the attribute cannot be found, and raises and
exception. Exception other than ``AttributeError`` will cause failure of super's
attribute resolution.

The implementation of the slot for the :class:`object` type is
``PyObject_GenericGetAttrSuper``, which peeks in the ``tp_dict`` for *cls*.

Note that *owner* and *object* will be the same object when using a
class-mode super.


In Python code
--------------

A Python class can contain a definition for a static method
``__getattribute_super__`` with the following prototype::

   def __getattribute_super__(cls, name, object, owner): pass

The method should perform attribute lookup for *name* on instance *self* while
only looking at *cls* (it should not look in super classes or the instance
*__dict__*

XXX: I haven't got a clue at the moment if the method should be an
instance-, class- or staticmethod. The prototype uses a staticmethod.

XXX: My prototype automagicly makes this a static method, just like __new__ is
made into a static method. That's more convenient, but also (too?) magical.

XXX: Should this raise AttributeError or return a magic value to signal that
an attribute cannot be found (such as NotImplemented, used in the comparison
operators)? I'm currently using an exception, a magical return value would
be slightly more efficient because the exception machinery is not invoked.


Alternative proposals
---------------------

Reuse ``tp_getattro``
.....................

It would be nice to avoid adding a new slot, thus keeping the API simpler and
easier to understand.  A comment on `Issue 18181`_ asked about reusing the
``tp_getattro`` slot, that is super could call the ``tp_getattro`` slot of all
methods along the MRO.

AFAIK that won't work because ``tp_getattro`` will look in the instance
``__dict__`` before it tries to resolve attributes using classes in the MRO.
This would mean that using ``tp_getattro`` instead of peeking the class
dictionaries changes the semantics of the `super class`_.


Open Issues
===========

* The names of the new slot and magic method are far from settled.

* I'm not too happy with the prototype for the new hook.

* Should ``__getattribute_super__`` be a class method instead?

  -> Yes? The method looks up a named attribute name of an object in
     a specific class. Is also likely needed to deal with @classmethod
     and super(Class, Class)

* Should ``__getattribute_super__`` be defined on object?

  -> Yes: makes it easier to delegate to the default implementation

* This doesn't necessarily work for class method super class
   (e.g. super(object, object))...


References
==========

* `Issue 18181`_ contains a prototype implementation


Copyright
=========

This document has been placed in the public domain.

.. _`Issue 18181`: http://bugs.python.org/issue18181

.. _`super class`: http://docs.python.org/3/library/functions.html?highlight=super#super


From cf.natali at gmail.com  Sat Jul  6 11:23:26 2013
From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=)
Date: Sat, 6 Jul 2013 11:23:26 +0200
Subject: [Python-Dev] PEP 446: Add new parameters to configure the
 inherance of files and for non-blocking sockets
In-Reply-To: <20130706004352.GA66371@cskk.homeip.net>
References: <CAMpsgwYjVGxyF4PT=g33R+SoU1tRfXT5yFwgUt3KTpnn4cHcXA@mail.gmail.com>
	<20130706004352.GA66371@cskk.homeip.net>
Message-ID: <CAH_1eM1RCDSHeaaBQT62Cjf2V9HAjSUJZS3zvZoidJ9Pyfk2FQ@mail.gmail.com>

> I've read your "Rejected Alternatives" more closely and Ulrich
> Drepper's article, though I think the article also supports adding
> a blocking (default True) parameter to open() and os.open(). If you
> try to change that default on a platform where it doesn't work, an
> exception should be raised.

Contrarily to close-on-exec, non-blocking only applies to a limited
type of files (e.g. it doesn't work for regular files, which represent
90% of open() use cases).

Also, one of the main reasons for exposing close-on-exec in
open()/socket() etc is to make it possible to create file descriptors
with the close-on-exec flag atomically, to prevent unwanted FD
inheritance especially in multi-threaded code. And that's not
necessary for the non-blocking parameter.

Those are two reasons why IMO "blocking" doesn't have to receive the
same treatment as close-on-exec (there's also the Windows issue but
I'm not familiar with it).

cf

From rdmurray at bitdance.com  Sat Jul  6 13:59:07 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Sat, 06 Jul 2013 07:59:07 -0400
Subject: [Python-Dev]
	=?utf-8?q?=5BPython-checkins=5D_cpython_=283=2E3=29?=
	=?utf-8?q?=3A_Issue_=2317860=3A_explicitly_mention_that_std*_strea?=
	=?utf-8?q?ms_are_opened_in_binary_mode_by?=
In-Reply-To: <3bnQwH1z6Vz7LjW@mail.python.org>
References: <3bnQwH1z6Vz7LjW@mail.python.org>
Message-ID: <20130706115908.2072B250185@webabinitio.net>

On Sat, 06 Jul 2013 10:25:19 +0200, ronald.oussoren <python-checkins at python.org> wrote:
> http://hg.python.org/cpython/rev/a2c2ffa1a41c
> changeset:   84453:a2c2ffa1a41c
> branch:      3.3
> parent:      84449:df79735b21c1
> user:        Ronald Oussoren <ronaldoussoren at mac.com>
> date:        Sat Jul 06 10:23:59 2013 +0200
> summary:
>   Issue #17860: explicitly mention that std* streams are opened in binary mode by default.
> 
> The documentation does mention that the streams are opened in text mode
> when univeral_newlines is true, but not that that they are opened in
> binary mode when that argument is false and that seems to confuse at
> least some users.
> 
> files:
>   Doc/library/subprocess.rst |  6 ++++--
>   1 files changed, 4 insertions(+), 2 deletions(-)
> 
> 
> diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst
> --- a/Doc/library/subprocess.rst
> +++ b/Doc/library/subprocess.rst
> @@ -293,7 +293,8 @@
>     If *universal_newlines* is ``True``, the file objects *stdin*, *stdout* and
>     *stderr* will be opened as text streams in :term:`universal newlines` mode
>     using the encoding returned by :func:`locale.getpreferredencoding(False)
> -   <locale.getpreferredencoding>`.  For *stdin*, line ending characters
> +   <locale.getpreferredencoding>`, otherwise these streams will be opened
> +   as binary streams.  For *stdin*, line ending characters
>     ``'\n'`` in the input will be converted to the default line separator
>     :data:`os.linesep`.  For *stdout* and *stderr*, all line endings in the
>     output will be converted to ``'\n'``.  For more information see the

IMO, either the default should be mentioned first, or the default
should be mentioned in a parenthetical.  Otherwise it sounds like
newline translation is being done in both modes.  Logically that makes
no sense, so the above construction will likely lead to, at a minimum,
an interruption in the flow for the reader, and at worse even more
confusion than not mentioning it at all.

--David

From victor.stinner at gmail.com  Sat Jul  6 14:04:13 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sat, 6 Jul 2013 14:04:13 +0200
Subject: [Python-Dev] PEP 446: Add new parameters to configure the
 inherance of files and for non-blocking sockets
In-Reply-To: <CAH_1eM1RCDSHeaaBQT62Cjf2V9HAjSUJZS3zvZoidJ9Pyfk2FQ@mail.gmail.com>
References: <CAMpsgwYjVGxyF4PT=g33R+SoU1tRfXT5yFwgUt3KTpnn4cHcXA@mail.gmail.com>
	<20130706004352.GA66371@cskk.homeip.net>
	<CAH_1eM1RCDSHeaaBQT62Cjf2V9HAjSUJZS3zvZoidJ9Pyfk2FQ@mail.gmail.com>
Message-ID: <CAMpsgwbRAkGccVY63oeT_LhJEuY1UWk2uE_SZ_17XeODzp3JpA@mail.gmail.com>

2013/7/6 Charles-Fran?ois Natali <cf.natali at gmail.com>:
>> I've read your "Rejected Alternatives" more closely and Ulrich
>> Drepper's article, though I think the article also supports adding
>> a blocking (default True) parameter to open() and os.open(). If you
>> try to change that default on a platform where it doesn't work, an
>> exception should be raised.
>
> Contrarily to close-on-exec, non-blocking only applies to a limited
> type of files (e.g. it doesn't work for regular files, which represent
> 90% of open() use cases).

What do you mean by "does not work"? On Linux, O_NONBLOCK flag can be
set on regular files, sockets, pipes, etc.

Example with a regular file on Linux 3.9:

smithers$ python3
Python 3.3.0 (default, Sep 29 2012, 22:07:38)
>>> from fcntl import *
>>> import os
>>> f=open("/etc/issue", "rb")
>>> fd=f.fileno()
>>> flags=fcntl(fd, F_GETFL)
>>> fcntl(fd, F_SETFL, flags|os.O_NONBLOCK)
0
>>> fcntl(fd, F_GETFL) & os.O_NONBLOCK
2048
>>> f.read(10)
b'Fedora rel'

For example, asyncore.file_dispatcher() uses fcntl to set the
O_NONBLOCK flag. (The asyncore module is probably the only module of
the stdlib which would benefit of a new os.set_blocking() function.)

> Also, one of the main reasons for exposing close-on-exec in
> open()/socket() etc is to make it possible to create file descriptors
> with the close-on-exec flag atomically, to prevent unwanted FD
> inheritance especially in multi-threaded code. And that's not
> necessary for the non-blocking parameter.

In the review of PEP 433 on this mailing list, Martin von Loewis said
that the PEP does guarantee the atomicity. The implementation of the
PEP falls back on ioctl or fcntl to set the flag on old Linux versions
(or if the OS does not support setting O_CLOEXEC flag when creating a
file descriptor). The GIL is released during system calls, so another
thread can call execv().

That's why I mentionned explicitly in the PEP 446 that it does not
offer any atomicity guarantee.

But open(filename, blocking=False) on Linux has at least one advantage
over f=open(filename); os.set_blocking(f.fileno()): it only uses one
syscall (versus 2 or 3 syscalls).

Victor

From ronaldoussoren at mac.com  Sat Jul  6 14:09:38 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Sat, 6 Jul 2013 14:09:38 +0200
Subject: [Python-Dev] [Python-checkins] cpython (3.3): Issue #17860:
	explicitly mention that std* streams are opened in binary mode by
In-Reply-To: <20130706115908.2072B250185@webabinitio.net>
References: <3bnQwH1z6Vz7LjW@mail.python.org>
	<20130706115908.2072B250185@webabinitio.net>
Message-ID: <A6E4F77E-3D82-437B-B37F-DEA6E0719A55@mac.com>


On 6 Jul, 2013, at 13:59, R. David Murray <rdmurray at bitdance.com> wrote:

> On Sat, 06 Jul 2013 10:25:19 +0200, ronald.oussoren <python-checkins at python.org> wrote:
>> http://hg.python.org/cpython/rev/a2c2ffa1a41c
>> changeset:   84453:a2c2ffa1a41c
>> branch:      3.3
>> parent:      84449:df79735b21c1
>> user:        Ronald Oussoren <ronaldoussoren at mac.com>
>> date:        Sat Jul 06 10:23:59 2013 +0200
>> summary:
>>  Issue #17860: explicitly mention that std* streams are opened in binary mode by default.
>> 
>> The documentation does mention that the streams are opened in text mode
>> when univeral_newlines is true, but not that that they are opened in
>> binary mode when that argument is false and that seems to confuse at
>> least some users.
>> 
>> files:
>>  Doc/library/subprocess.rst |  6 ++++--
>>  1 files changed, 4 insertions(+), 2 deletions(-)
>> 
>> 
>> diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst
>> --- a/Doc/library/subprocess.rst
>> +++ b/Doc/library/subprocess.rst
>> @@ -293,7 +293,8 @@
>>    If *universal_newlines* is ``True``, the file objects *stdin*, *stdout* and
>>    *stderr* will be opened as text streams in :term:`universal newlines` mode
>>    using the encoding returned by :func:`locale.getpreferredencoding(False)
>> -   <locale.getpreferredencoding>`.  For *stdin*, line ending characters
>> +   <locale.getpreferredencoding>`, otherwise these streams will be opened
>> +   as binary streams.  For *stdin*, line ending characters
>>    ``'\n'`` in the input will be converted to the default line separator
>>    :data:`os.linesep`.  For *stdout* and *stderr*, all line endings in the
>>    output will be converted to ``'\n'``.  For more information see the
> 
> IMO, either the default should be mentioned first, or the default
> should be mentioned in a parenthetical.  Otherwise it sounds like
> newline translation is being done in both modes.  Logically that makes
> no sense, so the above construction will likely lead to, at a minimum,
> an interruption in the flow for the reader, and at worse even more
> confusion than not mentioning it at all.

You've got a point there. Converting the next text (", otherwise ...") to a parententical
seems to be the cleanest fix, creating a separate sentence for the ``False`` case introduces
duplication unless I restructure the text.

Ronald

> 
> --David
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com


From ronaldoussoren at mac.com  Sat Jul  6 14:11:52 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Sat, 6 Jul 2013 14:11:52 +0200
Subject: [Python-Dev] PEP 446: Add new parameters to configure the
	inherance of files and for non-blocking sockets
In-Reply-To: <CAMpsgwbRAkGccVY63oeT_LhJEuY1UWk2uE_SZ_17XeODzp3JpA@mail.gmail.com>
References: <CAMpsgwYjVGxyF4PT=g33R+SoU1tRfXT5yFwgUt3KTpnn4cHcXA@mail.gmail.com>
	<20130706004352.GA66371@cskk.homeip.net>
	<CAH_1eM1RCDSHeaaBQT62Cjf2V9HAjSUJZS3zvZoidJ9Pyfk2FQ@mail.gmail.com>
	<CAMpsgwbRAkGccVY63oeT_LhJEuY1UWk2uE_SZ_17XeODzp3JpA@mail.gmail.com>
Message-ID: <95ACDB99-FBD5-4289-A592-C892A6651106@mac.com>


On 6 Jul, 2013, at 14:04, Victor Stinner <victor.stinner at gmail.com> wrote:

> 2013/7/6 Charles-Fran?ois Natali <cf.natali at gmail.com>:
>>> I've read your "Rejected Alternatives" more closely and Ulrich
>>> Drepper's article, though I think the article also supports adding
>>> a blocking (default True) parameter to open() and os.open(). If you
>>> try to change that default on a platform where it doesn't work, an
>>> exception should be raised.
>> 
>> Contrarily to close-on-exec, non-blocking only applies to a limited
>> type of files (e.g. it doesn't work for regular files, which represent
>> 90% of open() use cases).
> 
> What do you mean by "does not work"? On Linux, O_NONBLOCK flag can be
> set on regular files, sockets, pipes, etc.

I guess he means that O_NONBLOCK doesn't actually do anything for regular
files, regular files are always ready for I/O as far as select et. al. are
concerned and I/O will block when data has to be read from disk (or in 
the case of a network filesystem, from another machine).

Ronald


From victor.stinner at gmail.com  Sat Jul  6 14:43:24 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sat, 6 Jul 2013 14:43:24 +0200
Subject: [Python-Dev] PEP 446: Add new parameters to configure the
 inherance of files and for non-blocking sockets
In-Reply-To: <20130706004352.GA66371@cskk.homeip.net>
References: <CAMpsgwYjVGxyF4PT=g33R+SoU1tRfXT5yFwgUt3KTpnn4cHcXA@mail.gmail.com>
	<20130706004352.GA66371@cskk.homeip.net>
Message-ID: <CAMpsgwYN3j_3Vih7681qChb=jOwdP5zBQ=T-_fYDNrj9Xij=Ag@mail.gmail.com>

2013/7/6 Cameron Simpson <cs at zip.com.au>:
> Yes. Please forget I mentioned fork(); it is only relevant if you
> were offering some facility to undo the addition of cloexec to a
> Popen passed file descriptor. Which you are not.

Oh... gotcha. I now understood your concern.

There is a little "trick" here: at fork, file descriptors are
duplicated in the child process and almost all properties (open state,
flags, offset, etc.) are shared. There is one little exception: file
attributes are not shared, and there is only one file attribute:
O_CLOEXEC. Setting O_CLOEXEC in a child process does not affect the
flag in the parent process ;-) I will add a unit test to check this.

I modified the PEP to explain that, and I also mentioned the name of
the close-on-exec and blocking flags:
http://hg.python.org/peps/rev/425f831fddf7

Victor

From fuzzyman at voidspace.org.uk  Sat Jul  6 14:48:26 2013
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Sat, 6 Jul 2013 14:48:26 +0200
Subject: [Python-Dev] lament for the demise of unbound methods
In-Reply-To: <51D69AD3.7050003@v.loewis.de>
References: <51D55ACB.4050900@simplistix.co.uk>
	<CAP7+vJLVwzg0y-H3TbbbAbfsL8Bcmx3Y3=57uBA+O0hARhO8OA@mail.gmail.com>
	<51D5A5DF.3000103@simplistix.co.uk>
	<CAP7+vJLeHPkq-UgFreNBHR67UVx7Tih3jZz-VjEdbtBZ2mp3tw@mail.gmail.com>
	<761E2333-74A7-4F50-9943-E8148189507B@voidspace.org.uk>
	<51D69AD3.7050003@v.loewis.de>
Message-ID: <5C366E9D-507E-44C5-8C9A-AD14933ECC79@voidspace.org.uk>


On 5 Jul 2013, at 12:07, "Martin v. L?wis" <martin at v.loewis.de> wrote:

> Am 05.07.13 11:23, schrieb Michael Foord:
>> I've also lamented the death of bound methods in Python 3 for mock
>> "autospeccing". Autospec introspects objects and provides mock
>> objects with the same attributes - and with the same method
>> signatures.
> 
> I wonder why you need to figure out the signatures in advance.
> Can you just wait until the function is actually used, and then
> process the parameters as you get them?
> 

How does that solve the problem? Given a call and a reference to the original "function object" I need to know whether or not to trim the first argument from the original signature or not (remove self if the corresponding function object is actually a method).

Michael

> Regards,
> Martin


--
http://www.voidspace.org.uk/


May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing 
http://www.sqlite.org/different.html






From fuzzyman at voidspace.org.uk  Sat Jul  6 14:49:27 2013
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Sat, 6 Jul 2013 14:49:27 +0200
Subject: [Python-Dev] lament for the demise of unbound methods
In-Reply-To: <076A4BF7-C8CC-4325-985F-BB905B174F01@langa.pl>
References: <51D55ACB.4050900@simplistix.co.uk>
	<CAP7+vJLVwzg0y-H3TbbbAbfsL8Bcmx3Y3=57uBA+O0hARhO8OA@mail.gmail.com>
	<51D5A5DF.3000103@simplistix.co.uk>
	<CAP7+vJLeHPkq-UgFreNBHR67UVx7Tih3jZz-VjEdbtBZ2mp3tw@mail.gmail.com>
	<761E2333-74A7-4F50-9943-E8148189507B@voidspace.org.uk>
	<51D69AD3.7050003@v.loewis.de>
	<076A4BF7-C8CC-4325-985F-BB905B174F01@langa.pl>
Message-ID: <5647F6A3-787D-4A24-9769-61CAB99C27E7@voidspace.org.uk>


On 5 Jul 2013, at 12:26, ?ukasz Langa <lukasz at langa.pl> wrote:

> On 5 lip 2013, at 12:07, Martin v. L?wis <martin at v.loewis.de> wrote:
> 
>> I wonder why you need to figure out the signatures in advance.
>> Can you just wait until the function is actually used, and then
>> process the parameters as you get them?
>> 
> 
> My guess is that Michael's design lets mock objects be introspected as well, i.e. they don't appear as magical as they really are to the user code.
> 

This is also true. Doing it up front has some conveniences - for example dir(...) works correctly.

Michael

> -- 
> Best regards,
> ?ukasz Langa
> 
> WWW: http://lukasz.langa.pl/
> Twitter: @llanga
> IRC: ambv on #python-dev
> 


--
http://www.voidspace.org.uk/


May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing 
http://www.sqlite.org/different.html






From ronaldoussoren at mac.com  Sat Jul  6 15:03:13 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Sat, 6 Jul 2013 15:03:13 +0200
Subject: [Python-Dev] [Python-checkins] cpython (3.3): Issue #17860:
	explicitly mention that std* streams are opened in binary mode by
In-Reply-To: <A6E4F77E-3D82-437B-B37F-DEA6E0719A55@mac.com>
References: <3bnQwH1z6Vz7LjW@mail.python.org>
	<20130706115908.2072B250185@webabinitio.net>
	<A6E4F77E-3D82-437B-B37F-DEA6E0719A55@mac.com>
Message-ID: <1ADEC7C5-5557-4DBC-B220-1EECF6DB9A58@mac.com>


On 6 Jul, 2013, at 14:09, Ronald Oussoren <ronaldoussoren at mac.com> wrote:

> 
> On 6 Jul, 2013, at 13:59, R. David Murray <rdmurray at bitdance.com> wrote:
>> 
>> IMO, either the default should be mentioned first, or the default
>> should be mentioned in a parenthetical.  Otherwise it sounds like
>> newline translation is being done in both modes.  Logically that makes
>> no sense, so the above construction will likely lead to, at a minimum,
>> an interruption in the flow for the reader, and at worse even more
>> confusion than not mentioning it at all.
> 
> You've got a point there. Converting the next text (", otherwise ...") to a parententical
> seems to be the cleanest fix, creating a separate sentence for the ``False`` case introduces
> duplication unless I restructure the text.

I didn't like the parenthentical after all. Would this work for you?:

 
-   If *universal_newlines* is ``True``, the file objects *stdin*, *stdout* and
-   *stderr* will be opened as text streams in :term:`universal newlines` mode
+   If *universal_newlines* is ``False`` the file objects *stdin*, *stdout* and
+   *stderr* will be opened as binary streams, and no line ending conversion is done.
+
+   If *universal_newlines* is ``True``, these file objects
+   will be opened as text streams in :term:`universal newlines` mode
    using the encoding returned by :func:`locale.getpreferredencoding(False)
-   <locale.getpreferredencoding>`, otherwise these streams will be opened
-   as binary streams.  For *stdin*, line ending characters
+   <locale.getpreferredencoding>`.  For *stdin*, line ending characters
    ``'\n'`` in the input will be converted to the default line separator
    :data:`os.linesep`.  For *stdout* and *stderr*, all line endings in the
    output will be converted to ``'\n'``.  For more information see the

That is, a new paragraph is added before the existing one to explain the behavior of
"not universal_newlines".

Ronald


> 
> Ronald
> 
>> 
>> --David
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com


From greg at krypto.org  Sat Jul  6 17:14:26 2013
From: greg at krypto.org (Gregory P. Smith)
Date: Sat, 6 Jul 2013 08:14:26 -0700
Subject: [Python-Dev] [Python-checkins] cpython (3.3): Issue #17860:
 explicitly mention that std* streams are opened in binary mode by
In-Reply-To: <1ADEC7C5-5557-4DBC-B220-1EECF6DB9A58@mac.com>
References: <3bnQwH1z6Vz7LjW@mail.python.org>
	<20130706115908.2072B250185@webabinitio.net>
	<A6E4F77E-3D82-437B-B37F-DEA6E0719A55@mac.com>
	<1ADEC7C5-5557-4DBC-B220-1EECF6DB9A58@mac.com>
Message-ID: <CAGE7PNJL=G9Oda71384c12aRismNb5+1d0oMijNmm=zYEBVPbQ@mail.gmail.com>

Please update the docstring in subprocess.py with the wording improvements
that you settle on while you're at it.


On Sat, Jul 6, 2013 at 6:03 AM, Ronald Oussoren <ronaldoussoren at mac.com>wrote:

>
> On 6 Jul, 2013, at 14:09, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
>
> >
> > On 6 Jul, 2013, at 13:59, R. David Murray <rdmurray at bitdance.com> wrote:
> >>
> >> IMO, either the default should be mentioned first, or the default
> >> should be mentioned in a parenthetical.  Otherwise it sounds like
> >> newline translation is being done in both modes.  Logically that makes
> >> no sense, so the above construction will likely lead to, at a minimum,
> >> an interruption in the flow for the reader, and at worse even more
> >> confusion than not mentioning it at all.
> >
> > You've got a point there. Converting the next text (", otherwise ...")
> to a parententical
> > seems to be the cleanest fix, creating a separate sentence for the
> ``False`` case introduces
> > duplication unless I restructure the text.
>
> I didn't like the parenthentical after all. Would this work for you?:
>
>
> -   If *universal_newlines* is ``True``, the file objects *stdin*,
> *stdout* and
> -   *stderr* will be opened as text streams in :term:`universal newlines`
> mode
> +   If *universal_newlines* is ``False`` the file objects *stdin*,
> *stdout* and
> +   *stderr* will be opened as binary streams, and no line ending
> conversion is done.
> +
> +   If *universal_newlines* is ``True``, these file objects
> +   will be opened as text streams in :term:`universal newlines` mode
>     using the encoding returned by
> :func:`locale.getpreferredencoding(False)
> -   <locale.getpreferredencoding>`, otherwise these streams will be opened
> -   as binary streams.  For *stdin*, line ending characters
> +   <locale.getpreferredencoding>`.  For *stdin*, line ending characters
>     ``'\n'`` in the input will be converted to the default line separator
>     :data:`os.linesep`.  For *stdout* and *stderr*, all line endings in the
>     output will be converted to ``'\n'``.  For more information see the
>
> That is, a new paragraph is added before the existing one to explain the
> behavior of
> "not universal_newlines".
>
> Ronald
>
>
> >
> > Ronald
> >
> >>
> >> --David
> >> _______________________________________________
> >> Python-Dev mailing list
> >> Python-Dev at python.org
> >> http://mail.python.org/mailman/listinfo/python-dev
> >> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com
> >
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > http://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130706/72680176/attachment.html>

From larry at hastings.org  Sat Jul  6 19:33:17 2013
From: larry at hastings.org (Larry Hastings)
Date: Sat, 06 Jul 2013 19:33:17 +0200
Subject: [Python-Dev] Rough idea for adding introspection information
 for builtins
In-Reply-To: <5148374D.6050402@hastings.org>
References: <5147ED55.30605@hastings.org> <ki8vdd$ne3$1@ger.gmane.org>
	<CADiSq7d-BzzbpYSr4nnyPJyDcnigLWvV8X40V=Pd_onwBL4yYQ@mail.gmail.com>
	<5148374D.6050402@hastings.org>
Message-ID: <51D854DD.7080807@hastings.org>


Revisiting a four-month-old discussion:

On 03/19/2013 11:00 AM, Larry Hastings wrote:
> As for handling optional argument groups, my gut feeling is that we're 
> better off not leaking it out of Argument Clinic--don't expose it in 
> this string we're talking about, and don't add support for it in the 
> inspect.Parameter object.  I'm not going to debate range(), the syntax 
> of which predates one of our release managers.  But I suggest option 
> groups are simply a misfeature of the curses module.  There are some 
> other possible uses in builtins (I forgot to dig those out this 
> evening) but so far we're talking adding complexity to an array of 
> technologies (this representation, the parser, the Parameter object) 
> to support a handful of uses of something we shouldn't have done in 
> the first place, for consumers who I think won't care and won't 
> appreciate the added conceptual complexity.

I'm sad to say I've just about changed my mind on this.

This is what help(os.stat) looks like in my dev branch for Argument Clinic:

     >>> help(os.stat)
    Help on built-in function stat in module posix:

    stat(...)
         os.stat(path, *, dir_fd=None, follow_symlinks=True)
    ...


Argument Clinic added the line starting with "os.stat(path, ". pydoc 
generated the "stat(...)" line.  It doesn't have any info because of the 
lack of introspection information.

Once builtins have introspection information, pydoc can do a better job, 
and Argument Clinic can stop generating its redundant prototype line.  
But if pydoc doesn't have argument group information, it won't be able 
to tell where one group ends and the next begins, and it won't be able 
to render the prototype for the help text correctly.  I fear misleading 
text is even worse than no text at all.

I also suggest that fancy editors (PyCharm etc) want as much information 
as we can give them.  If we give them argument group information, they 
can flag malformed calls (e.g. "there's no way to legally call this 
function with exactly three arguments").

I therefore have two credible consumers of this information.  That's 
enough for me: I propose we amend the Parameter object to add option 
group information for positional-only parameters.


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130706/a6635df9/attachment-0001.html>

From solipsis at pitrou.net  Sat Jul  6 22:54:54 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 6 Jul 2013 22:54:54 +0200
Subject: [Python-Dev] Accepting PEP 445
Message-ID: <20130706225454.0932f6ba@fsol>


Hello,

I'm accepting PEP 445 (A C API to customize memory allocators) by
Victor.  There is probably some grammar to correct here and there
(neither Victor nor I are native English speakers), but I don't want
this to hold back acceptance.  The PEP is an obvious improvement in
flexibility for everyone embedding Python in some external application,
or even recompiling their own Python runtime for specific uses.

The implementation is tracked in http://bugs.python.org/issue3329

Thanks to Victor for his patience :-)

Regards

Antoine.



From a.cavallo at cavallinux.eu  Sat Jul  6 23:07:40 2013
From: a.cavallo at cavallinux.eu (Antonio Cavallo)
Date: Sat, 6 Jul 2013 22:07:40 +0100
Subject: [Python-Dev] Accepting PEP 445
In-Reply-To: <20130706225454.0932f6ba@fsol>
References: <20130706225454.0932f6ba@fsol>
Message-ID: <BD4C1108-3778-407B-931A-D0369F03D56D@cavallinux.eu>

Could that remove the need for the --with-pydebug flag?




On 6 Jul 2013, at 21:54, Antoine Pitrou <solipsis at pitrou.net> wrote:

> 
> Hello,
> 
> I'm accepting PEP 445 (A C API to customize memory allocators) by
> Victor.  There is probably some grammar to correct here and there
> (neither Victor nor I are native English speakers), but I don't want
> this to hold back acceptance.  The PEP is an obvious improvement in
> flexibility for everyone embedding Python in some external application,
> or even recompiling their own Python runtime for specific uses.
> 
> The implementation is tracked in http://bugs.python.org/issue3329
> 
> Thanks to Victor for his patience :-)
> 
> Regards
> 
> Antoine.
> 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/a.cavallo%40cavallinux.eu


From solipsis at pitrou.net  Sat Jul  6 23:43:26 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 6 Jul 2013 23:43:26 +0200
Subject: [Python-Dev] Accepting PEP 445
In-Reply-To: <BD4C1108-3778-407B-931A-D0369F03D56D@cavallinux.eu>
References: <20130706225454.0932f6ba@fsol>
	<BD4C1108-3778-407B-931A-D0369F03D56D@cavallinux.eu>
Message-ID: <20130706234326.0b94776b@fsol>

On Sat, 6 Jul 2013 22:07:40 +0100
Antonio Cavallo <a.cavallo at cavallinux.eu> wrote:
> Could that remove the need for the --with-pydebug flag?

No, --with-pydebug enables much more than just memory debug checks.

Regards

Antoine.


> 
> 
> 
> 
> On 6 Jul 2013, at 21:54, Antoine Pitrou <solipsis at pitrou.net> wrote:
> 
> > 
> > Hello,
> > 
> > I'm accepting PEP 445 (A C API to customize memory allocators) by
> > Victor.  There is probably some grammar to correct here and there
> > (neither Victor nor I are native English speakers), but I don't want
> > this to hold back acceptance.  The PEP is an obvious improvement in
> > flexibility for everyone embedding Python in some external application,
> > or even recompiling their own Python runtime for specific uses.
> > 
> > The implementation is tracked in http://bugs.python.org/issue3329
> > 
> > Thanks to Victor for his patience :-)
> > 
> > Regards
> > 
> > Antoine.
> > 
> > 
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > http://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe: http://mail.python.org/mailman/options/python-dev/a.cavallo%40cavallinux.eu
> 


From ncoghlan at gmail.com  Sun Jul  7 00:18:52 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 7 Jul 2013 08:18:52 +1000
Subject: [Python-Dev] lament for the demise of unbound methods
In-Reply-To: <5647F6A3-787D-4A24-9769-61CAB99C27E7@voidspace.org.uk>
References: <51D55ACB.4050900@simplistix.co.uk>
	<CAP7+vJLVwzg0y-H3TbbbAbfsL8Bcmx3Y3=57uBA+O0hARhO8OA@mail.gmail.com>
	<51D5A5DF.3000103@simplistix.co.uk>
	<CAP7+vJLeHPkq-UgFreNBHR67UVx7Tih3jZz-VjEdbtBZ2mp3tw@mail.gmail.com>
	<761E2333-74A7-4F50-9943-E8148189507B@voidspace.org.uk>
	<51D69AD3.7050003@v.loewis.de>
	<076A4BF7-C8CC-4325-985F-BB905B174F01@langa.pl>
	<5647F6A3-787D-4A24-9769-61CAB99C27E7@voidspace.org.uk>
Message-ID: <CADiSq7cEE_VGS_r0su=pJgL2NzjmSO+AgYgo1OYenAeu1G960Q@mail.gmail.com>

On 6 Jul 2013 22:52, "Michael Foord" <fuzzyman at voidspace.org.uk> wrote:
>
>
> On 5 Jul 2013, at 12:26, ?ukasz Langa <lukasz at langa.pl> wrote:
>
> > On 5 lip 2013, at 12:07, Martin v. L?wis <martin at v.loewis.de> wrote:
> >
> >> I wonder why you need to figure out the signatures in advance.
> >> Can you just wait until the function is actually used, and then
> >> process the parameters as you get them?
> >>
> >
> > My guess is that Michael's design lets mock objects be introspected as
well, i.e. they don't appear as magical as they really are to the user code.
> >
>
> This is also true. Doing it up front has some conveniences - for example
dir(...) works correctly.

Doesn't this really mean that in Py3, introspection APIs need a class *and*
an instance of that class for robust analysis of expected descriptor
results? (which was technically always true, the removal of unbound methods
just makes it significantly more important to handle descriptors with no
special behaviour on the class side of things).

And yes, I'm aware that idea poses a significant challenge for correct
handling of ABCs :P

Cheers,
Nick.

>
> Michael
>
> > --
> > Best regards,
> > ?ukasz Langa
> >
> > WWW: http://lukasz.langa.pl/
> > Twitter: @llanga
> > IRC: ambv on #python-dev
> >
>
>
> --
> http://www.voidspace.org.uk/
>
>
> May you do good and not evil
> May you find forgiveness for yourself and forgive others
> May you share freely, never taking more than you give.
> -- the sqlite blessing
> http://www.sqlite.org/different.html
>
>
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130707/f8c1c0cd/attachment.html>

From ncoghlan at gmail.com  Sun Jul  7 00:32:46 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 7 Jul 2013 08:32:46 +1000
Subject: [Python-Dev] Rough idea for adding introspection information
	for builtins
In-Reply-To: <51D854DD.7080807@hastings.org>
References: <5147ED55.30605@hastings.org> <ki8vdd$ne3$1@ger.gmane.org>
	<CADiSq7d-BzzbpYSr4nnyPJyDcnigLWvV8X40V=Pd_onwBL4yYQ@mail.gmail.com>
	<5148374D.6050402@hastings.org> <51D854DD.7080807@hastings.org>
Message-ID: <CADiSq7eYz0KDxNKtiih9RVBgLEei8tA=uiXP1AnWKMd450ep7g@mail.gmail.com>

On 7 Jul 2013 05:22, "Larry Hastings" <larry at hastings.org> wrote:
>
>
> Revisiting a four-month-old discussion:
>
>
> On 03/19/2013 11:00 AM, Larry Hastings wrote:
>>
>> As for handling optional argument groups, my gut feeling is that we're
better off not leaking it out of Argument Clinic--don't expose it in this
string we're talking about, and don't add support for it in the
inspect.Parameter object.  I'm not going to debate range(), the syntax of
which predates one of our release managers.  But I suggest option groups
are simply a misfeature of the curses module.  There are some other
possible uses in builtins (I forgot to dig those out this evening) but so
far we're talking adding complexity to an array of technologies (this
representation, the parser, the Parameter object) to support a handful of
uses of something we shouldn't have done in the first place, for consumers
who I think won't care and won't appreciate the added conceptual complexity.
>
>
> I'm sad to say I've just about changed my mind on this.
>
> This is what help(os.stat) looks like in my dev branch for Argument
Clinic:
>>
>> >>> help(os.stat)
>> Help on built-in function stat in module posix:
>>
>> stat(...)
>>     os.stat(path, *, dir_fd=None, follow_symlinks=True)
>> ...
>
>
> Argument Clinic added the line starting with "os.stat(path, ".  pydoc
generated the "stat(...)" line.  It doesn't have any info because of the
lack of introspection information.
>
> Once builtins have introspection information, pydoc can do a better job,
and Argument Clinic can stop generating its redundant prototype line.  But
if pydoc doesn't have argument group information, it won't be able to tell
where one group ends and the next begins, and it won't be able to render
the prototype for the help text correctly.  I fear misleading text is even
worse than no text at all.
>
> I also suggest that fancy editors (PyCharm etc) want as much information
as we can give them.  If we give them argument group information, they can
flag malformed calls (e.g. "there's no way to legally call this function
with exactly three arguments").
>
> I therefore have two credible consumers of this information.  That's
enough for me: I propose we amend the Parameter object to add option group
information for positional-only parameters.

Rather than perpetuating unwanted complexity, can't we just add a single
"incomplete signature" flag to handle the legacy cases, and leave those to
the docstrings?

As in, if the flag is set, pydoc displays the "..." because it knows the
signature data isn't quite right.

Alternatively (and even more simply), is it really so bad if argument
clinic doesn't support introspection of such functions at all, and avoids
setting __signature__ for such cases?

As a third option, we could add an "alternative signatures" attribute to
capture multiple orthogonal signatures that should be presented on separate
lines.

All of those possibilities sound more appealing to me than adding direct
support for parameter groups at the Python level (with my preference being
to postpone the question to 3.5 by not allowing introspection of affected
functions in this initial iteration).

Cheers,
Nick.

>
>
> /arry
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130707/109b6d4d/attachment.html>

From ncoghlan at gmail.com  Sun Jul  7 00:35:37 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 7 Jul 2013 08:35:37 +1000
Subject: [Python-Dev] Accepting PEP 445
In-Reply-To: <20130706225454.0932f6ba@fsol>
References: <20130706225454.0932f6ba@fsol>
Message-ID: <CADiSq7ejwdR+74p_rTt5z6O73AVv6uxM7H6ydXd-8DM8ja1ooA@mail.gmail.com>

On 7 Jul 2013 07:01, "Antoine Pitrou" <solipsis at pitrou.net> wrote:
>
>
> Hello,
>
> I'm accepting PEP 445 (A C API to customize memory allocators) by
> Victor.  There is probably some grammar to correct here and there
> (neither Victor nor I are native English speakers), but I don't want
> this to hold back acceptance.  The PEP is an obvious improvement in
> flexibility for everyone embedding Python in some external application,
> or even recompiling their own Python runtime for specific uses.
>
> The implementation is tracked in http://bugs.python.org/issue3329
>
> Thanks to Victor for his patience :-)

Yay! And thanks to you for suggesting this API would benefit from being run
through the PEP process - I think the end result is a significant
improvement over what we originally planned to commit.

Cheers,
Nick.

>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130707/71cb2126/attachment.html>

From victor.stinner at gmail.com  Sun Jul  7 01:04:38 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sun, 7 Jul 2013 01:04:38 +0200
Subject: [Python-Dev] Accepting PEP 445
In-Reply-To: <BD4C1108-3778-407B-931A-D0369F03D56D@cavallinux.eu>
References: <20130706225454.0932f6ba@fsol>
	<BD4C1108-3778-407B-931A-D0369F03D56D@cavallinux.eu>
Message-ID: <CAMpsgwZ6ddHVjhFobg5MsACCZUXQwONdQSyrU8EWBuiWLdZysQ@mail.gmail.com>

2013/7/6 Antonio Cavallo <a.cavallo at cavallinux.eu>:
> Could that remove the need for the --with-pydebug flag?

With the PEP 445, you still have to recompile Python with
--with-debug, but you don't have to recompile Python extensions
anymore. Debug checks on memory allocators are now implemented as
hooks, instead of using C macros.

I see this as an improvment ;-)

Victor

From larry at hastings.org  Sun Jul  7 02:22:46 2013
From: larry at hastings.org (Larry Hastings)
Date: Sun, 07 Jul 2013 02:22:46 +0200
Subject: [Python-Dev] Rough idea for adding introspection information
 for builtins
In-Reply-To: <CADiSq7eYz0KDxNKtiih9RVBgLEei8tA=uiXP1AnWKMd450ep7g@mail.gmail.com>
References: <5147ED55.30605@hastings.org> <ki8vdd$ne3$1@ger.gmane.org>
	<CADiSq7d-BzzbpYSr4nnyPJyDcnigLWvV8X40V=Pd_onwBL4yYQ@mail.gmail.com>
	<5148374D.6050402@hastings.org> <51D854DD.7080807@hastings.org>
	<CADiSq7eYz0KDxNKtiih9RVBgLEei8tA=uiXP1AnWKMd450ep7g@mail.gmail.com>
Message-ID: <51D8B4D6.3080107@hastings.org>

On 07/07/2013 12:32 AM, Nick Coghlan wrote:
>
> Rather than perpetuating unwanted complexity, can't we just add a 
> single "incomplete signature" flag to handle the legacy cases, and 
> leave those to the docstrings?
>
> As in, if the flag is set, pydoc displays the "..." because it knows 
> the signature data isn't quite right.
>
> Alternatively (and even more simply), is it really so bad if argument 
> clinic doesn't support introspection of such functions at all, and 
> avoids setting __signature__ for such cases?
>
> As a third option, we could add an "alternative signatures" attribute 
> to capture multiple orthogonal signatures that should be presented on 
> separate lines.
>
> All of those possibilities sound more appealing to me than adding 
> direct support for parameter groups at the Python level (with my 
> preference being to postpone the question to 3.5 by not allowing 
> introspection of affected functions in this initial iteration).
>

First, I think the PyCharm case is compelling enough on its own.  I 
realized after I sent it that there's a related class of tools that are 
interested: PyFlakes, PyLint, and the like.  I'm sure the static 
correctness analyzers would like to be able to automatically determine 
"this is an illegal number of parameters for this function" for 
builtins--particularly for third-party builtins!  The fact that we 
wouldn't need to special-case pydoc suggests it's the superior 
approach.  ("Special cases aren't special enough to break the rules.")

Second, the added complexity would be a single new member on the 
Parameter object.  Let me propose such a parameter here, in the style of 
the Parameter class documentation:

    group

        If not None, represents which "optional parameter group" this
        parameter belongs to.  Optional parameter groups are contiguous
        sequences of parameters that must either all be specified or all
        be unspecified.  For example, if a function takes four
        parameters but the last two are in an optional parameter group,
        you could specify either two or four arguments to that
        function--it would be illegal to specify three arguments. 
        Parameter groups can only contain positional-only parameters;
        therefore group will only be a non-None value when kind is
        POSITIONAL_ONLY.


I suggest that is a manageable level of complexity.  And that the 
tooling projects would very much like to have this information.

Third, your proposals are respectively: 1) a hack which fixes the 
docstring but doesn't fix the introspection information (so we'd be 
providing incorrect introspection information to tools), 2) a small 
cop-out (which I think would also probably require a hack to pydoc), and 
3) way more complicated than doing it the right way (so I don't see how 
it's an improvement).  Of your three suggestions I dislike 2) least.

This facet of call signatures has existed in Python since the addition 
of range().  I concede that it's legacy, but it's not going away.  
Ever.  I now think we're better off embracing this complexity than 
trying to sweep it under the rug.


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130707/a432f205/attachment-0001.html>

From cs at zip.com.au  Sun Jul  7 02:29:12 2013
From: cs at zip.com.au (Cameron Simpson)
Date: Sun, 7 Jul 2013 10:29:12 +1000
Subject: [Python-Dev] PEP 446: Add new parameters to configure the
 inherance of files and for non-blocking sockets
In-Reply-To: <CAMpsgwYN3j_3Vih7681qChb=jOwdP5zBQ=T-_fYDNrj9Xij=Ag@mail.gmail.com>
References: <CAMpsgwYN3j_3Vih7681qChb=jOwdP5zBQ=T-_fYDNrj9Xij=Ag@mail.gmail.com>
Message-ID: <20130707002912.GA49468@cskk.homeip.net>

On 06Jul2013 14:43, Victor Stinner <victor.stinner at gmail.com> wrote:
| 2013/7/6 Cameron Simpson <cs at zip.com.au>:
| > Yes. Please forget I mentioned fork(); it is only relevant if you
| > were offering some facility to undo the addition of cloexec to a
| > Popen passed file descriptor. Which you are not.
| 
| Oh... gotcha. I now understood your concern.
| 
| There is a little "trick" here: at fork, file descriptors are
| duplicated in the child process and almost all properties (open state,
| flags, offset, etc.) are shared. There is one little exception: file
| attributes are not shared, and there is only one file attribute:
| O_CLOEXEC. Setting O_CLOEXEC in a child process does not affect the
| flag in the parent process ;-) I will add a unit test to check this.

That's news to me. Interesting.

I can't find UNIX doco to support this, though I haven't finished
looking. None of the POSIX, Linux or MacOSX fork() manual pages
seem to describe this.

Can you point me at doco for this? I thought file descriptors were
a direct index into a per-process table of file handle structures,
and did not think the latter got part copies, part shared over a
fork.

Also, is O_CLOEXEC really the only file attribute? So O_NONBLOCK
is a flag and not an attribute (guessing the terminology distinguishes
these)? Now you mention it, ISTR noticing that there were distinct
ioctls to adjust flags and something else.

| I modified the PEP to explain that, and I also mentioned the name of
| the close-on-exec and blocking flags:
| http://hg.python.org/peps/rev/425f831fddf7

Thanks.

Your change says:

  On Windows, the close-on-exec flag is ``HANDLE_FLAG_INHERIT``.

I feel it would be clearer to say:

  On Windows, the close-on-exec flag is the inverse of ``HANDLE_FLAG_INHERIT``.

or

  On Windows, the close-on-exec flag implemented using ``HANDLE_FLAG_INHERIT``.

(This latter is fine because the rest of the paragraph explains the
meaning of HANDLE_FLAG_INHERIT.)

The bare "is" in your current wording suggests to me that the
truthiness (ick!) of close-on-exec is the same as for HANDLE_FLAG_INHERIT,
which it isn't.
-- 
Cameron Simpson <cs at zip.com.au>

I've always been a big Greenaway fan - I've seen and enjoyed "The Falls" for
crying out loud.        - Peter Alexander Merel <pete at extro.ucc.su.OZ.AU>

From cs at zip.com.au  Sun Jul  7 02:30:56 2013
From: cs at zip.com.au (Cameron Simpson)
Date: Sun, 7 Jul 2013 10:30:56 +1000
Subject: [Python-Dev] PEP 446: Add new parameters to configure the
 inherance of files and for non-blocking sockets
In-Reply-To: <CAH_1eM1RCDSHeaaBQT62Cjf2V9HAjSUJZS3zvZoidJ9Pyfk2FQ@mail.gmail.com>
References: <CAH_1eM1RCDSHeaaBQT62Cjf2V9HAjSUJZS3zvZoidJ9Pyfk2FQ@mail.gmail.com>
Message-ID: <20130707003056.GA55329@cskk.homeip.net>

On 06Jul2013 11:23, Charles-Fran?ois Natali <cf.natali at gmail.com> wrote:
| > I've read your "Rejected Alternatives" more closely and Ulrich
| > Drepper's article, though I think the article also supports adding
| > a blocking (default True) parameter to open() and os.open(). If you
| > try to change that default on a platform where it doesn't work, an
| > exception should be raised.
| 
| Contrarily to close-on-exec, non-blocking only applies to a limited
| type of files (e.g. it doesn't work for regular files, which represent
| 90% of open() use cases).

sockets, pipes, serial devices, ...

And you can set it on anything. Just because some things don't block
anyway isn't really a counter argument.

Cheers,
-- 
Cameron Simpson <cs at zip.com.au>

Hofstadter's Law: It always takes longer than you expect, even when you take
into account Hofstadter's Law.
- Douglas Hosfstadter, Godel, Escher, Bach: an Eternal Golden Braid

From ncoghlan at gmail.com  Sun Jul  7 03:04:12 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 7 Jul 2013 11:04:12 +1000
Subject: [Python-Dev] Rough idea for adding introspection information
	for builtins
In-Reply-To: <51D8B4D6.3080107@hastings.org>
References: <5147ED55.30605@hastings.org> <ki8vdd$ne3$1@ger.gmane.org>
	<CADiSq7d-BzzbpYSr4nnyPJyDcnigLWvV8X40V=Pd_onwBL4yYQ@mail.gmail.com>
	<5148374D.6050402@hastings.org> <51D854DD.7080807@hastings.org>
	<CADiSq7eYz0KDxNKtiih9RVBgLEei8tA=uiXP1AnWKMd450ep7g@mail.gmail.com>
	<51D8B4D6.3080107@hastings.org>
Message-ID: <CADiSq7e7zbGMaUp0+KZBa+dHjhpq-=UWsHdteb6vVVr3rLCFEw@mail.gmail.com>

On 7 Jul 2013 10:25, "Larry Hastings" <larry at hastings.org> wrote:
>
> On 07/07/2013 12:32 AM, Nick Coghlan wrote:
>>
>> Rather than perpetuating unwanted complexity, can't we just add a single
"incomplete signature" flag to handle the legacy cases, and leave those to
the docstrings?
>>
>> As in, if the flag is set, pydoc displays the "..." because it knows the
signature data isn't quite right.
>>
>> Alternatively (and even more simply), is it really so bad if argument
clinic doesn't support introspection of such functions at all, and avoids
setting __signature__ for such cases?
>>
>> As a third option, we could add an "alternative signatures" attribute to
capture multiple orthogonal signatures that should be presented on separate
lines.
>>
>> All of those possibilities sound more appealing to me than adding direct
support for parameter groups at the Python level (with my preference being
to postpone the question to 3.5 by not allowing introspection of affected
functions in this initial iteration).
>
>
> First, I think the PyCharm case is compelling enough on its own.  I
realized after I sent it that there's a related class of tools that are
interested: PyFlakes, PyLint, and the like.  I'm sure the static
correctness analyzers would like to be able to automatically determine
"this is an illegal number of parameters for this function" for
builtins--particularly for third-party builtins!  The fact that we wouldn't
need to special-case pydoc suggests it's the superior approach.  ("Special
cases aren't special enough to break the rules.")
>
> Second, the added complexity would be a single new member on the
Parameter object.  Let me propose such a parameter here, in the style of
the Parameter class documentation:
>>
>> group
>>>
>>> If not None, represents which "optional parameter group" this parameter
belongs to.  Optional parameter groups are contiguous sequences of
parameters that must either all be specified or all be unspecified.  For
example, if a function takes four parameters but the last two are in an
optional parameter group, you could specify either two or four arguments to
that function--it would be illegal to specify three arguments.  Parameter
groups can only contain positional-only parameters; therefore group will
only be a non-None value when kind is POSITIONAL_ONLY.
>
>
> I suggest that is a manageable level of complexity.  And that the tooling
projects would very much like to have this information.
>
> Third, your proposals are respectively: 1) a hack which fixes the
docstring but doesn't fix the introspection information (so we'd be
providing incorrect introspection information to tools), 2) a small cop-out
(which I think would also probably require a hack to pydoc), and 3) way
more complicated than doing it the right way (so I don't see how it's an
improvement).  Of your three suggestions I dislike 2) least.
>
> This facet of call signatures has existed in Python since the addition of
range().  I concede that it's legacy, but it's not going away.  Ever.  I
now think we're better off embracing this complexity than trying to sweep
it under the rug.

The "group" attribute sounds reasonable to me, with the proviso that we use
"multiple signature lines" as the way to represent them in pydoc (rather
than inventing a notation for showing them in a single signature line).

It occurs to me that "type" is itself an example of this kind of dual
signature.

Cheers,
Nick.

>
>
> /arry
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130707/afa3d53d/attachment.html>

From larry at hastings.org  Sun Jul  7 04:48:18 2013
From: larry at hastings.org (Larry Hastings)
Date: Sun, 07 Jul 2013 04:48:18 +0200
Subject: [Python-Dev] Rough idea for adding introspection information
 for builtins
In-Reply-To: <CADiSq7e7zbGMaUp0+KZBa+dHjhpq-=UWsHdteb6vVVr3rLCFEw@mail.gmail.com>
References: <5147ED55.30605@hastings.org> <ki8vdd$ne3$1@ger.gmane.org>
	<CADiSq7d-BzzbpYSr4nnyPJyDcnigLWvV8X40V=Pd_onwBL4yYQ@mail.gmail.com>
	<5148374D.6050402@hastings.org> <51D854DD.7080807@hastings.org>
	<CADiSq7eYz0KDxNKtiih9RVBgLEei8tA=uiXP1AnWKMd450ep7g@mail.gmail.com>
	<51D8B4D6.3080107@hastings.org>
	<CADiSq7e7zbGMaUp0+KZBa+dHjhpq-=UWsHdteb6vVVr3rLCFEw@mail.gmail.com>
Message-ID: <51D8D6F2.80007@hastings.org>

On 07/07/2013 03:04 AM, Nick Coghlan wrote:
>
> On 7 Jul 2013 10:25, "Larry Hastings" <larry at hastings.org 
> <mailto:larry at hastings.org>> wrote:
> >>
> >> group
> >>>
> >>> If not None, represents which "optional parameter group" this 
> parameter belongs to.  Optional parameter groups are contiguous 
> sequences of parameters that must either all be specified or all be 
> unspecified.  For example, if a function takes four parameters but the 
> last two are in an optional parameter group, you could specify either 
> two or four arguments to that function--it would be illegal to specify 
> three arguments.  Parameter groups can only contain positional-only 
> parameters; therefore group will only be a non-None value when kind is 
> POSITIONAL_ONLY.
>
> The "group" attribute sounds reasonable to me, with the proviso that 
> we use "multiple signature lines" as the way to represent them in 
> pydoc (rather than inventing a notation for showing them in a single 
> signature line).
>
> It occurs to me that "type" is itself an example of this kind of dual 
> signature.
>

We don't have to invent a notation--because we already have one. It's 
square brackets enclosing the optional parameter groups.  This is the 
syntax Guido dictated for Argument Clinic to use in its input DSL back 
at PyCon US 2013.  And the Python standard library documentation has 
been using this convention since the 90s. (Admittedly as a documentation 
convention, not in code.  But what we're talking about is documentation 
so I claim it's totally relevant.)

If we combine that with the admittedly-new "/" indicating "all previous 
parameters are positional-only", which we're also already using in the 
Argument Clinic input DSL syntax (at your suggestion!), we have a 
complete, crisp syntax.  I suppose "/" isn't really necessary, the 
Python documentation has survived without it for a long time.  But I 
think it'd would be a nice clarification; it would convey that you can't 
splat in **kwargs into functions specifying it, for example.

I expect this to be the format of the signature-for-builtins static data 
too, as well as the Argument Clinic input syntax.  Sure seems like a 
nice idea to use the same syntax everywhere.  Particularly allowing as 
how it's so nice and readable.


An admission: the Python standard library documentation actually uses 
*both* square-brackets-for-optional-groups and multiple lines. Sometimes 
even for the same function!  An example:

    http://docs.python.org/3/library/curses.html#curses.window.addch

Of the two, I believe the square brackets syntax is far more common.


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130707/8dd791be/attachment.html>

From raymond.hettinger at gmail.com  Sun Jul  7 05:30:44 2013
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Sat, 6 Jul 2013 17:30:44 -1000
Subject: [Python-Dev] lament for the demise of unbound methods
In-Reply-To: <CAP1=2W5--OmzkQDoS58=vXzmWZH0zasF2yNqZhxyo48JPvg8eQ@mail.gmail.com>
References: <51D55ACB.4050900@simplistix.co.uk> <51D5639B.4060604@python.org>
	<51D56705.1000908@simplistix.co.uk>
	<CAP1=2W5--OmzkQDoS58=vXzmWZH0zasF2yNqZhxyo48JPvg8eQ@mail.gmail.com>
Message-ID: <1F06580E-1BD5-4986-814C-0E2572F4F0C5@gmail.com>


On Jul 4, 2013, at 2:34 AM, Brett Cannon <brett at python.org> wrote:

> The loss of the ability to figure out the class from an unbound method seems quite an annoying step back from an introspection point of view.
> 
> It's only annoying if you take the perspective that methods are somehow special compared to functions. With the removal of bound class methods that makes methods == functions that are an attribute on a class. And when you take that perspective it makes having anything special about methods seem wrong. It also makes adding a function to a class post-class creation make more sense since there is no difference technically.

Well said, Brett.  
This is a nice summary.


Raymond
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130706/8effd0ec/attachment.html>

From rdmurray at bitdance.com  Sun Jul  7 06:59:18 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Sun, 07 Jul 2013 00:59:18 -0400
Subject: [Python-Dev] [Python-checkins] cpython (3.3): Issue #17860:
	explicitly mention that std* streams are opened in binary mode by
In-Reply-To: <CAGE7PNJL=G9Oda71384c12aRismNb5+1d0oMijNmm=zYEBVPbQ@mail.gmail.com>
References: <3bnQwH1z6Vz7LjW@mail.python.org>
	<20130706115908.2072B250185@webabinitio.net>
	<A6E4F77E-3D82-437B-B37F-DEA6E0719A55@mac.com>
	<1ADEC7C5-5557-4DBC-B220-1EECF6DB9A58@mac.com>
	<CAGE7PNJL=G9Oda71384c12aRismNb5+1d0oMijNmm=zYEBVPbQ@mail.gmail.com>
Message-ID: <20130707045918.BF3BE2502CD@webabinitio.net>

On Sat, 06 Jul 2013 08:14:26 -0700, "Gregory P. Smith" <greg at krypto.org> wrote:
> Please update the docstring in subprocess.py with the wording improvements
> that you settle on while you're at it.
> 
> On Sat, Jul 6, 2013 at 6:03 AM, Ronald Oussoren <ronaldoussoren at mac.com>wrote:
> > I didn't like the parenthentical after all. Would this work for you?:
> >
> > -   If *universal_newlines* is ``True``, the file objects *stdin*,
> > *stdout* and
> > -   *stderr* will be opened as text streams in :term:`universal newlines`
> > mode
> > +   If *universal_newlines* is ``False`` the file objects *stdin*,
> > *stdout* and
> > +   *stderr* will be opened as binary streams, and no line ending
> > conversion is done.
> > +
> > +   If *universal_newlines* is ``True``, these file objects
> > +   will be opened as text streams in :term:`universal newlines` mode
> >     using the encoding returned by
> > :func:`locale.getpreferredencoding(False)
> > -   <locale.getpreferredencoding>`, otherwise these streams will be opened
> > -   as binary streams.  For *stdin*, line ending characters
> > +   <locale.getpreferredencoding>`.  For *stdin*, line ending characters
> >     ``'\n'`` in the input will be converted to the default line separator
> >     :data:`os.linesep`.  For *stdout* and *stderr*, all line endings in the
> >     output will be converted to ``'\n'``.  For more information see the
> >
> > That is, a new paragraph is added before the existing one to explain the
> > behavior of "not universal_newlines".

Looks good to me.

--David

From ronaldoussoren at mac.com  Sun Jul  7 07:19:11 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Sun, 7 Jul 2013 07:19:11 +0200
Subject: [Python-Dev] Rough idea for adding introspection information
	for builtins
In-Reply-To: <51D854DD.7080807@hastings.org>
References: <5147ED55.30605@hastings.org> <ki8vdd$ne3$1@ger.gmane.org>
	<CADiSq7d-BzzbpYSr4nnyPJyDcnigLWvV8X40V=Pd_onwBL4yYQ@mail.gmail.com>
	<5148374D.6050402@hastings.org> <51D854DD.7080807@hastings.org>
Message-ID: <BCE3C554-9046-48ED-897A-B4B7D99AD212@mac.com>


On 6 Jul, 2013, at 19:33, Larry Hastings <larry at hastings.org> wrote:
> 
> Once builtins have introspection information, pydoc can do a better job, and Argument Clinic can stop generating its redundant prototype line.

Not entirely on topic, but close enough: pydoc currently doesn't use the __signature__ information at all. Adding such support would be easy enough, see #17053 for an implementation ;-)

Ronald

From rdmurray at bitdance.com  Sun Jul  7 07:25:04 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Sun, 07 Jul 2013 01:25:04 -0400
Subject: [Python-Dev] Rough idea for adding introspection information
	for builtins
In-Reply-To: <51D8D6F2.80007@hastings.org>
References: <5147ED55.30605@hastings.org> <ki8vdd$ne3$1@ger.gmane.org>
	<CADiSq7d-BzzbpYSr4nnyPJyDcnigLWvV8X40V=Pd_onwBL4yYQ@mail.gmail.com>
	<5148374D.6050402@hastings.org> <51D854DD.7080807@hastings.org>
	<CADiSq7eYz0KDxNKtiih9RVBgLEei8tA=uiXP1AnWKMd450ep7g@mail.gmail.com>
	<51D8B4D6.3080107@hastings.org>
	<CADiSq7e7zbGMaUp0+KZBa+dHjhpq-=UWsHdteb6vVVr3rLCFEw@mail.gmail.com>
	<51D8D6F2.80007@hastings.org>
Message-ID: <20130707052505.0CAD32502D1@webabinitio.net>

On Sun, 07 Jul 2013 04:48:18 +0200, Larry Hastings <larry at hastings.org> wrote:
> On 07/07/2013 03:04 AM, Nick Coghlan wrote:
> >
> > On 7 Jul 2013 10:25, "Larry Hastings" <larry at hastings.org 
> > <mailto:larry at hastings.org>> wrote:
> > >>
> > >> group
> > >>>
> > >>> If not None, represents which "optional parameter group" this 
> > parameter belongs to.  Optional parameter groups are contiguous 
> > sequences of parameters that must either all be specified or all be 
> > unspecified.  For example, if a function takes four parameters but the 
> > last two are in an optional parameter group, you could specify either 
> > two or four arguments to that function--it would be illegal to specify 
> > three arguments.  Parameter groups can only contain positional-only 
> > parameters; therefore group will only be a non-None value when kind is 
> > POSITIONAL_ONLY.
> >
> > The "group" attribute sounds reasonable to me, with the proviso that 
> > we use "multiple signature lines" as the way to represent them in 
> > pydoc (rather than inventing a notation for showing them in a single 
> > signature line).
> >
> > It occurs to me that "type" is itself an example of this kind of dual 
> > signature.
> >
> 
> We don't have to invent a notation--because we already have one. It's 
> square brackets enclosing the optional parameter groups.  This is the 
> syntax Guido dictated for Argument Clinic to use in its input DSL back 
> at PyCon US 2013.  And the Python standard library documentation has 
> been using this convention since the 90s. (Admittedly as a documentation 
> convention, not in code.  But what we're talking about is documentation 
> so I claim it's totally relevant.)
> 
> If we combine that with the admittedly-new "/" indicating "all previous 
> parameters are positional-only", which we're also already using in the 
> Argument Clinic input DSL syntax (at your suggestion!), we have a 
> complete, crisp syntax.  I suppose "/" isn't really necessary, the 
> Python documentation has survived without it for a long time.  But I 
> think it'd would be a nice clarification; it would convey that you can't 
> splat in **kwargs into functions specifying it, for example.
> 
> I expect this to be the format of the signature-for-builtins static data 
> too, as well as the Argument Clinic input syntax.  Sure seems like a 
> nice idea to use the same syntax everywhere.  Particularly allowing as 
> how it's so nice and readable.
> 
> 
> An admission: the Python standard library documentation actually uses 
> *both* square-brackets-for-optional-groups and multiple lines. Sometimes 
> even for the same function!  An example:
> 
>     http://docs.python.org/3/library/curses.html#curses.window.addch
> 
> Of the two, I believe the square brackets syntax is far more common.

Sorry to make your life more complicated, but unless I'm misunderstanding
something, issue 18220 (http://bugs.python.org/issue18220) throws another
monkey-wrench in to this.  If I'm understanding this discussion correctly,
that example:

    islice(stop)
    islice(start, stop [, step])

requires the multiple-signature approach.

Note also that the python3 documentation has moved away from the []
notation wherever possible.

--David

From ronaldoussoren at mac.com  Sun Jul  7 07:25:23 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Sun, 7 Jul 2013 07:25:23 +0200
Subject: [Python-Dev] Rough idea for adding introspection information
	for builtins
In-Reply-To: <51D8D6F2.80007@hastings.org>
References: <5147ED55.30605@hastings.org> <ki8vdd$ne3$1@ger.gmane.org>
	<CADiSq7d-BzzbpYSr4nnyPJyDcnigLWvV8X40V=Pd_onwBL4yYQ@mail.gmail.com>
	<5148374D.6050402@hastings.org> <51D854DD.7080807@hastings.org>
	<CADiSq7eYz0KDxNKtiih9RVBgLEei8tA=uiXP1AnWKMd450ep7g@mail.gmail.com>
	<51D8B4D6.3080107@hastings.org>
	<CADiSq7e7zbGMaUp0+KZBa+dHjhpq-=UWsHdteb6vVVr3rLCFEw@mail.gmail.com>
	<51D8D6F2.80007@hastings.org>
Message-ID: <B2BD18DF-87E4-43AC-83D7-DCD88D321FDC@mac.com>


On 7 Jul, 2013, at 4:48, Larry Hastings <larry at hastings.org> wrote:

> 
> If we combine that with the admittedly-new "/" indicating "all previous parameters are positional-only",

Signature objects use a name in angled brackets to indicate that a parameter is positional only, for example "input(<prompt>)". That might be an alternative to adding a "/" in the argument list in pydoc's output.

Ronald

From chris at simplistix.co.uk  Sun Jul  7 08:40:01 2013
From: chris at simplistix.co.uk (Chris Withers)
Date: Sun, 07 Jul 2013 07:40:01 +0100
Subject: [Python-Dev] lament for the demise of unbound methods
In-Reply-To: <51D69F3E.8060103@v.loewis.de>
References: <51D55ACB.4050900@simplistix.co.uk>
	<CAP7+vJLVwzg0y-H3TbbbAbfsL8Bcmx3Y3=57uBA+O0hARhO8OA@mail.gmail.com>
	<51D5A5DF.3000103@simplistix.co.uk> <51D69F3E.8060103@v.loewis.de>
Message-ID: <51D90D41.1080302@simplistix.co.uk>

On 05/07/2013 11:26, "Martin v. L?wis" wrote:
> ...
>>>> A.__getattribute__(A,'s')
> <staticmethod object at 0x100937828>
>>>> A.__getattribute__(A,'c')
> <classmethod object at 0x100937860>
>>>> A.__getattribute__(A,'r')
> <function A.r at 0x100938378>

Okay, but with this line:

found = found.__getattribute__(found, n)

I get a tonne of failures like this:

       File "<doctest 
testfixtures.tests.test_replacer.TestReplacer.test_with_statement[3]>", 
line 2, in <module>
         r.replace('testfixtures.tests.sample1.z',test_z)
       File 
"/Users/chris/LocalGIT/testfixtures/testfixtures/replace.py", line 50, 
in replace
         container, method, attribute, t_obj = resolve(target)
       File 
"/Users/chris/LocalGIT/testfixtures/testfixtures/resolve.py", line 17, 
in resolve
         found = found.__getattribute__(found, n)
     TypeError: expected 1 arguments, got 2

If I change it to :

found = found.__getattribute__(n)

I get fewer failures, but still plenty, now of the form:

       File "<doctest 
testfixtures.tests.test_replacer.TestReplacer.test_multiple_replace[7]>", line 
1, in <module>
         r.replace('testfixtures.tests.sample1.X.y',test_y)
       File 
"/Users/chris/LocalGIT/testfixtures/testfixtures/replace.py", line 50, 
in replace
         container, method, attribute, t_obj = resolve(target)
       File 
"/Users/chris/LocalGIT/testfixtures/testfixtures/resolve.py", line 17, 
in resolve
         found = found.__getattribute__(n)
     TypeError: expected 1 arguments, got 0

...so I'm back to:

found = found.__dict__[n]

...and having to catch both KeyError and AttributeError.

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From cf.natali at gmail.com  Sun Jul  7 09:30:11 2013
From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=)
Date: Sun, 7 Jul 2013 09:30:11 +0200
Subject: [Python-Dev] PEP 446: Add new parameters to configure the
 inherance of files and for non-blocking sockets
In-Reply-To: <20130707003056.GA55329@cskk.homeip.net>
References: <CAH_1eM1RCDSHeaaBQT62Cjf2V9HAjSUJZS3zvZoidJ9Pyfk2FQ@mail.gmail.com>
	<20130707003056.GA55329@cskk.homeip.net>
Message-ID: <CAH_1eM19BzWhgy=g8XtRaJdT0DXckxYHFdY=8pvPhbYHmgzzSg@mail.gmail.com>

2013/7/7 Cameron Simpson <cs at zip.com.au>:
> On 06Jul2013 11:23, Charles-Fran?ois Natali <cf.natali at gmail.com> wrote:
> | > I've read your "Rejected Alternatives" more closely and Ulrich
> | > Drepper's article, though I think the article also supports adding
> | > a blocking (default True) parameter to open() and os.open(). If you
> | > try to change that default on a platform where it doesn't work, an
> | > exception should be raised.
> |
> | Contrarily to close-on-exec, non-blocking only applies to a limited
> | type of files (e.g. it doesn't work for regular files, which represent
> | 90% of open() use cases).
>
> sockets, pipes, serial devices, ...

How do you use open() on a socket (which are already covered by
socket(blocking=...)? Also, I said *regular files* - for which
O_NONBLOCK doesn't make sense - represent 90% of io.open() use cases,
and stand by this claim. Nothing prevents you from setting the FD
non-blocking manually.

> And you can set it on anything. Just because some things don't block
> anyway isn't really a counter argument.

Well, it complicates the signature and implementation.
If we go the same way, why stop there and not expose O_DSYNC, O_SYNC,
O_DIRECT...

When using a high-level API like io.open(), I think we should only
expose portable flags, which are supported both on all operating
systems (like the 'x' O_EXCL flag added in 3.3) and file types.

If you want precise control over the open() sementics, os.open() is
the way to go (that's also the rationale behind io.open() opener
argument, see http://bugs.python.org/issue12105)

cf

From a.cavallo at cavallinux.eu  Sun Jul  7 12:17:20 2013
From: a.cavallo at cavallinux.eu (Antonio Cavallo)
Date: Sun, 7 Jul 2013 11:17:20 +0100
Subject: [Python-Dev] Accepting PEP 445
In-Reply-To: <CAMpsgwZ6ddHVjhFobg5MsACCZUXQwONdQSyrU8EWBuiWLdZysQ@mail.gmail.com>
References: <20130706225454.0932f6ba@fsol>
	<BD4C1108-3778-407B-931A-D0369F03D56D@cavallinux.eu>
	<CAMpsgwZ6ddHVjhFobg5MsACCZUXQwONdQSyrU8EWBuiWLdZysQ@mail.gmail.com>
Message-ID: <DAEB7768-CB5A-4745-9F08-0CF2620E3E09@cavallinux.eu>

Yes that is indeed a great news. 

Having to debug some binary only extension modules that will make my (rather selfish) life so much easier ;)

Thanks


On 7 Jul 2013, at 00:04, Victor Stinner <victor.stinner at gmail.com> wrote:

> 2013/7/6 Antonio Cavallo <a.cavallo at cavallinux.eu>:
>> Could that remove the need for the --with-pydebug flag?
> 
> With the PEP 445, you still have to recompile Python with
> --with-debug, but you don't have to recompile Python extensions
> anymore. Debug checks on memory allocators are now implemented as
> hooks, instead of using C macros.
> 
> I see this as an improvment ;-)
> 
> Victor


From larry at hastings.org  Sun Jul  7 13:35:30 2013
From: larry at hastings.org (Larry Hastings)
Date: Sun, 07 Jul 2013 13:35:30 +0200
Subject: [Python-Dev] Rough idea for adding introspection information
 for builtins
In-Reply-To: <B2BD18DF-87E4-43AC-83D7-DCD88D321FDC@mac.com>
References: <5147ED55.30605@hastings.org> <ki8vdd$ne3$1@ger.gmane.org>
	<CADiSq7d-BzzbpYSr4nnyPJyDcnigLWvV8X40V=Pd_onwBL4yYQ@mail.gmail.com>
	<5148374D.6050402@hastings.org> <51D854DD.7080807@hastings.org>
	<CADiSq7eYz0KDxNKtiih9RVBgLEei8tA=uiXP1AnWKMd450ep7g@mail.gmail.com>
	<51D8B4D6.3080107@hastings.org>
	<CADiSq7e7zbGMaUp0+KZBa+dHjhpq-=UWsHdteb6vVVr3rLCFEw@mail.gmail.com>
	<51D8D6F2.80007@hastings.org>
	<B2BD18DF-87E4-43AC-83D7-DCD88D321FDC@mac.com>
Message-ID: <51D95282.8030401@hastings.org>



On 07/07/2013 07:19 AM, Ronald Oussoren wrote:
> Not entirely on topic, but close enough: pydoc currently doesn't use 
> the __signature__ information at all. Adding such support would be 
> easy enough, see #17053 for an implementation ;-) 

True, it doesn't use inspect.signature, it uses inspect.getfullargspec.  
Since I don't propose modifying inspect.getfullargspec to add the 
optional parameter group information, 17053 or something like it would 
have to happen.


On 07/07/2013 07:25 AM, R. David Murray wrote:
> Sorry to make your life more complicated, but unless I'm misunderstanding
> something, issue 18220 (http://bugs.python.org/issue18220) throws another
> monkey-wrench in to this.  If I'm understanding this discussion correctly,
> that example:
>
>      islice(stop)
>      islice(start, stop [, step])
>
> requires the multiple-signature approach.
>
> Note also that the python3 documentation has moved away from the []
> notation wherever possible.

It depends on what problem you're addressing.  In terms of the Argument 
Clinic DSL, and in terms of the static introspection information stored 
for builtins, someone (Nick?) suggested a refinement to the semantics: 
in the face of ambiguity, prefer the leftmost group(s) first.  That 
means that range() and islice() could be specified as follows:

    range([start,] stop, [step])


In terms of the documentation, it might be better to preserve the 
multiple-lines approach, as perhaps that's more obvious to the reader.  
On the other hand: in Python 3, help(itertools.islice) uses solely the 
optional group syntax, on one line.


On 07/07/2013 07:25 AM, Ronald Oussoren wrote:
> Signature objects use a name in angled brackets to indicate that a parameter is positional only, for example "input(<prompt>)". That might be an alternative to adding a "/" in the argument list in pydoc's output.

I wasn't aware that Signature objects currently had any support 
whatsoever for positional-only parameters.  Yes, in theory they do, but 
in practice they have never seen one, because positional-only parameters 
only occur in builtins and Signature objects have no metadata for 
builtins.  (The very problem Argument Clinic eventually hopes to solve!)

Can you cite an example of this, so I may examine it?


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130707/31bc3903/attachment.html>

From ronaldoussoren at mac.com  Sun Jul  7 13:42:30 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Sun, 7 Jul 2013 13:42:30 +0200
Subject: [Python-Dev] Rough idea for adding introspection information
	for builtins
In-Reply-To: <51D95282.8030401@hastings.org>
References: <5147ED55.30605@hastings.org> <ki8vdd$ne3$1@ger.gmane.org>
	<CADiSq7d-BzzbpYSr4nnyPJyDcnigLWvV8X40V=Pd_onwBL4yYQ@mail.gmail.com>
	<5148374D.6050402@hastings.org> <51D854DD.7080807@hastings.org>
	<CADiSq7eYz0KDxNKtiih9RVBgLEei8tA=uiXP1AnWKMd450ep7g@mail.gmail.com>
	<51D8B4D6.3080107@hastings.org>
	<CADiSq7e7zbGMaUp0+KZBa+dHjhpq-=UWsHdteb6vVVr3rLCFEw@mail.gmail.com>
	<51D8D6F2.80007@hastings.org>
	<B2BD18DF-87E4-43AC-83D7-DCD88D321FDC@mac.com>
	<51D95282.8030401@hastings.org>
Message-ID: <5272EBDB-81E1-4B68-B899-242AC8A25D9F@mac.com>


On 7 Jul, 2013, at 13:35, Larry Hastings <larry at hastings.org> wrote:
> 
> On 07/07/2013 07:25 AM, Ronald Oussoren wrote:
>> Signature objects use a name in angled brackets to indicate that a parameter is positional only, for example "input(<prompt>)". That might be an alternative to adding a "/" in the argument list in pydoc's output.
>> 
> 
> I wasn't aware that Signature objects currently had any support whatsoever for positional-only parameters.  Yes, in theory they do, but in practice they have never seen one, because positional-only parameters only occur in builtins and Signature objects have no metadata for builtins.  (The very problem Argument Clinic eventually hopes to solve!)
> 
> Can you cite an example of this, so I may examine it?

I have a branch of PyObjC that uses this: <https://bitbucket.org/ronaldoussoren/pyobjc-3.0-unstable/overview>. That branch isn't quite stable yet, but does add a __signature__ slot to objc.selector and objc.function (basicly methods of Cocoa classes and automaticly wrapped global functions), both of which only have positional-only arguments. With the patch for pydoc/inspect I mentioned earlier I can then generate somewhat useful documentation for Cocoa classes using pydoc.

A word of warning though: the PyObjC source code isn't the most approachable, the code that generates the Signature object is actually in python (callable_signature in pyobjc-core/Lib/objc/_callable_docstr.py)

Ronald

> 
> 
> /arry


From victor.stinner at gmail.com  Sun Jul  7 15:21:01 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sun, 7 Jul 2013 15:21:01 +0200
Subject: [Python-Dev] PEP 446: Add new parameters to configure the
 inherance of files and for non-blocking sockets
In-Reply-To: <CAH_1eM19BzWhgy=g8XtRaJdT0DXckxYHFdY=8pvPhbYHmgzzSg@mail.gmail.com>
References: <CAH_1eM1RCDSHeaaBQT62Cjf2V9HAjSUJZS3zvZoidJ9Pyfk2FQ@mail.gmail.com>
	<20130707003056.GA55329@cskk.homeip.net>
	<CAH_1eM19BzWhgy=g8XtRaJdT0DXckxYHFdY=8pvPhbYHmgzzSg@mail.gmail.com>
Message-ID: <CAMpsgwYzTCbNTw_AT3RY2qWxWbf48ZG1F14+1EemaX=9i6gABQ@mail.gmail.com>

2013/7/7 Charles-Fran?ois Natali <cf.natali at gmail.com>:
> 2013/7/7 Cameron Simpson <cs at zip.com.au>:
>> On 06Jul2013 11:23, Charles-Fran?ois Natali <cf.natali at gmail.com> wrote:
>> | > I've read your "Rejected Alternatives" more closely and Ulrich
>> | > Drepper's article, though I think the article also supports adding
>> | > a blocking (default True) parameter to open() and os.open(). If you
>> | > try to change that default on a platform where it doesn't work, an
>> | > exception should be raised.
>> |
>> | Contrarily to close-on-exec, non-blocking only applies to a limited
>> | type of files (e.g. it doesn't work for regular files, which represent
>> | 90% of open() use cases).
>>
>> sockets, pipes, serial devices, ...
>
> How do you use open() on a socket (which are already covered by
> socket(blocking=...)? Also, I said *regular files* - for which
> O_NONBLOCK doesn't make sense - represent 90% of io.open() use cases,
> and stand by this claim. Nothing prevents you from setting the FD
> non-blocking manually.

Ok, I think that the best consensus here is to add a new
os.set_blocking() function. The function would not be available on
Windows, but should be available on all UNIX platforms. The advantage
of os.set_blocking() over fcntl is that it may be implemented using
ioctl (FIONBIO) to use 1 syscall instead of 2. It is also easier to
remember calling "os.set_blocking(fd, True)" instead of
"fcntl.fcntl(fd, fcntl.F_SETFL, fcntl.fcntl(fcntl.F_GETFL, fd) |
os.O_NONBLOCK)".

See the diff:
http://hg.python.org/peps/rev/ce61588d244c

@Charles-Fran?ois and Cameron: Do you prefer to new version of the PEP?
http://www.python.org/dev/peps/pep-0446/
(will be updated in a few minutes)

I'm not sure that many people are concerned by the number of syscalls,
but there is at least Peter Portante who opened a thread
"Modules/socketmodule.c: avoiding second fcntl() call worth the
effort?" on this mailing list, last January:
http://mail.python.org/pipermail/python-dev/2013-January/123661.html
"OpenStack Swift using the Eventlet module, which sets the accepted socket
non-blocking, resulting in twice the number of fcntl() calls. Not a killer
on performance, but it seems simple enough to save a system call here."

(This specific issue will be solved by the new blocking parameter
added to socket constructor ;-))

Victor

From victor.stinner at gmail.com  Sun Jul  7 16:49:20 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sun, 7 Jul 2013 16:49:20 +0200
Subject: [Python-Dev] Accepting PEP 445
In-Reply-To: <20130706225454.0932f6ba@fsol>
References: <20130706225454.0932f6ba@fsol>
Message-ID: <CAMpsgwZVt6vesNJEw3kLHGYuZwBrjcSPjAcJrQ=hOtn1iGjZag@mail.gmail.com>

2013/7/6 Antoine Pitrou <solipsis at pitrou.net>:
> I'm accepting PEP 445 (A C API to customize memory allocators) by
> Victor.  There is probably some grammar to correct here and there
> (neither Victor nor I are native English speakers), but I don't want
> this to hold back acceptance.  The PEP is an obvious improvement in
> flexibility for everyone embedding Python in some external application,
> or even recompiling their own Python runtime for specific uses.
>
> The implementation is tracked in http://bugs.python.org/issue3329

Thanks Antoine for you review :-)

I pushed the implementation of the PEP, splitted in different
changesets: new functions and then changes to use these new functions.

You can now play with the new hooks to hack your own memory debug tool
for Python 3.4! You can easily implement your own tool in Python! (if
you hook PYMEM_DOMAIN_MEM and PYMEM_DOMAIN_OBJ, but not
PYMEM_DOMAIN_RAW) I will share with you my experiment of these new
APIs.

I'm preparing a new version of pytracemalloc to adapt it to the API of
the PEP 445.
https://pypi.python.org/pypi/pytracemalloc

Victor

From Steve.Dower at microsoft.com  Sun Jul  7 17:17:21 2013
From: Steve.Dower at microsoft.com (Steve Dower)
Date: Sun, 7 Jul 2013 15:17:21 +0000
Subject: [Python-Dev] Hooking into super() attribute resolution
In-Reply-To: <46EC256E-9ACE-4C4C-BE3D-325D7DB87B85@mac.com>
References: <DCEB2B77-B9DD-405A-A084-EFB761B947BB@mac.com>,
	<46EC256E-9ACE-4C4C-BE3D-325D7DB87B85@mac.com>
Message-ID: <0fe31345247846658b331aa358c38c66@BLUPR03MB035.namprd03.prod.outlook.com>

Could the same result be achieved by hooking the MRO that super uses and returning a list of proxy objects? And then wouldn't you only really need a __getattribute__ that doesn't recurse (__getlocalattribute__)? The end result may be conceptually simpler, but you've thought through the edge cases better than I have.

(Apologies for the HTML top-post)

Sent from my Windows Phone
________________________________
From: Ronald Oussoren<mailto:ronaldoussoren at mac.com>
Sent: ?7/?6/?2013 0:47
To: Ronald Oussoren<mailto:ronaldoussoren at mac.com>
Cc: python-dev at python.org Dev<mailto:python-dev at python.org>
Subject: Re: [Python-Dev] Hooking into super() attribute resolution

I've updated the implementation in issue 18181 <http://bugs.python.org/issue18181> while adding some tests, and have updated the proposal as well.

The proposal has some open issues at the moment, most important of which is the actual signature for the new special method; in particular I haven't been able to decide if this should be an instance-, class- or static method. It is a static method in the proposal and prototype, but I'm not convinced that that is the right solution.

Ronald




PEP: TODO
Title: Hooking into super attribute resolution
Version: $Revision$
Last-Modified: $Date$
Author: Ronald Oussoren <ronaldoussoren at mac.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 12-Jun-2013
Post-History: 2-Jul-2013, ?


Abstract
========

In current python releases the attribute resolution of the `super class`_
peeks in the ``__dict__`` attribute of classes on the MRO to look
for attributes. This PEP introduces a hook that classes can use
to override that behavior for specific classes.


Rationale
=========

Peeking in the class ``__dict__`` works for regular classes, but can
cause problems when a class dynamicly looks up attributes in a
``__getattribute__`` method.

The new hook makes it possible to introduce the same customization for
attribute lookup through the `super class`_.


The superclass attribute lookup hook
====================================

In C code
---------

A new slot ``tp_getattro_super`` is added to the ``PyTypeObject`` struct. The
``tp_getattro`` slot for super will call this slot when it is not ``NULL``,
and will raise an exception when it is not set (which shouldn't happen because
the method is implemented for :class:`object`).

The slot has the following prototype::

    PyObject* (*getattrosuperfunc)(PyTypeObject* cls, PyObject* name,
        PyObject* object, PyObject* owner);

The function should perform attribute lookup on *object* for *name*, but only
looking in type *tp* (which will be one of the types on the MRO for *self*)
and without looking in the instance *__dict__*.

The function returns ``NULL`` when the attribute cannot be found, and raises and
exception. Exception other than ``AttributeError`` will cause failure of super's
attribute resolution.

The implementation of the slot for the :class:`object` type is
``PyObject_GenericGetAttrSuper``, which peeks in the ``tp_dict`` for *cls*.

Note that *owner* and *object* will be the same object when using a
class-mode super.


In Python code
--------------

A Python class can contain a definition for a static method
``__getattribute_super__`` with the following prototype::

   def __getattribute_super__(cls, name, object, owner): pass

The method should perform attribute lookup for *name* on instance *self* while
only looking at *cls* (it should not look in super classes or the instance
*__dict__*

XXX: I haven't got a clue at the moment if the method should be an
instance-, class- or staticmethod. The prototype uses a staticmethod.

XXX: My prototype automagicly makes this a static method, just like __new__ is
made into a static method. That's more convenient, but also (too?) magical.

XXX: Should this raise AttributeError or return a magic value to signal that
an attribute cannot be found (such as NotImplemented, used in the comparison
operators)? I'm currently using an exception, a magical return value would
be slightly more efficient because the exception machinery is not invoked.


Alternative proposals
---------------------

Reuse ``tp_getattro``
.....................

It would be nice to avoid adding a new slot, thus keeping the API simpler and
easier to understand.  A comment on `Issue 18181`_ asked about reusing the
``tp_getattro`` slot, that is super could call the ``tp_getattro`` slot of all
methods along the MRO.

AFAIK that won't work because ``tp_getattro`` will look in the instance
``__dict__`` before it tries to resolve attributes using classes in the MRO.
This would mean that using ``tp_getattro`` instead of peeking the class
dictionaries changes the semantics of the `super class`_.


Open Issues
===========

* The names of the new slot and magic method are far from settled.

* I'm not too happy with the prototype for the new hook.

* Should ``__getattribute_super__`` be a class method instead?

  -> Yes? The method looks up a named attribute name of an object in
     a specific class. Is also likely needed to deal with @classmethod
     and super(Class, Class)

* Should ``__getattribute_super__`` be defined on object?

  -> Yes: makes it easier to delegate to the default implementation

* This doesn't necessarily work for class method super class
   (e.g. super(object, object))...


References
==========

* `Issue 18181`_ contains a prototype implementation


Copyright
=========

This document has been placed in the public domain.

.. _`Issue 18181`: http://bugs.python.org/issue18181

.. _`super class`: http://docs.python.org/3/library/functions.html?highlight=super#super

_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130707/cd3f2216/attachment.html>

From larry at hastings.org  Sun Jul  7 19:20:02 2013
From: larry at hastings.org (Larry Hastings)
Date: Sun, 07 Jul 2013 19:20:02 +0200
Subject: [Python-Dev] Rough idea for adding introspection information
 for builtins
In-Reply-To: <5272EBDB-81E1-4B68-B899-242AC8A25D9F@mac.com>
References: <5147ED55.30605@hastings.org> <ki8vdd$ne3$1@ger.gmane.org>
	<CADiSq7d-BzzbpYSr4nnyPJyDcnigLWvV8X40V=Pd_onwBL4yYQ@mail.gmail.com>
	<5148374D.6050402@hastings.org> <51D854DD.7080807@hastings.org>
	<CADiSq7eYz0KDxNKtiih9RVBgLEei8tA=uiXP1AnWKMd450ep7g@mail.gmail.com>
	<51D8B4D6.3080107@hastings.org>
	<CADiSq7e7zbGMaUp0+KZBa+dHjhpq-=UWsHdteb6vVVr3rLCFEw@mail.gmail.com>
	<51D8D6F2.80007@hastings.org>
	<B2BD18DF-87E4-43AC-83D7-DCD88D321FDC@mac.com>
	<51D95282.8030401@hastings.org>
	<5272EBDB-81E1-4B68-B899-242AC8A25D9F@mac.com>
Message-ID: <51D9A342.4060404@hastings.org>

On 07/07/2013 01:42 PM, Ronald Oussoren wrote:
> On 7 Jul, 2013, at 13:35, Larry Hastings <larry at hastings.org> wrote:
>> On 07/07/2013 07:25 AM, Ronald Oussoren wrote:
>>> Signature objects use a name in angled brackets to indicate that a parameter is positional only, for example "input(<prompt>)". That might be an alternative to adding a "/" in the argument list in pydoc's output.
>>>
>> I wasn't aware that Signature objects currently had any support whatsoever for positional-only parameters.  Yes, in theory they do, but in practice they have never seen one, because positional-only parameters only occur in builtins and Signature objects have no metadata for builtins.  (The very problem Argument Clinic eventually hopes to solve!)
>>
>> Can you cite an example of this, so I may examine it?
> I have a branch of PyObjC that uses this: <https://bitbucket.org/ronaldoussoren/pyobjc-3.0-unstable/overview>. That branch isn't quite stable yet, but does add a __signature__ slot to objc.selector and objc.function (basicly methods of Cocoa classes and automaticly wrapped global functions), both of which only have positional-only arguments. With the patch for pydoc/inspect I mentioned earlier I can then generate somewhat useful documentation for Cocoa classes using pydoc.
>
> A word of warning though: the PyObjC source code isn't the most approachable, the code that generates the Signature object is actually in python (callable_signature in pyobjc-core/Lib/objc/_callable_docstr.py)

Ah.  In other words, you have proposed it yourself in an external 
project.  I thought you were saying this was something Python itself 
already did.

In that case, I think I will stick with Guido's suggested syntax. 
Consider window.border in the curses module: eight positional-only 
parameters, each in its own optional parameter group.  Adding sixteen 
angle-brackets to that already unreadable morass will make it even 
worse.  But with "/" we add only a single extra character, in an 
easy-to-find place (the end).


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130707/3def6f7c/attachment-0001.html>

From ronaldoussoren at mac.com  Sun Jul  7 21:15:23 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Sun, 7 Jul 2013 21:15:23 +0200
Subject: [Python-Dev] Rough idea for adding introspection information
	for builtins
In-Reply-To: <51D9A342.4060404@hastings.org>
References: <5147ED55.30605@hastings.org> <ki8vdd$ne3$1@ger.gmane.org>
	<CADiSq7d-BzzbpYSr4nnyPJyDcnigLWvV8X40V=Pd_onwBL4yYQ@mail.gmail.com>
	<5148374D.6050402@hastings.org> <51D854DD.7080807@hastings.org>
	<CADiSq7eYz0KDxNKtiih9RVBgLEei8tA=uiXP1AnWKMd450ep7g@mail.gmail.com>
	<51D8B4D6.3080107@hastings.org>
	<CADiSq7e7zbGMaUp0+KZBa+dHjhpq-=UWsHdteb6vVVr3rLCFEw@mail.gmail.com>
	<51D8D6F2.80007@hastings.org>
	<B2BD18DF-87E4-43AC-83D7-DCD88D321FDC@mac.com>
	<51D95282.8030401@hastings.org>
	<5272EBDB-81E1-4B68-B899-242AC8A25D9F@mac.com>
	<51D9A342.4060404@hastings.org>
Message-ID: <476B98A3-4701-4964-B2BE-AE0ABF75E29E@mac.com>


On 7 Jul, 2013, at 19:20, Larry Hastings <larry at hastings.org> wrote:

> On 07/07/2013 01:42 PM, Ronald Oussoren wrote:
>> On 7 Jul, 2013, at 13:35, Larry Hastings <larry at hastings.org>
>>  wrote:
>> 
>>> On 07/07/2013 07:25 AM, Ronald Oussoren wrote:
>>> 
>>>> Signature objects use a name in angled brackets to indicate that a parameter is positional only, for example "input(<prompt>)". That might be an alternative to adding a "/" in the argument list in pydoc's output.
>>>> 
>>>> 
>>> I wasn't aware that Signature objects currently had any support whatsoever for positional-only parameters.  Yes, in theory they do, but in practice they have never seen one, because positional-only parameters only occur in builtins and Signature objects have no metadata for builtins.  (The very problem Argument Clinic eventually hopes to solve!)
>>> 
>>> Can you cite an example of this, so I may examine it?
>>> 
>> I have a branch of PyObjC that uses this: <https://bitbucket.org/ronaldoussoren/pyobjc-3.0-unstable/overview>
>> . That branch isn't quite stable yet, but does add a __signature__ slot to objc.selector and objc.function (basicly methods of Cocoa classes and automaticly wrapped global functions), both of which only have positional-only arguments. With the patch for pydoc/inspect I mentioned earlier I can then generate somewhat useful documentation for Cocoa classes using pydoc.
>> 
>> A word of warning though: the PyObjC source code isn't the most approachable, the code that generates the Signature object is actually in python (callable_signature in pyobjc-core/Lib/objc/_callable_docstr.py)
>> 
> 
> Ah.  In other words, you have proposed it yourself in an external project.  I thought you were saying this was something Python itself already did.

I wasn't clear enough in what I wrote.  The stdlib contains support for positional-only arguments in Signature objects (see Lib/inspect.py, line 1472, which says "_POSITIONAL_ONLY        = _ParameterKind(0, name='POSITIONAL_ONLY')".  The __str__ of Parameter amongst other says:

        if kind == _POSITIONAL_ONLY:
            if formatted is None:
                formatted = ''
            formatted = '<{}>'.format(formatted)

That is, it adds angled brackets around the names of positional-only parameters.  I pointed to PyObjC as an example of code that actually creates Signature objects with positional-only arguments, as far as I know the stdlib never does this because the stdlib can only create signatures for plain python functions and those cannot have such arguments.

> In that case, I think I will stick with Guido's suggested syntax.  Consider window.border in the curses module: eight positional-only parameters, each in its own optional parameter group.  Adding sixteen angle-brackets to that already unreadable morass will make it even worse.  But with "/" we add only a single extra character, in an easy-to-find place (the end).

Using Guido's suggestion is fine by me, I agree that there is a clear risk of angle-bracket overload for functions with a lot of arguments. I do think that the __str__ for Signatures should be changed to match the convention. 

And to be clear: I'm looking forward to having Argument Clinic and __signature__ objects on built-in functions, "funcname(...)" in the output pydoc is somewhat annoying, especially for extensions where the author hasn't bothered to provide a docstring. That's one reason I wrote the __signature__ support in PyObjC in the first place (and the patch for pydoc to actually use the signature information)

Ronald


From ronaldoussoren at mac.com  Sun Jul  7 21:36:43 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Sun, 7 Jul 2013 21:36:43 +0200
Subject: [Python-Dev] Hooking into super() attribute resolution
In-Reply-To: <0fe31345247846658b331aa358c38c66@BLUPR03MB035.namprd03.prod.outlook.com>
References: <DCEB2B77-B9DD-405A-A084-EFB761B947BB@mac.com>
	<46EC256E-9ACE-4C4C-BE3D-325D7DB87B85@mac.com>
	<0fe31345247846658b331aa358c38c66@BLUPR03MB035.namprd03.prod.outlook.com>
Message-ID: <CCA2B7DA-C3B7-44C1-B488-9D91B26F8C61@mac.com>


On 7 Jul, 2013, at 17:17, Steve Dower <Steve.Dower at microsoft.com> wrote:

> Could the same result be achieved by hooking the MRO that super uses and returning a list of proxy objects?

What is the advantage over adding a hook to the class itself? That seems to be the right place to add such a hook, super already looks in the classes along the MRO and my proposal would add a formal interface for that instead of having super peek into the class __dict__. I have thought of using a custom mapping object for the tp_dict slot to intercept this, but that won't work because super assumes that tp_dict is an actual PyDictObject (and likely other parts of the interpreter do so as well).


> And then wouldn't you only really need a __getattribute__ that doesn't recurse (__getlocalattribute__)? The end result may be conceptually simpler, but you've thought through the edge cases better than I have. 

__getattribute_super__ already is a kind of __getlocalattribute__, the primairy difference being __getattribute_super__ is a staticmethod instead of an instance method. To be honest I'm not sure if a staticmethod is the right solution, I'm having a hard time to determine if this should be a class, instance or static method. 

Currently super(StartClass, x) basicly does (assuming x is an instance method):


def __getattribute__(self, name):
    mro = type(x).mro()
    idx = mro.index(StartClass)
    while idx < len(mro):
       dct = mro[idx].__dict__
       try:
          result = dct[name]
	  # deal with descriptors here
          return result

       except KeyError:
           continue
    return object.__getattribute__(self, name)

With my proposal 'dct' would no longer be needed and 'result = dct[name]' would be 'mro[idx].__getattribute_super__(mro[idx], name, x, StartClass)' (I may have the last argument for the call to __getattribute_super__ wrong, but that's the idea). Given that the first argument of __get...super__ is the same as the object the method get getattr-ed from I guess the method should be a classmethod instead of an staticmethod. Changing that would be easy enough.

I'm still interested in feedback on the basic idea, I'd love to here that my proposal isn't necessary because there is already a way to get the behavior I'm looking for although that's nog going to happen ;-).

Ronald


> 
> (Apologies for the HTML top-post)

I don't mind.

PS. Does anyone know if the pep editors are away (conferences, holidays, ...)? I could just check in my proposal in the peps repository, but as this is my first PEP I'd prefer to follow the documented procedure and have someone that knows what he's doing look at the metadata before checking in.

> 
> Sent from my Windows Phone
> From: Ronald Oussoren
> Sent: ?7/?6/?2013 0:47
> To: Ronald Oussoren
> Cc: python-dev at python.org Dev
> Subject: Re: [Python-Dev] Hooking into super() attribute resolution
> 
> I've updated the implementation in issue 18181 <http://bugs.python.org/issue18181> while adding some tests, and have updated the proposal as well. 
> 
> The proposal has some open issues at the moment, most important of which is the actual signature for the new special method; in particular I haven't been able to decide if this should be an instance-, class- or static method. It is a static method in the proposal and prototype, but I'm not convinced that that is the right solution.
> 
> Ronald
> 
> 
> 
> 
> PEP: TODO
> Title: Hooking into super attribute resolution
> Version: $Revision$
> Last-Modified: $Date$
> Author: Ronald Oussoren <ronaldoussoren at mac.com>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 12-Jun-2013
> Post-History: 2-Jul-2013, ?
> 
> 
> Abstract
> ========
> 
> In current python releases the attribute resolution of the `super class`_
> peeks in the ``__dict__`` attribute of classes on the MRO to look
> for attributes. This PEP introduces a hook that classes can use
> to override that behavior for specific classes.
> 
> 
> Rationale
> =========
> 
> Peeking in the class ``__dict__`` works for regular classes, but can
> cause problems when a class dynamicly looks up attributes in a
> ``__getattribute__`` method.
> 
> The new hook makes it possible to introduce the same customization for
> attribute lookup through the `super class`_.
> 
> 
> The superclass attribute lookup hook
> ====================================
> 
> In C code
> ---------
> 
> A new slot ``tp_getattro_super`` is added to the ``PyTypeObject`` struct. The
> ``tp_getattro`` slot for super will call this slot when it is not ``NULL``,
> and will raise an exception when it is not set (which shouldn't happen because
> the method is implemented for :class:`object`).
> 
> The slot has the following prototype::
> 
>     PyObject* (*getattrosuperfunc)(PyTypeObject* cls, PyObject* name,
>         PyObject* object, PyObject* owner);
> 
> The function should perform attribute lookup on *object* for *name*, but only
> looking in type *tp* (which will be one of the types on the MRO for *self*)
> and without looking in the instance *__dict__*.
> 
> The function returns ``NULL`` when the attribute cannot be found, and raises and
> exception. Exception other than ``AttributeError`` will cause failure of super's
> attribute resolution.
> 
> The implementation of the slot for the :class:`object` type is
> ``PyObject_GenericGetAttrSuper``, which peeks in the ``tp_dict`` for *cls*.
> 
> Note that *owner* and *object* will be the same object when using a
> class-mode super.
> 
> 
> In Python code
> --------------
> 
> A Python class can contain a definition for a static method
> ``__getattribute_super__`` with the following prototype::
> 
>    def __getattribute_super__(cls, name, object, owner): pass
> 
> The method should perform attribute lookup for *name* on instance *self* while
> only looking at *cls* (it should not look in super classes or the instance
> *__dict__*
> 
> XXX: I haven't got a clue at the moment if the method should be an
> instance-, class- or staticmethod. The prototype uses a staticmethod.
> 
> XXX: My prototype automagicly makes this a static method, just like __new__ is
> made into a static method. That's more convenient, but also (too?) magical.
> 
> XXX: Should this raise AttributeError or return a magic value to signal that
> an attribute cannot be found (such as NotImplemented, used in the comparison
> operators)? I'm currently using an exception, a magical return value would
> be slightly more efficient because the exception machinery is not invoked.
> 
> 
> Alternative proposals
> ---------------------
> 
> Reuse ``tp_getattro``
> .....................
> 
> It would be nice to avoid adding a new slot, thus keeping the API simpler and
> easier to understand.  A comment on `Issue 18181`_ asked about reusing the
> ``tp_getattro`` slot, that is super could call the ``tp_getattro`` slot of all
> methods along the MRO.
> 
> AFAIK that won't work because ``tp_getattro`` will look in the instance
> ``__dict__`` before it tries to resolve attributes using classes in the MRO.
> This would mean that using ``tp_getattro`` instead of peeking the class
> dictionaries changes the semantics of the `super class`_.
> 
> 
> Open Issues
> ===========
> 
> * The names of the new slot and magic method are far from settled.
> 
> * I'm not too happy with the prototype for the new hook.
> 
> * Should ``__getattribute_super__`` be a class method instead?
> 
>   -> Yes? The method looks up a named attribute name of an object in
>      a specific class. Is also likely needed to deal with @classmethod
>      and super(Class, Class)
> 
> * Should ``__getattribute_super__`` be defined on object?
> 
>   -> Yes: makes it easier to delegate to the default implementation
> 
> * This doesn't necessarily work for class method super class
>    (e.g. super(object, object))...
> 
> 
> References
> ==========
> 
> * `Issue 18181`_ contains a prototype implementation
> 
> 
> Copyright
> =========
> 
> This document has been placed in the public domain.
> 
> .. _`Issue 18181`: http://bugs.python.org/issue18181
> 
> .. _`super class`: http://docs.python.org/3/library/functions.html?highlight=super#super
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
> 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com


From tjreedy at udel.edu  Sun Jul  7 23:37:55 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 07 Jul 2013 17:37:55 -0400
Subject: [Python-Dev] Rough idea for adding introspection information
	for builtins
In-Reply-To: <51D95282.8030401@hastings.org>
References: <5147ED55.30605@hastings.org> <ki8vdd$ne3$1@ger.gmane.org>
	<CADiSq7d-BzzbpYSr4nnyPJyDcnigLWvV8X40V=Pd_onwBL4yYQ@mail.gmail.com>
	<5148374D.6050402@hastings.org> <51D854DD.7080807@hastings.org>
	<CADiSq7eYz0KDxNKtiih9RVBgLEei8tA=uiXP1AnWKMd450ep7g@mail.gmail.com>
	<51D8B4D6.3080107@hastings.org>
	<CADiSq7e7zbGMaUp0+KZBa+dHjhpq-=UWsHdteb6vVVr3rLCFEw@mail.gmail.com>
	<51D8D6F2.80007@hastings.org>
	<B2BD18DF-87E4-43AC-83D7-DCD88D321FDC@mac.com>
	<51D95282.8030401@hastings.org>
Message-ID: <krcn3i$79p$1@ger.gmane.org>

On 7/7/2013 7:35 AM, Larry Hastings wrote:
>
>
> On 07/07/2013 07:19 AM, Ronald Oussoren wrote:
>> Not entirely on topic, but close enough: pydoc currently doesn't use
>> the __signature__ information at all. Adding such support would be
>> easy enough, see #17053 for an implementation ;-)
>
> True, it doesn't use inspect.signature, it uses inspect.getfullargspec.

This is currently true of Idle calltips. But with 3.2 out of the way, I 
plan to switch sometime and not worry about 2.7.

> Since I don't propose modifying inspect.getfullargspec to add the
> optional parameter group information, 17053 or something like it would
> have to happen.
>
>
> On 07/07/2013 07:25 AM, R. David Murray wrote:
>> Sorry to make your life more complicated, but unless I'm misunderstanding
>> something, issue 18220 (http://bugs.python.org/issue18220) throws another
>> monkey-wrench in to this.  If I'm understanding this discussion correctly,
>> that example:
>>
>>      islice(stop)
>>      islice(start, stop [, step])
>>
>> requires the multiple-signature approach.
>>
>> Note also that the python3 documentation has moved away from the []
>> notation wherever possible.
>
> It depends on what problem you're addressing.  In terms of the Argument
> Clinic DSL, and in terms of the static introspection information stored
> for builtins, someone (Nick?) suggested a refinement to the semantics:
> in the face of ambiguity, prefer the leftmost group(s) first.  That
> means that range() and islice() could be specified as follows:
>
>     range([start,] stop, [step])

This is how it was until last September. See #15831, which also changed 
max, min, and slice entries to use two lines. The multiple lines for 
bytes and str signatures in the docstrings were another issue.

> In terms of the documentation, it might be better to preserve the
> multiple-lines approach, as perhaps that's more obvious to the reader.

It seems that signatures that do not match what one can do with a def 
statement are confusing. The header line for the Python version of the 
above is
	def range(start_or_stop, stop=None, step=1):
My suggestion to use this, which is the actual signature, was rejected 
in favor of using two lines. This is fine with me, as it documents the 
calling signatures rather than the hybrid definition signature used to 
implement the call signatures.

> On the other hand: in Python 3, help(itertools.islice) uses solely the
> optional group syntax, on one line.

Because it has not yet been changed ;-).

> On 07/07/2013 07:25 AM, Ronald Oussoren wrote:
>> Signature objects use a name in angled brackets to indicate that a parameter is positional only, for example "input(<prompt>)". That might be an alternative to adding a "/" in the argument list in pydoc's output.
>
> I wasn't aware that Signature objects currently had any support
> whatsoever for positional-only parameters.  Yes, in theory they do, but
> in practice they have never seen one, because positional-only parameters
> only occur in builtins and Signature objects have no metadata for
> builtins.  (The very problem Argument Clinic eventually hopes to solve!)

I look forward to the day when accurate (auto-generated) call data is 
also available for C-coded functions.

-- 
Terry Jan Reedy


From Steve.Dower at microsoft.com  Mon Jul  8 17:19:42 2013
From: Steve.Dower at microsoft.com (Steve Dower)
Date: Mon, 8 Jul 2013 15:19:42 +0000
Subject: [Python-Dev] Hooking into super() attribute resolution
In-Reply-To: <CCA2B7DA-C3B7-44C1-B488-9D91B26F8C61@mac.com>
References: <DCEB2B77-B9DD-405A-A084-EFB761B947BB@mac.com>
	<46EC256E-9ACE-4C4C-BE3D-325D7DB87B85@mac.com>
	<0fe31345247846658b331aa358c38c66@BLUPR03MB035.namprd03.prod.outlook.com>,
	<CCA2B7DA-C3B7-44C1-B488-9D91B26F8C61@mac.com>
Message-ID: <5fe3649f4c3c41ba927ca62fd762699c@BLUPR03MB035.namprd03.prod.outlook.com>

The only real advantage is a simpler signature and more easily explained use (assuming the person you're explaining it to is familiar with metaclasses, so most of the hard explaining has been done).

I'm still not sure that this isn't simply a bug in super. If the superclass's metaclass provides a __getattr__ then it should probably use it and abandon it's own MRO traversal.

I still haven't thought the edge cases through, and it seems like there'd be some with that change, so that's where __getattribute_super__ comes in - super can call it without abandoning its MRO traversal.

AFAICT, the difference between that and __getlocalattribute__ is that the latter would be implemented on a metaclass while the former takes extra parameters. I think this functionality is advanced enough that requiring a metaclass isn't unreasonable.

(The proxy objects idea was a red herring, sorry :) )

Steve

Sent from my Windows Phone
________________________________
From: Ronald Oussoren<mailto:ronaldoussoren at mac.com>
Sent: ?7/?7/?2013 12:37
To: Steve Dower<mailto:Steve.Dower at microsoft.com>
Cc: python-dev at python.org<mailto:python-dev at python.org>
Subject: Re: [Python-Dev] Hooking into super() attribute resolution


On 7 Jul, 2013, at 17:17, Steve Dower <Steve.Dower at microsoft.com> wrote:

> Could the same result be achieved by hooking the MRO that super uses and returning a list of proxy objects?

What is the advantage over adding a hook to the class itself? That seems to be the right place to add such a hook, super already looks in the classes along the MRO and my proposal would add a formal interface for that instead of having super peek into the class __dict__. I have thought of using a custom mapping object for the tp_dict slot to intercept this, but that won't work because super assumes that tp_dict is an actual PyDictObject (and likely other parts of the interpreter do so as well).


> And then wouldn't you only really need a __getattribute__ that doesn't recurse (__getlocalattribute__)? The end result may be conceptually simpler, but you've thought through the edge cases better than I have.

__getattribute_super__ already is a kind of __getlocalattribute__, the primairy difference being __getattribute_super__ is a staticmethod instead of an instance method. To be honest I'm not sure if a staticmethod is the right solution, I'm having a hard time to determine if this should be a class, instance or static method.

Currently super(StartClass, x) basicly does (assuming x is an instance method):


def __getattribute__(self, name):
    mro = type(x).mro()
    idx = mro.index(StartClass)
    while idx < len(mro):
       dct = mro[idx].__dict__
       try:
          result = dct[name]
          # deal with descriptors here
          return result

       except KeyError:
           continue
    return object.__getattribute__(self, name)

With my proposal 'dct' would no longer be needed and 'result = dct[name]' would be 'mro[idx].__getattribute_super__(mro[idx], name, x, StartClass)' (I may have the last argument for the call to __getattribute_super__ wrong, but that's the idea). Given that the first argument of __get...super__ is the same as the object the method get getattr-ed from I guess the method should be a classmethod instead of an staticmethod. Changing that would be easy enough.

I'm still interested in feedback on the basic idea, I'd love to here that my proposal isn't necessary because there is already a way to get the behavior I'm looking for although that's nog going to happen ;-).

Ronald


>
> (Apologies for the HTML top-post)

I don't mind.

PS. Does anyone know if the pep editors are away (conferences, holidays, ...)? I could just check in my proposal in the peps repository, but as this is my first PEP I'd prefer to follow the documented procedure and have someone that knows what he's doing look at the metadata before checking in.

>
> Sent from my Windows Phone
> From: Ronald Oussoren
> Sent: ?7/?6/?2013 0:47
> To: Ronald Oussoren
> Cc: python-dev at python.org Dev
> Subject: Re: [Python-Dev] Hooking into super() attribute resolution
>
> I've updated the implementation in issue 18181 <http://bugs.python.org/issue18181> while adding some tests, and have updated the proposal as well.
>
> The proposal has some open issues at the moment, most important of which is the actual signature for the new special method; in particular I haven't been able to decide if this should be an instance-, class- or static method. It is a static method in the proposal and prototype, but I'm not convinced that that is the right solution.
>
> Ronald
>
>
>
>
> PEP: TODO
> Title: Hooking into super attribute resolution
> Version: $Revision$
> Last-Modified: $Date$
> Author: Ronald Oussoren <ronaldoussoren at mac.com>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 12-Jun-2013
> Post-History: 2-Jul-2013, ?
>
>
> Abstract
> ========
>
> In current python releases the attribute resolution of the `super class`_
> peeks in the ``__dict__`` attribute of classes on the MRO to look
> for attributes. This PEP introduces a hook that classes can use
> to override that behavior for specific classes.
>
>
> Rationale
> =========
>
> Peeking in the class ``__dict__`` works for regular classes, but can
> cause problems when a class dynamicly looks up attributes in a
> ``__getattribute__`` method.
>
> The new hook makes it possible to introduce the same customization for
> attribute lookup through the `super class`_.
>
>
> The superclass attribute lookup hook
> ====================================
>
> In C code
> ---------
>
> A new slot ``tp_getattro_super`` is added to the ``PyTypeObject`` struct. The
> ``tp_getattro`` slot for super will call this slot when it is not ``NULL``,
> and will raise an exception when it is not set (which shouldn't happen because
> the method is implemented for :class:`object`).
>
> The slot has the following prototype::
>
>     PyObject* (*getattrosuperfunc)(PyTypeObject* cls, PyObject* name,
>         PyObject* object, PyObject* owner);
>
> The function should perform attribute lookup on *object* for *name*, but only
> looking in type *tp* (which will be one of the types on the MRO for *self*)
> and without looking in the instance *__dict__*.
>
> The function returns ``NULL`` when the attribute cannot be found, and raises and
> exception. Exception other than ``AttributeError`` will cause failure of super's
> attribute resolution.
>
> The implementation of the slot for the :class:`object` type is
> ``PyObject_GenericGetAttrSuper``, which peeks in the ``tp_dict`` for *cls*.
>
> Note that *owner* and *object* will be the same object when using a
> class-mode super.
>
>
> In Python code
> --------------
>
> A Python class can contain a definition for a static method
> ``__getattribute_super__`` with the following prototype::
>
>    def __getattribute_super__(cls, name, object, owner): pass
>
> The method should perform attribute lookup for *name* on instance *self* while
> only looking at *cls* (it should not look in super classes or the instance
> *__dict__*
>
> XXX: I haven't got a clue at the moment if the method should be an
> instance-, class- or staticmethod. The prototype uses a staticmethod.
>
> XXX: My prototype automagicly makes this a static method, just like __new__ is
> made into a static method. That's more convenient, but also (too?) magical.
>
> XXX: Should this raise AttributeError or return a magic value to signal that
> an attribute cannot be found (such as NotImplemented, used in the comparison
> operators)? I'm currently using an exception, a magical return value would
> be slightly more efficient because the exception machinery is not invoked.
>
>
> Alternative proposals
> ---------------------
>
> Reuse ``tp_getattro``
> .....................
>
> It would be nice to avoid adding a new slot, thus keeping the API simpler and
> easier to understand.  A comment on `Issue 18181`_ asked about reusing the
> ``tp_getattro`` slot, that is super could call the ``tp_getattro`` slot of all
> methods along the MRO.
>
> AFAIK that won't work because ``tp_getattro`` will look in the instance
> ``__dict__`` before it tries to resolve attributes using classes in the MRO.
> This would mean that using ``tp_getattro`` instead of peeking the class
> dictionaries changes the semantics of the `super class`_.
>
>
> Open Issues
> ===========
>
> * The names of the new slot and magic method are far from settled.
>
> * I'm not too happy with the prototype for the new hook.
>
> * Should ``__getattribute_super__`` be a class method instead?
>
>   -> Yes? The method looks up a named attribute name of an object in
>      a specific class. Is also likely needed to deal with @classmethod
>      and super(Class, Class)
>
> * Should ``__getattribute_super__`` be defined on object?
>
>   -> Yes: makes it easier to delegate to the default implementation
>
> * This doesn't necessarily work for class method super class
>    (e.g. super(object, object))...
>
>
> References
> ==========
>
> * `Issue 18181`_ contains a prototype implementation
>
>
> Copyright
> =========
>
> This document has been placed in the public domain.
>
> .. _`Issue 18181`: http://bugs.python.org/issue18181
>
> .. _`super class`: http://docs.python.org/3/library/functions.html?highlight=super#super
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130708/ec8c9d0b/attachment.html>

From ronaldoussoren at mac.com  Mon Jul  8 17:57:47 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Mon, 8 Jul 2013 17:57:47 +0200
Subject: [Python-Dev] Hooking into super() attribute resolution
In-Reply-To: <5fe3649f4c3c41ba927ca62fd762699c@BLUPR03MB035.namprd03.prod.outlook.com>
References: <DCEB2B77-B9DD-405A-A084-EFB761B947BB@mac.com>
	<46EC256E-9ACE-4C4C-BE3D-325D7DB87B85@mac.com>
	<0fe31345247846658b331aa358c38c66@BLUPR03MB035.namprd03.prod.outlook.com>
	<CCA2B7DA-C3B7-44C1-B488-9D91B26F8C61@mac.com>
	<5fe3649f4c3c41ba927ca62fd762699c@BLUPR03MB035.namprd03.prod.outlook.com>
Message-ID: <7946F238-3DA7-4198-9CED-23B33246B26F@mac.com>


On 8 Jul, 2013, at 17:19, Steve Dower <Steve.Dower at microsoft.com> wrote:

> The only real advantage is a simpler signature and more easily explained use (assuming the person you're explaining it to is familiar with metaclasses, so most of the hard explaining has been done).

The signature is as complex as it is to be able to call descr.__get__ with the correct arguments. I ended up with the current signature when I added __getattribute_super__ to object and removed the tp_dict peeking code from super's tp_getattro.

A way to get a simpler interface again would be a method that returns an attribute *without* performing calls to descr.__get__. That could then be used for both __getattribute__ and super.__getattribute__, instead of peeking in a class' dictionary. I must admit that I haven't thought about the ramifactions of this (both functionally and performance wise).  This might end up being easier to explain: both normal attribute resolution and super's resolution would end up using the same mechanism, with the differences being that super doesn't begin resolution at the start of the mro and ignores the instance __dict__.  The disadvantage is introducing a new way to affect attribute resolution (do I use "__getattribute__" or this new method?). 

The new interface would be something like:

    @classmethod
    def __getlocalname__(cls, object, name):
        pass

Or as you mentioned later as a __getlocalname__ method on the metaclass. The "object" argument wouldn't be necessary to reproduce current functionality, and isn't necessary for my usecase as well, but a hook for attribute resolution on an instance that doesn't have access to that instance feels wrong.

> 
> I'm still not sure that this isn't simply a bug in super. If the superclass's metaclass provides a __getattr__ then it should probably use it and abandon it's own MRO traversal.

I'd have to think about this, but on first glance this would mean a change in the semantics that a metaclass' __getattr__ currently has.

> 
> I still haven't thought the edge cases through, and it seems like there'd be some with that change, so that's where __getattribute_super__ comes in - super can call it without abandoning its MRO traversal.
> 
> AFAICT, the difference between that and __getlocalattribute__ is that the latter would be implemented on a metaclass while the former takes extra parameters. I think this functionality is advanced enough that requiring a metaclass isn't unreasonable.

I'm not necessarily oppossed to a solution that requires using a metaclass, I already have metaclasses with custom metaclasses in PyObjC and this wouldn't add that much complexity to that :-)

Ronald


From ncoghlan at gmail.com  Tue Jul  9 00:59:03 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 9 Jul 2013 08:59:03 +1000
Subject: [Python-Dev] Rough idea for adding introspection information
	for builtins
In-Reply-To: <476B98A3-4701-4964-B2BE-AE0ABF75E29E@mac.com>
References: <5147ED55.30605@hastings.org> <ki8vdd$ne3$1@ger.gmane.org>
	<CADiSq7d-BzzbpYSr4nnyPJyDcnigLWvV8X40V=Pd_onwBL4yYQ@mail.gmail.com>
	<5148374D.6050402@hastings.org> <51D854DD.7080807@hastings.org>
	<CADiSq7eYz0KDxNKtiih9RVBgLEei8tA=uiXP1AnWKMd450ep7g@mail.gmail.com>
	<51D8B4D6.3080107@hastings.org>
	<CADiSq7e7zbGMaUp0+KZBa+dHjhpq-=UWsHdteb6vVVr3rLCFEw@mail.gmail.com>
	<51D8D6F2.80007@hastings.org>
	<B2BD18DF-87E4-43AC-83D7-DCD88D321FDC@mac.com>
	<51D95282.8030401@hastings.org>
	<5272EBDB-81E1-4B68-B899-242AC8A25D9F@mac.com>
	<51D9A342.4060404@hastings.org>
	<476B98A3-4701-4964-B2BE-AE0ABF75E29E@mac.com>
Message-ID: <CADiSq7c+fRotZWgnkAV3g9KJnUQo1SjWwYs+dF7EjbX=94i9+A@mail.gmail.com>

Since it's relevant: my recollection us that the current use of angle
brackets in inspect.Signature is just the default use of them for "there is
no canonical representation of this, but leaving them out would be
misleading" (I haven't checked if the PEP says that explicitly).

I previously forgot Larry, Guido & I discussed the appropriate use of
square brackets and the slash in the definition format at PyCon, so I now
think having the Argument Clinic PEP also cover their appropriate use in
the inspect.Signature string output is a good idea.

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130709/df2d267b/attachment.html>

From Steve.Dower at microsoft.com  Tue Jul  9 01:21:52 2013
From: Steve.Dower at microsoft.com (Steve Dower)
Date: Mon, 8 Jul 2013 23:21:52 +0000
Subject: [Python-Dev] Hooking into super() attribute resolution
In-Reply-To: <7946F238-3DA7-4198-9CED-23B33246B26F@mac.com>
References: <DCEB2B77-B9DD-405A-A084-EFB761B947BB@mac.com>
	<46EC256E-9ACE-4C4C-BE3D-325D7DB87B85@mac.com>
	<0fe31345247846658b331aa358c38c66@BLUPR03MB035.namprd03.prod.outlook.com>
	<CCA2B7DA-C3B7-44C1-B488-9D91B26F8C61@mac.com>
	<5fe3649f4c3c41ba927ca62fd762699c@BLUPR03MB035.namprd03.prod.outlook.com>
	<7946F238-3DA7-4198-9CED-23B33246B26F@mac.com>
Message-ID: <5339a8b72f264faa99a8720b5d1d161d@BLUPR03MB035.namprd03.prod.outlook.com>

> From: Ronald Oussoren [mailto:ronaldoussoren at mac.com]
> Sent: Monday, July 8, 2013 0858
>
> On 8 Jul, 2013, at 17:19, Steve Dower <Steve.Dower at microsoft.com> wrote:
> 
> > The only real advantage is a simpler signature and more easily explained
> use (assuming the person you're explaining it to is familiar with metaclasses,
> so most of the hard explaining has been done).
> 
> The signature is as complex as it is to be able to call descr.__get__ with the
> correct arguments. I ended up with the current signature when I added
> __getattribute_super__ to object and removed the tp_dict peeking code
> from super's tp_getattro.
> 
> A way to get a simpler interface again would be a method that returns an
> attribute *without* performing calls to descr.__get__. That could then be
> used for both __getattribute__ and super.__getattribute__, instead of
> peeking in a class' dictionary. I must admit that I haven't thought about the
> ramifactions of this (both functionally and performance wise).  This might
> end up being easier to explain: both normal attribute resolution and super's
> resolution would end up using the same mechanism, with the differences
> being that super doesn't begin resolution at the start of the mro and ignores
> the instance __dict__.  The disadvantage is introducing a new way to affect
> attribute resolution (do I use "__getattribute__" or this new method?).
> 
> The new interface would be something like:
> 
>     @classmethod
>     def __getlocalname__(cls, object, name):
>         pass
> 
> Or as you mentioned later as a __getlocalname__ method on the metaclass.
> The "object" argument wouldn't be necessary to reproduce current
> functionality, and isn't necessary for my usecase as well, but a hook for
> attribute resolution on an instance that doesn't have access to that instance
> feels wrong.

Except that if it's on a metaclass, the 'instance' it has access to is cls. The descriptor side of things is more interesting, but I see no reason why super can't do that itself, since it knows the actual instance to call __get__ with. (Presumably it already does this with the __dict__ lookup, since that won't call __get__ either.)

Explaining the new method is easiest if the default implementation is (literally):

def __getlocalname__(self, name):
    try:
        return self.__dict__[name]
    except KeyError:
        raise AttributeError(name)

which does not do any descriptor resolution (and is only a small step from simply replacing __dict__ with a custom object, which is basically where we started). The only change I've really suggested is making it an instance method that can be implemented on a metaclass if you want it for class members.

> >
> > I'm still not sure that this isn't simply a bug in super. If the superclass's
> metaclass provides a __getattr__ then it should probably use it and abandon
> it's own MRO traversal.
> 
> I'd have to think about this, but on first glance this would mean a change in
> the semantics that a metaclass' __getattr__ currently has.

Exactly. Probably not a great idea to change that.

> >
> > I still haven't thought the edge cases through, and it seems like there'd be
> some with that change, so that's where __getattribute_super__ comes in -
> super can call it without abandoning its MRO traversal.
> >
> > AFAICT, the difference between that and __getlocalattribute__ is that the
> latter would be implemented on a metaclass while the former takes extra
> parameters. I think this functionality is advanced enough that requiring a
> metaclass isn't unreasonable.
> 
> I'm not necessarily oppossed to a solution that requires using a metaclass, I
> already have metaclasses with custom metaclasses in PyObjC and this
> wouldn't add that much complexity to that :-)

I assumed you were - when I was working on similar sort of code they made life extremely easy. 

> Ronald

Steve



From victor.stinner at gmail.com  Tue Jul  9 01:39:14 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 9 Jul 2013 01:39:14 +0200
Subject: [Python-Dev] pyfailmalloc: new debug tool to test how your
	applications handles MemoryError
Message-ID: <CAMpsgwb1uoK46trEa+s9WOQ4PUQPeX1HCgqOifJn5CbXwjoAVA@mail.gmail.com>

Hi,

The PEP 445 (Add new APIs to customize Python memory allocators) has
been accepted, I commited its implementation. So it's time to have fun
with this API.

I developed a small Python module (150 lines of C code) to inject
memory allocation failures:
https://pypi.python.org/pypi/pyfailmalloc

This module can be used to test your application under very low memory
condition, test how your application handles MemoryError. I ran it in
the Python test suite: Python core and stdlib do not handle
MemoryError correctly everywhere...

I opened the following issue to track all bugs found by pytracemalloc
and related commits to fix these bugs:
http://bugs.python.org/issue18408

I already fixed bugs in PyObject_GC_NewVar(), PyDict_New(),
list.pop(), type.__bases__ setter, etc. I also found a bug in
_PyMem_DebugRealloc(): a function supposed to detect bugs :-)

Have fun with this new tool on your applications and Python modules.
Tell me if you would like to help me on issue #18408, there are still
a lot of bugs to fix.

Victor

From victor.stinner at gmail.com  Tue Jul  9 01:41:23 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 9 Jul 2013 01:41:23 +0200
Subject: [Python-Dev] pyfailmalloc: new debug tool to test how your
 applications handles MemoryError
In-Reply-To: <CAMpsgwb1uoK46trEa+s9WOQ4PUQPeX1HCgqOifJn5CbXwjoAVA@mail.gmail.com>
References: <CAMpsgwb1uoK46trEa+s9WOQ4PUQPeX1HCgqOifJn5CbXwjoAVA@mail.gmail.com>
Message-ID: <CAMpsgwaCayXw4QxuFf6Fm-wtpLg6FYAgUva0JBRML74=xAeHBg@mail.gmail.com>

2013/7/9 Victor Stinner <victor.stinner at gmail.com>:
> I developed a small Python module (150 lines of C code) to inject
> memory allocation failures:
> https://pypi.python.org/pypi/pyfailmalloc

Bitbucket was down, so I was unable to give the link to its source
code. The server is back, here is the C code:
https://bitbucket.org/haypo/pyfailmalloc/src/tip/failmalloc.c

You can see that it's now easy to develop such tools thanks to the PEP 445.

Victor

From ronaldoussoren at mac.com  Tue Jul  9 10:25:41 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Tue, 9 Jul 2013 10:25:41 +0200
Subject: [Python-Dev] Hooking into super() attribute resolution
In-Reply-To: <5339a8b72f264faa99a8720b5d1d161d@BLUPR03MB035.namprd03.prod.outlook.com>
References: <DCEB2B77-B9DD-405A-A084-EFB761B947BB@mac.com>
	<46EC256E-9ACE-4C4C-BE3D-325D7DB87B85@mac.com>
	<0fe31345247846658b331aa358c38c66@BLUPR03MB035.namprd03.prod.outlook.com>
	<CCA2B7DA-C3B7-44C1-B488-9D91B26F8C61@mac.com>
	<5fe3649f4c3c41ba927ca62fd762699c@BLUPR03MB035.namprd03.prod.outlook.com>
	<7946F238-3DA7-4198-9CED-23B33246B26F@mac.com>
	<5339a8b72f264faa99a8720b5d1d161d@BLUPR03MB035.namprd03.prod.outlook.com>
Message-ID: <07A27E22-AC02-46E9-8B02-1A25B94A061D@mac.com>


On 9 Jul, 2013, at 1:21, Steve Dower <Steve.Dower at microsoft.com> wrote:
>> 
> 
> Except that if it's on a metaclass, the 'instance' it has access to is cls. The descriptor side of things is more interesting, but I see no reason why super can't do that itself, since it knows the actual instance to call __get__ with. (Presumably it already does this with the __dict__ lookup, since that won't call __get__ either.)
> 
> Explaining the new method is easiest if the default implementation is (literally):
> 
> def __getlocalname__(self, name):
>    try:
>        return self.__dict__[name]
>    except KeyError:
>        raise AttributeError(name)
> 
> which does not do any descriptor resolution (and is only a small step from simply replacing __dict__ with a custom object, which is basically where we started). The only change I've really suggested is making it an instance method that can be implemented on a metaclass if you want it for class members.

I like this idea and will experiment with implementing this later this week.  The only thing I'm not sure about is how to indicate that the name could not be found, raising an exception could end up being to expensive if the __getlocalname__ hook gets used in object.__getattribute__ as well. I guess I'll have to run benchmarks to determine if this really is a problem.

Ronald

From martin at v.loewis.de  Tue Jul  9 20:15:13 2013
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 09 Jul 2013 20:15:13 +0200
Subject: [Python-Dev] Accepting PEP 445
In-Reply-To: <CAMpsgwZ6ddHVjhFobg5MsACCZUXQwONdQSyrU8EWBuiWLdZysQ@mail.gmail.com>
References: <20130706225454.0932f6ba@fsol>
	<BD4C1108-3778-407B-931A-D0369F03D56D@cavallinux.eu>
	<CAMpsgwZ6ddHVjhFobg5MsACCZUXQwONdQSyrU8EWBuiWLdZysQ@mail.gmail.com>
Message-ID: <51DC5331.8010409@v.loewis.de>

Am 07.07.13 01:04, schrieb Victor Stinner:
> 2013/7/6 Antonio Cavallo <a.cavallo at cavallinux.eu>:
>> Could that remove the need for the --with-pydebug flag?
> 
> With the PEP 445, you still have to recompile Python with
> --with-debug, but you don't have to recompile Python extensions
> anymore. 

Really? What about _PyObject_HEAD_EXTRA? The object layout
still changes under --with-pydebug, AFAICT.

Regards,
Martin



From christian at python.org  Fri Jul 12 01:58:19 2013
From: christian at python.org (Christian Heimes)
Date: Fri, 12 Jul 2013 01:58:19 +0200
Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL
Message-ID: <51DF469B.6010007@python.org>

Hi,

how do you feel about dropping Windows XP support for Python 3.4? It
would enable us to use some features that are only available on Windows
Vista and newer, for example http://bugs.python.org/issue6926 and
http://bugs.python.org/issue1763 .

PEP 11 says:
  A new feature release X.Y.0 will support all Windows releases
  whose extended support phase is not yet expired.

For Python 3.4 is going to be a very close call. According to PEP 429
3.4.0 final is scheduled for February 22, 2014. The extended support
phase of Windows XP ends merely 45 days later on April 8, 2014. Do we
really have to restrict ourselves to an API that is going to become
deprecated 45 days after the estimated release of 3.4.0?

Christian

From Steve.Dower at microsoft.com  Fri Jul 12 02:11:55 2013
From: Steve.Dower at microsoft.com (Steve Dower)
Date: Fri, 12 Jul 2013 00:11:55 +0000
Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL
In-Reply-To: <51DF469B.6010007@python.org>
References: <51DF469B.6010007@python.org>
Message-ID: <a11abcf4949a4e31b7989c7372088475@BLUPR03MB199.namprd03.prod.outlook.com>

+1. And maybe amend PEP 11 to specify "whose extended support phase does not expire within 6 months of release"? (I picked 6 for no particular reason.)

I don't see any good reason for Python to support an OS that Microsoft doesn't, but once 3.4.0 has been released with XP support it can't really be taken out for 3.4.1. Since 3.4.1 is almost certainly going to be after the end of extended support, better to drop it for .0

Steve

> From: Christian Heimes
> Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL
> 
> Hi,
> 
> how do you feel about dropping Windows XP support for Python 3.4? It
> would enable us to use some features that are only available on Windows
> Vista and newer, for example http://bugs.python.org/issue6926 and
> http://bugs.python.org/issue1763 .
> 
> PEP 11 says:
>   A new feature release X.Y.0 will support all Windows releases
>   whose extended support phase is not yet expired.
> 
> For Python 3.4 is going to be a very close call. According to PEP 429
> 3.4.0 final is scheduled for February 22, 2014. The extended support
> phase of Windows XP ends merely 45 days later on April 8, 2014. Do we
> really have to restrict ourselves to an API that is going to become
> deprecated 45 days after the estimated release of 3.4.0?
> 
> Christian



From ethan at stoneleaf.us  Fri Jul 12 02:06:33 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Thu, 11 Jul 2013 17:06:33 -0700
Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL
In-Reply-To: <51DF469B.6010007@python.org>
References: <51DF469B.6010007@python.org>
Message-ID: <51DF4889.4030007@stoneleaf.us>

On 07/11/2013 04:58 PM, Christian Heimes wrote:
>
> how do you feel about dropping Windows XP support for Python 3.4? It
> would enable us to use some features that are only available on Windows
> Vista and newer, for example http://bugs.python.org/issue6926 and
> http://bugs.python.org/issue1763 .
>
> PEP 11 says:
>    A new feature release X.Y.0 will support all Windows releases
>    whose extended support phase is not yet expired.
>
> For Python 3.4 is going to be a very close call. According to PEP 429
> 3.4.0 final is scheduled for February 22, 2014. The extended support
> phase of Windows XP ends merely 45 days later on April 8, 2014. Do we
> really have to restrict ourselves to an API that is going to become
> deprecated 45 days after the estimated release of 3.4.0?

Just because Microsoft no longer supports it doesn't mean there aren't still a lot of machines using it.  Of course, I 
suppose those folks can just stick with 3.3.  ;)

--
~Ethan~

From benhoyt at gmail.com  Fri Jul 12 03:49:54 2013
From: benhoyt at gmail.com (Ben Hoyt)
Date: Fri, 12 Jul 2013 13:49:54 +1200
Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL
In-Reply-To: <51DF469B.6010007@python.org>
References: <51DF469B.6010007@python.org>
Message-ID: <CAL9jXCH9MSarDGxjR-oUFyBV1Gp2AxaBExdBQfNFHmJ5rF+qwA@mail.gmail.com>

I guess it has to be dropped at some stage, but with Windows XP it's a case
of "XP is dead. Long live XP!" There are still an awful lot of XP boxes out
there, and I'd kind hate to see support dropped completely. We still use it
here at home.

Wikipedia/Net Applications says that Windows XP has still has a full 37% of
market share! (http://en.wikipedia.org/wiki/Usage_share_of_operating_systems
)

What about just have these attributes/functions on OSes that support it,
for example os.kill on Python 2.6 vs 2.7?

-Ben


On Fri, Jul 12, 2013 at 11:58 AM, Christian Heimes <christian at python.org>wrote:

> Hi,
>
> how do you feel about dropping Windows XP support for Python 3.4? It
> would enable us to use some features that are only available on Windows
> Vista and newer, for example http://bugs.python.org/issue6926 and
> http://bugs.python.org/issue1763 .
>
> PEP 11 says:
>   A new feature release X.Y.0 will support all Windows releases
>   whose extended support phase is not yet expired.
>
> For Python 3.4 is going to be a very close call. According to PEP 429
> 3.4.0 final is scheduled for February 22, 2014. The extended support
> phase of Windows XP ends merely 45 days later on April 8, 2014. Do we
> really have to restrict ourselves to an API that is going to become
> deprecated 45 days after the estimated release of 3.4.0?
>
> Christian
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/benhoyt%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130712/331518fd/attachment.html>

From rdmurray at bitdance.com  Fri Jul 12 04:28:49 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Thu, 11 Jul 2013 22:28:49 -0400
Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL
In-Reply-To: <CAL9jXCH9MSarDGxjR-oUFyBV1Gp2AxaBExdBQfNFHmJ5rF+qwA@mail.gmail.com>
References: <51DF469B.6010007@python.org>
	<CAL9jXCH9MSarDGxjR-oUFyBV1Gp2AxaBExdBQfNFHmJ5rF+qwA@mail.gmail.com>
Message-ID: <20130712022850.CB07A250066@webabinitio.net>

On Fri, 12 Jul 2013 13:49:54 +1200, Ben Hoyt <benhoyt at gmail.com> wrote:
> I guess it has to be dropped at some stage, but with Windows XP it's a case
> of "XP is dead. Long live XP!" There are still an awful lot of XP boxes out
> there, and I'd kind hate to see support dropped completely. We still use it
> here at home.
> 
> Wikipedia/Net Applications says that Windows XP has still has a full 37% of
> market share! (http://en.wikipedia.org/wiki/Usage_share_of_operating_systems
> )
> 
> What about just have these attributes/functions on OSes that support it,
> for example os.kill on Python 2.6 vs 2.7?

I'm afraid it's not that simple.  The issue (as I understand it from Crys)
is that we compile using setting that prevent the advanced features being
used, and that's really the only way to do it.  That is, you can only get
the advanced features by using certain settings, and if you use those,
the compiled code won't run on XP.  So it is not practical to decide only
at runtime to support the advanced feature, meaning there would have to
be a differently compiled version of Python specifically for Windows XP
(and doubtless new XP-specific ifdefs *as well*), and I doubt the core
team is going to go there.

The older versions of Python won't be going away.  Those can still be
used on XP.  Of course, they won't get bug fixes...just like XP itself.

--David

From benhoyt at gmail.com  Fri Jul 12 04:59:40 2013
From: benhoyt at gmail.com (Ben Hoyt)
Date: Fri, 12 Jul 2013 14:59:40 +1200
Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL
In-Reply-To: <20130712022850.CB07A250066@webabinitio.net>
References: <51DF469B.6010007@python.org>
	<CAL9jXCH9MSarDGxjR-oUFyBV1Gp2AxaBExdBQfNFHmJ5rF+qwA@mail.gmail.com>
	<20130712022850.CB07A250066@webabinitio.net>
Message-ID: <CAL9jXCFUvz7-HiQu5+9+mnBS8tU4qs1f3iv3zCZnBgW_DAZGLQ@mail.gmail.com>

Ah, yeah, that makes sense -- thanks for the further explanation. True
about older versions of Python "not going away".

> What about just have these attributes/functions on OSes that support it,

> > for example os.kill on Python 2.6 vs 2.7?
>
> I'm afraid it's not that simple.  The issue (as I understand it from Crys)
> is that we compile using setting that prevent the advanced features being
> used, and that's really the only way to do it.  That is, you can only get
> the advanced features by using certain settings, and if you use those,
> the compiled code won't run on XP.  So it is not practical to decide only
> at runtime to support the advanced feature, meaning there would have to
> be a differently compiled version of Python specifically for Windows XP
> (and doubtless new XP-specific ifdefs *as well*), and I doubt the core
> team is going to go there.
>
> The older versions of Python won't be going away.  Those can still be
> used on XP.  Of course, they won't get bug fixes...just like XP itself.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130712/01e3765c/attachment.html>

From stephen at xemacs.org  Fri Jul 12 05:16:31 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 12 Jul 2013 12:16:31 +0900
Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL
In-Reply-To: <a11abcf4949a4e31b7989c7372088475@BLUPR03MB199.namprd03.prod.outlook.com>
References: <51DF469B.6010007@python.org>
	<a11abcf4949a4e31b7989c7372088475@BLUPR03MB199.namprd03.prod.outlook.com>
Message-ID: <87k3kwtqls.fsf@uwakimon.sk.tsukuba.ac.jp>

Steve Dower writes:

 > I don't see any good reason for Python to support an OS that
 > Microsoft doesn't,

How about the *users* of that OS?

I don't see any good reason to take into account what Microsoft does
or doesn't support.  If that lack of support leads to Python users
dropping XP like hot potatoes, that will be visible in itself.  I
doubt it will, though.  EOL for XP has been coming a long long time,
far longer than Microsoft anticipated ;-), yet usage persists (most
small businesses I know in Japan are still using XP-based apps, small
sample, I admit).

If Python support for XP leads to significant pain for the majority of
Python users, or the majority on Windows, that's a good reason to drop
it, (which needs to be balanced against users who still need support).

Regards,



From regebro at gmail.com  Fri Jul 12 05:27:28 2013
From: regebro at gmail.com (Lennart Regebro)
Date: Fri, 12 Jul 2013 05:27:28 +0200
Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL
In-Reply-To: <a11abcf4949a4e31b7989c7372088475@BLUPR03MB199.namprd03.prod.outlook.com>
References: <51DF469B.6010007@python.org>
	<a11abcf4949a4e31b7989c7372088475@BLUPR03MB199.namprd03.prod.outlook.com>
Message-ID: <CAL0kPAVzP7yE+56EVJoQB1XhX6CswzB4CNBMpG=8TRYgb2sw_w@mail.gmail.com>

On Fri, Jul 12, 2013 at 2:11 AM, Steve Dower <Steve.Dower at microsoft.com> wrote:
> +1. And maybe amend PEP 11 to specify "whose extended support phase does not expire within 6 months of release"? (I picked 6 for no particular reason.)

Why have the specification in PEP 11 if we feel we can change the
rules arbitrarily when we feel like it?

//Lennart

From ben+python at benfinney.id.au  Fri Jul 12 07:23:00 2013
From: ben+python at benfinney.id.au (Ben Finney)
Date: Fri, 12 Jul 2013 15:23:00 +1000
Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL
References: <51DF469B.6010007@python.org>
	<a11abcf4949a4e31b7989c7372088475@BLUPR03MB199.namprd03.prod.outlook.com>
	<87k3kwtqls.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <7wmwps1he3.fsf@benfinney.id.au>

"Stephen J. Turnbull" <stephen at xemacs.org> writes:

> I don't see any good reason to take into account what Microsoft does
> or doesn't support.

It seems you're advocating a position quite ad odds with
<URL:http://www.python.org/dev/peps/pep-0011/#id7>. Can you propose an
amendment to PEP 11 that would remove that consideration?

-- 
 \           ?If you do not trust the source do not use this program.? |
  `\                                ?Microsoft Vista security dialogue |
_o__)                                                                  |
Ben Finney


From ncoghlan at gmail.com  Fri Jul 12 07:32:27 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 12 Jul 2013 15:32:27 +1000
Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL
In-Reply-To: <CAL0kPAVzP7yE+56EVJoQB1XhX6CswzB4CNBMpG=8TRYgb2sw_w@mail.gmail.com>
References: <51DF469B.6010007@python.org>
	<a11abcf4949a4e31b7989c7372088475@BLUPR03MB199.namprd03.prod.outlook.com>
	<CAL0kPAVzP7yE+56EVJoQB1XhX6CswzB4CNBMpG=8TRYgb2sw_w@mail.gmail.com>
Message-ID: <CADiSq7c_egbrFXtWAmLHMxBsZZMGGktioi4_eyHe84G9BgVfVQ@mail.gmail.com>

On 12 July 2013 13:27, Lennart Regebro <regebro at gmail.com> wrote:

> On Fri, Jul 12, 2013 at 2:11 AM, Steve Dower <Steve.Dower at microsoft.com>
> wrote:
> > +1. And maybe amend PEP 11 to specify "whose extended support phase does
> not expire within 6 months of release"? (I picked 6 for no particular
> reason.)
>
> Why have the specification in PEP 11 if we feel we can change the
> rules arbitrarily when we feel like it?
>

Because process PEPs are documentation of community practice, not an
inviolable constraint (e.g. PEP 1 has frequently lagged behind what we
*actually* do, and only been updated once we noticed we had drifted away
from the nominal procedures). In this case, the question of "What do we do
when a Windows version goes EOL shortly after a Python release?" hasn't
come up before, so PEP 11 has never had to take it into account.

That doesn't mean we *should* change it, it just means the option is one we
have available to us.

Fixing issue 6926 only requires setting the minimum API version to Windows
*XP*, so it isn't actually relevant to the question of whether or not to
drop support for XP (only W2k, which I thought we already dropped, but we
mustn't have bumped the minimum Windows API version at the time).

Issue 1763 looks like it could be better solved through pywin32 than
through standard library changes. It certainly doesn't appear to be worth
the cost of dropping Windows XP support.

Unless there are more compelling examples of APIs that we can't access
through Windows XP compatible interfaces, -1 on the change for 3.4.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130712/966f43a6/attachment.html>

From solipsis at pitrou.net  Fri Jul 12 09:45:11 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 12 Jul 2013 09:45:11 +0200
Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL
References: <51DF469B.6010007@python.org>
	<CAL9jXCH9MSarDGxjR-oUFyBV1Gp2AxaBExdBQfNFHmJ5rF+qwA@mail.gmail.com>
	<20130712022850.CB07A250066@webabinitio.net>
Message-ID: <20130712094511.16be8157@fsol>

On Thu, 11 Jul 2013 22:28:49 -0400
"R. David Murray" <rdmurray at bitdance.com> wrote:
> On Fri, 12 Jul 2013 13:49:54 +1200, Ben Hoyt <benhoyt at gmail.com> wrote:
> > I guess it has to be dropped at some stage, but with Windows XP it's a case
> > of "XP is dead. Long live XP!" There are still an awful lot of XP boxes out
> > there, and I'd kind hate to see support dropped completely. We still use it
> > here at home.
> > 
> > Wikipedia/Net Applications says that Windows XP has still has a full 37% of
> > market share! (http://en.wikipedia.org/wiki/Usage_share_of_operating_systems
> > )
> > 
> > What about just have these attributes/functions on OSes that support it,
> > for example os.kill on Python 2.6 vs 2.7?
> 
> I'm afraid it's not that simple.  The issue (as I understand it from Crys)
> is that we compile using setting that prevent the advanced features being
> used, and that's really the only way to do it.  That is, you can only get
> the advanced features by using certain settings, and if you use those,
> the compiled code won't run on XP.  So it is not practical to decide only
> at runtime to support the advanced feature,

It's not practical, but we already do it.  See
check_GetFinalPathNameByHandle in Modules/posixmodule.c.

Regards

Antoine.



From christian at python.org  Fri Jul 12 13:32:52 2013
From: christian at python.org (Christian Heimes)
Date: Fri, 12 Jul 2013 13:32:52 +0200
Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL
In-Reply-To: <CAL9jXCH9MSarDGxjR-oUFyBV1Gp2AxaBExdBQfNFHmJ5rF+qwA@mail.gmail.com>
References: <51DF469B.6010007@python.org>
	<CAL9jXCH9MSarDGxjR-oUFyBV1Gp2AxaBExdBQfNFHmJ5rF+qwA@mail.gmail.com>
Message-ID: <51DFE964.9020006@python.org>

Am 12.07.2013 03:49, schrieb Ben Hoyt:
> I guess it has to be dropped at some stage, but with Windows XP it's a
> case of "XP is dead. Long live XP!" There are still an awful lot of XP
> boxes out there, and I'd kind hate to see support dropped completely. We
> still use it here at home.
> 
> Wikipedia/Net Applications says that Windows XP has still has a full 37%
> of market share!
> (http://en.wikipedia.org/wiki/Usage_share_of_operating_systems)

I'm not planing to shut Windows XP out from Python completely. Users of
Python can still use Python 3.3 or 2.7. Python 3.3 will get a final bug
fix release after the release of Python 3.4.0 and security fixes until 2017.

Windows XP is really, *really* old. It has been released almost 12 years
ago. Linux kernel 2.4.0 was released about the same time. Its mainstream
support has ended 4 years ago. Do people really expect that they can run
the latest version of a program on a decommissioned operating system?

Christian

From murman at gmail.com  Fri Jul 12 16:05:59 2013
From: murman at gmail.com (Michael Urman)
Date: Fri, 12 Jul 2013 09:05:59 -0500
Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL
In-Reply-To: <51DF469B.6010007@python.org>
References: <51DF469B.6010007@python.org>
Message-ID: <CAOpBPYUvxVro0s8Zwv=EZJLMQy_0X8aDCKu72KCKndbo94CyuQ@mail.gmail.com>

On Thu, Jul 11, 2013 at 6:58 PM, Christian Heimes <christian at python.org> wrote:
> For Python 3.4 is going to be a very close call. According to PEP 429
> 3.4.0 final is scheduled for February 22, 2014. The extended support
> phase of Windows XP ends merely 45 days later on April 8, 2014. Do we
> really have to restrict ourselves to an API that is going to become
> deprecated 45 days after the estimated release of 3.4.0?

If your motivation is to ease the use of APIs only available on
Windows Vista and later, you've got another year to wait: Windows
Server 2003 R2 extended support lasts through until July 2015.
http://support.microsoft.com/lifecycle/search/default.aspx?alpha=Windows+Server+2003+R2

Michael

From guido at python.org  Fri Jul 12 17:50:49 2013
From: guido at python.org (Guido van Rossum)
Date: Fri, 12 Jul 2013 08:50:49 -0700
Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL
In-Reply-To: <51DFE964.9020006@python.org>
References: <51DF469B.6010007@python.org>
	<CAL9jXCH9MSarDGxjR-oUFyBV1Gp2AxaBExdBQfNFHmJ5rF+qwA@mail.gmail.com>
	<51DFE964.9020006@python.org>
Message-ID: <CAP7+vJK=Y=_mbC1CUAenvrTip=X8y4NQZHfsXfbZ8y4kUijq1g@mail.gmail.com>

You underestimate the reach of XP. For older or underpowered hardware
outside the developed world it is still the de facto choice. And it
definitely is the best version of Windows ever. None of the Win98 crap and
none of the Vista junk.

Telling people to go install Ubuntu is not really fair if others around
them don't or if they need certain software.

--Guido van Rossum (sent from Android phone)
On Jul 12, 2013 4:34 AM, "Christian Heimes" <christian at python.org> wrote:

> Am 12.07.2013 03:49, schrieb Ben Hoyt:
> > I guess it has to be dropped at some stage, but with Windows XP it's a
> > case of "XP is dead. Long live XP!" There are still an awful lot of XP
> > boxes out there, and I'd kind hate to see support dropped completely. We
> > still use it here at home.
> >
> > Wikipedia/Net Applications says that Windows XP has still has a full 37%
> > of market share!
> > (http://en.wikipedia.org/wiki/Usage_share_of_operating_systems)
>
> I'm not planing to shut Windows XP out from Python completely. Users of
> Python can still use Python 3.3 or 2.7. Python 3.3 will get a final bug
> fix release after the release of Python 3.4.0 and security fixes until
> 2017.
>
> Windows XP is really, *really* old. It has been released almost 12 years
> ago. Linux kernel 2.4.0 was released about the same time. Its mainstream
> support has ended 4 years ago. Do people really expect that they can run
> the latest version of a program on a decommissioned operating system?
>
> Christian
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130712/ffa01ec9/attachment.html>

From status at bugs.python.org  Fri Jul 12 18:07:25 2013
From: status at bugs.python.org (Python tracker)
Date: Fri, 12 Jul 2013 18:07:25 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20130712160725.F31FD56A2D@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2013-07-05 - 2013-07-12)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    4067 ( -5)
  closed 26140 (+74)
  total  30207 (+69)

Open issues with patches: 1834 


Issues opened (49)
==================

#7457: Adding a read_pkg_file to DistributionMetadata
http://bugs.python.org/issue7457  reopened by jason.coombs

#18366: Look into using setuptools 0.8 with devinabox
http://bugs.python.org/issue18366  opened by brett.cannon

#18367: See if a venv setup can be used for devinabox for coverage
http://bugs.python.org/issue18367  opened by brett.cannon

#18368: PyOS_StdioReadline() leaks memory when realloc() fails
http://bugs.python.org/issue18368  opened by christian.heimes

#18369: X509 cert class for ssl module
http://bugs.python.org/issue18369  opened by christian.heimes

#18372: _Pickler_New() doesn't call PyObject_GC_Track(self)
http://bugs.python.org/issue18372  opened by christian.heimes

#18373: implement sys.get/setbyteswarningflag()
http://bugs.python.org/issue18373  opened by dholth

#18374: ast.parse gives wrong position (col_offset) for some BinOp-s
http://bugs.python.org/issue18374  opened by Aivar.Annamaa

#18375: python -m test --randseed 1234 does not randomize tests
http://bugs.python.org/issue18375  opened by flox

#18376: show the effective count of process when running the testsuite
http://bugs.python.org/issue18376  opened by flox

#18378: locale.getdefaultlocale() fails on Mac OS X with default langu
http://bugs.python.org/issue18378  opened by Dmitry.Jemerov

#18379: SSLSocket.getpeercert(): OCSP and CRL DP URIs
http://bugs.python.org/issue18379  opened by christian.heimes

#18381: unittest warnings counter
http://bugs.python.org/issue18381  opened by giampaolo.rodola

#18382: multiprocessing's overlapped PipeConnection issues on Windows 
http://bugs.python.org/issue18382  opened by wprins

#18383: test_warnings modifies warnings.filters when running with "-W 
http://bugs.python.org/issue18383  opened by flox

#18384: Add devhelp build instructions to the documentation makefile
http://bugs.python.org/issue18384  opened by brouberol

#18387: Add 'symbols' link to pydoc's html menu bar.
http://bugs.python.org/issue18387  opened by ron_adam

#18389: document that os.path.relpath does not interrogate the file sy
http://bugs.python.org/issue18389  opened by jveldridge

#18391: socket.fromfd()'s API is difficult or impossible to use correc
http://bugs.python.org/issue18391  opened by cks

#18392: Doc: PyObject_Malloc() is not documented
http://bugs.python.org/issue18392  opened by haypo

#18393: Gestalt() is deprecated on OSX 10.8, remove support?
http://bugs.python.org/issue18393  opened by ronaldoussoren

#18394: cgi.FieldStorage triggers ResourceWarning sometimes
http://bugs.python.org/issue18394  opened by flox

#18395: Make _Py_char2wchar() and _Py_wchar2char() public
http://bugs.python.org/issue18395  opened by haypo

#18396: test_signal.test_issue9324() fails on buildbot AMD64 Windows7 
http://bugs.python.org/issue18396  opened by haypo

#18397: Python with MinGW
http://bugs.python.org/issue18397  opened by Friedrich.Spee.von.Langenfeld

#18400: Minor increase to Pickle test coverage
http://bugs.python.org/issue18400  opened by davidsackett

#18401: Tests for pdb import ~/.pdbrc
http://bugs.python.org/issue18401  opened by sptonkin

#18402: Finding perl64
http://bugs.python.org/issue18402  opened by mjdorma

#18404: Memory leak in gdbmmodule
http://bugs.python.org/issue18404  opened by bkabrda

#18405: crypt.mksalt() result has unnecessarily low entropy
http://bugs.python.org/issue18405  opened by christian.heimes

#18406: unicodedata.itergraphemes / str.itergraphemes / str.graphemes
http://bugs.python.org/issue18406  opened by dpk

#18407: Fix compiler warnings in pythoncore for Win64
http://bugs.python.org/issue18407  opened by jkloth

#18408: Fixes crashes found by pyfailmalloc
http://bugs.python.org/issue18408  opened by haypo

#18409: IDLE Improvements: Unit test for AutoComplete.py
http://bugs.python.org/issue18409  opened by philwebster

#18410: IDLE Improvements: Unit test for SearchDialog.py
http://bugs.python.org/issue18410  opened by philwebster

#18416: Move to absolute file paths for module.__file__
http://bugs.python.org/issue18416  opened by brett.cannon

#18417: urlopen() has a hidden default for its timeout argument
http://bugs.python.org/issue18417  opened by barry

#18418: Thread.isAlive() sometimes True after fork
http://bugs.python.org/issue18418  opened by emptysquare

#18421: Refactor call_with_frame() function of pyexpat.c
http://bugs.python.org/issue18421  opened by haypo

#18423: Document limitations on -m
http://bugs.python.org/issue18423  opened by acooke

#18425: IDLE Unit test for IdleHistory.py
http://bugs.python.org/issue18425  opened by JayKrish

#18426: Crash when extension does not use PyModule_Create()
http://bugs.python.org/issue18426  opened by Padowan

#18428: IDLE startup error
http://bugs.python.org/issue18428  opened by mwei11

#18429: IDLE: Format Paragraph doesn't function with comment blocks
http://bugs.python.org/issue18429  opened by Todd.Rovito

#18430: gzip, bz2, lzma: peek advances file position of existing file 
http://bugs.python.org/issue18430  opened by nbargnesi

#18431: Encoded words in structured headers are not decoded by the new
http://bugs.python.org/issue18431  opened by r.david.murray

#18432: sched modules queue property should return a list, not an iter
http://bugs.python.org/issue18432  opened by rhettinger

#18433: Clarify venv docs about which methods check for pre-existing p
http://bugs.python.org/issue18433  opened by brett.cannon

#18434: Update venv example to use setuptools
http://bugs.python.org/issue18434  opened by brett.cannon



Most recent 15 issues with no replies (15)
==========================================

#18434: Update venv example to use setuptools
http://bugs.python.org/issue18434

#18433: Clarify venv docs about which methods check for pre-existing p
http://bugs.python.org/issue18433

#18431: Encoded words in structured headers are not decoded by the new
http://bugs.python.org/issue18431

#18429: IDLE: Format Paragraph doesn't function with comment blocks
http://bugs.python.org/issue18429

#18425: IDLE Unit test for IdleHistory.py
http://bugs.python.org/issue18425

#18423: Document limitations on -m
http://bugs.python.org/issue18423

#18421: Refactor call_with_frame() function of pyexpat.c
http://bugs.python.org/issue18421

#18410: IDLE Improvements: Unit test for SearchDialog.py
http://bugs.python.org/issue18410

#18409: IDLE Improvements: Unit test for AutoComplete.py
http://bugs.python.org/issue18409

#18405: crypt.mksalt() result has unnecessarily low entropy
http://bugs.python.org/issue18405

#18401: Tests for pdb import ~/.pdbrc
http://bugs.python.org/issue18401

#18400: Minor increase to Pickle test coverage
http://bugs.python.org/issue18400

#18395: Make _Py_char2wchar() and _Py_wchar2char() public
http://bugs.python.org/issue18395

#18394: cgi.FieldStorage triggers ResourceWarning sometimes
http://bugs.python.org/issue18394

#18392: Doc: PyObject_Malloc() is not documented
http://bugs.python.org/issue18392



Most recent 15 issues waiting for review (15)
=============================================

#18432: sched modules queue property should return a list, not an iter
http://bugs.python.org/issue18432

#18430: gzip, bz2, lzma: peek advances file position of existing file 
http://bugs.python.org/issue18430

#18429: IDLE: Format Paragraph doesn't function with comment blocks
http://bugs.python.org/issue18429

#18425: IDLE Unit test for IdleHistory.py
http://bugs.python.org/issue18425

#18418: Thread.isAlive() sometimes True after fork
http://bugs.python.org/issue18418

#18408: Fixes crashes found by pyfailmalloc
http://bugs.python.org/issue18408

#18407: Fix compiler warnings in pythoncore for Win64
http://bugs.python.org/issue18407

#18405: crypt.mksalt() result has unnecessarily low entropy
http://bugs.python.org/issue18405

#18404: Memory leak in gdbmmodule
http://bugs.python.org/issue18404

#18402: Finding perl64
http://bugs.python.org/issue18402

#18400: Minor increase to Pickle test coverage
http://bugs.python.org/issue18400

#18396: test_signal.test_issue9324() fails on buildbot AMD64 Windows7 
http://bugs.python.org/issue18396

#18393: Gestalt() is deprecated on OSX 10.8, remove support?
http://bugs.python.org/issue18393

#18389: document that os.path.relpath does not interrogate the file sy
http://bugs.python.org/issue18389

#18387: Add 'symbols' link to pydoc's html menu bar.
http://bugs.python.org/issue18387



Top 10 most discussed issues (10)
=================================

#18408: Fixes crashes found by pyfailmalloc
http://bugs.python.org/issue18408  14 msgs

#10042: functools.total_ordering fails to handle NotImplemented correc
http://bugs.python.org/issue10042  12 msgs

#18257: Two copies of python-config
http://bugs.python.org/issue18257  12 msgs

#18378: locale.getdefaultlocale() fails on Mac OS X with default langu
http://bugs.python.org/issue18378  10 msgs

#18100: socket.sendall() cannot send buffers of 2GB or more
http://bugs.python.org/issue18100   9 msgs

#18365: Idle: mock Text class and test thereof
http://bugs.python.org/issue18365   9 msgs

#18430: gzip, bz2, lzma: peek advances file position of existing file 
http://bugs.python.org/issue18430   9 msgs

#18389: document that os.path.relpath does not interrogate the file sy
http://bugs.python.org/issue18389   8 msgs

#18397: Python with MinGW
http://bugs.python.org/issue18397   8 msgs

#18426: Crash when extension does not use PyModule_Create()
http://bugs.python.org/issue18426   8 msgs



Issues closed (72)
==================

#2554: test_ioctl failed Python 2.6a2 Solaris 10 SUN C
http://bugs.python.org/issue2554  closed by christian.heimes

#3093: Namespace pollution from multiprocessing
http://bugs.python.org/issue3093  closed by christian.heimes

#3266: Python-2.5.2/Modules/mmapmodule.c:915: error: `O_RDWR' undecla
http://bugs.python.org/issue3266  closed by christian.heimes

#3329: API for setting the memory allocator used by Python
http://bugs.python.org/issue3329  closed by haypo

#5060: gcc profile guided optimization
http://bugs.python.org/issue5060  closed by christian.heimes

#7573: Position independent include of Python.h
http://bugs.python.org/issue7573  closed by christian.heimes

#7686: redundant open modes 'rbb', 'wbb', 'abb' no longer work on Win
http://bugs.python.org/issue7686  closed by christian.heimes

#9838: Inadequate C-API to Python 3 I/O objects
http://bugs.python.org/issue9838  closed by christian.heimes

#11012: Add log1p(), exp1m(), gamma(), and lgamma() to cmath
http://bugs.python.org/issue11012  closed by rhettinger

#11366: xml.etree.ElementTree.Element: docs should mention immediate s
http://bugs.python.org/issue11366  closed by christian.heimes

#11908: Weird `slice.stop or sys.maxint`
http://bugs.python.org/issue11908  closed by rhettinger

#12115: some tests need to be skipped on threadless systems
http://bugs.python.org/issue12115  closed by christian.heimes

#12272: Python 2.7.1 version conflict for package "Tcl" on Windows 7
http://bugs.python.org/issue12272  closed by christian.heimes

#12984: XML NamedNodeMap ( attribName in NamedNodeMap fails )
http://bugs.python.org/issue12984  closed by christian.heimes

#12990: launcher can't work on path including tradition chinese	char
http://bugs.python.org/issue12990  closed by ronaldoussoren

#13396: new method random.getrandbytes()
http://bugs.python.org/issue13396  closed by rhettinger

#13418: Embedded Python memory leak
http://bugs.python.org/issue13418  closed by christian.heimes

#13871: namedtuple does not normalize field names when checking for du
http://bugs.python.org/issue13871  closed by rhettinger

#14139: test_ftplib: segfault
http://bugs.python.org/issue14139  closed by christian.heimes

#14268: _move_file is broken because of a bad mock
http://bugs.python.org/issue14268  closed by christian.heimes

#14620: Fatal Python error: Cannot recover from stack overflow.
http://bugs.python.org/issue14620  closed by christian.heimes

#14748: spwd.getspall() is returning LDAP (non local) users too
http://bugs.python.org/issue14748  closed by christian.heimes

#15197: test_gettext failure on Win64 buildbot
http://bugs.python.org/issue15197  closed by christian.heimes

#15553: Segfault in test_6_daemon_threads() of test_threading, on Mac 
http://bugs.python.org/issue15553  closed by ronaldoussoren

#17198: dbm.whichdb references unitialized 'ndbm' variable
http://bugs.python.org/issue17198  closed by ezio.melotti

#17221: Resort Misc/NEWS
http://bugs.python.org/issue17221  closed by christian.heimes

#17845: Clarify successful build message
http://bugs.python.org/issue17845  closed by brett.cannon

#17860: subprocess docs lack info how to use output result
http://bugs.python.org/issue17860  closed by ronaldoussoren

#17987: test.support.captured_stderr, captured_stdin not documented
http://bugs.python.org/issue17987  closed by r.david.murray

#18013: cgi.FieldStorage does not parse W3C sample
http://bugs.python.org/issue18013  closed by flox

#18020: html.escape 10x slower than cgi.escape
http://bugs.python.org/issue18020  closed by ezio.melotti

#18044: Email headers do not properly decode to unicode.
http://bugs.python.org/issue18044  closed by r.david.murray

#18101: Tk.split() doesn't work with nested Unicode strings
http://bugs.python.org/issue18101  closed by serhiy.storchaka

#18106: There are unused variables in Lib/test/test_collections.py
http://bugs.python.org/issue18106  closed by ezio.melotti

#18116: getpass.unix_getpass() always fallback to sys.stdin
http://bugs.python.org/issue18116  closed by r.david.murray

#18136: Put local build paths before system build paths in configure.a
http://bugs.python.org/issue18136  closed by brett.cannon

#18203: Replace direct calls to malloc() with PyMem_Malloc() or PyMem_
http://bugs.python.org/issue18203  closed by haypo

#18231: What's new in Python should explain what's new in UCD
http://bugs.python.org/issue18231  closed by belopolsky

#18289: python.org Interactive interpreter linked with libedit can seg
http://bugs.python.org/issue18289  closed by loewis

#18308: checkRecvmsgAddress wrong in test_socket.py (AIX failures)
http://bugs.python.org/issue18308  closed by neologix

#18338: python --version should send output to STDOUT
http://bugs.python.org/issue18338  closed by serhiy.storchaka

#18342: Use the repr of a module name for ImportError in ceval.c
http://bugs.python.org/issue18342  closed by brett.cannon

#18351: Incorrect variable name in importlib._bootstrap._get_sourcefil
http://bugs.python.org/issue18351  closed by brett.cannon

#18352: collections.Counter with added attributes are not deepcopied p
http://bugs.python.org/issue18352  closed by rhettinger

#18354: http://www.python.org/doc/ has outdated note
http://bugs.python.org/issue18354  closed by brian.curtin

#18360: Won't install.  Keeps telling me DLL is missing.
http://bugs.python.org/issue18360  closed by terry.reedy

#18361: Move dev-in-a-box to os.cpu_count()
http://bugs.python.org/issue18361  closed by brett.cannon

#18362: Make build_cpython.py from dev-in-a-box work outside of a box
http://bugs.python.org/issue18362  closed by brett.cannon

#18363: Change use of acronym tag in devinabox index.html to abbr
http://bugs.python.org/issue18363  closed by brett.cannon

#18364: Remove _not_found hack from importlib
http://bugs.python.org/issue18364  closed by brett.cannon

#18370: ast.parse produces wrong lineno and col_offset for triple-quot
http://bugs.python.org/issue18370  closed by Aivar.Annamaa

#18371: Should be able to specify SSL version for smtplib
http://bugs.python.org/issue18371  closed by christian.heimes

#18377: Python Launcher code cleanup
http://bugs.python.org/issue18377  closed by ronaldoussoren

#18380: email/quoprimime.py doesn't use re.sub correctly
http://bugs.python.org/issue18380  closed by ezio.melotti

#18385: Add devhelp build instructions to the documentation makefile (
http://bugs.python.org/issue18385  closed by ezio.melotti

#18386: Better random number generator
http://bugs.python.org/issue18386  closed by rhettinger

#18388: Link to the contributor agreement from devinabox
http://bugs.python.org/issue18388  closed by brett.cannon

#18390: Better documenting of ``hg revert -r`` when merging with anoth
http://bugs.python.org/issue18390  closed by brett.cannon

#18398: Debug Python build fails on OS X:  Fatal Python error: bad ID:
http://bugs.python.org/issue18398  closed by haypo

#18399: Fix grammar in comment in python.c
http://bugs.python.org/issue18399  closed by r.david.murray

#18403: Minor bug in The Python Tutorial
http://bugs.python.org/issue18403  closed by ezio.melotti

#18411: signal.SIGINT in windows cause process exit directly.
http://bugs.python.org/issue18411  closed by r.david.murray

#18412: Minor problem in PEP 8
http://bugs.python.org/issue18412  closed by barry

#18413: logging.error('asdf', extra={'msg':123}) raise KeyError except
http://bugs.python.org/issue18413  closed by christian.heimes

#18414: random.choices(seq, k)
http://bugs.python.org/issue18414  closed by christian.heimes

#18415: Normalize single/double quote usage in importlib
http://bugs.python.org/issue18415  closed by brett.cannon

#18419: For reasons I can't explain Python in some cases says that get
http://bugs.python.org/issue18419  closed by ezio.melotti

#18420: os.rename FileNotFound complaining about dst, but it is src wh
http://bugs.python.org/issue18420  closed by ned.deily

#18422: is_package missing so can't use -m
http://bugs.python.org/issue18422  closed by brett.cannon

#18424: sum() does not return only the sum of a sequence of numbers + 
http://bugs.python.org/issue18424  closed by r.david.murray

#18427: str.replace causes segfault for long strings
http://bugs.python.org/issue18427  closed by ronaldoussoren

#979407: urllib2 digest auth totally broken
http://bugs.python.org/issue979407  closed by christian.heimes

From v+python at g.nevcal.com  Fri Jul 12 21:45:03 2013
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Fri, 12 Jul 2013 12:45:03 -0700
Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL
In-Reply-To: <CAP7+vJK=Y=_mbC1CUAenvrTip=X8y4NQZHfsXfbZ8y4kUijq1g@mail.gmail.com>
References: <51DF469B.6010007@python.org>
	<CAL9jXCH9MSarDGxjR-oUFyBV1Gp2AxaBExdBQfNFHmJ5rF+qwA@mail.gmail.com>
	<51DFE964.9020006@python.org>
	<CAP7+vJK=Y=_mbC1CUAenvrTip=X8y4NQZHfsXfbZ8y4kUijq1g@mail.gmail.com>
Message-ID: <51E05CBF.9080107@g.nevcal.com>

On 7/12/2013 8:50 AM, Guido van Rossum wrote:
>
> You underestimate the reach of XP. For older or underpowered hardware 
> outside the developed world it is still the de facto choice. And it 
> definitely is the best version of Windows ever. None of the Win98 crap 
> and none of the Vista junk.
>
> Telling people to go install Ubuntu is not really fair if others 
> around them don't or if they need certain software.
>
> --Guido van Rossum (sent from Android phone)
>
Well said. +100
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130712/8414309e/attachment.html>

From cs at zip.com.au  Sat Jul 13 04:57:02 2013
From: cs at zip.com.au (Cameron Simpson)
Date: Sat, 13 Jul 2013 12:57:02 +1000
Subject: [Python-Dev] PEP 446: Add new parameters to configure the
 inherance of files and for non-blocking sockets
In-Reply-To: <CAMpsgwYzTCbNTw_AT3RY2qWxWbf48ZG1F14+1EemaX=9i6gABQ@mail.gmail.com>
References: <CAMpsgwYzTCbNTw_AT3RY2qWxWbf48ZG1F14+1EemaX=9i6gABQ@mail.gmail.com>
Message-ID: <20130713025702.GA23103@cskk.homeip.net>

Sorry for the delay; work got in the way.

On 07Jul2013 15:21, Victor Stinner <victor.stinner at gmail.com> wrote:
| Ok, I think that the best consensus here is to add a new
| os.set_blocking() function. The function would not be available on
| Windows, but should be available on all UNIX platforms.

Thanks.

| See the diff:
| http://hg.python.org/peps/rev/ce61588d244c

This sentence:

  The flag is cleared in the child process before executing the
  program, the change does not change the flag in the parent process.

needs a semicolon, not a comma.

| @Charles-Fran?ois and Cameron: Do you prefer to new version of the PEP?
| http://www.python.org/dev/peps/pep-0446/
| (will be updated in a few minutes)

I'm happy with it.

Thank you,
-- 
Cameron Simpson <cs at zip.com.au>

From ericsnowcurrently at gmail.com  Sat Jul 13 06:10:34 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Fri, 12 Jul 2013 22:10:34 -0600
Subject: [Python-Dev] PLY in stdlib (was cffi in stdlib)
In-Reply-To: <AC6A1BA1-3169-472C-BCB1-2C57A1CF3169@voidspace.org.uk>
References: <mailman.13132.1361899302.2938.python-dev@python.org>
	<F6FAB3A2-6040-42CE-A4FA-93077756C2DC@dabeaz.com>
	<AC6A1BA1-3169-472C-BCB1-2C57A1CF3169@voidspace.org.uk>
Message-ID: <CALFfu7BxLEDOYwjCSO8=+Xh==1Ef8Eq2N2B2TnvCbwGfGfZG_Q@mail.gmail.com>

On Feb 27, 2013 4:31 AM, "Michael Foord" <fuzzyman at voidspace.org.uk> wrote:
>
>
> On 27 Feb 2013, at 11:00, David Beazley <dave at dabeaz.com> wrote:
>
> >>
> >> From: Eli Bendersky <eliben at gmail.com>
> >>
> >> I'll be the first one to admit that pycparser is almost certainly not
> >> generally useful enough to be exposed in the stdlib. So just using it
as an
> >> implementation detail is absolutely fine. PLY is a more interesting
> >> question, however, since PLY is somewhat more generally useful. That
said,
> >> I see all this as implementation details that shouldn't distract us
from
> >> the main point of whether cffi should be added.
> >>
> >
> > Regarding the inclusion of PLY or some subcomponent of it in the
standard library, it's not an entirely crazy idea in my opinion.
>
> +1 PLY is capable and well tried-and-tested. We used it in Resolver One
to implement a pretty large grammar and it is (in my opinion) best of breed
in the Python parser generator world. Being stable and widely used, with an
"available maintainer", makes it an ideal candidate for standard library
inclusion.

Is this still on the table?

-eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130712/1f07229d/attachment.html>

From tjreedy at udel.edu  Sat Jul 13 06:41:10 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 13 Jul 2013 00:41:10 -0400
Subject: [Python-Dev] PLY in stdlib (was cffi in stdlib)
In-Reply-To: <CALFfu7BxLEDOYwjCSO8=+Xh==1Ef8Eq2N2B2TnvCbwGfGfZG_Q@mail.gmail.com>
References: <mailman.13132.1361899302.2938.python-dev@python.org>
	<F6FAB3A2-6040-42CE-A4FA-93077756C2DC@dabeaz.com>
	<AC6A1BA1-3169-472C-BCB1-2C57A1CF3169@voidspace.org.uk>
	<CALFfu7BxLEDOYwjCSO8=+Xh==1Ef8Eq2N2B2TnvCbwGfGfZG_Q@mail.gmail.com>
Message-ID: <krqlp1$e25$1@ger.gmane.org>

On 7/13/2013 12:10 AM, Eric Snow wrote:
>
> On Feb 27, 2013 4:31 AM, "Michael Foord" <fuzzyman at voidspace.org.uk

>  > +1 PLY is capable and well tried-and-tested. We used it in Resolver
> One to implement a pretty large grammar and it is (in my opinion) best
> of breed in the Python parser generator world. Being stable and widely
> used, with an "available maintainer", makes it an ideal candidate for
> standard library inclusion.
>
> Is this still on the table?

Who is the maintainer and what is his opinion?

-- 
Terry Jan Reedy


From jdunck at gmail.com  Sat Jul 13 06:57:20 2013
From: jdunck at gmail.com (Jeremy Dunck)
Date: Fri, 12 Jul 2013 21:57:20 -0700
Subject: [Python-Dev] PLY in stdlib (was cffi in stdlib)
In-Reply-To: <krqlp1$e25$1@ger.gmane.org>
References: <mailman.13132.1361899302.2938.python-dev@python.org>
	<F6FAB3A2-6040-42CE-A4FA-93077756C2DC@dabeaz.com>
	<AC6A1BA1-3169-472C-BCB1-2C57A1CF3169@voidspace.org.uk>
	<CALFfu7BxLEDOYwjCSO8=+Xh==1Ef8Eq2N2B2TnvCbwGfGfZG_Q@mail.gmail.com>
	<krqlp1$e25$1@ger.gmane.org>
Message-ID: <CAM0i3f5BvP4pkS5Q9NdkbOGrEvRd212cKQ7O6nAJh0kmx4DEFw@mail.gmail.com>

David Beasley; see earlier in this same thread:

http://mail.python.org/pipermail/python-dev/2013-February/thread.html#124389



On Fri, Jul 12, 2013 at 9:41 PM, Terry Reedy <tjreedy at udel.edu> wrote:

> On 7/13/2013 12:10 AM, Eric Snow wrote:
>
>>
>> On Feb 27, 2013 4:31 AM, "Michael Foord" <fuzzyman at voidspace.org.uk
>>
>
>   > +1 PLY is capable and well tried-and-tested. We used it in Resolver
>> One to implement a pretty large grammar and it is (in my opinion) best
>> of breed in the Python parser generator world. Being stable and widely
>> used, with an "available maintainer", makes it an ideal candidate for
>> standard library inclusion.
>>
>> Is this still on the table?
>>
>
> Who is the maintainer and what is his opinion?
>
> --
> Terry Jan Reedy
>
>
> ______________________________**_________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/**mailman/listinfo/python-dev<http://mail.python.org/mailman/listinfo/python-dev>
> Unsubscribe: http://mail.python.org/**mailman/options/python-dev/**
> jdunck%40gmail.com<http://mail.python.org/mailman/options/python-dev/jdunck%40gmail.com>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130712/d11124c0/attachment-0001.html>

From agriff at tin.it  Sat Jul 13 08:41:55 2013
From: agriff at tin.it (Andrea Griffini)
Date: Sat, 13 Jul 2013 08:41:55 +0200
Subject: [Python-Dev] Doubt about closure/function generation in bytecode
Message-ID: <CALy7ODst91xMBuT0bxt7Q7tm3WxZd1ewBZ6JjV3GaTaVoKocRQ@mail.gmail.com>

Is there any semantic difference between

    BUILD_TUPLE 0
    LOAD_CONST <code object ... >
    MAKE_CLOSURE 0

and

    LOAD_CONST <code object ...>
    MAKE_FUNCTION 0

?

In other words is there any difference between a function and a closure
that doesn't capture anything? Is it just a speed optimization for a common
case?

Andrea Griffini
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130713/1989c40d/attachment.html>

From fuzzyman at voidspace.org.uk  Sat Jul 13 10:24:02 2013
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Sat, 13 Jul 2013 11:24:02 +0300
Subject: [Python-Dev] PLY in stdlib (was cffi in stdlib)
In-Reply-To: <krqlp1$e25$1@ger.gmane.org>
References: <mailman.13132.1361899302.2938.python-dev@python.org>
	<F6FAB3A2-6040-42CE-A4FA-93077756C2DC@dabeaz.com>
	<AC6A1BA1-3169-472C-BCB1-2C57A1CF3169@voidspace.org.uk>
	<CALFfu7BxLEDOYwjCSO8=+Xh==1Ef8Eq2N2B2TnvCbwGfGfZG_Q@mail.gmail.com>
	<krqlp1$e25$1@ger.gmane.org>
Message-ID: <96746EE7-E96A-4363-B0F7-D7950207C702@voidspace.org.uk>


On 13 Jul 2013, at 07:41, Terry Reedy <tjreedy at udel.edu> wrote:

> On 7/13/2013 12:10 AM, Eric Snow wrote:
>> 
>> On Feb 27, 2013 4:31 AM, "Michael Foord" <fuzzyman at voidspace.org.uk
> 
>> > +1 PLY is capable and well tried-and-tested. We used it in Resolver
>> One to implement a pretty large grammar and it is (in my opinion) best
>> of breed in the Python parser generator world. Being stable and widely
>> used, with an "available maintainer", makes it an ideal candidate for
>> standard library inclusion.
>> 
>> Is this still on the table?
> 
> Who is the maintainer and what is his opinion?


The maintainer is David Beazley and as far as I recall he has not expressed an opinion on this particular question. It would obviously need his agreement (and maintenance commitment) if it is to fly.

Michael

> 
> -- 
> Terry Jan Reedy
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk


--
http://www.voidspace.org.uk/


May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing 
http://www.sqlite.org/different.html






From brett at python.org  Sat Jul 13 15:12:42 2013
From: brett at python.org (Brett Cannon)
Date: Sat, 13 Jul 2013 09:12:42 -0400
Subject: [Python-Dev] PLY in stdlib (was cffi in stdlib)
In-Reply-To: <96746EE7-E96A-4363-B0F7-D7950207C702@voidspace.org.uk>
References: <mailman.13132.1361899302.2938.python-dev@python.org>
	<F6FAB3A2-6040-42CE-A4FA-93077756C2DC@dabeaz.com>
	<AC6A1BA1-3169-472C-BCB1-2C57A1CF3169@voidspace.org.uk>
	<CALFfu7BxLEDOYwjCSO8=+Xh==1Ef8Eq2N2B2TnvCbwGfGfZG_Q@mail.gmail.com>
	<krqlp1$e25$1@ger.gmane.org>
	<96746EE7-E96A-4363-B0F7-D7950207C702@voidspace.org.uk>
Message-ID: <CAP1=2W5brJjW0==ou1+NQ=wntRm11mkduzXQY_eDnizxAXHF0g@mail.gmail.com>

On Sat, Jul 13, 2013 at 4:24 AM, Michael Foord <fuzzyman at voidspace.org.uk>wrote:

>
> On 13 Jul 2013, at 07:41, Terry Reedy <tjreedy at udel.edu> wrote:
>
> > On 7/13/2013 12:10 AM, Eric Snow wrote:
> >>
> >> On Feb 27, 2013 4:31 AM, "Michael Foord" <fuzzyman at voidspace.org.uk
> >
> >> > +1 PLY is capable and well tried-and-tested. We used it in Resolver
> >> One to implement a pretty large grammar and it is (in my opinion) best
> >> of breed in the Python parser generator world. Being stable and widely
> >> used, with an "available maintainer", makes it an ideal candidate for
> >> standard library inclusion.
> >>
> >> Is this still on the table?
> >
> > Who is the maintainer and what is his opinion?
>
>
> The maintainer is David Beazley and as far as I recall he has not
> expressed an opinion on this particular question. It would obviously need
> his agreement (and maintenance commitment) if it is to fly.
>

Just because we have now had two conflicting replies on this: David is down
with PLY being added, but Alex Gaynor was working on a cleanup called RPLY
for RPython. Basically David said the two of them should work together to
clean up PLY and then it should be good to proposing for the stdlib (e.g.
there are some backwards-compatibility hacks which should be removed).

Below is David's original email on the topic from Feb 27:

-------------------------------

Regarding the inclusion of PLY or some subcomponent of it in the standard
library, it's not an entirely crazy idea in my opinion.  LALR(1) parsers
have been around for a long time, are generally known to anyone who's used
yacc/bison, and would be useful outside the context of cffi or pycparser.
 PLY has also been around for about 12 years and is what I would call
stable.  It gets an update about every year or two, but that's about it.
PLY is also relatively small--just two files and about 4300 lines of code
(much of which could probably be scaled down a bit).

The only downside to including PLY might be the fact that there are very
few people walking around who've actually had to *implement* an LALR(1)
parser generator.  Some of the code for that is extremely hairy and
mathematical.   At this time, I don't think there are any bugs in it, but
it's not the sort of thing that one wants to wander into casually.    Also,
there are some horrible hacks in PLY that I'd really like to get rid of,
but am currently stuck with due to backwards compatibility issues.

Alex Gaynor has been working on a PLY variant (RPLY) geared at RPython and
which has a slightly different programming interface.    I'd say if we were
to go down this route, he and I should work together to put together some
kind of more general "parsing.lalr" package (or similar) that  cleans it up
and makes it more suitable as a library for building different kinds of
parsing tools on top of.

Cheers,
Dave
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130713/580c892e/attachment.html>

From dave at dabeaz.com  Sat Jul 13 15:26:23 2013
From: dave at dabeaz.com (David Beazley)
Date: Sat, 13 Jul 2013 08:26:23 -0500
Subject: [Python-Dev] PLY in stdlib (was cffi in stdlib)
In-Reply-To: <CAP1=2W5brJjW0==ou1+NQ=wntRm11mkduzXQY_eDnizxAXHF0g@mail.gmail.com>
References: <mailman.13132.1361899302.2938.python-dev@python.org>
	<F6FAB3A2-6040-42CE-A4FA-93077756C2DC@dabeaz.com>
	<AC6A1BA1-3169-472C-BCB1-2C57A1CF3169@voidspace.org.uk>
	<CALFfu7BxLEDOYwjCSO8=+Xh==1Ef8Eq2N2B2TnvCbwGfGfZG_Q@mail.gmail.com>
	<krqlp1$e25$1@ger.gmane.org>
	<96746EE7-E96A-4363-B0F7-D7950207C702@voidspace.org.uk>
	<CAP1=2W5brJjW0==ou1+NQ=wntRm11mkduzXQY_eDnizxAXHF0g@mail.gmail.com>
Message-ID: <E680C312-42BC-42DD-8125-ED45D560B24A@dabeaz.com>

I'm in favor of PLY going into stdlib with the caveat that there are some things about it that should probably be cleaned up and modernized.   For instance, the method by which it writes the cached parsing tables needs to be cleaned up.  I still think putting the LALR(1) generator code into a common library usable by both PLY/RPLY would be a useful thing to do.  That code is really hairy and non-trivial to understand without something like the Dragon book nearby (and even then it's not easy).   

So, if I were to make any kind of proposal, I would say, make a standard library module for just the LALR(1) generator code.   If the PLY interface is needed to add pycparser or cffi to the standard library, that can be added too, but as a separate module that uses the more general LALR(1) library.

Cheers,
Dave

On Jul 13, 2013, at 8:12 AM, Brett Cannon wrote:

> 
> 
> 
> On Sat, Jul 13, 2013 at 4:24 AM, Michael Foord <fuzzyman at voidspace.org.uk> wrote:
> 
> On 13 Jul 2013, at 07:41, Terry Reedy <tjreedy at udel.edu> wrote:
> 
> > On 7/13/2013 12:10 AM, Eric Snow wrote:
> >>
> >> On Feb 27, 2013 4:31 AM, "Michael Foord" <fuzzyman at voidspace.org.uk
> >
> >> > +1 PLY is capable and well tried-and-tested. We used it in Resolver
> >> One to implement a pretty large grammar and it is (in my opinion) best
> >> of breed in the Python parser generator world. Being stable and widely
> >> used, with an "available maintainer", makes it an ideal candidate for
> >> standard library inclusion.
> >>
> >> Is this still on the table?
> >
> > Who is the maintainer and what is his opinion?
> 
> 
> The maintainer is David Beazley and as far as I recall he has not expressed an opinion on this particular question. It would obviously need his agreement (and maintenance commitment) if it is to fly.
> 
> Just because we have now had two conflicting replies on this: David is down with PLY being added, but Alex Gaynor was working on a cleanup called RPLY for RPython. Basically David said the two of them should work together to clean up PLY and then it should be good to proposing for the stdlib (e.g. there are some backwards-compatibility hacks which should be removed).
> 
> Below is David's original email on the topic from Feb 27:
> 
> -------------------------------
> 
> Regarding the inclusion of PLY or some subcomponent of it in the standard library, it's not an entirely crazy idea in my opinion.  LALR(1) parsers have been around for a long time, are generally known to anyone who's used yacc/bison, and would be useful outside the context of cffi or pycparser.  PLY has also been around for about 12 years and is what I would call stable.  It gets an update about every year or two, but that's about it.   PLY is also relatively small--just two files and about 4300 lines of code (much of which could probably be scaled down a bit).
> 
> The only downside to including PLY might be the fact that there are very few people walking around who've actually had to *implement* an LALR(1) parser generator.  Some of the code for that is extremely hairy and mathematical.   At this time, I don't think there are any bugs in it, but it's not the sort of thing that one wants to wander into casually.    Also, there are some horrible hacks in PLY that I'd really like to get rid of, but am currently stuck with due to backwards compatibility issues.
> 
> Alex Gaynor has been working on a PLY variant (RPLY) geared at RPython and which has a slightly different programming interface.    I'd say if we were to go down this route, he and I should work together to put together some kind of more general "parsing.lalr" package (or similar) that  cleans it up and makes it more suitable as a library for building different kinds of parsing tools on top of.
> 
> Cheers,
> Dave

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130713/80eb781a/attachment.html>

From ncoghlan at gmail.com  Sun Jul 14 08:54:33 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 14 Jul 2013 16:54:33 +1000
Subject: [Python-Dev] PLY in stdlib (was cffi in stdlib)
In-Reply-To: <E680C312-42BC-42DD-8125-ED45D560B24A@dabeaz.com>
References: <mailman.13132.1361899302.2938.python-dev@python.org>
	<F6FAB3A2-6040-42CE-A4FA-93077756C2DC@dabeaz.com>
	<AC6A1BA1-3169-472C-BCB1-2C57A1CF3169@voidspace.org.uk>
	<CALFfu7BxLEDOYwjCSO8=+Xh==1Ef8Eq2N2B2TnvCbwGfGfZG_Q@mail.gmail.com>
	<krqlp1$e25$1@ger.gmane.org>
	<96746EE7-E96A-4363-B0F7-D7950207C702@voidspace.org.uk>
	<CAP1=2W5brJjW0==ou1+NQ=wntRm11mkduzXQY_eDnizxAXHF0g@mail.gmail.com>
	<E680C312-42BC-42DD-8125-ED45D560B24A@dabeaz.com>
Message-ID: <CADiSq7dsk3vxU0XSUvHXp-AK+tExVSHQsBC+LH08PCXC4D4FOQ@mail.gmail.com>

On 13 July 2013 23:26, David Beazley <dave at dabeaz.com> wrote:

> I'm in favor of PLY going into stdlib with the caveat that there are some
> things about it that should probably be cleaned up and modernized.   For
> instance, the method by which it writes the cached parsing tables needs to
> be cleaned up.  I still think putting the LALR(1) generator code into a
> common library usable by both PLY/RPLY would be a useful thing to do.  That
> code is really hairy and non-trivial to understand without something like
> the Dragon book nearby (and even then it's not easy).
>
> So, if I were to make any kind of proposal, I would say, make a standard
> library module for just the LALR(1) generator code.   If the PLY interface
> is needed to add pycparser or cffi to the standard library, that can be
> added too, but as a separate module that uses the more general LALR(1)
> library.
>

lrparsing is a more recent entry in the LR parsing stakes:
https://pypi.python.org/pypi/lrparsing (although, as Russell put it in his
PyCon AU lightning talk, if PLY had shown up ranked higher than 506 in his
PyPI search for "parser", he probably would have just used that:
http://pyvideo.org/video/2222/sunday-lightning-talks at about 2:15)

(I plan to bug Russell about putting that up on one of the DVCS hosting
sites next time I see him at BrisPy - for the moment, the source is
available through the tarball/sdist)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130714/d09f24e5/attachment.html>

From ncoghlan at gmail.com  Sun Jul 14 10:11:18 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 14 Jul 2013 18:11:18 +1000
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
Message-ID: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>

Currently, the naming section of PEP 8 doesn't say very much about what a
leading underscore *means* in the Python standard library.

I would like to add a new "Private interfaces" subsection under "Naming
Conventions" to say the following:

=================
Private interfaces

Unless explicitly documented otherwise, a leading underscore on any name
indicates that it is an internal implementation detail and backwards
compatibility guarantees do not apply. It is strongly encouraged that
private APIs (whether modules, classes, functions, attributes or other
names) be clearly marked in this way, as many Python users rely on
introspection to identify available functionality and may be mislead into
believing an API without the leading underscore is in fact a public API
with the standard backwards compatibility guarantees.

All test modules are also considered private interfaces.

Even though they typically lack the leading underscore, modules imported by
another module are also considered an implementation detail. Other modules
*should* not rely on indirect access to such modules unless they are an
explicitly documented part of the API (such as ``os.path``).
=================

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130714/3464a429/attachment.html>

From stephen at xemacs.org  Sun Jul 14 10:30:22 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sun, 14 Jul 2013 17:30:22 +0900
Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL
In-Reply-To: <7wmwps1he3.fsf@benfinney.id.au>
References: <51DF469B.6010007@python.org>
	<a11abcf4949a4e31b7989c7372088475@BLUPR03MB199.namprd03.prod.outlook.com>
	<87k3kwtqls.fsf@uwakimon.sk.tsukuba.ac.jp>
	<7wmwps1he3.fsf@benfinney.id.au>
Message-ID: <87bo65tug1.fsf@uwakimon.sk.tsukuba.ac.jp>

Ben Finney writes:
 > "Stephen J. Turnbull" <stephen at xemacs.org> writes:
 > 
 > > I don't see any good reason to take into account what Microsoft does
 > > or doesn't support.
 > 
 > It seems you're advocating a position quite ad odds with
 > <URL:http://www.python.org/dev/peps/pep-0011/#id7>.

Not at all.  The first thing that the PEP says about unsupporting code
is:

    Unsupporting platforms

    If a certain platform that currently has special code in it is
    deemed to be without Python users, 

What a vendor supports is only a heuristic.  Existence of users comes
first.

Note that the policy says that some Windows platforms *will* be
supported.  It doesn't say others will be unsupported (except
implicitly: 3 years after the last version of Visual Studio capable of
building releases for that platform goes out of extended support, the
build infrastructure will be removed).

I don't see a good reason to change the PEP.


From ncoghlan at gmail.com  Sun Jul 14 13:09:02 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 14 Jul 2013 21:09:02 +1000
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
Message-ID: <CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>

On 14 July 2013 18:11, Nick Coghlan <ncoghlan at gmail.com> wrote:

> Currently, the naming section of PEP 8 doesn't say very much about what a
> leading underscore *means* in the Python standard library.
>
> I would like to add a new "Private interfaces" subsection under "Naming
> Conventions" to say the following:
>
> =================
> Private interfaces
>
> Unless explicitly documented otherwise, a leading underscore on any name
> indicates that it is an internal implementation detail and backwards
> compatibility guarantees do not apply. It is strongly encouraged that
> private APIs (whether modules, classes, functions, attributes or other
> names) be clearly marked in this way, as many Python users rely on
> introspection to identify available functionality and may be mislead into
> believing an API without the leading underscore is in fact a public API
> with the standard backwards compatibility guarantees.
>
> All test modules are also considered private interfaces.
>
> Even though they typically lack the leading underscore, modules imported
> by another module are also considered an implementation detail. Other
> modules *should* not rely on indirect access to such modules unless they
> are an explicitly documented part of the API (such as ``os.path``).
> =================
>

Slight adjustment to the proposed wording to ensure completely undocumented
modules are also considered private:

=================
Private interfaces

Unless explicitly documented otherwise, a leading underscore on any name
indicates that it is an internal implementation detail and backwards
compatibility guarantees do not apply. It is strongly encouraged that
private APIs (whether modules, classes, functions, attributes or other
names) be clearly marked in this way, as Python users may rely on
introspection to identify available functionality and may be misled into
believing an API without a leading underscore is in fact a public API with
the standard backwards compatibility guarantees.

Even when their names do not start with a leading underscore, all test
modules and all modules that are not covered in the documentation are also
considered private interfaces.

Similarly, the specific modules and external APIs imported by a module are
always considered an implementation detail. Other modules should not rely
on indirect access to such imported interfaces unless they are an
explicitly documented part of the containing module's API (such as
``os.path`` or a package ``__init__`` module exposing functionality from
submodules).
=================



-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130714/3017dc71/attachment.html>

From brett at python.org  Sun Jul 14 15:13:13 2013
From: brett at python.org (Brett Cannon)
Date: Sun, 14 Jul 2013 09:13:13 -0400
Subject: [Python-Dev] PLY in stdlib (was cffi in stdlib)
In-Reply-To: <CADiSq7dsk3vxU0XSUvHXp-AK+tExVSHQsBC+LH08PCXC4D4FOQ@mail.gmail.com>
References: <mailman.13132.1361899302.2938.python-dev@python.org>
	<F6FAB3A2-6040-42CE-A4FA-93077756C2DC@dabeaz.com>
	<AC6A1BA1-3169-472C-BCB1-2C57A1CF3169@voidspace.org.uk>
	<CALFfu7BxLEDOYwjCSO8=+Xh==1Ef8Eq2N2B2TnvCbwGfGfZG_Q@mail.gmail.com>
	<krqlp1$e25$1@ger.gmane.org>
	<96746EE7-E96A-4363-B0F7-D7950207C702@voidspace.org.uk>
	<CAP1=2W5brJjW0==ou1+NQ=wntRm11mkduzXQY_eDnizxAXHF0g@mail.gmail.com>
	<E680C312-42BC-42DD-8125-ED45D560B24A@dabeaz.com>
	<CADiSq7dsk3vxU0XSUvHXp-AK+tExVSHQsBC+LH08PCXC4D4FOQ@mail.gmail.com>
Message-ID: <CAP1=2W7-2NDWyCs6jkYNNraLBrAfOoftOT4PknU89V2KwJkMBg@mail.gmail.com>

On Sun, Jul 14, 2013 at 2:54 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 13 July 2013 23:26, David Beazley <dave at dabeaz.com> wrote:
>
>> I'm in favor of PLY going into stdlib with the caveat that there are some
>> things about it that should probably be cleaned up and modernized.   For
>> instance, the method by which it writes the cached parsing tables needs to
>> be cleaned up.  I still think putting the LALR(1) generator code into a
>> common library usable by both PLY/RPLY would be a useful thing to do.  That
>> code is really hairy and non-trivial to understand without something like
>> the Dragon book nearby (and even then it's not easy).
>>
>> So, if I were to make any kind of proposal, I would say, make a standard
>> library module for just the LALR(1) generator code.   If the PLY interface
>> is needed to add pycparser or cffi to the standard library, that can be
>> added too, but as a separate module that uses the more general LALR(1)
>> library.
>>
>
> lrparsing is a more recent entry in the LR parsing stakes:
> https://pypi.python.org/pypi/lrparsing (although, as Russell put it in
> his PyCon AU lightning talk, if PLY had shown up ranked higher than 506 in
> his PyPI search for "parser", he probably would have just used that:
> http://pyvideo.org/video/2222/sunday-lightning-talks at about 2:15)
>
> (I plan to bug Russell about putting that up on one of the DVCS hosting
> sites next time I see him at BrisPy - for the moment, the source is
> available through the tarball/sdist)
>

It seems a bit new compared to PLY's 15 years of existence to be considered
in the running. Plus Russell would need to change the license from GPL.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130714/6cf557e5/attachment.html>

From dave at dabeaz.com  Sun Jul 14 15:32:28 2013
From: dave at dabeaz.com (David Beazley)
Date: Sun, 14 Jul 2013 08:32:28 -0500
Subject: [Python-Dev] PLY in stdlib (was cffi in stdlib)
In-Reply-To: <CAP1=2W7-2NDWyCs6jkYNNraLBrAfOoftOT4PknU89V2KwJkMBg@mail.gmail.com>
References: <mailman.13132.1361899302.2938.python-dev@python.org>
	<F6FAB3A2-6040-42CE-A4FA-93077756C2DC@dabeaz.com>
	<AC6A1BA1-3169-472C-BCB1-2C57A1CF3169@voidspace.org.uk>
	<CALFfu7BxLEDOYwjCSO8=+Xh==1Ef8Eq2N2B2TnvCbwGfGfZG_Q@mail.gmail.com>
	<krqlp1$e25$1@ger.gmane.org>
	<96746EE7-E96A-4363-B0F7-D7950207C702@voidspace.org.uk>
	<CAP1=2W5brJjW0==ou1+NQ=wntRm11mkduzXQY_eDnizxAXHF0g@mail.gmail.com>
	<E680C312-42BC-42DD-8125-ED45D560B24A@dabeaz.com>
	<CADiSq7dsk3vxU0XSUvHXp-AK+tExVSHQsBC+LH08PCXC4D4FOQ@mail.gmail.com>
	<CAP1=2W7-2NDWyCs6jkYNNraLBrAfOoftOT4PknU89V2KwJkMBg@mail.gmail.com>
Message-ID: <8595C679-3D16-4312-98C8-146309FDF825@dabeaz.com>

On Jul 14, 2013, at 8:13 AM, Brett Cannon wrote:

> 
> 
> 
> On Sun, Jul 14, 2013 at 2:54 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 13 July 2013 23:26, David Beazley <dave at dabeaz.com> wrote:
> I'm in favor of PLY going into stdlib with the caveat that there are some things about it that should probably be cleaned up and modernized.   For instance, the method by which it writes the cached parsing tables needs to be cleaned up.  I still think putting the LALR(1) generator code into a common library usable by both PLY/RPLY would be a useful thing to do.  That code is really hairy and non-trivial to understand without something like the Dragon book nearby (and even then it's not easy).   
> 
> So, if I were to make any kind of proposal, I would say, make a standard library module for just the LALR(1) generator code.   If the PLY interface is needed to add pycparser or cffi to the standard library, that can be added too, but as a separate module that uses the more general LALR(1) library.
> 
> lrparsing is a more recent entry in the LR parsing stakes:https://pypi.python.org/pypi/lrparsing (although, as Russell put it in his PyCon AU lightning talk, if PLY had shown up ranked higher than 506 in his PyPI search for "parser", he probably would have just used that: http://pyvideo.org/video/2222/sunday-lightning-talks at about 2:15)
> 
> (I plan to bug Russell about putting that up on one of the DVCS hosting sites next time I see him at BrisPy - for the moment, the source is available through the tarball/sdist)
> 
> It seems a bit new compared to PLY's 15 years of existence to be considered in the running. Plus Russell would need to change the license from GPL. 

I honestly don't have any particular thoughts about PLY vs. other parser generators and the merits of their inclusion (or not) in the standard library.   My impression has always been that the main interest in PLY was due to interest in seeing CFFI in the standard library.  I'd say my main desire on the PLY side is that if it does go into the standard library, perhaps I could make it slightly less of mysterious black box and clean up a few bits.

Cheers,
Dave

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130714/5a02e24f/attachment.html>

From brett at python.org  Sun Jul 14 15:49:21 2013
From: brett at python.org (Brett Cannon)
Date: Sun, 14 Jul 2013 09:49:21 -0400
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
Message-ID: <CAP1=2W75YYks28w4qZ00R9Rpav_e4eec+roQifros0UNFBn39Q@mail.gmail.com>

On Sun, Jul 14, 2013 at 7:09 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 14 July 2013 18:11, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>> Currently, the naming section of PEP 8 doesn't say very much about what a
>> leading underscore *means* in the Python standard library.
>>
>> I would like to add a new "Private interfaces" subsection under "Naming
>> Conventions" to say the following:
>>
>> =================
>> Private interfaces
>>
>> Unless explicitly documented otherwise, a leading underscore on any name
>> indicates that it is an internal implementation detail and backwards
>> compatibility guarantees do not apply. It is strongly encouraged that
>> private APIs (whether modules, classes, functions, attributes or other
>> names) be clearly marked in this way, as many Python users rely on
>> introspection to identify available functionality and may be mislead into
>> believing an API without the leading underscore is in fact a public API
>> with the standard backwards compatibility guarantees.
>>
>> All test modules are also considered private interfaces.
>>
>> Even though they typically lack the leading underscore, modules imported
>> by another module are also considered an implementation detail. Other
>> modules *should* not rely on indirect access to such modules unless they
>> are an explicitly documented part of the API (such as ``os.path``).
>> =================
>>
>
> Slight adjustment to the proposed wording to ensure completely
> undocumented modules are also considered private:
>
> =================
> Private interfaces
>
> Unless explicitly documented otherwise, a leading underscore on any name
> indicates that it is an internal implementation detail and backwards
> compatibility guarantees do not apply. It is strongly encouraged that
> private APIs (whether modules, classes, functions, attributes or other
> names) be clearly marked in this way, as Python users may rely on
> introspection to identify available functionality and may be misled into
> believing an API without a leading underscore is in fact a public API with
> the standard backwards compatibility guarantees.
>
> Even when their names do not start with a leading underscore, all test
> modules and all modules that are not covered in the documentation are also
> considered private interfaces.
>
> Similarly, the specific modules and external APIs imported by a module are
> always considered an implementation detail. Other modules should not rely
> on indirect access to such imported interfaces unless they are an
> explicitly documented part of the containing module's API (such as
> ``os.path`` or a package ``__init__`` module exposing functionality from
> submodules).
> =================
>

+1
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130714/a04efc65/attachment.html>

From greg at krypto.org  Sun Jul 14 18:15:23 2013
From: greg at krypto.org (Gregory P. Smith)
Date: Sun, 14 Jul 2013 09:15:23 -0700
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
Message-ID: <CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>

+1  This is already how we've been behaving for years.


On Sun, Jul 14, 2013 at 4:09 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 14 July 2013 18:11, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>> Currently, the naming section of PEP 8 doesn't say very much about what a
>> leading underscore *means* in the Python standard library.
>>
>> I would like to add a new "Private interfaces" subsection under "Naming
>> Conventions" to say the following:
>>
>>
> Slight adjustment to the proposed wording to ensure completely
> undocumented modules are also considered private:
>
> =================
> Private interfaces
>
> Unless explicitly documented otherwise, a leading underscore on any name
> indicates that it is an internal implementation detail and backwards
> compatibility guarantees do not apply. It is strongly encouraged that
> private APIs (whether modules, classes, functions, attributes or other
> names) be clearly marked in this way,
>

<snip>


> as Python users may rely on introspection to identify available
> functionality and may be misled into believing an API without a leading
> underscore is in fact a public API with the standard backwards
> compatibility guarantees.
>

While true, I'm not sure the last part of the sentence is necessary. Once
we've established that a leading _ indicates something is private there
isn't much point in explaining the various ways people might find them.
 I'm happy regardless of this bit being there.


> Even when their names do not start with a leading underscore, all test
> modules and all modules that are not covered in the documentation are also
> considered private interfaces.
>
> Similarly, the specific modules and external APIs imported by a module are
> always considered an implementation detail. Other modules should not rely
> on indirect access to such imported interfaces unless they are an
> explicitly documented part of the containing module's API (such as
> ``os.path`` or a package ``__init__`` module exposing functionality from
> submodules).
> =================
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130714/4ac29086/attachment.html>

From steve at pearwood.info  Mon Jul 15 01:48:33 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 15 Jul 2013 09:48:33 +1000
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
Message-ID: <51E338D1.5070900@pearwood.info>

On 14/07/13 21:09, Nick Coghlan wrote:

> Slight adjustment to the proposed wording to ensure completely undocumented
> modules are also considered private:

-1 on this adjustment. If somebody cannot be bothered writing a one-line doc string:

"This module is private, don't touch."

then they certainly shouldn't be allowed to use up a public name for a private module. I don't think we should be encouraging more private, undocumented modules. (Documentation is valuable even for private modules.)

I'd go further, and say that no more private modules should be accepted for the std lib unless they have a leading underscore. I suppose for backwards compatibility reasons, we probably can't go through the std lib and rename private modules to make it clear they are private, but we don't have to accept new ones without the underscore.


-- 
Steven

From cs at zip.com.au  Mon Jul 15 02:01:17 2013
From: cs at zip.com.au (Cameron Simpson)
Date: Mon, 15 Jul 2013 10:01:17 +1000
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
 underscores
In-Reply-To: <51E338D1.5070900@pearwood.info>
References: <51E338D1.5070900@pearwood.info>
Message-ID: <20130715000117.GA46795@cskk.homeip.net>

On 15Jul2013 09:48, Steven D'Aprano <steve at pearwood.info> wrote:
| On 14/07/13 21:09, Nick Coghlan wrote:
| 
| >Slight adjustment to the proposed wording to ensure completely undocumented
| >modules are also considered private:
| 
| -1 on this adjustment. If somebody cannot be bothered writing a one-line doc string:
| 
| "This module is private, don't touch."
| 
| then they certainly shouldn't be allowed to use up a public name for a private module. I don't think we should be encouraging more private, undocumented modules. (Documentation is valuable even for private modules.)
| 
| I'd go further, and say that no more private modules should be accepted for the std lib unless they have a leading underscore. I suppose for backwards compatibility reasons, we probably can't go through the std lib and rename private modules to make it clear they are private, but we don't have to accept new ones without the underscore.

I disagree.

A private module is a perfectly sane way to implement the internals
of something, especially if it is subject to implementation change
in the future.

Clarification: is Nick classifying a module with docstrings by no
content in the "modules" section of the Python doco as undocumented?
That is what I would presume; I'd expect the code to be littered
with docstrings anyway, but the module as a whole is not presented
in the documentation and so should be private and not relied upon.

Cheers,
-- 
Cameron Simpson <cs at zip.com.au>

I sometimes wish that people would put a little more emphasis upon the
observance of the law than they do upon its enforcement. - Calvin Coolidge

From benhoyt at gmail.com  Mon Jul 15 02:21:38 2013
From: benhoyt at gmail.com (Ben Hoyt)
Date: Mon, 15 Jul 2013 12:21:38 +1200
Subject: [Python-Dev] Why is the return value of __contains__ coerced to
 boolean, but that of __lt__ and the like is not?
Message-ID: <CAL9jXCE2nK9ugrpo2_aoNqXKexTwjBoztvSh_1FEDWa0kmfKag@mail.gmail.com>

I'm curious why the return value of __contains__ is coerced to True or
False, whereas the return value of "normal" comparison operators like
__lt__ and the like are not. The latter return the value directly without
forcing it to be True or False.

This makes overriding __contains__ significantly less flexible, so I'm
wondering why it's designed or implemented that way. (I believe it's the
cmp_outcome() function in Python/ceval.c that does this:
http://hg.python.org/cpython/file/db9fe49069ed/Python/ceval.c#l4545)

For example, the "peewee" ORM overloads __lt__ and the like so it can map
Python expressions to SQL. But it can't do this with the "in" operator due
to the result of "x in y" always returning True or False in Python. So it
(ab)uses the "<<" operator to do this instead (See the peewee docs at
http://peewee.readthedocs.org/en/latest/peewee/querying.html#column-lookups
).

I'm sure there's a good reason for why "in" is different here, but I can't
see why right now.

-Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130715/e1126a2e/attachment.html>

From brett at python.org  Mon Jul 15 02:27:16 2013
From: brett at python.org (Brett Cannon)
Date: Sun, 14 Jul 2013 20:27:16 -0400
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <20130715000117.GA46795@cskk.homeip.net>
References: <51E338D1.5070900@pearwood.info>
	<20130715000117.GA46795@cskk.homeip.net>
Message-ID: <CAP1=2W4iXeEvmMdjzrp+TOyPN+Gk93JnpsWKd1ggC7aQaZ0Cbw@mail.gmail.com>

On Sun, Jul 14, 2013 at 8:01 PM, Cameron Simpson <cs at zip.com.au> wrote:

> On 15Jul2013 09:48, Steven D'Aprano <steve at pearwood.info> wrote:
> | On 14/07/13 21:09, Nick Coghlan wrote:
> |
> | >Slight adjustment to the proposed wording to ensure completely
> undocumented
> | >modules are also considered private:
> |
> | -1 on this adjustment. If somebody cannot be bothered writing a one-line
> doc string:
> |
> | "This module is private, don't touch."
> |
> | then they certainly shouldn't be allowed to use up a public name for a
> private module. I don't think we should be encouraging more private,
> undocumented modules. (Documentation is valuable even for private modules.)
> |
> | I'd go further, and say that no more private modules should be accepted
> for the std lib unless they have a leading underscore. I suppose for
> backwards compatibility reasons, we probably can't go through the std lib
> and rename private modules to make it clear they are private, but we don't
> have to accept new ones without the underscore.
>
> I disagree.
>
> A private module is a perfectly sane way to implement the internals
> of something, especially if it is subject to implementation change
> in the future.
>
> Clarification: is Nick classifying a module with docstrings by no
> content in the "modules" section of the Python doco as undocumented?
>

Yes. This has nothing to do with docstrings, just the official
documentation at docs.python.org.


> That is what I would presume; I'd expect the code to be littered
> with docstrings anyway, but the module as a whole is not presented
> in the documentation and so should be private and not relied upon.
>

Exactly.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130714/d466ee04/attachment.html>

From benhoyt at gmail.com  Mon Jul 15 02:29:46 2013
From: benhoyt at gmail.com (Ben Hoyt)
Date: Mon, 15 Jul 2013 12:29:46 +1200
Subject: [Python-Dev] Why is the return value of __contains__ coerced to
 boolean, but that of __lt__ and the like is not?
In-Reply-To: <CAL9jXCE2nK9ugrpo2_aoNqXKexTwjBoztvSh_1FEDWa0kmfKag@mail.gmail.com>
References: <CAL9jXCE2nK9ugrpo2_aoNqXKexTwjBoztvSh_1FEDWa0kmfKag@mail.gmail.com>
Message-ID: <CAL9jXCHYLX3dn_1jcdLNm1R4vudHLJYz5T47yRaHm9ihetJqsw@mail.gmail.com>

Oh, just to show what I mean here with some code (same in both Python 2.x
and 3.x):

-----
from __future__ import print_function

class C(object):
    def __contains__(self, x):
        print('__contains__({0!r})'.format(x))
        return x

    def __lt__(self, x):
        print('__lt__({0!r})'.format(x))
        return x

c = C()
print(42 in c)
print(0 in c)
print(c < 42)
print(c < 0)
-----

This prints the following:

__contains__(42)
True
__contains__(0)
False
__lt__(42)
42
__lt__(0)
0

Whereas I'd kinda expect it to print:

__contains__(42)
42
__contains__(0)
0
__lt__(42)
42
__lt__(0)
0

-Ben



On Mon, Jul 15, 2013 at 12:21 PM, Ben Hoyt <benhoyt at gmail.com> wrote:

> I'm curious why the return value of __contains__ is coerced to True or
> False, whereas the return value of "normal" comparison operators like
> __lt__ and the like are not. The latter return the value directly without
> forcing it to be True or False.
>
> This makes overriding __contains__ significantly less flexible, so I'm
> wondering why it's designed or implemented that way. (I believe it's the
> cmp_outcome() function in Python/ceval.c that does this:
> http://hg.python.org/cpython/file/db9fe49069ed/Python/ceval.c#l4545)
>
> For example, the "peewee" ORM overloads __lt__ and the like so it can map
> Python expressions to SQL. But it can't do this with the "in" operator due
> to the result of "x in y" always returning True or False in Python. So it
> (ab)uses the "<<" operator to do this instead (See the peewee docs at
> http://peewee.readthedocs.org/en/latest/peewee/querying.html#column-lookups
> ).
>
> I'm sure there's a good reason for why "in" is different here, but I can't
> see why right now.
>
> -Ben
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130715/13d195a3/attachment.html>

From ncoghlan at gmail.com  Mon Jul 15 02:30:02 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 15 Jul 2013 10:30:02 +1000
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <51E338D1.5070900@pearwood.info>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<51E338D1.5070900@pearwood.info>
Message-ID: <CADiSq7cpDyhC_HNggMnwpSYPs9XVNkipu2ud9aTmscp+=PZoaQ@mail.gmail.com>

On 15 July 2013 09:48, Steven D'Aprano <steve at pearwood.info> wrote:
> On 14/07/13 21:09, Nick Coghlan wrote:
>
>> Slight adjustment to the proposed wording to ensure completely
>> undocumented
>> modules are also considered private:
>
>
> -1 on this adjustment. If somebody cannot be bothered writing a one-line doc
> string:
>
> "This module is private, don't touch."
>
> then they certainly shouldn't be allowed to use up a public name for a
> private module. I don't think we should be encouraging more private,
> undocumented modules. (Documentation is valuable even for private modules.)

For context, this arose when I checked PEP 8 after the point was
raised on distutils-sig that pip uses a public name for its
implementation module (also pip, same as the CLI), but officially
exposes no stable public programmatic API.

At the moment, according to what little PEP 8 has to say about public
vs private interfaces, that means a bundled pip would represent a new
public API, despite the fact that the only documented interface for
pip is the CLI, not the Python module API.

Since we've been burned by people assuming private APIs are public in
the past, I figured it made sense to get the actual standards we
follow in practice documented properly, *before* anyone further gets
the wrong idea from "import pip; help(pip)".

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Mon Jul 15 02:45:20 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 15 Jul 2013 10:45:20 +1000
Subject: [Python-Dev] Why is the return value of __contains__ coerced to
 boolean, but that of __lt__ and the like is not?
In-Reply-To: <CAL9jXCE2nK9ugrpo2_aoNqXKexTwjBoztvSh_1FEDWa0kmfKag@mail.gmail.com>
References: <CAL9jXCE2nK9ugrpo2_aoNqXKexTwjBoztvSh_1FEDWa0kmfKag@mail.gmail.com>
Message-ID: <CADiSq7fsUiRq6XZ8ySe3XzWF7qUVHua0-24PDXB+YO18kCqdbg@mail.gmail.com>

On 15 July 2013 10:21, Ben Hoyt <benhoyt at gmail.com> wrote:
> I'm sure there's a good reason for why "in" is different here, but I can't
> see why right now.

It depends on what you mean by "good reason" - PEP 207 (which is what
allows arbitrary objects to be returned from comparison operations)
was entirely about replacing __cmp__ with the rich comparison methods,
it doesn't mention __contains__ at all.

At this point the main limitations are backwards compatibility (having
existing containment tests suddenly start returning anything other
than True or False would be problematic), along with the signature of
CPython's sq_contains slot (it returns an integer rather than a
PyObject pointer).

Accordingly, to convert containment testing to a rich comparison
operation would require a new protocol. That said, there is potential
value in redefining containment in terms of a symmetric protocol
(rather than the current only-controlled-by-the-container behaviour),
so such a PEP may be worth writing. (it would initially be a topic for
python-ideas rather than python-dev, though)

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From tjreedy at udel.edu  Mon Jul 15 04:10:16 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 14 Jul 2013 22:10:16 -0400
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
Message-ID: <krvlm5$tjo$1@ger.gmane.org>

On 7/14/2013 7:09 AM, Nick Coghlan wrote:

> Slight adjustment to the proposed wording to ensure completely
> undocumented modules are also considered private:
>
> =================
> Private interfaces
>
> Unless explicitly documented otherwise, a leading underscore on any name
> indicates that it is an internal implementation detail and backwards
> compatibility guarantees do not apply. It is strongly encouraged that
> private APIs (whether modules, classes, functions, attributes or other
> names) be clearly marked in this way, as Python users may rely on
> introspection to identify available functionality and may be misled into
> believing an API without a leading underscore is in fact a public API
> with the standard backwards compatibility guarantees.
>
> Even when their names do not start with a leading underscore, all test
> modules and all modules that are not covered in the documentation are
> also considered private interfaces.

I was going to suggest adding 'and most idlelib ' between 'test' and 
'modules', but the broader addition covers idlelib, which is not 
mentioned yet in the docs, even in the (unindexed) Idle chapter. When it 
is, I will try to remember to make explicit which names and interfaces 
are public and that the rest are private.

-- 
Terry Jan Reedy


From benhoyt at gmail.com  Mon Jul 15 05:34:08 2013
From: benhoyt at gmail.com (Ben Hoyt)
Date: Mon, 15 Jul 2013 15:34:08 +1200
Subject: [Python-Dev] Why is the return value of __contains__ coerced to
 boolean, but that of __lt__ and the like is not?
In-Reply-To: <CADiSq7fsUiRq6XZ8ySe3XzWF7qUVHua0-24PDXB+YO18kCqdbg@mail.gmail.com>
References: <CAL9jXCE2nK9ugrpo2_aoNqXKexTwjBoztvSh_1FEDWa0kmfKag@mail.gmail.com>
	<CADiSq7fsUiRq6XZ8ySe3XzWF7qUVHua0-24PDXB+YO18kCqdbg@mail.gmail.com>
Message-ID: <CAL9jXCGLN7JFwFNngF6tCkn9Y4VDSzUoZtCCOsAQ-pdHQssJPw@mail.gmail.com>

Thanks, Nick -- that's helpful info. Writing such a PEP is a nice idea, but
I think it'd be beyond me (I'm not familiar enough with CPython internals,
protocols, etc).

Can you explain what you mean by "symmetric protocol rather than the
current only-controlled-by-the-container behaviour"?

-Ben


On Mon, Jul 15, 2013 at 12:45 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 15 July 2013 10:21, Ben Hoyt <benhoyt at gmail.com> wrote:
> > I'm sure there's a good reason for why "in" is different here, but I
> can't
> > see why right now.
>
> It depends on what you mean by "good reason" - PEP 207 (which is what
> allows arbitrary objects to be returned from comparison operations)
> was entirely about replacing __cmp__ with the rich comparison methods,
> it doesn't mention __contains__ at all.
>
> At this point the main limitations are backwards compatibility (having
> existing containment tests suddenly start returning anything other
> than True or False would be problematic), along with the signature of
> CPython's sq_contains slot (it returns an integer rather than a
> PyObject pointer).
>
> Accordingly, to convert containment testing to a rich comparison
> operation would require a new protocol. That said, there is potential
> value in redefining containment in terms of a symmetric protocol
> (rather than the current only-controlled-by-the-container behaviour),
> so such a PEP may be worth writing. (it would initially be a topic for
> python-ideas rather than python-dev, though)
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130715/4d5d5834/attachment.html>

From steve at pearwood.info  Mon Jul 15 05:37:15 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 15 Jul 2013 13:37:15 +1000
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <CADiSq7cpDyhC_HNggMnwpSYPs9XVNkipu2ud9aTmscp+=PZoaQ@mail.gmail.com>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<51E338D1.5070900@pearwood.info>
	<CADiSq7cpDyhC_HNggMnwpSYPs9XVNkipu2ud9aTmscp+=PZoaQ@mail.gmail.com>
Message-ID: <20130715033714.GA21977@ando>

On Mon, Jul 15, 2013 at 10:30:02AM +1000, Nick Coghlan wrote:
> On 15 July 2013 09:48, Steven D'Aprano <steve at pearwood.info> wrote:
> > On 14/07/13 21:09, Nick Coghlan wrote:
> >
> >> Slight adjustment to the proposed wording to ensure completely
> >> undocumented
> >> modules are also considered private:
> >
> >
> > -1 on this adjustment. If somebody cannot be bothered writing a one-line doc
> > string:
> >
> > "This module is private, don't touch."
> >
> > then they certainly shouldn't be allowed to use up a public name for a
> > private module. I don't think we should be encouraging more private,
> > undocumented modules. (Documentation is valuable even for private modules.)
> 
> For context, this arose when I checked PEP 8 after the point was
> raised on distutils-sig that pip uses a public name for its
> implementation module (also pip, same as the CLI), but officially
> exposes no stable public programmatic API.

I don't think we should bless this as official policy, certainly not for 
the standard library. We can't tell external modules what to do, and 
possibly there's nothing we can do about existing undocumented private 
modules, short of renaming them, but I think that at least for new 
additions to the std lib module names should follow the same 
single-underscore convention as functions, classes etc. A single leading 
underscore introduces the private namespace, everything else is public.

I've seen too much undocumented public code to ever assume that lack of 
documentation implies it is private.


> At the moment, according to what little PEP 8 has to say about public
> vs private interfaces, that means a bundled pip would represent a new
> public API, despite the fact that the only documented interface for
> pip is the CLI, not the Python module API.
> 
> Since we've been burned by people assuming private APIs are public in
> the past, I figured it made sense to get the actual standards we
> follow in practice documented properly, *before* anyone further gets
> the wrong idea from "import pip; help(pip)".

I can tell you that people will do that regardless of what PEP 8 has to 
say about "not documented == private". Especially as worded. I'm sure I 
won't be the only person to reason that if a module has a docstring, it 
is therefore documented.

It takes one character to explicitly document a module as private. 
That's much better than expecting people to treat their failure to 
find documentation as an implicit warning.



-- 
Steven

From steve at pearwood.info  Mon Jul 15 05:44:16 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 15 Jul 2013 13:44:16 +1000
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <20130715000117.GA46795@cskk.homeip.net>
References: <51E338D1.5070900@pearwood.info>
	<20130715000117.GA46795@cskk.homeip.net>
Message-ID: <20130715034416.GB21977@ando>

On Mon, Jul 15, 2013 at 10:01:17AM +1000, Cameron Simpson wrote:
> On 15Jul2013 09:48, Steven D'Aprano <steve at pearwood.info> wrote:

> | I'd go further, and say that no more private modules should be 
> | accepted for the std lib unless they have a leading underscore. I 
> | suppose for backwards compatibility reasons, we probably can't go 
> | through the std lib and rename private modules to make it clear they 
> | are private, but we don't have to accept new ones without the 
> | underscore.
> 
> I disagree.
> 
> A private module is a perfectly sane way to implement the internals
> of something, especially if it is subject to implementation change
> in the future.

Of course private modules are sane. I never suggested "no new private 
modules at all". But putting them in the same namespace as public 
modules is not, just to save a leading underscore in the file name.

You don't even have to use the underscore in your own code:

import _stuff as stuff

is allowed, and doesn't make _stuff.py public since imported modules are 
considered implementation details by default.


-- 
Steven

From steve at pearwood.info  Mon Jul 15 05:47:56 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 15 Jul 2013 13:47:56 +1000
Subject: [Python-Dev] Why is the return value of __contains__ coerced to
	boolean, but that of __lt__ and the like is not?
In-Reply-To: <CAL9jXCGLN7JFwFNngF6tCkn9Y4VDSzUoZtCCOsAQ-pdHQssJPw@mail.gmail.com>
References: <CAL9jXCE2nK9ugrpo2_aoNqXKexTwjBoztvSh_1FEDWa0kmfKag@mail.gmail.com>
	<CADiSq7fsUiRq6XZ8ySe3XzWF7qUVHua0-24PDXB+YO18kCqdbg@mail.gmail.com>
	<CAL9jXCGLN7JFwFNngF6tCkn9Y4VDSzUoZtCCOsAQ-pdHQssJPw@mail.gmail.com>
Message-ID: <20130715034756.GC21977@ando>

On Mon, Jul 15, 2013 at 03:34:08PM +1200, Ben Hoyt wrote:
> Thanks, Nick -- that's helpful info. Writing such a PEP is a nice idea, but
> I think it'd be beyond me (I'm not familiar enough with CPython internals,
> protocols, etc).
> 
> Can you explain what you mean by "symmetric protocol rather than the
> current only-controlled-by-the-container behaviour"?

Most operators can be controlled by either the left-hand or right-hand 
operand. For example, x + y can end up calling either x.__add__(y) or 
y.__radd_(x). The `in` operator is an exception, it only ever calls the 
container:

x in y => y.__contains__(x)

but never x.__contained_by__(y)



-- 
Steven

From ncoghlan at gmail.com  Mon Jul 15 05:53:56 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 15 Jul 2013 13:53:56 +1000
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <20130715033714.GA21977@ando>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<51E338D1.5070900@pearwood.info>
	<CADiSq7cpDyhC_HNggMnwpSYPs9XVNkipu2ud9aTmscp+=PZoaQ@mail.gmail.com>
	<20130715033714.GA21977@ando>
Message-ID: <CADiSq7chih-wfwfKD=kjATU+QPG2ZjcwD8_SyDpSUWge5CBZng@mail.gmail.com>

On 15 Jul 2013 13:43, "Steven D'Aprano" <steve at pearwood.info> wrote:
>
> On Mon, Jul 15, 2013 at 10:30:02AM +1000, Nick Coghlan wrote:
> > On 15 July 2013 09:48, Steven D'Aprano <steve at pearwood.info> wrote:
> > > On 14/07/13 21:09, Nick Coghlan wrote:
> > >
> > >> Slight adjustment to the proposed wording to ensure completely
> > >> undocumented
> > >> modules are also considered private:
> > >
> > >
> > > -1 on this adjustment. If somebody cannot be bothered writing a
one-line doc
> > > string:
> > >
> > > "This module is private, don't touch."
> > >
> > > then they certainly shouldn't be allowed to use up a public name for a
> > > private module. I don't think we should be encouraging more private,
> > > undocumented modules. (Documentation is valuable even for private
modules.)
> >
> > For context, this arose when I checked PEP 8 after the point was
> > raised on distutils-sig that pip uses a public name for its
> > implementation module (also pip, same as the CLI), but officially
> > exposes no stable public programmatic API.
>
> I don't think we should bless this as official policy, certainly not for
> the standard library. We can't tell external modules what to do, and
> possibly there's nothing we can do about existing undocumented private
> modules, short of renaming them, but I think that at least for new
> additions to the std lib module names should follow the same
> single-underscore convention as functions, classes etc. A single leading
> underscore introduces the private namespace, everything else is public.
>
> I've seen too much undocumented public code to ever assume that lack of
> documentation implies it is private.
>
>
> > At the moment, according to what little PEP 8 has to say about public
> > vs private interfaces, that means a bundled pip would represent a new
> > public API, despite the fact that the only documented interface for
> > pip is the CLI, not the Python module API.
> >
> > Since we've been burned by people assuming private APIs are public in
> > the past, I figured it made sense to get the actual standards we
> > follow in practice documented properly, *before* anyone further gets
> > the wrong idea from "import pip; help(pip)".
>
> I can tell you that people will do that regardless of what PEP 8 has to
> say about "not documented == private". Especially as worded. I'm sure I
> won't be the only person to reason that if a module has a docstring, it
> is therefore documented.
>
> It takes one character to explicitly document a module as private.
> That's much better than expecting people to treat their failure to
> find documentation as an implicit warning.

We're considering bundling the pip CLI for the convenience of educators and
beginners, not adding their implementation module to the standard library.
That means we need a bright shining line for what constitutes "in the
standard library". Yes, it's better if people properly use a leading
underscore, but the fact is many don't (including idlelib, as Terry noted)
and it becomes a disruptive change to add it later if people don't start
with it. The pip and IDLE developers have better things to be doing than
changing the names of undocumented APIs just because the application is
distributed along with CPython.

Cheers,
Nick.

>
>
>
> --
> Steven
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130715/7a182b78/attachment.html>

From ncoghlan at gmail.com  Mon Jul 15 05:55:37 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 15 Jul 2013 13:55:37 +1000
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <20130715034416.GB21977@ando>
References: <51E338D1.5070900@pearwood.info>
	<20130715000117.GA46795@cskk.homeip.net>
	<20130715034416.GB21977@ando>
Message-ID: <CADiSq7fcx7CZHmAdHoGMHzD+Kk_j32=T5bgCKiYnbV_OsFNTsA@mail.gmail.com>

On 15 Jul 2013 13:46, "Steven D'Aprano" <steve at pearwood.info> wrote:
>
> On Mon, Jul 15, 2013 at 10:01:17AM +1000, Cameron Simpson wrote:
> > On 15Jul2013 09:48, Steven D'Aprano <steve at pearwood.info> wrote:
>
> > | I'd go further, and say that no more private modules should be
> > | accepted for the std lib unless they have a leading underscore. I
> > | suppose for backwards compatibility reasons, we probably can't go
> > | through the std lib and rename private modules to make it clear they
> > | are private, but we don't have to accept new ones without the
> > | underscore.
> >
> > I disagree.
> >
> > A private module is a perfectly sane way to implement the internals
> > of something, especially if it is subject to implementation change
> > in the future.
>
> Of course private modules are sane. I never suggested "no new private
> modules at all". But putting them in the same namespace as public
> modules is not, just to save a leading underscore in the file name.

It's not to save a leading underscore - it's to avoid renaming existing
packages like pip and idlelib.

Cheers,
Nick.

>
> You don't even have to use the underscore in your own code:
>
> import _stuff as stuff
>
> is allowed, and doesn't make _stuff.py public since imported modules are
> considered implementation details by default.
>
>
> --
> Steven
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130715/3d115c1f/attachment.html>

From ncoghlan at gmail.com  Mon Jul 15 06:17:55 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 15 Jul 2013 14:17:55 +1000
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
Message-ID: <CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>

On 15 July 2013 02:15, Gregory P. Smith <greg at krypto.org> wrote:
> On Sun, Jul 14, 2013 at 4:09 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> as Python users may rely on introspection to identify available
>> functionality and may be misled into believing an API without a leading
>> underscore is in fact a public API with the standard backwards compatibility
>> guarantees.
>
>
> While true, I'm not sure the last part of the sentence is necessary. Once
> we've established that a leading _ indicates something is private there
> isn't much point in explaining the various ways people might find them.  I'm
> happy regardless of this bit being there.

You'd be surprised how many  non-core devs react with astonishment
when I suggest that not documenting something isn't enough to avoid
having users consider it a supported public API - they usually get it
after I point out how far you can usually get just by using dir() and
help() to play with a new library at the interactive prompt instead of
looking at out-of-band docs. I figure including the second part will
help prevent some "But why?" reactions in the future.

I think Steven has a reasonable point about being clearer that an
explicit leading underscore is the preferred solution for any new
private modules, though, so here's an updated proposal:

=================
Private interfaces

Unless explicitly documented otherwise, a leading underscore on any
name indicates that it is an internal implementation detail and any
backwards compatibility guarantees do not apply. It is strongly
encouraged that private APIs (whether modules, classes, functions,
attributes or other names) be clearly marked in this way, as Python
users may rely on introspection to identify available functionality
and may be misled into believing an API without a leading underscore
is in fact a public API with the standard backwards compatibility
guarantees.

While the explicit use of a leading underscore in the names of private
modules is preferred, all test modules and all modules that are not
explicitly covered in the documentation are also considered private
interfaces, even when their names do not start with a leading
underscore and even if they include a module level documentation
string. This includes submodules of packages that are documented as if
they were a single module.

Similarly, the specific modules and external APIs imported by a module
are always considered an implementation detail. Other modules should
not rely on indirect access to such imported interfaces unless they
are an explicitly documented part of the containing module's API (such
as ``os.path`` or a package ``__init__`` module that exposes
functionality from submodules).
=================



--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From regebro at gmail.com  Mon Jul 15 07:12:50 2013
From: regebro at gmail.com (Lennart Regebro)
Date: Mon, 15 Jul 2013 07:12:50 +0200
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
Message-ID: <CAL0kPAUDwruWuR=-f4vH3kU_Pj5sN0d=aE59avcpTaKmHmVBpg@mail.gmail.com>

On Mon, Jul 15, 2013 at 6:17 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> =================
> Private interfaces
>
> Unless explicitly documented otherwise, a leading underscore on any
> name indicates that it is an internal implementation detail and any
> backwards compatibility guarantees do not apply. It is strongly
> encouraged that private APIs (whether modules, classes, functions,
> attributes or other names) be clearly marked in this way, as Python
> users may rely on introspection to identify available functionality
> and may be misled into believing an API without a leading underscore
> is in fact a public API with the standard backwards compatibility
> guarantees.

Very good.

> While the explicit use of a leading underscore in the names of private
> modules is preferred, all test modules and all modules that are not
> explicitly covered in the documentation are also considered private
> interfaces, even when their names do not start with a leading
> underscore and even if they include a module level documentation
> string. This includes submodules of packages that are documented as if
> they were a single module.

But wait, aren't this about how to use other peoples modules, more
than a style guide of how to write your own modules?
I don't think this belongs in PEP 8 at all.

//Lennart

From stephen at xemacs.org  Mon Jul 15 07:16:37 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Mon, 15 Jul 2013 14:16:37 +0900
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of
	leading	underscores
In-Reply-To: <CADiSq7fcx7CZHmAdHoGMHzD+Kk_j32=T5bgCKiYnbV_OsFNTsA@mail.gmail.com>
References: <51E338D1.5070900@pearwood.info>
	<20130715000117.GA46795@cskk.homeip.net>
	<20130715034416.GB21977@ando>
	<CADiSq7fcx7CZHmAdHoGMHzD+Kk_j32=T5bgCKiYnbV_OsFNTsA@mail.gmail.com>
Message-ID: <871u70tnbe.fsf@uwakimon.sk.tsukuba.ac.jp>

Nick Coghlan writes:

 >> Of course private modules are sane. I never suggested "no new private
 >> modules at all". But putting them in the same namespace as public
 >> modules is not, just to save a leading underscore in the file name.

 > It's not to save a leading underscore - it's to avoid renaming
 > existing packages like pip and idlelib.

"existing" is not a subset of "new."  Am I missing something?<wink/>

What Steven is suggesting is that there are two kinds of private
package in the stdlib: those that can conveniently be given names that
indicate that they are private (generally, "new" packages), and those
that can't ("existing" packages).

For the latter IIRC he suggested adding a line at the top of the module
documentation "This module is *private*, and you cannot depend on API
compatibility with future or past releases of this module.  The public
API is in ... [alternatively, There is no public API]."

I really don't see how one can object to this suggestion, given an
appropriate definition of "convenient enough to get a _name".

From raymond.hettinger at gmail.com  Mon Jul 15 08:01:21 2013
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Sun, 14 Jul 2013 23:01:21 -0700
Subject: [Python-Dev] PLY in stdlib (was cffi in stdlib)
In-Reply-To: <8595C679-3D16-4312-98C8-146309FDF825@dabeaz.com>
References: <mailman.13132.1361899302.2938.python-dev@python.org>
	<F6FAB3A2-6040-42CE-A4FA-93077756C2DC@dabeaz.com>
	<AC6A1BA1-3169-472C-BCB1-2C57A1CF3169@voidspace.org.uk>
	<CALFfu7BxLEDOYwjCSO8=+Xh==1Ef8Eq2N2B2TnvCbwGfGfZG_Q@mail.gmail.com>
	<krqlp1$e25$1@ger.gmane.org>
	<96746EE7-E96A-4363-B0F7-D7950207C702@voidspace.org.uk>
	<CAP1=2W5brJjW0==ou1+NQ=wntRm11mkduzXQY_eDnizxAXHF0g@mail.gmail.com>
	<E680C312-42BC-42DD-8125-ED45D560B24A@dabeaz.com>
	<CADiSq7dsk3vxU0XSUvHXp-AK+tExVSHQsBC+LH08PCXC4D4FOQ@mail.gmail.com>
	<CAP1=2W7-2NDWyCs6jkYNNraLBrAfOoftOT4PknU89V2KwJkMBg@mail.gmail.com>
	<8595C679-3D16-4312-98C8-146309FDF825@dabeaz.com>
Message-ID: <268FFB76-3125-4BB1-9706-4BFE86696652@gmail.com>


On Jul 14, 2013, at 6:32 AM, David Beazley <dave at dabeaz.com> wrote:

> I honestly don't have any particular thoughts about PLY vs. other parser generators and the merits of their inclusion (or not) in the standard library. 

I would love to have PLY in the standard library.
It would open up a whole new world to some users
and be the basis for tool generation for others.


Raymond

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130714/1015ab94/attachment.html>

From benjamin at python.org  Mon Jul 15 08:06:28 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Sun, 14 Jul 2013 23:06:28 -0700
Subject: [Python-Dev] Doubt about closure/function generation in bytecode
In-Reply-To: <CALy7ODst91xMBuT0bxt7Q7tm3WxZd1ewBZ6JjV3GaTaVoKocRQ@mail.gmail.com>
References: <CALy7ODst91xMBuT0bxt7Q7tm3WxZd1ewBZ6JjV3GaTaVoKocRQ@mail.gmail.com>
Message-ID: <CAPZV6o_kDrq6WfKHKtiO9GP8tnORqmfEyx9=2+4oLDCQSbpm8w@mail.gmail.com>

2013/7/12 Andrea Griffini <agriff at tin.it>:
> Is there any semantic difference between
>
>     BUILD_TUPLE 0
>     LOAD_CONST <code object ... >
>     MAKE_CLOSURE 0
>
> and
>
>     LOAD_CONST <code object ...>
>     MAKE_FUNCTION 0
>
> ?
>
> In other words is there any difference between a function and a closure that
> doesn't capture anything? Is it just a speed optimization for a common case?

It probably has more to do with the fact that nested scopes with added
much later in Python's history than functions.
>



--
Regards,
Benjamin

From tjreedy at udel.edu  Mon Jul 15 09:31:03 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 15 Jul 2013 03:31:03 -0400
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
Message-ID: <ks08fk$vnu$1@ger.gmane.org>

On 7/15/2013 12:17 AM, Nick Coghlan wrote:

> You'd be surprised how many  non-core devs react with astonishment
> when I suggest that not documenting something isn't enough to avoid
> having users consider it a supported public API - they usually get it
> after I point out how far you can usually get just by using dir() and
> help() to play with a new library at the interactive prompt instead of
> looking at out-of-band docs. I figure including the second part will
> help prevent some "But why?" reactions in the future.

I agree with the goal of preventing such reactions, but I suggest 
slightly different means to do so.

I just looked and idlelib.__init__.py consists of
"""# Dummy file to make this a package.
"""
help(idlelib) prints the comment (which surprised me, but it seems to be 
smart enough to look for a 'doc comment' when it does not find a doc 
string). I now plan to change the comment to a docstring so 
'idlelib.__doc__' also works and change the text to something like

"""A package of private modules that implement the IDLE integrated shell 
and editor application. See the manual for the public entry points.""" 
(or maybe list them briefly here in the doc string).

This should properly inform people who introspect.

Also, the manual needs a small, indexed, section on idlelib that says 
much the same thing along with documenting what *is* public (which is 
not completely clear to me at the moment).

> I think Steven has a reasonable point about being clearer that an
> explicit leading underscore is the preferred solution for any new
> private modules, though, so here's an updated proposal:

The public/private distinction is not always as clean-cut as seems to be 
assumed in this discussion. Test and idlelib both have public entry 
points to an application based on private files. ('Running the Python 
test suite' is a batch application.)

So I do not think either should be _ prefixed. For both, and anything 
similar, a proper docstring and manual entry explaining the nuances 
should be completely acceptable as a substitute for a blunt, 
all-or-nothing flag.

I also think that marking something private, either way, or partially 
private with words, should extend to its contents. In particular, the 
100s of implementation files in test and idlelib need not be _ marked as 
as they would be covered once test and idlelib are properly documented. 
The special _ mark makes sense when it is the exception, but not in a 
context where it would be the overwhelming norm, because everything but 
one or two files is private.

> =================
> Private interfaces
>
> Unless explicitly documented otherwise, a leading underscore on any
> name indicates that it is an internal implementation detail and any
> backwards compatibility guarantees do not apply. It is strongly
> encouraged that private APIs (whether modules, classes, functions,
> attributes or other names) be clearly marked in this way, as Python
> users may rely on introspection to identify available functionality
> and may be misled into believing an API without a leading underscore
> is in fact a public API with the standard backwards compatibility
> guarantees.
>
> While the explicit use of a leading underscore in the names of private
> modules is preferred, all test modules and all modules that are not
> explicitly covered in the documentation are also considered private
> interfaces, even when their names do not start with a leading
> underscore and even if they include a module level documentation
> string. This includes submodules of packages that are documented as if
> they were a single module.

Since I am planning to explicitly cover idlelib in the doc, to list the 
public interface, just as is done for test, I would like it explicitly 
mentioned along with test.

I do not think absence of documentation is a good signal. I would rather 
say that any private or partially private package that does not have a _ 
name *must* have a docstring saying that it is all or mostly  private. 
(Or be a subpackage or module of something so marked.) And presuming 
that it is associated with something that is documented, then the doc 
should state that the implementation files are private.

In other words, I suggest that "modules that are not explicitly covered 
in the documentation ... [whose] names do not start with a leading 
underscore" should be empty. And I think "even if they include a module 
level documentation string." is backwards. Such files *should* have a 
docstring that serves as a substitute for a _ prefix.

> Similarly, the specific modules and external APIs imported by a module
> are always considered an implementation detail. Other modules should
> not rely on indirect access to such imported interfaces unless they
> are an explicitly documented part of the containing module's API (such
> as ``os.path`` or a package ``__init__`` module that exposes
> functionality from submodules).
> =================

-- 
Terry Jan Reedy


From ncoghlan at gmail.com  Mon Jul 15 09:32:53 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 15 Jul 2013 17:32:53 +1000
Subject: [Python-Dev] PLY in stdlib (was cffi in stdlib)
In-Reply-To: <268FFB76-3125-4BB1-9706-4BFE86696652@gmail.com>
References: <mailman.13132.1361899302.2938.python-dev@python.org>
	<F6FAB3A2-6040-42CE-A4FA-93077756C2DC@dabeaz.com>
	<AC6A1BA1-3169-472C-BCB1-2C57A1CF3169@voidspace.org.uk>
	<CALFfu7BxLEDOYwjCSO8=+Xh==1Ef8Eq2N2B2TnvCbwGfGfZG_Q@mail.gmail.com>
	<krqlp1$e25$1@ger.gmane.org>
	<96746EE7-E96A-4363-B0F7-D7950207C702@voidspace.org.uk>
	<CAP1=2W5brJjW0==ou1+NQ=wntRm11mkduzXQY_eDnizxAXHF0g@mail.gmail.com>
	<E680C312-42BC-42DD-8125-ED45D560B24A@dabeaz.com>
	<CADiSq7dsk3vxU0XSUvHXp-AK+tExVSHQsBC+LH08PCXC4D4FOQ@mail.gmail.com>
	<CAP1=2W7-2NDWyCs6jkYNNraLBrAfOoftOT4PknU89V2KwJkMBg@mail.gmail.com>
	<8595C679-3D16-4312-98C8-146309FDF825@dabeaz.com>
	<268FFB76-3125-4BB1-9706-4BFE86696652@gmail.com>
Message-ID: <CADiSq7cFN8bSAD4ymmxPth+RYAZMRMgHeKAhnJUdV=zy=5S_Ew@mail.gmail.com>

On 15 July 2013 16:01, Raymond Hettinger <raymond.hettinger at gmail.com> wrote:
>
> On Jul 14, 2013, at 6:32 AM, David Beazley <dave at dabeaz.com> wrote:
>
> I honestly don't have any particular thoughts about PLY vs. other parser
> generators and the merits of their inclusion (or not) in the standard
> library.
>
>
> I would love to have PLY in the standard library.
> It would open up a whole new world to some users
> and be the basis for tool generation for others.

Agreed (and the fact it's a prerequisite for pycparsing and cffi just
seals the deal).

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From p.f.moore at gmail.com  Mon Jul 15 09:37:01 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 15 Jul 2013 08:37:01 +0100
Subject: [Python-Dev] PLY in stdlib (was cffi in stdlib)
In-Reply-To: <268FFB76-3125-4BB1-9706-4BFE86696652@gmail.com>
References: <mailman.13132.1361899302.2938.python-dev@python.org>
	<F6FAB3A2-6040-42CE-A4FA-93077756C2DC@dabeaz.com>
	<AC6A1BA1-3169-472C-BCB1-2C57A1CF3169@voidspace.org.uk>
	<CALFfu7BxLEDOYwjCSO8=+Xh==1Ef8Eq2N2B2TnvCbwGfGfZG_Q@mail.gmail.com>
	<krqlp1$e25$1@ger.gmane.org>
	<96746EE7-E96A-4363-B0F7-D7950207C702@voidspace.org.uk>
	<CAP1=2W5brJjW0==ou1+NQ=wntRm11mkduzXQY_eDnizxAXHF0g@mail.gmail.com>
	<E680C312-42BC-42DD-8125-ED45D560B24A@dabeaz.com>
	<CADiSq7dsk3vxU0XSUvHXp-AK+tExVSHQsBC+LH08PCXC4D4FOQ@mail.gmail.com>
	<CAP1=2W7-2NDWyCs6jkYNNraLBrAfOoftOT4PknU89V2KwJkMBg@mail.gmail.com>
	<8595C679-3D16-4312-98C8-146309FDF825@dabeaz.com>
	<268FFB76-3125-4BB1-9706-4BFE86696652@gmail.com>
Message-ID: <CACac1F_+ExjzR0GuZmiyxYg_CjECGknsHbAaLetGY8aNXee_fw@mail.gmail.com>

On 15 July 2013 07:01, Raymond Hettinger <raymond.hettinger at gmail.com>wrote:

> I would love to have PLY in the standard library.
> It would open up a whole new world to some users
> and be the basis for tool generation for others.
>

+1. Parser generators are useful tools - parsers are right on the boundary
of "easy enough to understand why you'd like one, but hard enough to put
you off implementing our own". And there is value in tools like this being
in the stdlib as opposed to a 3rd party dependency.

Paul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130715/462c6e89/attachment.html>

From ncoghlan at gmail.com  Mon Jul 15 09:48:59 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 15 Jul 2013 17:48:59 +1000
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <ks08fk$vnu$1@ger.gmane.org>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
	<ks08fk$vnu$1@ger.gmane.org>
Message-ID: <CADiSq7fAS+-keOjBmOEs-UvV=bV4qB9=ikiN5dkvRW7LSrge6g@mail.gmail.com>

On 15 July 2013 17:31, Terry Reedy <tjreedy at udel.edu> wrote:
> I do not think absence of documentation is a good signal. I would rather say
> that any private or partially private package that does not have a _ name
> *must* have a docstring saying that it is all or mostly  private. (Or be a
> subpackage or module of something so marked.) And presuming that it is
> associated with something that is documented, then the doc should state that
> the implementation files are private.
>
> In other words, I suggest that "modules that are not explicitly covered in
> the documentation ... [whose] names do not start with a leading underscore"
> should be empty. And I think "even if they include a module level
> documentation string." is backwards. Such files *should* have a docstring
> that serves as a substitute for a _ prefix.

I (now) agree - a disclaimer in the docstring is a better signal than
an absence of documentation, and one we can reasonably make a
requirement even for external applications that we bundle or otherwise
bless as the "obvious way to do it" (as is happening with pip).

=================
Private interfaces

Unless explicitly documented otherwise, a leading underscore on any
name indicates that it is an internal implementation detail and any
backwards compatibility guarantees do not apply. It is strongly
encouraged that private APIs (whether modules, classes, functions,
attributes or other names) be clearly marked in this way, as Python
users may rely on introspection to identify available functionality
and may be misled into believing an API without a leading underscore
is in fact a public API with the standard backwards compatibility
guarantees.

While the explicit use of a leading underscore is the preferred solution,
the names of some private (or partially private) modules (such as ``test``
and ``idlelib``) lack the leading underscore either for historical reasons
or because they expose a public command line interface through the
``-m`` switch. Such modules should include an explicit disclaimer in
their module docstring, indicating that they do not use the leading
underscore convention and noting where the definition of the public
API (if any) can be found (the public API definition may also be part
of the module docstring).

As a general principle, the specific modules and external APIs imported by
a module are always considered an implementation detail. Other modules
should not rely on indirect access to such imported interfaces unless they
are an explicitly documented part of the containing module's API (such
as ``os.path`` or a package ``__init__`` module that exposes functionality
from submodules).
=================

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From p.f.moore at gmail.com  Mon Jul 15 10:26:50 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 15 Jul 2013 09:26:50 +0100
Subject: [Python-Dev] Adding Python scripts to PATHEXT on Windows
Message-ID: <CACac1F9WtTj69gBv2SKR3chxFdduhNtxNOmPqrt=mK5kHb_ZMA@mail.gmail.com>

With the addition of the Python launcher (PEP 397), Python scripts on
Windows are executable in much the same way as on Unix. However, "out of
the box" it is still not quite possible to use Python scripts. This is
because the .py extension is not registered with Windows in the PATHEXT
environment variable (which holds all file extensions which should be
looked for on PATH at the command line).

There are two main consequences to this:

1. Under powershell, ".py" files on PATH are not found at all (this is not
the case with cmd.exe, and may be technically a bug in powershell - but
it's unlikely to change nevertheless).
2. Under both powershell and cmd, users have to explicitly include the
extension when invoking the command.

I would like to propose that the .py extension be added to PATHEXT as part
of the "Register Python Extensions" option in the Python MSI installer.
This would mean that Python users had a clean way of writing cross-platform
"wrapper scripts" and command-line applications out of the box, just by
writing a normal Python script with a #! line at the top.

Does anyone have any objections to this? I could try to write a patch, but
I know next to nothing about building MSIs, so if any of the installer
experts could help that would be fantastic.

Paul

PS There would still be one difference, in that scripts intended to be used
as commands would be named with '.py' on Windows and with no extension on
Unix, but that's a pretty trivial difference, and reflects fundamental
platform differences rather than configuration. So I don't propose worrying
about this.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130715/da10c602/attachment.html>

From ncoghlan at gmail.com  Mon Jul 15 10:40:15 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 15 Jul 2013 18:40:15 +1000
Subject: [Python-Dev] Adding Python scripts to PATHEXT on Windows
In-Reply-To: <CACac1F9WtTj69gBv2SKR3chxFdduhNtxNOmPqrt=mK5kHb_ZMA@mail.gmail.com>
References: <CACac1F9WtTj69gBv2SKR3chxFdduhNtxNOmPqrt=mK5kHb_ZMA@mail.gmail.com>
Message-ID: <CADiSq7cdn3sqKDUkQ-nuOyasMrRR4g5=gMMUTJyhRYMfWRCqYQ@mail.gmail.com>

Sounds good to me - in the long run it may allow us to drop the Windows exe
wrappers when installing scripts on Windows.

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130715/9a5db819/attachment.html>

From python.00 at klix.ch  Mon Jul 15 10:32:12 2013
From: python.00 at klix.ch (Gerald Klix)
Date: Mon, 15 Jul 2013 10:32:12 +0200
Subject: [Python-Dev] Adding Python scripts to PATHEXT on Windows
In-Reply-To: <CACac1F9WtTj69gBv2SKR3chxFdduhNtxNOmPqrt=mK5kHb_ZMA@mail.gmail.com>
References: <CACac1F9WtTj69gBv2SKR3chxFdduhNtxNOmPqrt=mK5kHb_ZMA@mail.gmail.com>
Message-ID: <8FC77D8F-B447-4ACD-87DF-42A04ABB29C4@klix.ch>

+1

BTW: I use lots of "*.py-executables" on Linux and OSX, so different extension don't bother me.

Gerald


Am 15.07.2013 um 10:26 schrieb Paul Moore <p.f.moore at gmail.com>:

> With the addition of the Python launcher (PEP 397), Python scripts on
> Windows are executable in much the same way as on Unix. However, "out of
> the box" it is still not quite possible to use Python scripts. This is
> because the .py extension is not registered with Windows in the PATHEXT
> environment variable (which holds all file extensions which should be
> looked for on PATH at the command line).
> 
> There are two main consequences to this:
> 
> 1. Under powershell, ".py" files on PATH are not found at all (this is not
> the case with cmd.exe, and may be technically a bug in powershell - but
> it's unlikely to change nevertheless).
> 2. Under both powershell and cmd, users have to explicitly include the
> extension when invoking the command.
> 
> I would like to propose that the .py extension be added to PATHEXT as part
> of the "Register Python Extensions" option in the Python MSI installer.
> This would mean that Python users had a clean way of writing cross-platform
> "wrapper scripts" and command-line applications out of the box, just by
> writing a normal Python script with a #! line at the top.
> 
> Does anyone have any objections to this? I could try to write a patch, but
> I know next to nothing about building MSIs, so if any of the installer
> experts could help that would be fantastic.
> 
> Paul
> 
> PS There would still be one difference, in that scripts intended to be used
> as commands would be named with '.py' on Windows and with no extension on
> Unix, but that's a pretty trivial difference, and reflects fundamental
> platform differences rather than configuration. So I don't propose worrying
> about this.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/python.00%40klix.ch


From p.f.moore at gmail.com  Mon Jul 15 10:42:33 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 15 Jul 2013 09:42:33 +0100
Subject: [Python-Dev] Adding Python scripts to PATHEXT on Windows
In-Reply-To: <CADiSq7cdn3sqKDUkQ-nuOyasMrRR4g5=gMMUTJyhRYMfWRCqYQ@mail.gmail.com>
References: <CACac1F9WtTj69gBv2SKR3chxFdduhNtxNOmPqrt=mK5kHb_ZMA@mail.gmail.com>
	<CADiSq7cdn3sqKDUkQ-nuOyasMrRR4g5=gMMUTJyhRYMfWRCqYQ@mail.gmail.com>
Message-ID: <CACac1F_vX5kPW6s=W9k6GhEtq-GS1Ao5U_UtD2szU7pQ7_rQww@mail.gmail.com>

On 15 July 2013 09:40, Nick Coghlan <ncoghlan at gmail.com> wrote:

> Sounds good to me - in the long run it may allow us to drop the Windows
> exe wrappers when installing scripts on Windows.


That's my motivation. In particular removing pip's exe wrappers to ease
some of the issues being flagged on distutils-sig.
Paul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130715/9a41db7f/attachment.html>

From ncoghlan at gmail.com  Mon Jul 15 12:04:54 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 15 Jul 2013 20:04:54 +1000
Subject: [Python-Dev] Adding Python scripts to PATHEXT on Windows
In-Reply-To: <CACac1F_vX5kPW6s=W9k6GhEtq-GS1Ao5U_UtD2szU7pQ7_rQww@mail.gmail.com>
References: <CACac1F9WtTj69gBv2SKR3chxFdduhNtxNOmPqrt=mK5kHb_ZMA@mail.gmail.com>
	<CADiSq7cdn3sqKDUkQ-nuOyasMrRR4g5=gMMUTJyhRYMfWRCqYQ@mail.gmail.com>
	<CACac1F_vX5kPW6s=W9k6GhEtq-GS1Ao5U_UtD2szU7pQ7_rQww@mail.gmail.com>
Message-ID: <CADiSq7fY0Rbt4xniRg3HZ8QA3KuYNHrt=ev6-woSQ1o32d-rUA@mail.gmail.com>

On 15 July 2013 18:42, Paul Moore <p.f.moore at gmail.com> wrote:
> On 15 July 2013 09:40, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> Sounds good to me - in the long run it may allow us to drop the Windows
>> exe wrappers when installing scripts on Windows.
>
>
> That's my motivation. In particular removing pip's exe wrappers to ease some
> of the issues being flagged on distutils-sig.

Ah, true. For some reason I was thinking we kept __main__.__file__
open, but we don't - we close it as soon as we have read and compiled
it.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Mon Jul 15 12:22:57 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 15 Jul 2013 20:22:57 +1000
Subject: [Python-Dev] Refactoring test.support into a subpackage
Message-ID: <CADiSq7f3nxGaEk2nUztGOHim4sk6oG9ZyFT1maOxnX7H5Y=0ig@mail.gmail.com>

At the PyCon AU sprints, some of the sprinters worked on a plan Chris
Jerdonek and I came up with months ago to convert test.support from a
module into a subpackage.

This plan arose from some nasty test suite hacks in the lead up to the
release of Python 3.3, where some of the pkgutil tests ended up in
test.test_runpy so they could use the package creation infrastructure
I have in there without needing to extract a helper module with a
release imminent.

The scope of the plan covers a few distinct steps:

1. Move Lib/test/support.py to Lib/test/support/__init__.py
(http://bugs.python.org/issue15494)
2. Add a new (documented) test.support.pkg_helper submodule with the
package creation support code (http://bugs.python.org/issue15403 and
http://bugs.python.org/issue15376)
3. Move the pkgutil tests to test.test_pkgutil where they belong
(http://bugs.python.org/issue15358)
4. Move the existing test.script_helper and test.bytecode_helper
support modules into the test.support subpackage and document them (no
issue for that as yet)

These changes would be made on both 3.3 and default, to avoid future
merge conflicts (note that Indra also posted an updated patch for one
pending test suite update to cope with the relocation:
http://bugs.python.org/issue15415)

My main motivation for doing this is to give a bit more visibility to
the helpers that already exist, as well as making them easier to find
when we want to look into the details of what a test is doing. At the
moment, they tend to get lost in the sea of actual test files in the
main test directory.

(Chris and I initially held off on this while waiting for the final
3.2 release, and then that got delayed and it wasn't until Indra,
Michael and Stephen started looking into it at the sprints that I
started paying attention again)

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ronaldoussoren at mac.com  Mon Jul 15 13:13:13 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Mon, 15 Jul 2013 13:13:13 +0200
Subject: [Python-Dev] Hooking into super() attribute resolution
In-Reply-To: <5339a8b72f264faa99a8720b5d1d161d@BLUPR03MB035.namprd03.prod.outlook.com>
References: <DCEB2B77-B9DD-405A-A084-EFB761B947BB@mac.com>
	<46EC256E-9ACE-4C4C-BE3D-325D7DB87B85@mac.com>
	<0fe31345247846658b331aa358c38c66@BLUPR03MB035.namprd03.prod.outlook.com>
	<CCA2B7DA-C3B7-44C1-B488-9D91B26F8C61@mac.com>
	<5fe3649f4c3c41ba927ca62fd762699c@BLUPR03MB035.namprd03.prod.outlook.com>
	<7946F238-3DA7-4198-9CED-23B33246B26F@mac.com>
	<5339a8b72f264faa99a8720b5d1d161d@BLUPR03MB035.namprd03.prod.outlook.com>
Message-ID: <C7C64F30-D50D-4BD5-8040-2DEC5606D130@mac.com>


On 9 Jul, 2013, at 1:21, Steve Dower <Steve.Dower at microsoft.com> wrote:

> 
> Except that if it's on a metaclass, the 'instance' it has access to is cls. The descriptor side of things is more interesting, but I see no reason why super can't do that itself, since it knows the actual instance to call __get__ with. (Presumably it already does this with the __dict__ lookup, since that won't call __get__ either.)
> 
> Explaining the new method is easiest if the default implementation is (literally):
> 
> def __getlocalname__(self, name):
>    try:
>        return self.__dict__[name]
>    except KeyError:
>        raise AttributeError(name)
> 
> which does not do any descriptor resolution (and is only a small step from simply replacing __dict__ with a custom object, which is basically where we started). The only change I've really suggested is making it an instance method that can be implemented on a metaclass if you want it for class members.

I've documented this (with a different name) in the current PEP draf (<http://www.python.org/dev/peps/pep-0447/>) and am working on an implementation.

Using a lookup method on the metaclass has a nice side-effect: the same method could be used by object.__getattribute__ (aka PyObject_GenericGetAttr) as well which could simplify my primary usecase for this new API. 

There is a problem with that though: the type attribute cache in Object/typeobject.c, using that cache isn't valid if _PyType_Lookup calls a method instead of peeking in tp_dict (the cache cannot be in the default __getlocalname__ implementation because a primary usecase for the cache appears to be avoiding walking the MRO [1]). I haven't decided yet what to do about this, a number of options:

* Don't use __getlocalname__ in _PyType_Lookup

* Use __getlocalname__ as a fallback after peeking in tp_dict (that is, use it like __getattr__ instead of __getattribute__) and only cache the attribute when it is fetched from tp_dict.

* Don't add a default __getlocalname__ and disable the attribute cache for types with a metatype that does have this method (this might be non-trivial because a metatype might grow a __getlocalname__ slot after instances of the metatype have been created; that would be ugly code but is possbile)

Ronald

[1] "appears" because I haven't found documentation for the cache yet

From ronaldoussoren at mac.com  Mon Jul 15 17:12:45 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Mon, 15 Jul 2013 17:12:45 +0200
Subject: [Python-Dev] PEP 447: hooking into super() attribute resolution
Message-ID: <2055CD40-2081-45DC-8B9A-889085FC7CEE@mac.com>

Hi,

I've posted a new update of my proposal to add a way to override the attribute resolution proces by super(). I've rewritten the PEP and implementation based on feedback by Steve Dower.

In the current edition of the proposal the hook is a method on the type, defined in a metaclass for types. A simple demo:

    class meta (type):
        def __locallookup__(cls, name):
            return type.__locallookup__(cls, name.upper())

    class S (metaclass=meta):
        def m(self):
            return 42

        def M(self):
             return "fortytwo"

    class SS (S):
        def m(self):
            return 21

        def M(self):
           return "twentyone"

    o = SS()
    print("self", o.m())
    print("super", super(SS, o).m())

The last line currently prints "super 42" and would print "super fortytwo" with my proposal.

A major open issue: the __locallookup__ method could also be used for normal attribute resolution, but that probably causes issues with attribute caching (see the PEP for details).  I haven't worked out yet if it is worthwhile to tweak the proposal to fix the caching issues (even though the proposal currently says that PyObject_GenericGetAttr will use the new method). Fixing the caching issue definitly would help in my primary use case by reducing code duplication, but might end up making the API unnecessarily complex.

Ronald



PEP: 447
Title: Hooking into super attribute resolution
Version: $Revision$
Last-Modified: $Date$
Author: Ronald Oussoren <ronaldoussoren at mac.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 12-Jun-2013
Post-History: 2-Jul-2013, ?


Abstract
========

In current python releases the attribute resolution of the `super class`_
peeks in the ``__dict__`` attribute of classes on the MRO to look
for attributes. This PEP introduces a hook that classes can use
to override that behavior for specific classes.


Rationale
=========

Peeking in the class ``__dict__`` works for regular classes, but can
cause problems when a class dynamicly looks up attributes in a
``__getattribute__`` method.

This new hook makes it possible to affect attribute lookup for both normal
attribute lookup and lookup through the `super class`_.


The superclass attribute lookup hook
====================================

Both ``super.__getattribute__`` and ``object.__getattribute__`` (or
`PyObject_GenericGetAttr`_ in C code) walk an object's MRO and peek in the
class' ``__dict__`` to look up attributes. A way to affect this lookup is
using a method on the meta class for the type, that by default looks up
the name in the class ``__dict__``.

In Python code
--------------

A meta type can define a method ``__locallookup__`` that is called during
attribute resolution by both ``super.__getattribute__`` and ``object.__getattribute``::

    class MetaType (type):

        def __locallookup__(cls, name):
            try:
                return cls.__dict__[name]
            except KeyError:
                raise AttributeError(name) from None

The example method above is pseudo code for the implementation of this method on
`type`_ (the actual implementation is in C, and doesn't suffer from the recursion
problem in this example).

The ``__locallookup__`` method has as its arguments a class and the name of the attribute
that is looked up. It should return the value of the attribute without invoking descriptors,
or raise `AttributeError`_ when the name cannot be found.


In C code
---------

A new slot ``tp_locallookup`` is added to the ``PyTypeObject`` struct, this slot
corresponds to the ``__locallookup__`` method on `type`_.

The slot has the following prototype::

    PyObject* (*locallookupfunc)(PyTypeObject* cls, PyObject* name);

This method should lookup *name* in the namespace of *cls*, without looking at superclasses,
and should not invoke descriptors. The method returns ``NULL`` without setting an exception
when the *name* cannot be found, and returns a new reference otherwise (not a borrowed reference).


Usage of this hook
------------------

The new method will be defined for `type`_, and will peek in the class dictionary::

    static PyObject*
    type_locallookup(PyTypeObject* cls, PyObject* name)
    {
        PyObject* res;
        if (!cls->tp_dict) {
	    return NULL;
	}

	res = PyDict_GetItem(cls, name);
	Py_XINCREF(res);
	return res;
    }

The new method will be used by both `PyObject_GenericGetAttr`_ and
``super.__getattribute__`` instead of peeking in a type's ``tp_dict``.


Alternative proposals
---------------------

``__getattribute_super__``
..........................

An earlier version of this PEP used the following static method on classes::

    def __getattribute_super__(cls, name, object, owner): pass

This method performed name lookup as well as invoking descriptors and was necessarily
limited to working only with ``super.__getattribute__``.


Reuse ``tp_getattro``
.....................

It would be nice to avoid adding a new slot, thus keeping the API simpler and
easier to understand.  A comment on `Issue 18181`_ asked about reusing the
``tp_getattro`` slot, that is super could call the ``tp_getattro`` slot of all
methods along the MRO.

That won't work because ``tp_getattro`` will look in the instance
``__dict__`` before it tries to resolve attributes using classes in the MRO.
This would mean that using ``tp_getattro`` instead of peeking the class
dictionaries changes the semantics of the `super class`_.


Open Issues
===========

* The names of the new slot and magic method are far from settled.

* Should the python method raise an exception or return a magic value (such as the
  `NotImplemented`_ return value used by comparison operators). The latter could be
  slightly faster because it doesn't have to overhead of setting up exception state, but
  makes it impossible to use that value as an attribute on a class.

* The proposed change to `PyObject_GenericGetAttr`_ will probably cause problems with the
  attribute lookup cache (MCACHE):

  1. That code stores borrowed references, which won't work when the hook is present. That
     is mostly fixable, but at the cost of possibly keeping garbage alive.

  2. Caching isn't an option when a hook might execute arbitrary code (and there hence is
     no reason to assume that the hooks return value won't change later on).

     The only workaround I could find for this is to make the hook a fallback (that is,
     more like ``__getattr__`` than ``__getattribute__``).


References
==========

* `Issue 18181`_ contains a prototype implementation


Copyright
=========

This document has been placed in the public domain.

.. _`Issue 18181`: http://bugs.python.org/issue18181

.. _`super class`: http://docs.python.org/3/library/functions.html#super

.. _`NotImplemented`: http://docs.python.org/3/library/constants.html#NotImplemented

.. _`PyObject_GenericGetAttr`: http://docs.python.org/3/c-api/object.html#PyObject_GenericGetAttr

.. _`type`: http://docs.python.org/3/library/functions.html#type

.. _`AttributeError`: http://docs.python.org/3/library/exceptions.html#AttributeError

From guido at python.org  Mon Jul 15 18:49:04 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 15 Jul 2013 09:49:04 -0700
Subject: [Python-Dev] PEP 447: hooking into super() attribute resolution
In-Reply-To: <2055CD40-2081-45DC-8B9A-889085FC7CEE@mac.com>
References: <2055CD40-2081-45DC-8B9A-889085FC7CEE@mac.com>
Message-ID: <CAP7+vJ+zaOHd2LR8jCyDXubmHRXTYy=gsBPJUouEr-3FwymR9w@mail.gmail.com>

On Mon, Jul 15, 2013 at 8:12 AM, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
> I've posted a new update of my proposal to add a way to override the attribute resolution proces by super(). I've rewritten the PEP and implementation based on feedback by Steve Dower.
>
> In the current edition of the proposal the hook is a method on the type, defined in a metaclass for types. A simple demo:
>
>     class meta (type):
>         def __locallookup__(cls, name):
>             return type.__locallookup__(cls, name.upper())
>
>     class S (metaclass=meta):
>         def m(self):
>             return 42
>
>         def M(self):
>              return "fortytwo"
>
>     class SS (S):
>         def m(self):
>             return 21
>
>         def M(self):
>            return "twentyone"
>
>     o = SS()
>     print("self", o.m())
>     print("super", super(SS, o).m())
>
> The last line currently prints "super 42" and would print "super fortytwo" with my proposal.

I would suggest adding that motivation to the PEP, since you clearly
thought it was necessary to get people to consider your proposal. :-)

> A major open issue: the __locallookup__ method could also be used for normal attribute resolution, but that probably causes issues with attribute caching (see the PEP for details).  I haven't worked out yet if it is worthwhile to tweak the proposal to fix the caching issues (even though the proposal currently says that PyObject_GenericGetAttr will use the new method). Fixing the caching issue definitly would help in my primary use case by reducing code duplication, but might end up making the API unnecessarily complex.

Hm. It looks like the functionality you actually want to hook into is
in _PyType_Lookup().

I think that in this case the PEP's acceptance will be highly
correlated with your ability to produce an actual patch that (a)
implements exactly the functionality you want (never mind whether it
matches the exact API currently proposed), and (b) doesn't slow down
classes that don't provide this hook.

Other than that, I think that it's a laudable attempt at generalizing,
and I hope you solve the implementation conundrum.

-- 
--Guido van Rossum (python.org/~guido)

From ronaldoussoren at mac.com  Mon Jul 15 18:56:56 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Mon, 15 Jul 2013 18:56:56 +0200
Subject: [Python-Dev] PEP 447: hooking into super() attribute resolution
In-Reply-To: <CAP7+vJ+zaOHd2LR8jCyDXubmHRXTYy=gsBPJUouEr-3FwymR9w@mail.gmail.com>
References: <2055CD40-2081-45DC-8B9A-889085FC7CEE@mac.com>
	<CAP7+vJ+zaOHd2LR8jCyDXubmHRXTYy=gsBPJUouEr-3FwymR9w@mail.gmail.com>
Message-ID: <23580FF6-CE74-4461-85BE-4333A894B552@mac.com>


On 15 Jul, 2013, at 18:49, Guido van Rossum <guido at python.org> wrote:
> 
> 
>> A major open issue: the __locallookup__ method could also be used for normal attribute resolution, but that probably causes issues with attribute caching (see the PEP for details).  I haven't worked out yet if it is worthwhile to tweak the proposal to fix the caching issues (even though the proposal currently says that PyObject_GenericGetAttr will use the new method). Fixing the caching issue definitly would help in my primary use case by reducing code duplication, but might end up making the API unnecessarily complex.
> 
> Hm. It looks like the functionality you actually want to hook into is
> in _PyType_Lookup().

That's right. I didn't want to get too technical, but forgot to consider
who are reading this :-)

> 
> I think that in this case the PEP's acceptance will be highly
> correlated with your ability to produce an actual patch that (a)
> implements exactly the functionality you want (never mind whether it
> matches the exact API currently proposed), and (b) doesn't slow down
> classes that don't provide this hook.

I'd only reall need the functional change to super(), but I am working on
a patch that also changes _PyType_Lookup. I think I can avoid slowdown
by making the tp_locallookup optional and only "punishing" those classes
that use the new slot.  A minor complication is that I'll have to change
the interface of _PyType_Lookup, it currently returns a borrowed reference
and will return a new reference. That's just careful bookkeeping though.

> 
> Other than that, I think that it's a laudable attempt at generalizing,
> and I hope you solve the implementation conundrum.

I was pleasantly surprised in how the changed API was cleaner and applicable
to _PyType_Lookup as well. I guess that means I'm on the right path.

Ronald


From zachary.ware at gmail.com  Mon Jul 15 18:43:46 2013
From: zachary.ware at gmail.com (Zachary Ware)
Date: Mon, 15 Jul 2013 11:43:46 -0500
Subject: [Python-Dev] [Python-checkins] cpython: Issue #18393: Remove
 use of deprecated API on OSX
In-Reply-To: <3bv9J80Cjxz7LjQ@mail.python.org>
References: <3bv9J80Cjxz7LjQ@mail.python.org>
Message-ID: <CAKJDb-N4QOZDEYdUG5j34Z+S82+z-Ux+72T6BZ6Jc7w0eEOdNg@mail.gmail.com>

On Mon, Jul 15, 2013 at 11:32 AM, ronald.oussoren
<python-checkins at python.org> wrote:
> http://hg.python.org/cpython/rev/ccbaf6762b54
> changeset:   84634:ccbaf6762b54
> user:        Ronald Oussoren <ronaldoussoren at mac.com>
> date:        Mon Jul 15 18:32:09 2013 +0200
> summary:
>   Issue #18393: Remove use of deprecated API on OSX
>
> The "Gestalt" function on OSX is deprecated (starting with OSX 10.8),
> remove its usage from the stdlib. The patch removes a number of private

I believe this means that Lib/test/leakers/test_gestalt.py can be
removed as well.

> functions and a private module, but does not change the public API.
>
> The removed code was effectively dead, the platform module has used
> other code to fetch the OSX version for years and could only use
> on the Gestalt-based code as a fallback. That fallback can only trigger
> on broken OSX installs (that is, someone has removed parts of the system
> install)
>
> files:
>   Lib/platform.py    |  64 +-------------------------
>   Misc/NEWS          |   4 +
>   Modules/_gestalt.c |  84 ----------------------------------
>   setup.py           |   4 -
>   4 files changed, 5 insertions(+), 151 deletions(-)
>
>
> <snip diffs>
>
> --
> Repository URL: http://hg.python.org/cpython
>

--
Zach

From guido at python.org  Mon Jul 15 19:14:10 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 15 Jul 2013 10:14:10 -0700
Subject: [Python-Dev] PEP 447: hooking into super() attribute resolution
In-Reply-To: <23580FF6-CE74-4461-85BE-4333A894B552@mac.com>
References: <2055CD40-2081-45DC-8B9A-889085FC7CEE@mac.com>
	<CAP7+vJ+zaOHd2LR8jCyDXubmHRXTYy=gsBPJUouEr-3FwymR9w@mail.gmail.com>
	<23580FF6-CE74-4461-85BE-4333A894B552@mac.com>
Message-ID: <CAP7+vJJ7n6Vfm3RrPYSsW2-a_xH3RuOQPZtiHGfk+uZPXq0rrw@mail.gmail.com>

On Mon, Jul 15, 2013 at 9:56 AM, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
>
> On 15 Jul, 2013, at 18:49, Guido van Rossum <guido at python.org> wrote:
>>
>>
>>> A major open issue: the __locallookup__ method could also be used for normal attribute resolution, but that probably causes issues with attribute caching (see the PEP for details).  I haven't worked out yet if it is worthwhile to tweak the proposal to fix the caching issues (even though the proposal currently says that PyObject_GenericGetAttr will use the new method). Fixing the caching issue definitly would help in my primary use case by reducing code duplication, but might end up making the API unnecessarily complex.
>>
>> Hm. It looks like the functionality you actually want to hook into is
>> in _PyType_Lookup().
>
> That's right. I didn't want to get too technical, but forgot to consider
> who are reading this :-)

Heh. :-)

>> I think that in this case the PEP's acceptance will be highly
>> correlated with your ability to produce an actual patch that (a)
>> implements exactly the functionality you want (never mind whether it
>> matches the exact API currently proposed), and (b) doesn't slow down
>> classes that don't provide this hook.
>
> I'd only reall need the functional change to super(), but I am working on
> a patch that also changes _PyType_Lookup. I think I can avoid slowdown
> by making the tp_locallookup optional and only "punishing" those classes
> that use the new slot.  A minor complication is that I'll have to change
> the interface of _PyType_Lookup, it currently returns a borrowed reference
> and will return a new reference. That's just careful bookkeeping though.

Hm. That sounds scary. I'd rename it, because this is actually an
important (and scary) signature change, that type checkers otherwise
won't see. Don't trust that you can find all call sites (there may be
some in 3rd party code even though it starts with _). You don't want
to suddenly start causing leaks all over the place.

FWIW I agree that the change should affect _PyType_Lookup, because it
would be very surprising if super() used a different override than
regular attribute lookup.

>> Other than that, I think that it's a laudable attempt at generalizing,
>> and I hope you solve the implementation conundrum.
>
> I was pleasantly surprised in how the changed API was cleaner and applicable
> to _PyType_Lookup as well. I guess that means I'm on the right path.

There's a Zen of Python line about that. :-)

-- 
--Guido van Rossum (python.org/~guido)

From barry at python.org  Mon Jul 15 21:51:31 2013
From: barry at python.org (Barry Warsaw)
Date: Mon, 15 Jul 2013 15:51:31 -0400
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
 underscores
In-Reply-To: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
Message-ID: <20130715155131.43c7a519@anarchist>

On Jul 14, 2013, at 06:11 PM, Nick Coghlan wrote:

>Private interfaces

PEP 8 does say:

    _single_leading_underscore: weak "internal use" indicator. E.g. from M
    import * does not import objects whose name starts with an underscore.

I'm in favor of making this a stronger recommendation, but I have a small
semantic quibble.  Instead of "private interface" it should be "non-public
interface".  The two aren't quite the same thing; for example, often
single-leading underscores are used for methods that subclasses are supposed
to override, e.g. akin to "protected" in C++.

Besides, Python doesn't really have any strong notion of privateness, so
saying "non-public" means "just because you can, doesn't mean you should".

-Barry

From tjreedy at udel.edu  Mon Jul 15 21:54:59 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 15 Jul 2013 15:54:59 -0400
Subject: [Python-Dev] Refactoring test.support into a subpackage
In-Reply-To: <CADiSq7f3nxGaEk2nUztGOHim4sk6oG9ZyFT1maOxnX7H5Y=0ig@mail.gmail.com>
References: <CADiSq7f3nxGaEk2nUztGOHim4sk6oG9ZyFT1maOxnX7H5Y=0ig@mail.gmail.com>
Message-ID: <ks1k2g$g25$1@ger.gmane.org>

On 7/15/2013 6:22 AM, Nick Coghlan wrote:
> At the PyCon AU sprints, some of the sprinters worked on a plan Chris
> Jerdonek and I came up with months ago to convert test.support from a
> module into a subpackage.
>
> This plan arose from some nasty test suite hacks in the lead up to the
> release of Python 3.3, where some of the pkgutil tests ended up in
> test.test_runpy so they could use the package creation infrastructure
> I have in there without needing to extract a helper module with a
> release imminent.
>
> The scope of the plan covers a few distinct steps:
>
> 1. Move Lib/test/support.py to Lib/test/support/__init__.py
> (http://bugs.python.org/issue15494)
> 2. Add a new (documented) test.support.pkg_helper submodule with the
> package creation support code (http://bugs.python.org/issue15403 and
> http://bugs.python.org/issue15376)
> 3. Move the pkgutil tests to test.test_pkgutil where they belong
> (http://bugs.python.org/issue15358)
> 4. Move the existing test.script_helper and test.bytecode_helper
> support modules into the test.support subpackage and document them (no
> issue for that as yet)
>
> These changes would be made on both 3.3 and default, to avoid future
> merge conflicts (note that Indra also posted an updated patch for one
> pending test suite update to cope with the relocation:
> http://bugs.python.org/issue15415)
>
> My main motivation for doing this is to give a bit more visibility to
> the helpers that already exist, as well as making them easier to find

Bundling together all the support code I should know about sounds good 
to me. I only became aware of script_helper, recently, when I saw it 
imported in a test_xxx file. I still have no idea what it does.

> when we want to look into the details of what a test is doing. At the
> moment, they tend to get lost in the sea of actual test files in the
> main test directory.

-- 
Terry Jan Reedy


From solipsis at pitrou.net  Mon Jul 15 21:56:45 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 15 Jul 2013 21:56:45 +0200
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<20130715155131.43c7a519@anarchist>
Message-ID: <20130715215645.421c2eed@fsol>

On Mon, 15 Jul 2013 15:51:31 -0400
Barry Warsaw <barry at python.org> wrote:
> On Jul 14, 2013, at 06:11 PM, Nick Coghlan wrote:
> 
> >Private interfaces
> 
> PEP 8 does say:
> 
>     _single_leading_underscore: weak "internal use" indicator. E.g. from M
>     import * does not import objects whose name starts with an underscore.
> 
> I'm in favor of making this a stronger recommendation, but I have a small
> semantic quibble.  Instead of "private interface" it should be "non-public
> interface".  The two aren't quite the same thing; for example, often
> single-leading underscores are used for methods that subclasses are supposed
> to override, e.g. akin to "protected" in C++.

C++ doesn't have a monopoly over the definition of "private".

> Besides, Python doesn't really have any strong notion of privateness, so
> saying "non-public" means "just because you can, doesn't mean you should".

If it doesn't have any notion of privateness, then it can't have a
notion of "publicness" either.
If you really want another word (I am personally fine with "private"),
"internal" it should be.

Regards

Antoine.



From barry at python.org  Mon Jul 15 22:10:45 2013
From: barry at python.org (Barry Warsaw)
Date: Mon, 15 Jul 2013 16:10:45 -0400
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
 underscores
In-Reply-To: <20130715215645.421c2eed@fsol>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<20130715155131.43c7a519@anarchist> <20130715215645.421c2eed@fsol>
Message-ID: <20130715161045.7a0c6bcd@anarchist>

On Jul 15, 2013, at 09:56 PM, Antoine Pitrou wrote:

>If you really want another word (I am personally fine with "private"),
>"internal" it should be.

I would be fine with "internal", since that's how PEP 8 already classifies
such names. :)

-Barry

From ncoghlan at gmail.com  Tue Jul 16 00:28:36 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 16 Jul 2013 08:28:36 +1000
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <20130715161045.7a0c6bcd@anarchist>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<20130715155131.43c7a519@anarchist> <20130715215645.421c2eed@fsol>
	<20130715161045.7a0c6bcd@anarchist>
Message-ID: <CADiSq7eeLexw=4ZSWTnWdKbs2c95KhUS1rpwY+-DVkJJFHCLmg@mail.gmail.com>

On 16 Jul 2013 06:13, "Barry Warsaw" <barry at python.org> wrote:
>
> On Jul 15, 2013, at 09:56 PM, Antoine Pitrou wrote:
>
> >If you really want another word (I am personally fine with "private"),
> >"internal" it should be.
>
> I would be fine with "internal", since that's how PEP 8 already classifies
> such names. :)

Yeah, internal is a better word for this.

Cheers,
Nick.

>
> -Barry
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130716/52e9113d/attachment.html>

From barry at python.org  Tue Jul 16 00:40:39 2013
From: barry at python.org (Barry Warsaw)
Date: Mon, 15 Jul 2013 18:40:39 -0400
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
 underscores
In-Reply-To: <CADiSq7fAS+-keOjBmOEs-UvV=bV4qB9=ikiN5dkvRW7LSrge6g@mail.gmail.com>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
	<ks08fk$vnu$1@ger.gmane.org>
	<CADiSq7fAS+-keOjBmOEs-UvV=bV4qB9=ikiN5dkvRW7LSrge6g@mail.gmail.com>
Message-ID: <20130715184039.60590e62@anarchist>

Working from what I think is the latest version.

In general, i'd rather be prescriptive of future conventions than descriptive
of current conventions.  It's okay to exempt existing code, and as a general
rule we've never been fond of rewriting existing code to update it to new
standards or APIs.  We don't need to do so here either.

On Jul 15, 2013, at 05:48 PM, Nick Coghlan wrote:

>Private interfaces

"Internal" or "Non-public"

>Unless explicitly documented otherwise, a leading underscore on any
>name indicates that it is an internal implementation detail and any
>backwards compatibility guarantees do not apply. It is strongly
>encouraged that private APIs (whether modules, classes, functions,
>attributes or other names) be clearly marked in this way, as Python
>users may rely on introspection to identify available functionality
>and may be misled into believing an API without a leading underscore
>is in fact a public API with the standard backwards compatibility
>guarantees.

How about:

"All internal interfaces (modules, classes, functions, attributes or other
names) should be prefixed with a single leading underscore.  Such names are
internal implementation details for which no backward compatibility guarantees
are made, unless otherwise specified.

Existing code and other narrowly accepted exceptions may override this
recommendation, in which case the docstrings and/or documentation for such
code must clearly and explicitly state the internal status of the APIs.

Imported names should always be considered an implementation detail.  Other
modules must not rely on indirect access to such imported names unless they
are an explicitly documented part of the containing module's API, such as
``os.path`` or a package's ``__init__`` module that exposes functionality from
submodules.  Public names exported by a module should be include in the
module's ``__all__`` attribute."

>While the explicit use of a leading underscore is the preferred solution,
>the names of some private (or partially private) modules (such as ``test``
>and ``idlelib``) lack the leading underscore either for historical reasons
>or because they expose a public command line interface through the
>``-m`` switch. Such modules should include an explicit disclaimer in
>their module docstring, indicating that they do not use the leading
>underscore convention and noting where the definition of the public
>API (if any) can be found (the public API definition may also be part
>of the module docstring).
>
>As a general principle, the specific modules and external APIs imported by
>a module are always considered an implementation detail. Other modules
>should not rely on indirect access to such imported interfaces unless they
>are an explicitly documented part of the containing module's API (such
>as ``os.path`` or a package ``__init__`` module that exposes functionality
>from submodules).

Cheers,
-Barry

From guido at python.org  Tue Jul 16 01:14:53 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 15 Jul 2013 16:14:53 -0700
Subject: [Python-Dev] Misc re.match() complaint
Message-ID: <CAP7+vJKdLzJ3BOgD9e=Nw8KT0v4apdgutu7nKLPAgtEGCapA5A@mail.gmail.com>

In a discussion about mypy I discovered that the Python 3 version of
the re module's Match object behaves subtly different from the Python
2 version when the target string (i.e. the haystack, not the needle)
is a buffer object.

In Python 2, the type of the return value of group() is always either
a Unicode string or an 8-bit string, and the type is determined by
looking at the target string -- if the target is unicode, group()
returns a unicode string, otherwise, group() returns an 8-bit string.
In particular, if the target is a buffer object, group() returns an
8-bit string. I think this is the appropriate behavior: otherwise
using regular expression matching to extract a small substring from a
large target string would unnecessarily keep the large target string
alive as long as the substring is alive.

But in Python 3, the behavior of group() has changed so that its
return type always matches that of the target string. I think this is
bad -- apart from the lifetime concern, it means that if your target
happens to be a bytearray, the return value isn't even hashable!

Does anyone remember whether this was a conscious decision? Is it too
late to fix?

--
--Guido van Rossum (python.org/~guido)

From greg at krypto.org  Tue Jul 16 01:30:17 2013
From: greg at krypto.org (Gregory P. Smith)
Date: Mon, 15 Jul 2013 16:30:17 -0700
Subject: [Python-Dev] Misc re.match() complaint
In-Reply-To: <CAP7+vJKdLzJ3BOgD9e=Nw8KT0v4apdgutu7nKLPAgtEGCapA5A@mail.gmail.com>
References: <CAP7+vJKdLzJ3BOgD9e=Nw8KT0v4apdgutu7nKLPAgtEGCapA5A@mail.gmail.com>
Message-ID: <CAGE7PNJs5qhUiWhv46HBzwpPHsk6Mt_N3+h3XSL2CoNOg7KrYw@mail.gmail.com>

On Mon, Jul 15, 2013 at 4:14 PM, Guido van Rossum <guido at python.org> wrote:

> In a discussion about mypy I discovered that the Python 3 version of
> the re module's Match object behaves subtly different from the Python
> 2 version when the target string (i.e. the haystack, not the needle)
> is a buffer object.
>
> In Python 2, the type of the return value of group() is always either
> a Unicode string or an 8-bit string, and the type is determined by
> looking at the target string -- if the target is unicode, group()
> returns a unicode string, otherwise, group() returns an 8-bit string.
> In particular, if the target is a buffer object, group() returns an
> 8-bit string. I think this is the appropriate behavior: otherwise
> using regular expression matching to extract a small substring from a
> large target string would unnecessarily keep the large target string
> alive as long as the substring is alive.
>
> But in Python 3, the behavior of group() has changed so that its
> return type always matches that of the target string. I think this is
> bad -- apart from the lifetime concern, it means that if your target
> happens to be a bytearray, the return value isn't even hashable!
>
> Does anyone remember whether this was a conscious decision? Is it too
> late to fix?


Hmm, that is not what I'd expect either. I would never expect it to return
a bytearray; I'd normally assume that .group() returned a bytes object if
the input was binary data and a str object if the input was unicode data
(str) regardless of specific types containing the input target data.

I'm going to hazard a guess that not much, if anything, would be depending
on getting a bytearray out of that. Fix this in 3.4? 3.3 and earlier users
are stuck with an extra bytes() call and data copy in these cases I guess.

-gps
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130715/1019a7a8/attachment.html>

From guido at python.org  Tue Jul 16 01:37:27 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 15 Jul 2013 16:37:27 -0700
Subject: [Python-Dev] Misc re.match() complaint
In-Reply-To: <CAGE7PNJs5qhUiWhv46HBzwpPHsk6Mt_N3+h3XSL2CoNOg7KrYw@mail.gmail.com>
References: <CAP7+vJKdLzJ3BOgD9e=Nw8KT0v4apdgutu7nKLPAgtEGCapA5A@mail.gmail.com>
	<CAGE7PNJs5qhUiWhv46HBzwpPHsk6Mt_N3+h3XSL2CoNOg7KrYw@mail.gmail.com>
Message-ID: <CAP7+vJLy7Jb_MB9yp8FTYd-QYGJPA7yu5Eoi-N8ZXENsbgoj6w@mail.gmail.com>

Ok, created http://bugs.python.org/issue18468.

On Mon, Jul 15, 2013 at 4:30 PM, Gregory P. Smith <greg at krypto.org> wrote:
>
> On Mon, Jul 15, 2013 at 4:14 PM, Guido van Rossum <guido at python.org> wrote:
>>
>> In a discussion about mypy I discovered that the Python 3 version of
>> the re module's Match object behaves subtly different from the Python
>> 2 version when the target string (i.e. the haystack, not the needle)
>> is a buffer object.
>>
>> In Python 2, the type of the return value of group() is always either
>> a Unicode string or an 8-bit string, and the type is determined by
>> looking at the target string -- if the target is unicode, group()
>> returns a unicode string, otherwise, group() returns an 8-bit string.
>> In particular, if the target is a buffer object, group() returns an
>> 8-bit string. I think this is the appropriate behavior: otherwise
>> using regular expression matching to extract a small substring from a
>> large target string would unnecessarily keep the large target string
>> alive as long as the substring is alive.
>>
>> But in Python 3, the behavior of group() has changed so that its
>> return type always matches that of the target string. I think this is
>> bad -- apart from the lifetime concern, it means that if your target
>> happens to be a bytearray, the return value isn't even hashable!
>>
>> Does anyone remember whether this was a conscious decision? Is it too
>> late to fix?
>
>
> Hmm, that is not what I'd expect either. I would never expect it to return a
> bytearray; I'd normally assume that .group() returned a bytes object if the
> input was binary data and a str object if the input was unicode data (str)
> regardless of specific types containing the input target data.
>
> I'm going to hazard a guess that not much, if anything, would be depending
> on getting a bytearray out of that. Fix this in 3.4? 3.3 and earlier users
> are stuck with an extra bytes() call and data copy in these cases I guess.
>
> -gps
>



-- 
--Guido van Rossum (python.org/~guido)

From python at mrabarnett.plus.com  Tue Jul 16 02:10:51 2013
From: python at mrabarnett.plus.com (MRAB)
Date: Tue, 16 Jul 2013 01:10:51 +0100
Subject: [Python-Dev] Misc re.match() complaint
In-Reply-To: <CAGE7PNJs5qhUiWhv46HBzwpPHsk6Mt_N3+h3XSL2CoNOg7KrYw@mail.gmail.com>
References: <CAP7+vJKdLzJ3BOgD9e=Nw8KT0v4apdgutu7nKLPAgtEGCapA5A@mail.gmail.com>
	<CAGE7PNJs5qhUiWhv46HBzwpPHsk6Mt_N3+h3XSL2CoNOg7KrYw@mail.gmail.com>
Message-ID: <51E48F8B.1090905@mrabarnett.plus.com>

On 16/07/2013 00:30, Gregory P. Smith wrote:
>
> On Mon, Jul 15, 2013 at 4:14 PM, Guido van Rossum <guido at python.org
> <mailto:guido at python.org>> wrote:
>
>     In a discussion about mypy I discovered that the Python 3 version of
>     the re module's Match object behaves subtly different from the Python
>     2 version when the target string (i.e. the haystack, not the needle)
>     is a buffer object.
>
>     In Python 2, the type of the return value of group() is always either
>     a Unicode string or an 8-bit string, and the type is determined by
>     looking at the target string -- if the target is unicode, group()
>     returns a unicode string, otherwise, group() returns an 8-bit string.
>     In particular, if the target is a buffer object, group() returns an
>     8-bit string. I think this is the appropriate behavior: otherwise
>     using regular expression matching to extract a small substring from a
>     large target string would unnecessarily keep the large target string
>     alive as long as the substring is alive.
>
>     But in Python 3, the behavior of group() has changed so that its
>     return type always matches that of the target string. I think this is
>     bad -- apart from the lifetime concern, it means that if your target
>     happens to be a bytearray, the return value isn't even hashable!
>
>     Does anyone remember whether this was a conscious decision? Is it too
>     late to fix?
>
>
> Hmm, that is not what I'd expect either. I would never expect it to
> return a bytearray; I'd normally assume that .group() returned a bytes
> object if the input was binary data and a str object if the input was
> unicode data (str) regardless of specific types containing the input
> target data.
>
> I'm going to hazard a guess that not much, if anything, would be
> depending on getting a bytearray out of that. Fix this in 3.4? 3.3 and
> earlier users are stuck with an extra bytes() call and data copy in
> these cases I guess.
>
I'm not sure I understand the complaint.

I get this for Python 2.7:

Python 2.7.5 (default, May 15 2013, 22:43:36) [MSC v.1500 32 bit 
(Intel)] on win
32
Type "help", "copyright", "credits" or "license" for more information.
 >>> import array
 >>> import re
 >>> re.match(r"a", array.array("b", "a")).group()
array('b', [97])

It's the same even in Python 2.4.


From ncoghlan at gmail.com  Tue Jul 16 02:10:40 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 16 Jul 2013 10:10:40 +1000
Subject: [Python-Dev] Misc re.match() complaint
In-Reply-To: <CAP7+vJKdLzJ3BOgD9e=Nw8KT0v4apdgutu7nKLPAgtEGCapA5A@mail.gmail.com>
References: <CAP7+vJKdLzJ3BOgD9e=Nw8KT0v4apdgutu7nKLPAgtEGCapA5A@mail.gmail.com>
Message-ID: <CADiSq7c25ZgmTcSy4V07VsYqPXUHxHmPuf0g=XtU2+vN16oZVw@mail.gmail.com>

On 16 Jul 2013 09:17, "Guido van Rossum" <guido at python.org> wrote:
>
> Does anyone remember whether this was a conscious decision?

I doubt it was a conscious decision - an unfortunate amount of the standard
library's handling of the text model change falls into the category of
"implementation accident" :(

> Is it too
> late to fix?

Like Greg, I'm comfortable with the idea of calling "bug" on this one,
fixing it in 3.4 and making a note in the "Porting to Python 3.4" section
of the What's New guide.

Cheers,
Nick.

>
> --
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130716/95c3cc26/attachment.html>

From guido at python.org  Tue Jul 16 02:25:11 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 15 Jul 2013 17:25:11 -0700
Subject: [Python-Dev] Misc re.match() complaint
In-Reply-To: <51E48F8B.1090905@mrabarnett.plus.com>
References: <CAP7+vJKdLzJ3BOgD9e=Nw8KT0v4apdgutu7nKLPAgtEGCapA5A@mail.gmail.com>
	<CAGE7PNJs5qhUiWhv46HBzwpPHsk6Mt_N3+h3XSL2CoNOg7KrYw@mail.gmail.com>
	<51E48F8B.1090905@mrabarnett.plus.com>
Message-ID: <CAP7+vJKRuKL6vsPRmvo-+9gPtB=usp57F31_1H--sVN0_qi6GQ@mail.gmail.com>

On Mon, Jul 15, 2013 at 5:10 PM, MRAB <python at mrabarnett.plus.com> wrote:
> On 16/07/2013 00:30, Gregory P. Smith wrote:
>>
>>
>> On Mon, Jul 15, 2013 at 4:14 PM, Guido van Rossum <guido at python.org
>> <mailto:guido at python.org>> wrote:
>>
>>     In a discussion about mypy I discovered that the Python 3 version of
>>     the re module's Match object behaves subtly different from the Python
>>     2 version when the target string (i.e. the haystack, not the needle)
>>     is a buffer object.
>>
>>     In Python 2, the type of the return value of group() is always either
>>     a Unicode string or an 8-bit string, and the type is determined by
>>     looking at the target string -- if the target is unicode, group()
>>     returns a unicode string, otherwise, group() returns an 8-bit string.
>>     In particular, if the target is a buffer object, group() returns an
>>     8-bit string. I think this is the appropriate behavior: otherwise
>>     using regular expression matching to extract a small substring from a
>>     large target string would unnecessarily keep the large target string
>>     alive as long as the substring is alive.
>>
>>     But in Python 3, the behavior of group() has changed so that its
>>     return type always matches that of the target string. I think this is
>>     bad -- apart from the lifetime concern, it means that if your target
>>     happens to be a bytearray, the return value isn't even hashable!
>>
>>     Does anyone remember whether this was a conscious decision? Is it too
>>     late to fix?
>>
>>
>> Hmm, that is not what I'd expect either. I would never expect it to
>> return a bytearray; I'd normally assume that .group() returned a bytes
>> object if the input was binary data and a str object if the input was
>> unicode data (str) regardless of specific types containing the input
>> target data.
>>
>> I'm going to hazard a guess that not much, if anything, would be
>> depending on getting a bytearray out of that. Fix this in 3.4? 3.3 and
>> earlier users are stuck with an extra bytes() call and data copy in
>> these cases I guess.
>>
> I'm not sure I understand the complaint.
>
> I get this for Python 2.7:
>
> Python 2.7.5 (default, May 15 2013, 22:43:36) [MSC v.1500 32 bit (Intel)] on
> win
> 32
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import array
>>>> import re
>>>> re.match(r"a", array.array("b", "a")).group()
> array('b', [97])
>
> It's the same even in Python 2.4.

Ah, but now try it with buffer():

>> re.search('yz+', buffer('xyzzy')).group()
'yzz'
>>>

The equivalent in Python 3 (using memoryview) returns a memoryview:

>>> re.search(b'yz+', memoryview(b'xyzzy')).group()
<memory at 0x10d03a688>
>>>

And I still think that any return type for group() except bytes or str
is wrong. (Except possibly a subclass of these.)

-- 
--Guido van Rossum (python.org/~guido)

From victor.stinner at gmail.com  Tue Jul 16 02:34:49 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 16 Jul 2013 02:34:49 +0200
Subject: [Python-Dev] Is it safe to call PyEval_EvalFrameEx() with an
	exception set?
Message-ID: <CAMpsgwbsDHwGhyAmaxwzhPzXtW8cQ6ifxHD2RSe2Swrpws7APA@mail.gmail.com>

Hi,

I'm working on the issue #18408 (fix issues found by my pyfailmalloc
tool). To analyze some bugs, I have to add debug traces in various
functions to find which function returned NULL without setting an
error, or the opposite: returned a valid object, but with an exception
set.

I would like to add assertions in Python/ceval.c to detect such bugs
earlier. The problem is that some functions rely on the ability to
call PyEval_EvalFrameEx() with an exception set. Is it expected?
Should it be avoided? The current exception can be replaced with a new
exception.


Example 1:

    module = PyImport_Import(module_name);
    if (module == NULL) {
        PyErr_Format(PicklingError,
                     "Can't pickle %R: import of module %R failed",
                     obj, module_name);
        goto error;
    }

Formatting the obj argument calls indirectly PyEval_EvalFrameEx(),
whereas an ImportError was raised. The original ImportError is
replaced with a new PickleError exception.

To not call PyEval_EvalFrameEx() with an exception set, PyErr_Format()
can call PyErr_Clear() before calling PyUnicode_FromFormatV(), maybe
only in debug mode.


Example 2:

test_sqlite uses sqlite.connection.create_aggregate() to create a
Python callback called by the C sqlite3 library. The "callback" is a
class with 3 methods: constructor, step() and finalize(). The test
AggregateTests.CheckAggrNoStep() uses a class with no step() method:
it ensures that an AttributeError is raised.

The problem is that the finalize() method is called even if the class
has no step method, and so there is a current AttributeError
exception.

I don't know if finalize() should be called in this case. It may be a
bug in the sqlite module. More generally,


Victor

From chrism at plope.com  Tue Jul 16 02:23:13 2013
From: chrism at plope.com (Chris McDonough)
Date: Mon, 15 Jul 2013 20:23:13 -0400
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
 underscores
In-Reply-To: <20130715184039.60590e62@anarchist>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
	<ks08fk$vnu$1@ger.gmane.org>
	<CADiSq7fAS+-keOjBmOEs-UvV=bV4qB9=ikiN5dkvRW7LSrge6g@mail.gmail.com>
	<20130715184039.60590e62@anarchist>
Message-ID: <1373934193.30382.59.camel@thinko>

On Mon, 2013-07-15 at 18:40 -0400, Barry Warsaw wrote:
> Working from what I think is the latest version.
> 
> In general, i'd rather be prescriptive of future conventions than descriptive
> of current conventions.  It's okay to exempt existing code, and as a general
> rule we've never been fond of rewriting existing code to update it to new
> standards or APIs.  We don't need to do so here either.

FWIW, I'm very skeptical of a PEP 8 guideline that would try to
proscribe that the "non-internal" API of a module or class would be
defined solely by a naming convention.

If what's being described here does become a rule, there is reason to
believe that future users who treat this PEP as the word-of-god (and
there are a *lot* of them; I hear from people literally every week who
want to "PEP8-ify" my code in some limited-value-added way) will be
harmed.  They'll be living in a fantasy world where every
non-underscore-prefixed thing is now a defacto API.  But I have lived in
a world where that has not been the case since 1998, and the chance that
I'll go back and change all my public code to satisfy a questionable
introspection convention is pretty slim.

- C


> 
> On Jul 15, 2013, at 05:48 PM, Nick Coghlan wrote:
> 
> >Private interfaces
> 
> "Internal" or "Non-public"
> 
> >Unless explicitly documented otherwise, a leading underscore on any
> >name indicates that it is an internal implementation detail and any
> >backwards compatibility guarantees do not apply. It is strongly
> >encouraged that private APIs (whether modules, classes, functions,
> >attributes or other names) be clearly marked in this way, as Python
> >users may rely on introspection to identify available functionality
> >and may be misled into believing an API without a leading underscore
> >is in fact a public API with the standard backwards compatibility
> >guarantees.
> 
> How about:
> 
> "All internal interfaces (modules, classes, functions, attributes or other
> names) should be prefixed with a single leading underscore.  Such names are
> internal implementation details for which no backward compatibility guarantees
> are made, unless otherwise specified.
> 
> Existing code and other narrowly accepted exceptions may override this
> recommendation, in which case the docstrings and/or documentation for such
> code must clearly and explicitly state the internal status of the APIs.
> 
> Imported names should always be considered an implementation detail.  Other
> modules must not rely on indirect access to such imported names unless they
> are an explicitly documented part of the containing module's API, such as
> ``os.path`` or a package's ``__init__`` module that exposes functionality from
> submodules.  Public names exported by a module should be include in the
> module's ``__all__`` attribute."
> 
> >While the explicit use of a leading underscore is the preferred solution,
> >the names of some private (or partially private) modules (such as ``test``
> >and ``idlelib``) lack the leading underscore either for historical reasons
> >or because they expose a public command line interface through the
> >``-m`` switch. Such modules should include an explicit disclaimer in
> >their module docstring, indicating that they do not use the leading
> >underscore convention and noting where the definition of the public
> >API (if any) can be found (the public API definition may also be part
> >of the module docstring).
> >
> >As a general principle, the specific modules and external APIs imported by
> >a module are always considered an implementation detail. Other modules
> >should not rely on indirect access to such imported interfaces unless they
> >are an explicitly documented part of the containing module's API (such
> >as ``os.path`` or a package ``__init__`` module that exposes functionality
> >from submodules).
> 
> Cheers,
> -Barry
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40plope.com



From steve at pearwood.info  Tue Jul 16 03:25:03 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 16 Jul 2013 11:25:03 +1000
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <1373934193.30382.59.camel@thinko>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
	<ks08fk$vnu$1@ger.gmane.org>
	<CADiSq7fAS+-keOjBmOEs-UvV=bV4qB9=ikiN5dkvRW7LSrge6g@mail.gmail.com>
	<20130715184039.60590e62@anarchist>
	<1373934193.30382.59.camel@thinko>
Message-ID: <51E4A0EF.8070205@pearwood.info>

On 16/07/13 10:23, Chris McDonough wrote:

> If what's being described here does become a rule, there is reason to
> believe that future users who treat this PEP as the word-of-god (and
> there are a *lot* of them; I hear from people literally every week who
> want to "PEP8-ify" my code in some limited-value-added way) will be
> harmed.

I sympathise with your pain, but is that not true of every PEP 8 naming convention?


> They'll be living in a fantasy world where every
> non-underscore-prefixed thing is now a defacto API.

If your code has no obvious, documented convention at all for what's internal and what is not, they are no worse off.

If you do have a documented convention for internal implementation details, then you are no worse off. "I have better things to do than PEP8-ify old, working, stable code" is a perfectly acceptable answer. "I have better things to do than PEP9-ify old, working, stable code, but if you want to provide regression tests and a working patch, I'll let you do so" might be an even better one :-)


-- 
Steven

From ncoghlan at gmail.com  Tue Jul 16 03:30:40 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 16 Jul 2013 11:30:40 +1000
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <1373934193.30382.59.camel@thinko>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
	<ks08fk$vnu$1@ger.gmane.org>
	<CADiSq7fAS+-keOjBmOEs-UvV=bV4qB9=ikiN5dkvRW7LSrge6g@mail.gmail.com>
	<20130715184039.60590e62@anarchist>
	<1373934193.30382.59.camel@thinko>
Message-ID: <CADiSq7e2NGhuY=9+TOHy4RWVVXgKWxbgk3frES2coJHrG9WT5Q@mail.gmail.com>

On 16 July 2013 10:23, Chris McDonough <chrism at plope.com> wrote:
> On Mon, 2013-07-15 at 18:40 -0400, Barry Warsaw wrote:
>> Working from what I think is the latest version.
>>
>> In general, i'd rather be prescriptive of future conventions than descriptive
>> of current conventions.  It's okay to exempt existing code, and as a general
>> rule we've never been fond of rewriting existing code to update it to new
>> standards or APIs.  We don't need to do so here either.
>
> FWIW, I'm very skeptical of a PEP 8 guideline that would try to
> proscribe that the "non-internal" API of a module or class would be
> defined solely by a naming convention.
>
> If what's being described here does become a rule, there is reason to
> believe that future users who treat this PEP as the word-of-god (and
> there are a *lot* of them; I hear from people literally every week who
> want to "PEP8-ify" my code in some limited-value-added way) will be
> harmed.  They'll be living in a fantasy world where every
> non-underscore-prefixed thing is now a defacto API.  But I have lived in
> a world where that has not been the case since 1998, and the chance that
> I'll go back and change all my public code to satisfy a questionable
> introspection convention is pretty slim.

Hence the "just say it's an internal API in the docstring" escape
clause. That's a pretty low bar to meet for PEP 8 compliance, and one
we need, too (for things like test, idlelib and a bundled pip).

I do need to figure out how to tweak the wording to make it clear that
internal nature of an API applies to all contained APIs as well.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From stephen at xemacs.org  Tue Jul 16 04:03:17 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Tue, 16 Jul 2013 11:03:17 +0900
Subject: [Python-Dev] Misc re.match() complaint
In-Reply-To: <CAP7+vJKRuKL6vsPRmvo-+9gPtB=usp57F31_1H--sVN0_qi6GQ@mail.gmail.com>
References: <CAP7+vJKdLzJ3BOgD9e=Nw8KT0v4apdgutu7nKLPAgtEGCapA5A@mail.gmail.com>
	<CAGE7PNJs5qhUiWhv46HBzwpPHsk6Mt_N3+h3XSL2CoNOg7KrYw@mail.gmail.com>
	<51E48F8B.1090905@mrabarnett.plus.com>
	<CAP7+vJKRuKL6vsPRmvo-+9gPtB=usp57F31_1H--sVN0_qi6GQ@mail.gmail.com>
Message-ID: <87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp>

Guido van Rossum writes:

 > And I still think that any return type for group() except bytes or str
 > is wrong. (Except possibly a subclass of these.)

I'm not sure I understand.  Do you mean in the context of the match
object API, where constructing "(target, match.start(), match.end())"
to get a group-like object that refers to the target rather than
copying the text is simple?  (Such objects are very useful in the
restricted application of constructing a programmable text editor.)

Or is this something deeper, that a group *is* a new object in
principle?


From python at mrabarnett.plus.com  Tue Jul 16 04:18:34 2013
From: python at mrabarnett.plus.com (MRAB)
Date: Tue, 16 Jul 2013 03:18:34 +0100
Subject: [Python-Dev] Misc re.match() complaint
In-Reply-To: <CAP7+vJKRuKL6vsPRmvo-+9gPtB=usp57F31_1H--sVN0_qi6GQ@mail.gmail.com>
References: <CAP7+vJKdLzJ3BOgD9e=Nw8KT0v4apdgutu7nKLPAgtEGCapA5A@mail.gmail.com>
	<CAGE7PNJs5qhUiWhv46HBzwpPHsk6Mt_N3+h3XSL2CoNOg7KrYw@mail.gmail.com>
	<51E48F8B.1090905@mrabarnett.plus.com>
	<CAP7+vJKRuKL6vsPRmvo-+9gPtB=usp57F31_1H--sVN0_qi6GQ@mail.gmail.com>
Message-ID: <51E4AD7A.2050505@mrabarnett.plus.com>

On 16/07/2013 01:25, Guido van Rossum wrote:
> On Mon, Jul 15, 2013 at 5:10 PM, MRAB <python at mrabarnett.plus.com> wrote:
>> On 16/07/2013 00:30, Gregory P. Smith wrote:
>>>
>>>
>>> On Mon, Jul 15, 2013 at 4:14 PM, Guido van Rossum <guido at python.org
>>> <mailto:guido at python.org>> wrote:
>>>
>>>     In a discussion about mypy I discovered that the Python 3 version of
>>>     the re module's Match object behaves subtly different from the Python
>>>     2 version when the target string (i.e. the haystack, not the needle)
>>>     is a buffer object.
>>>
>>>     In Python 2, the type of the return value of group() is always either
>>>     a Unicode string or an 8-bit string, and the type is determined by
>>>     looking at the target string -- if the target is unicode, group()
>>>     returns a unicode string, otherwise, group() returns an 8-bit string.
>>>     In particular, if the target is a buffer object, group() returns an
>>>     8-bit string. I think this is the appropriate behavior: otherwise
>>>     using regular expression matching to extract a small substring from a
>>>     large target string would unnecessarily keep the large target string
>>>     alive as long as the substring is alive.
>>>
>>>     But in Python 3, the behavior of group() has changed so that its
>>>     return type always matches that of the target string. I think this is
>>>     bad -- apart from the lifetime concern, it means that if your target
>>>     happens to be a bytearray, the return value isn't even hashable!
>>>
>>>     Does anyone remember whether this was a conscious decision? Is it too
>>>     late to fix?
>>>
>>>
>>> Hmm, that is not what I'd expect either. I would never expect it to
>>> return a bytearray; I'd normally assume that .group() returned a bytes
>>> object if the input was binary data and a str object if the input was
>>> unicode data (str) regardless of specific types containing the input
>>> target data.
>>>
>>> I'm going to hazard a guess that not much, if anything, would be
>>> depending on getting a bytearray out of that. Fix this in 3.4? 3.3 and
>>> earlier users are stuck with an extra bytes() call and data copy in
>>> these cases I guess.
>>>
>> I'm not sure I understand the complaint.
>>
>> I get this for Python 2.7:
>>
>> Python 2.7.5 (default, May 15 2013, 22:43:36) [MSC v.1500 32 bit (Intel)] on
>> win
>> 32
>> Type "help", "copyright", "credits" or "license" for more information.
>>>>> import array
>>>>> import re
>>>>> re.match(r"a", array.array("b", "a")).group()
>> array('b', [97])
>>
>> It's the same even in Python 2.4.
>
> Ah, but now try it with buffer():
>
>>> re.search('yz+', buffer('xyzzy')).group()
> 'yzz'
>>>>
>
> The equivalent in Python 3 (using memoryview) returns a memoryview:
>
>>>> re.search(b'yz+', memoryview(b'xyzzy')).group()
> <memory at 0x10d03a688>
>>>>
>
> And I still think that any return type for group() except bytes or str
> is wrong. (Except possibly a subclass of these.)
>
On the other hand, I think that it's not unreasonable that the output
is the same type as the input. You could reason that what it's doing is
returning a slice of the input, and that slice should be the same type
as its source.

Incidentally, the regex module does what Python 3's re module currently
does, even in Python 2. Nobody's complained!


From chrism at plope.com  Tue Jul 16 04:20:35 2013
From: chrism at plope.com (Chris McDonough)
Date: Mon, 15 Jul 2013 22:20:35 -0400
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
 underscores
In-Reply-To: <51E4A0EF.8070205@pearwood.info>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
	<ks08fk$vnu$1@ger.gmane.org>
	<CADiSq7fAS+-keOjBmOEs-UvV=bV4qB9=ikiN5dkvRW7LSrge6g@mail.gmail.com>
	<20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko>
	<51E4A0EF.8070205@pearwood.info>
Message-ID: <1373941235.30382.62.camel@thinko>

On Tue, 2013-07-16 at 11:25 +1000, Steven D'Aprano wrote:

> 
> If your code has no obvious, documented convention at all for what's internal and what is not, they are no worse off.
> 
> If you do have a documented convention for internal implementation details, then you are no worse off. "I have better things to do than PEP8-ify old, working, stable code" is a perfectly acceptable answer. "I have better things to do than PEP9-ify old, working, stable code, but if you want to provide regression tests and a working patch, I'll let you do so" might be an even better one :-)

Welp, I guess I'm logically boxed in then.  Thanks for showing me the
errors in my thinking.  Should be no problem to manage the updating of
that 500K lines of public code.

/scarcasm

- C



From guido at python.org  Tue Jul 16 04:20:29 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 15 Jul 2013 19:20:29 -0700
Subject: [Python-Dev] Misc re.match() complaint
In-Reply-To: <87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <CAP7+vJKdLzJ3BOgD9e=Nw8KT0v4apdgutu7nKLPAgtEGCapA5A@mail.gmail.com>
	<CAGE7PNJs5qhUiWhv46HBzwpPHsk6Mt_N3+h3XSL2CoNOg7KrYw@mail.gmail.com>
	<51E48F8B.1090905@mrabarnett.plus.com>
	<CAP7+vJKRuKL6vsPRmvo-+9gPtB=usp57F31_1H--sVN0_qi6GQ@mail.gmail.com>
	<87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <CAP7+vJ+WgZSed2zwiCgBpNDVcr1urvvctv8UK705dJ6iEBGQfg@mail.gmail.com>

On Mon, Jul 15, 2013 at 7:03 PM, Stephen J. Turnbull <stephen at xemacs.org> wrote:
> Guido van Rossum writes:
>
>  > And I still think that any return type for group() except bytes or str
>  > is wrong. (Except possibly a subclass of these.)
>
> I'm not sure I understand.  Do you mean in the context of the match
> object API, where constructing "(target, match.start(), match.end())"
> to get a group-like object that refers to the target rather than
> copying the text is simple?  (Such objects are very useful in the
> restricted application of constructing a programmable text editor.)

I'm not sure I understand you. :-(

The group() method on the match object returned by re.match() and
re.search() returns a string-ish object representing the matched
substring. (I'm using "string-ish" to allow for both unicode and
bytes, which are exactly the two matching modes supported be the re
module.) In most contexts (text editors excluded) the program will use
this string just as it would use any other string, perhaps using it to
open a file, perhaps as a key into some cache, and so on.

I can clearly see the reasons why you want the target string to allow
other types besides str and bytes, in particular other things that are
known to represent sequences of bytes, such as bytearray and
memoryview. These reasons primarily have to do with optimizing the
representation of the target string in case it takes up a large amount
of memory, or other situations where we'd like to reduce the number of
times each byte is copied before we see it.

But I don't see as much of a use case for group() returning an object
of the same type as the target string. In particular in the case of a
target string that is a bytearray, group() has to copy the bytes
regardless of whether it creates a bytes or a bytearray instance. And
I do see a use case for group() returning an immutable object.

> Or is this something deeper, that a group *is* a new object in
> principle?

No, I just think of it as returning "a string" and I think it's most
useful if that is always an immutable object, even if the target
string is some other bytes buffer.

FWIW, it feels as if the change in behavior is probably just due to
how slices work.

-- 
--Guido van Rossum (python.org/~guido)

From guido at python.org  Tue Jul 16 04:23:55 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 15 Jul 2013 19:23:55 -0700
Subject: [Python-Dev] Misc re.match() complaint
In-Reply-To: <51E4AD7A.2050505@mrabarnett.plus.com>
References: <CAP7+vJKdLzJ3BOgD9e=Nw8KT0v4apdgutu7nKLPAgtEGCapA5A@mail.gmail.com>
	<CAGE7PNJs5qhUiWhv46HBzwpPHsk6Mt_N3+h3XSL2CoNOg7KrYw@mail.gmail.com>
	<51E48F8B.1090905@mrabarnett.plus.com>
	<CAP7+vJKRuKL6vsPRmvo-+9gPtB=usp57F31_1H--sVN0_qi6GQ@mail.gmail.com>
	<51E4AD7A.2050505@mrabarnett.plus.com>
Message-ID: <CAP7+vJLhQEohqzO3h3h=4=iphO=0LV1Gfb_hsww0hjqjqakFfA@mail.gmail.com>

On Mon, Jul 15, 2013 at 7:18 PM, MRAB <python at mrabarnett.plus.com> wrote:
> On the other hand, I think that it's not unreasonable that the output
> is the same type as the input. You could reason that what it's doing is
> returning a slice of the input, and that slice should be the same type
> as its source.

By now I'm pretty sure that is why it changed.

But I am challenging how useful that is, compared to always returning
something immutable.

> Incidentally, the regex module does what Python 3's re module currently
> does, even in Python 2. Nobody's complained!

Well, you'd only see complaints from folks who (a) use the regex
module, (b) use it with a buffer object as the target string, and (c)
try to use the group() return value as a dict key. Each of these is
probably a small majority of all users.

-- 
--Guido van Rossum (python.org/~guido)

From ncoghlan at gmail.com  Tue Jul 16 04:34:00 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 16 Jul 2013 12:34:00 +1000
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <1373941235.30382.62.camel@thinko>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
	<ks08fk$vnu$1@ger.gmane.org>
	<CADiSq7fAS+-keOjBmOEs-UvV=bV4qB9=ikiN5dkvRW7LSrge6g@mail.gmail.com>
	<20130715184039.60590e62@anarchist>
	<1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info>
	<1373941235.30382.62.camel@thinko>
Message-ID: <CADiSq7cr8JrYJHtfxjgzdHjiJjC-ipcm98ejKTsTAHL6SPMPyg@mail.gmail.com>

On 16 July 2013 12:20, Chris McDonough <chrism at plope.com> wrote:
> On Tue, 2013-07-16 at 11:25 +1000, Steven D'Aprano wrote:
>
>>
>> If your code has no obvious, documented convention at all for what's internal and what is not, they are no worse off.
>>
>> If you do have a documented convention for internal implementation details, then you are no worse off. "I have better things to do than PEP8-ify old, working, stable code" is a perfectly acceptable answer. "I have better things to do than PEP9-ify old, working, stable code, but if you want to provide regression tests and a working patch, I'll let you do so" might be an even better one :-)
>
> Welp, I guess I'm logically boxed in then.  Thanks for showing me the
> errors in my thinking.  Should be no problem to manage the updating of
> that 500K lines of public code.
>
> /scarcasm

How do get from "If this doesn't apply to a module, just add something
like 'This is an internal API' or 'This module includes internal APIs,
consult the documentation for the public API' to the module docstring"
to "updating 500k lines of public code"? The version in Barry's email
that you replied to has that escape clause in it, so the fact it was
missing from my original text doesn't justify this reaction.

Cheers,
Nick.

P.S. Note that, while I'm trying to account for it in this particular
case, we're never going to let the fact that many people misuse PEP 8
by considering it as a holy standard that should be followed by all
Python code everywhere stop us from including updates that are valid
specifically for the standard library.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From victor.stinner at gmail.com  Tue Jul 16 04:38:23 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 16 Jul 2013 04:38:23 +0200
Subject: [Python-Dev] [Python-checkins] cpython (merge default -> 3.3):
	merge 3.3 (#18470)
In-Reply-To: <3bvQHF6dScz7Lp9@mail.python.org>
References: <3bvQHF6dScz7Lp9@mail.python.org>
Message-ID: <CAMpsgwYnkwjT_9QujxMyd_0Q0wJ4m+vaMp0qnJ+xs+746t7njw@mail.gmail.com>

2013/7/16 benjamin.peterson <python-checkins at python.org>:
> http://hg.python.org/cpython/rev/8889c9b5dd3a
> changeset:   84654:8889c9b5dd3a
> branch:      3.3
> parent:      84653:c3a510b22218
> parent:      84652:8a078bf3cf14
> user:        Benjamin Peterson <benjamin at python.org>
> date:        Mon Jul 15 19:15:49 2013 -0700
> summary:
>   merge 3.3 (#18470)

This commit merges branches default and 3.3 into 3.3. Usually, we
merge branches 3.3 and default into default.

Because of this changeset, when I try to merge 3.3 into default, I get
a long list of modifications...

Victor

From victor.stinner at gmail.com  Tue Jul 16 04:40:49 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 16 Jul 2013 04:40:49 +0200
Subject: [Python-Dev] [Python-checkins] cpython (3.3): check the return
 value of new_string() (closes #18470)
In-Reply-To: <3bvQHD4yjMz7LlT@mail.python.org>
References: <3bvQHD4yjMz7LlT@mail.python.org>
Message-ID: <CAMpsgwah995gcaRvpr5qGrHA5ksz=iZSZbZcCMzvmZgdqd6r_A@mail.gmail.com>

2013/7/16 benjamin.peterson <python-checkins at python.org>:
> http://hg.python.org/cpython/rev/c3a510b22218
> changeset:   84653:c3a510b22218
> branch:      3.3
> parent:      84651:e22dd5fda5a8
> user:        Benjamin Peterson <benjamin at python.org>
> date:        Mon Jul 15 19:15:34 2013 -0700
> summary:
>   check the return value of new_string() (closes #18470)
>
> ...
>
>
> diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c
> --- a/Parser/tokenizer.c
> +++ b/Parser/tokenizer.c
> @@ -250,17 +253,21 @@
>                  t++;
>
>              if (begin < t) {
> -                char* r = new_string(begin, t - begin);
> +                char* r = new_string(begin, t - begin, tok);
> +                if (!r)
> +                    return 0;
>                  char* q = get_normal_name(r);
> ...

Visual Studio does support instructions between declarations, and so
this changeset broke Windows buildbots.

Should we add the "-Werror=declaration-after-statement" compiler flag
to the 3.3 branch? (in debug mode?)

Victor

From barry at python.org  Tue Jul 16 04:48:15 2013
From: barry at python.org (Barry Warsaw)
Date: Mon, 15 Jul 2013 22:48:15 -0400
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
 underscores
In-Reply-To: <1373934193.30382.59.camel@thinko>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
	<ks08fk$vnu$1@ger.gmane.org>
	<CADiSq7fAS+-keOjBmOEs-UvV=bV4qB9=ikiN5dkvRW7LSrge6g@mail.gmail.com>
	<20130715184039.60590e62@anarchist>
	<1373934193.30382.59.camel@thinko>
Message-ID: <20130715224815.07b7a8f3@anarchist>

On Jul 15, 2013, at 08:23 PM, Chris McDonough wrote:

>On Mon, 2013-07-15 at 18:40 -0400, Barry Warsaw wrote:
>> Working from what I think is the latest version.
>> 
>> In general, i'd rather be prescriptive of future conventions than
>> descriptive of current conventions.  It's okay to exempt existing code, and
>> as a general rule we've never been fond of rewriting existing code to
>> update it to new standards or APIs.  We don't need to do so here either.
>
>FWIW, I'm very skeptical of a PEP 8 guideline that would try to
>proscribe that the "non-internal" API of a module or class would be
>defined solely by a naming convention.
>
>If what's being described here does become a rule, there is reason to
>believe that future users who treat this PEP as the word-of-god (and
>there are a *lot* of them; I hear from people literally every week who
>want to "PEP8-ify" my code in some limited-value-added way) will be
>harmed.  They'll be living in a fantasy world where every
>non-underscore-prefixed thing is now a defacto API.  But I have lived in
>a world where that has not been the case since 1998, and the chance that
>I'll go back and change all my public code to satisfy a questionable
>introspection convention is pretty slim.

I can sympathize with this.  What I tried to get at, but probably didn't
succeed, is to say "if it starts with a single underscore, treat it as
internal" but not necessarily the converse, i.o.w. if it *doesn't* start with
an underscore that does not imply that it's public.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130715/1152577e/attachment.pgp>

From chrism at plope.com  Tue Jul 16 05:02:42 2013
From: chrism at plope.com (Chris McDonough)
Date: Mon, 15 Jul 2013 23:02:42 -0400
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
 underscores
In-Reply-To: <CADiSq7cr8JrYJHtfxjgzdHjiJjC-ipcm98ejKTsTAHL6SPMPyg@mail.gmail.com>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
	<ks08fk$vnu$1@ger.gmane.org>
	<CADiSq7fAS+-keOjBmOEs-UvV=bV4qB9=ikiN5dkvRW7LSrge6g@mail.gmail.com>
	<20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko>
	<51E4A0EF.8070205@pearwood.info> <1373941235.30382.62.camel@thinko>
	<CADiSq7cr8JrYJHtfxjgzdHjiJjC-ipcm98ejKTsTAHL6SPMPyg@mail.gmail.com>
Message-ID: <1373943762.30382.88.camel@thinko>

On Tue, 2013-07-16 at 12:34 +1000, Nick Coghlan wrote:

> How do get from "If this doesn't apply to a module, just add something
> like 'This is an internal API' or 'This module includes internal APIs,
> consult the documentation for the public API' to the module docstring"
> to "updating 500k lines of public code"? The version in Barry's email
> that you replied to has that escape clause in it, so the fact it was
> missing from my original text doesn't justify this reaction.

I may be wrong, but as I see it this is a decision to help (although
only conditionally!) a theoretical IDE user or an agressive user of
dir() at the expense of effectively either 1) making existing
maintainers revisit existing code, or 2 ) making maintainers continually
explain to users that their codebase is not "PEP8 compliant".

> P.S. Note that, while I'm trying to account for it in this particular
> case, we're never going to let the fact that many people misuse PEP 8
> by considering it as a holy standard that should be followed by all
> Python code everywhere stop us from including updates that are valid
> specifically for the standard library.

Fair enough, technically.  But historical reality is that a lot of
people just haven't read much past the title: "Style Guide for Python
Code".  Or at least they appear to have no concept that it's about code
*in the stdlib only*.  People take it way, way too seriously for
non-stdlib code.  But they do, and it takes time to manage that.

Given that misunderstanding, is there a way to divorce stdlib-centric
guidelines like the one being discussed now from "PEP8"-ness?  I don't
think any amount of marketing effort or reasoned explanation is going to
separate "PEP8" from "correct code" in people's minds at this point.

- C

ps.. the real irritant behind my minirant is this:

OSS developers have spent many months jumping through bw incompat hoops
in Python over the last few years, and it has taken time away from doing
things that provide value.  The less I can do of that, the better, and
Python gets more value too.  That said, I realize that I'm in the
minority because I happen to have a metric ton of public code out there.
But it'd be nice if that was encouraged rather than effectively punished
on the hunch that it might provide some benefit for a theoretical new
user.



From ncoghlan at gmail.com  Tue Jul 16 05:03:29 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 16 Jul 2013 13:03:29 +1000
Subject: [Python-Dev] Misc re.match() complaint
In-Reply-To: <CAP7+vJ+WgZSed2zwiCgBpNDVcr1urvvctv8UK705dJ6iEBGQfg@mail.gmail.com>
References: <CAP7+vJKdLzJ3BOgD9e=Nw8KT0v4apdgutu7nKLPAgtEGCapA5A@mail.gmail.com>
	<CAGE7PNJs5qhUiWhv46HBzwpPHsk6Mt_N3+h3XSL2CoNOg7KrYw@mail.gmail.com>
	<51E48F8B.1090905@mrabarnett.plus.com>
	<CAP7+vJKRuKL6vsPRmvo-+9gPtB=usp57F31_1H--sVN0_qi6GQ@mail.gmail.com>
	<87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAP7+vJ+WgZSed2zwiCgBpNDVcr1urvvctv8UK705dJ6iEBGQfg@mail.gmail.com>
Message-ID: <CADiSq7dTdxBbEqtsV-fdWab5O90JdvnEZRLKXZY9tO86iArPSA@mail.gmail.com>

On 16 July 2013 12:20, Guido van Rossum <guido at python.org> wrote:
> On Mon, Jul 15, 2013 at 7:03 PM, Stephen J. Turnbull <stephen at xemacs.org> wrote:
>> Or is this something deeper, that a group *is* a new object in
>> principle?
>
> No, I just think of it as returning "a string" and I think it's most
> useful if that is always an immutable object, even if the target
> string is some other bytes buffer.
>
> FWIW, it feels as if the change in behavior is probably just due to
> how slices work.

I took a look at the way the 2.7 re code works, and the change does
indeed appear to be due to the difference in the way slices work for
buffer and memoryview objects:

Slicing a buffer creates an 8-bit string:

>>> buffer(b"abc")[0:1]
'a'

Slicing a memoryview creates another memoryview:

>>> memoryview(b"abc")[0:1]
<memory at 0x7f3320541b98>

Unfortunately, memoryview doesn't currently allow subclasses, so it
isn't easy to create a derivative that coerces to bytes on slicing :(

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Tue Jul 16 05:11:49 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 16 Jul 2013 13:11:49 +1000
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <1373943762.30382.88.camel@thinko>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
	<ks08fk$vnu$1@ger.gmane.org>
	<CADiSq7fAS+-keOjBmOEs-UvV=bV4qB9=ikiN5dkvRW7LSrge6g@mail.gmail.com>
	<20130715184039.60590e62@anarchist>
	<1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info>
	<1373941235.30382.62.camel@thinko>
	<CADiSq7cr8JrYJHtfxjgzdHjiJjC-ipcm98ejKTsTAHL6SPMPyg@mail.gmail.com>
	<1373943762.30382.88.camel@thinko>
Message-ID: <CADiSq7ck152x2_dPJ8TET6mJbk3maO9-q2FU5rt6pAaA_G0+fQ@mail.gmail.com>

On 16 July 2013 13:02, Chris McDonough <chrism at plope.com> wrote:
> OSS developers have spent many months jumping through bw incompat hoops
> in Python over the last few years, and it has taken time away from doing
> things that provide value.  The less I can do of that, the better, and
> Python gets more value too.  That said, I realize that I'm in the
> minority because I happen to have a metric ton of public code out there.
> But it'd be nice if that was encouraged rather than effectively punished
> on the hunch that it might provide some benefit for a theoretical new
> user.

You, Armin and everyone else that works on the bytes/text boundary are
indeed the hardest hit by the Python 3 transition, and I appreciate
the hard work you have all done to help make that transition as
successful as it has been so far.

However, the fact that people abuse PEP 8 by treating it as "all
Python code in the world should follow these rules" cannot, and will
not, stop us from continuing to use it to set appropriate guidelines
*for the standard library*.

I'll look into adding some stronger wording at the top making it clear
that while PEP 8 is a useful starting point and a good default if a
project doesn't have a defined style guide of it's own, it is *not*
the be-all-and-end-all for Python style guides. Treating it as such as
an abuse of the PEP, pure and simple.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From chrism at plope.com  Tue Jul 16 05:16:46 2013
From: chrism at plope.com (Chris McDonough)
Date: Mon, 15 Jul 2013 23:16:46 -0400
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
 underscores
In-Reply-To: <CADiSq7ck152x2_dPJ8TET6mJbk3maO9-q2FU5rt6pAaA_G0+fQ@mail.gmail.com>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
	<ks08fk$vnu$1@ger.gmane.org>
	<CADiSq7fAS+-keOjBmOEs-UvV=bV4qB9=ikiN5dkvRW7LSrge6g@mail.gmail.com>
	<20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko>
	<51E4A0EF.8070205@pearwood.info> <1373941235.30382.62.camel@thinko>
	<CADiSq7cr8JrYJHtfxjgzdHjiJjC-ipcm98ejKTsTAHL6SPMPyg@mail.gmail.com>
	<1373943762.30382.88.camel@thinko>
	<CADiSq7ck152x2_dPJ8TET6mJbk3maO9-q2FU5rt6pAaA_G0+fQ@mail.gmail.com>
Message-ID: <1373944606.30382.92.camel@thinko>

On Tue, 2013-07-16 at 13:11 +1000, Nick Coghlan wrote:
> On 16 July 2013 13:02, Chris McDonough <chrism at plope.com> wrote:
> > OSS developers have spent many months jumping through bw incompat hoops
> > in Python over the last few years, and it has taken time away from doing
> > things that provide value.  The less I can do of that, the better, and
> > Python gets more value too.  That said, I realize that I'm in the
> > minority because I happen to have a metric ton of public code out there.
> > But it'd be nice if that was encouraged rather than effectively punished
> > on the hunch that it might provide some benefit for a theoretical new
> > user.
> 
> You, Armin and everyone else that works on the bytes/text boundary are
> indeed the hardest hit by the Python 3 transition, and I appreciate
> the hard work you have all done to help make that transition as
> successful as it has been so far.
> 
> However, the fact that people abuse PEP 8 by treating it as "all
> Python code in the world should follow these rules" cannot, and will
> not, stop us from continuing to use it to set appropriate guidelines
> *for the standard library*.
> 
> I'll look into adding some stronger wording at the top making it clear
> that while PEP 8 is a useful starting point and a good default if a
> project doesn't have a defined style guide of it's own, it is *not*
> the be-all-and-end-all for Python style guides. Treating it as such as
> an abuse of the PEP, pure and simple.

I understand that.  Unfortunately the remainder of the world does not.
The same IDEs that would be helped via this proposed change have "PEP8
modes" turned on *by default*!
http://blog.jetbrains.com/pycharm/2013/02/long-awaited-pep-8-checks-on-the-fly-improved-doctest-support-and-more-in-pycharm-2-7/

It seems like an unwise step to continue stuffing things into the PEP8
brand bag as a result if that stuff is only meant to apply to the
stdlib.

- C



From benjamin at python.org  Tue Jul 16 05:55:16 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Mon, 15 Jul 2013 20:55:16 -0700
Subject: [Python-Dev] [Python-checkins] cpython (3.3): check the return
 value of new_string() (closes #18470)
In-Reply-To: <CAMpsgwah995gcaRvpr5qGrHA5ksz=iZSZbZcCMzvmZgdqd6r_A@mail.gmail.com>
References: <3bvQHD4yjMz7LlT@mail.python.org>
	<CAMpsgwah995gcaRvpr5qGrHA5ksz=iZSZbZcCMzvmZgdqd6r_A@mail.gmail.com>
Message-ID: <CAPZV6o8AwOMDNCqjvxFXz5oxwqwg7LEU_q-nyZNwR+xSAfNxwQ@mail.gmail.com>

2013/7/15 Victor Stinner <victor.stinner at gmail.com>:
> 2013/7/16 benjamin.peterson <python-checkins at python.org>:
>> http://hg.python.org/cpython/rev/c3a510b22218
>> changeset:   84653:c3a510b22218
>> branch:      3.3
>> parent:      84651:e22dd5fda5a8
>> user:        Benjamin Peterson <benjamin at python.org>
>> date:        Mon Jul 15 19:15:34 2013 -0700
>> summary:
>>   check the return value of new_string() (closes #18470)
>>
>> ...
>>
>>
>> diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c
>> --- a/Parser/tokenizer.c
>> +++ b/Parser/tokenizer.c
>> @@ -250,17 +253,21 @@
>>                  t++;
>>
>>              if (begin < t) {
>> -                char* r = new_string(begin, t - begin);
>> +                char* r = new_string(begin, t - begin, tok);
>> +                if (!r)
>> +                    return 0;
>>                  char* q = get_normal_name(r);
>> ...
>
> Visual Studio does support instructions between declarations, and so
> this changeset broke Windows buildbots.

Fixed.

>
> Should we add the "-Werror=declaration-after-statement" compiler flag
> to the 3.3 branch? (in debug mode?)

We've lived with this limitation for years. I'm sure we can wait until
3.4 becomes maintenance. :)


--
Regards,
Benjamin

From guido at python.org  Tue Jul 16 06:53:42 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 15 Jul 2013 21:53:42 -0700
Subject: [Python-Dev] Misc re.match() complaint
In-Reply-To: <CADiSq7dTdxBbEqtsV-fdWab5O90JdvnEZRLKXZY9tO86iArPSA@mail.gmail.com>
References: <CAP7+vJKdLzJ3BOgD9e=Nw8KT0v4apdgutu7nKLPAgtEGCapA5A@mail.gmail.com>
	<CAGE7PNJs5qhUiWhv46HBzwpPHsk6Mt_N3+h3XSL2CoNOg7KrYw@mail.gmail.com>
	<51E48F8B.1090905@mrabarnett.plus.com>
	<CAP7+vJKRuKL6vsPRmvo-+9gPtB=usp57F31_1H--sVN0_qi6GQ@mail.gmail.com>
	<87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAP7+vJ+WgZSed2zwiCgBpNDVcr1urvvctv8UK705dJ6iEBGQfg@mail.gmail.com>
	<CADiSq7dTdxBbEqtsV-fdWab5O90JdvnEZRLKXZY9tO86iArPSA@mail.gmail.com>
Message-ID: <CAP7+vJ+WCtJMjGX480QYB4-LJ2TtQ4jS8ee=H7SP4VsyuzvCpg@mail.gmail.com>

Hm. I'd still like to change this, but I understand it's debatable...
Is the group() method written in C or Python? If it's in C it should
be simple enough to let it just do a little bit of pointer math and
construct a bytes object from the given area of memory -- after all,
it must have a pointer to that memory area in order to do the matching
in the first place (although I realize the code may be separated by a
gulf of abstraction :-).

--Guido

On Mon, Jul 15, 2013 at 8:03 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 16 July 2013 12:20, Guido van Rossum <guido at python.org> wrote:
>> On Mon, Jul 15, 2013 at 7:03 PM, Stephen J. Turnbull <stephen at xemacs.org> wrote:
>>> Or is this something deeper, that a group *is* a new object in
>>> principle?
>>
>> No, I just think of it as returning "a string" and I think it's most
>> useful if that is always an immutable object, even if the target
>> string is some other bytes buffer.
>>
>> FWIW, it feels as if the change in behavior is probably just due to
>> how slices work.
>
> I took a look at the way the 2.7 re code works, and the change does
> indeed appear to be due to the difference in the way slices work for
> buffer and memoryview objects:
>
> Slicing a buffer creates an 8-bit string:
>
>>>> buffer(b"abc")[0:1]
> 'a'
>
> Slicing a memoryview creates another memoryview:
>
>>>> memoryview(b"abc")[0:1]
> <memory at 0x7f3320541b98>
>
> Unfortunately, memoryview doesn't currently allow subclasses, so it
> isn't easy to create a derivative that coerces to bytes on slicing :(
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia



-- 
--Guido van Rossum (python.org/~guido)

From ncoghlan at gmail.com  Tue Jul 16 07:44:47 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 16 Jul 2013 15:44:47 +1000
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <1373944606.30382.92.camel@thinko>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
	<ks08fk$vnu$1@ger.gmane.org>
	<CADiSq7fAS+-keOjBmOEs-UvV=bV4qB9=ikiN5dkvRW7LSrge6g@mail.gmail.com>
	<20130715184039.60590e62@anarchist>
	<1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info>
	<1373941235.30382.62.camel@thinko>
	<CADiSq7cr8JrYJHtfxjgzdHjiJjC-ipcm98ejKTsTAHL6SPMPyg@mail.gmail.com>
	<1373943762.30382.88.camel@thinko>
	<CADiSq7ck152x2_dPJ8TET6mJbk3maO9-q2FU5rt6pAaA_G0+fQ@mail.gmail.com>
	<1373944606.30382.92.camel@thinko>
Message-ID: <CADiSq7fxi8rMbcdVN_SsUycjNWmB_61KQomcowV+m8m9+Qp79A@mail.gmail.com>

On 16 July 2013 13:16, Chris McDonough <chrism at plope.com> wrote:
> I understand that.  Unfortunately the remainder of the world does not.
> The same IDEs that would be helped via this proposed change have "PEP8
> modes" turned on *by default*!
> http://blog.jetbrains.com/pycharm/2013/02/long-awaited-pep-8-checks-on-the-fly-improved-doctest-support-and-more-in-pycharm-2-7/
>
> It seems like an unwise step to continue stuffing things into the PEP8
> brand bag as a result if that stuff is only meant to apply to the
> stdlib.

Clarifying what constitutes an internal interface in a way that
doesn't require renaming anything is a necessary prerequisite for
bundling or bootstrapping the pip CLI in Python 3.4 (as pip exposes
its internal implemetnation API as "import pip" rather than "import
_pip" and renaming it would lead to a lot of pointless code churn).
Without that concern, the topic never would have come up.

Yes, this new section may give the PEP 8 prescriptivists more
ammunition to complain about existing projects with different coding
standards not complying with the latest standard library guidelines.
We're not unfamiliar with the complaint, given the number of standard
library modules (some dating from the 20th century) that don't comply
with the current incarnation of PEP 8 circa 2013.

However, PEP 8 is already quite explicit about what we think of that
kind of behaviour:
http://www.python.org/dev/peps/pep-0008/#a-foolish-consistency-is-the-hobgoblin-of-little-minds

PEP 8 is a tool. Like many of the other tools we provide as part of
Python, it can be abused. However, the likelihood of such abuse is
*not* an excuse to avoid using the tool appropriately, it's a reason
to continue the thankless task of trying to educate people on what's
appropriate and what isn't (and making wholesale changes to working
code just to comply with a style guide generally isn't appropriate).

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From stephen at xemacs.org  Tue Jul 16 07:59:11 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Tue, 16 Jul 2013 14:59:11 +0900
Subject: [Python-Dev] Misc re.match() complaint
In-Reply-To: <CAP7+vJ+WgZSed2zwiCgBpNDVcr1urvvctv8UK705dJ6iEBGQfg@mail.gmail.com>
References: <CAP7+vJKdLzJ3BOgD9e=Nw8KT0v4apdgutu7nKLPAgtEGCapA5A@mail.gmail.com>
	<CAGE7PNJs5qhUiWhv46HBzwpPHsk6Mt_N3+h3XSL2CoNOg7KrYw@mail.gmail.com>
	<51E48F8B.1090905@mrabarnett.plus.com>
	<CAP7+vJKRuKL6vsPRmvo-+9gPtB=usp57F31_1H--sVN0_qi6GQ@mail.gmail.com>
	<87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAP7+vJ+WgZSed2zwiCgBpNDVcr1urvvctv8UK705dJ6iEBGQfg@mail.gmail.com>
Message-ID: <87ppujrqog.fsf@uwakimon.sk.tsukuba.ac.jp>

Guido van Rossum writes:

 > I'm not sure I understand you. :-(

My apologies.  This:

 > > Or is this something deeper, that a group *is* a new object in
 > > principle?
 > 
 > No, I just think of it as returning "a string" and I think it's most
 > useful if that is always an immutable object, even if the target
 > string is some other bytes buffer.

is exactly the kind of answer I was looking for.


From chrism at plope.com  Tue Jul 16 08:01:22 2013
From: chrism at plope.com (Chris McDonough)
Date: Tue, 16 Jul 2013 02:01:22 -0400
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
 underscores
In-Reply-To: <CADiSq7fxi8rMbcdVN_SsUycjNWmB_61KQomcowV+m8m9+Qp79A@mail.gmail.com>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
	<ks08fk$vnu$1@ger.gmane.org>
	<CADiSq7fAS+-keOjBmOEs-UvV=bV4qB9=ikiN5dkvRW7LSrge6g@mail.gmail.com>
	<20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko>
	<51E4A0EF.8070205@pearwood.info> <1373941235.30382.62.camel@thinko>
	<CADiSq7cr8JrYJHtfxjgzdHjiJjC-ipcm98ejKTsTAHL6SPMPyg@mail.gmail.com>
	<1373943762.30382.88.camel@thinko>
	<CADiSq7ck152x2_dPJ8TET6mJbk3maO9-q2FU5rt6pAaA_G0+fQ@mail.gmail.com>
	<1373944606.30382.92.camel@thinko>
	<CADiSq7fxi8rMbcdVN_SsUycjNWmB_61KQomcowV+m8m9+Qp79A@mail.gmail.com>
Message-ID: <1373954482.30382.111.camel@thinko>

On Tue, 2013-07-16 at 15:44 +1000, Nick Coghlan wrote:
> On 16 July 2013 13:16, Chris McDonough <chrism at plope.com> wrote:
> > I understand that.  Unfortunately the remainder of the world does not.
> > The same IDEs that would be helped via this proposed change have "PEP8
> > modes" turned on *by default*!
> > http://blog.jetbrains.com/pycharm/2013/02/long-awaited-pep-8-checks-on-the-fly-improved-doctest-support-and-more-in-pycharm-2-7/
> >
> > It seems like an unwise step to continue stuffing things into the PEP8
> > brand bag as a result if that stuff is only meant to apply to the
> > stdlib.
> 
> Clarifying what constitutes an internal interface in a way that
> doesn't require renaming anything is a necessary prerequisite for
> bundling or bootstrapping the pip CLI in Python 3.4 (as pip exposes
> its internal implemetnation API as "import pip" rather than "import
> _pip" and renaming it would lead to a lot of pointless code churn).
> Without that concern, the topic never would have come up.
> 
> Yes, this new section may give the PEP 8 prescriptivists more
> ammunition to complain about existing projects with different coding
> standards not complying with the latest standard library guidelines.
> We're not unfamiliar with the complaint, given the number of standard
> library modules (some dating from the 20th century) that don't comply
> with the current incarnation of PEP 8 circa 2013.
> 
> However, PEP 8 is already quite explicit about what we think of that
> kind of behaviour:
> http://www.python.org/dev/peps/pep-0008/#a-foolish-consistency-is-the-hobgoblin-of-little-minds
> 
> PEP 8 is a tool. Like many of the other tools we provide as part of
> Python, it can be abused. However, the likelihood of such abuse is
> *not* an excuse to avoid using the tool appropriately, it's a reason
> to continue the thankless task of trying to educate people on what's
> appropriate and what isn't (and making wholesale changes to working
> code just to comply with a style guide generally isn't appropriate).

Well, I know it's arguing with the umpire at this point, but PEP8
prescriptionism (great word by the way!) is de rigeur these days.  It's
a python-dev perogative to ignore that, but it has consequences that
reverberate further than this maillist.

- C



From ncoghlan at gmail.com  Tue Jul 16 08:10:31 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 16 Jul 2013 16:10:31 +1000
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <1373954482.30382.111.camel@thinko>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
	<ks08fk$vnu$1@ger.gmane.org>
	<CADiSq7fAS+-keOjBmOEs-UvV=bV4qB9=ikiN5dkvRW7LSrge6g@mail.gmail.com>
	<20130715184039.60590e62@anarchist>
	<1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info>
	<1373941235.30382.62.camel@thinko>
	<CADiSq7cr8JrYJHtfxjgzdHjiJjC-ipcm98ejKTsTAHL6SPMPyg@mail.gmail.com>
	<1373943762.30382.88.camel@thinko>
	<CADiSq7ck152x2_dPJ8TET6mJbk3maO9-q2FU5rt6pAaA_G0+fQ@mail.gmail.com>
	<1373944606.30382.92.camel@thinko>
	<CADiSq7fxi8rMbcdVN_SsUycjNWmB_61KQomcowV+m8m9+Qp79A@mail.gmail.com>
	<1373954482.30382.111.camel@thinko>
Message-ID: <CADiSq7e1SFWMY-YG6FJMq-VG1M2e-Gv1RmPEySbcRKVqtUXwNA@mail.gmail.com>

On 16 July 2013 16:01, Chris McDonough <chrism at plope.com> wrote:
> Well, I know it's arguing with the umpire at this point, but PEP8
> prescriptionism (great word by the way!) is de rigeur these days.  It's
> a python-dev perogative to ignore that, but it has consequences that
> reverberate further than this maillist.

Yeah, we should probably be a bit more vocal in pointing out that PEP
8 is something to be applied judiciously, *not* treated as "sacred
writ, never to be questioned". It's very tempting to just sigh and
ignore it, instead of explicitly pointing out the problems with taking
it too literally for people's personal projects (or, worse, using it
to abuse developers of projects that long predate the recommendations
in more recent versions) :P

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Tue Jul 16 08:20:38 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 16 Jul 2013 16:20:38 +1000
Subject: [Python-Dev] Misc re.match() complaint
In-Reply-To: <CAP7+vJ+WCtJMjGX480QYB4-LJ2TtQ4jS8ee=H7SP4VsyuzvCpg@mail.gmail.com>
References: <CAP7+vJKdLzJ3BOgD9e=Nw8KT0v4apdgutu7nKLPAgtEGCapA5A@mail.gmail.com>
	<CAGE7PNJs5qhUiWhv46HBzwpPHsk6Mt_N3+h3XSL2CoNOg7KrYw@mail.gmail.com>
	<51E48F8B.1090905@mrabarnett.plus.com>
	<CAP7+vJKRuKL6vsPRmvo-+9gPtB=usp57F31_1H--sVN0_qi6GQ@mail.gmail.com>
	<87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAP7+vJ+WgZSed2zwiCgBpNDVcr1urvvctv8UK705dJ6iEBGQfg@mail.gmail.com>
	<CADiSq7dTdxBbEqtsV-fdWab5O90JdvnEZRLKXZY9tO86iArPSA@mail.gmail.com>
	<CAP7+vJ+WCtJMjGX480QYB4-LJ2TtQ4jS8ee=H7SP4VsyuzvCpg@mail.gmail.com>
Message-ID: <CADiSq7cTvQm_E=o0RrhBryNeSMTEjna3ZQSqSG_fZq5OCO9vzw@mail.gmail.com>

On 16 July 2013 14:53, Guido van Rossum <guido at python.org> wrote:
> Hm. I'd still like to change this, but I understand it's debatable...
> Is the group() method written in C or Python? If it's in C it should
> be simple enough to let it just do a little bit of pointer math and
> construct a bytes object from the given area of memory -- after all,
> it must have a pointer to that memory area in order to do the matching
> in the first place (although I realize the code may be separated by a
> gulf of abstraction :-).

It shouldn't be too bad - I tracked it down through sre_compile, and
everything seems to funnel into match_getslice_by_index [1], so it
should be possible to detect the non-bytes, non-strings there and
coerce them to bytes.

OTOH, you can already get the same effect by explicitly wrapping the
input in memoryview before passing it to re, and then converting the
output to bytes to release the reference to the underlying data, and
doing that doesn't raise ugly backwards compatibility concerns....

Cheers,
Nick.

[1] http://hg.python.org/cpython/file/daf9ea42b610/Modules/_sre.c#l3198

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ronaldoussoren at mac.com  Tue Jul 16 08:32:41 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Tue, 16 Jul 2013 08:32:41 +0200
Subject: [Python-Dev] [Python-checkins] cpython: Issue #18393: Remove
	use of deprecated API on OSX
In-Reply-To: <CAKJDb-N4QOZDEYdUG5j34Z+S82+z-Ux+72T6BZ6Jc7w0eEOdNg@mail.gmail.com>
References: <3bv9J80Cjxz7LjQ@mail.python.org>
	<CAKJDb-N4QOZDEYdUG5j34Z+S82+z-Ux+72T6BZ6Jc7w0eEOdNg@mail.gmail.com>
Message-ID: <DFFE10F8-C6A8-4AA0-9F53-81757A9B69EE@mac.com>


On 15 Jul, 2013, at 18:43, Zachary Ware <zachary.ware at gmail.com> wrote:

> On Mon, Jul 15, 2013 at 11:32 AM, ronald.oussoren
> <python-checkins at python.org> wrote:
>> http://hg.python.org/cpython/rev/ccbaf6762b54
>> changeset:   84634:ccbaf6762b54
>> user:        Ronald Oussoren <ronaldoussoren at mac.com>
>> date:        Mon Jul 15 18:32:09 2013 +0200
>> summary:
>>  Issue #18393: Remove use of deprecated API on OSX
>> 
>> The "Gestalt" function on OSX is deprecated (starting with OSX 10.8),
>> remove its usage from the stdlib. The patch removes a number of private
> 
> I believe this means that Lib/test/leakers/test_gestalt.py can be
> removed as well.

Interesting... test_gestalt.py cannot have worked in Py3k at all. I've removed the file.

Thanks,

   Ronald


From solipsis at pitrou.net  Tue Jul 16 09:55:51 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 16 Jul 2013 09:55:51 +0200
Subject: [Python-Dev] Misc re.match() complaint
References: <CAP7+vJKdLzJ3BOgD9e=Nw8KT0v4apdgutu7nKLPAgtEGCapA5A@mail.gmail.com>
	<CAGE7PNJs5qhUiWhv46HBzwpPHsk6Mt_N3+h3XSL2CoNOg7KrYw@mail.gmail.com>
	<51E48F8B.1090905@mrabarnett.plus.com>
	<CAP7+vJKRuKL6vsPRmvo-+9gPtB=usp57F31_1H--sVN0_qi6GQ@mail.gmail.com>
	<87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAP7+vJ+WgZSed2zwiCgBpNDVcr1urvvctv8UK705dJ6iEBGQfg@mail.gmail.com>
	<CADiSq7dTdxBbEqtsV-fdWab5O90JdvnEZRLKXZY9tO86iArPSA@mail.gmail.com>
	<CAP7+vJ+WCtJMjGX480QYB4-LJ2TtQ4jS8ee=H7SP4VsyuzvCpg@mail.gmail.com>
Message-ID: <20130716095551.5aca0285@pitrou.net>

Le Mon, 15 Jul 2013 21:53:42 -0700,
Guido van Rossum <guido at python.org> a ?crit :
> Hm. I'd still like to change this, but I understand it's debatable...
> Is the group() method written in C or Python?

Is there a strong enough use case to change it? I can't say the current
behaviour seems very useful either, but some people may depend on it.
I already find it a bit weird that you're passing a bytearray or
memoryview to re.match(), to be honest :-)

Regards

Antoine.


> If it's in C it should
> be simple enough to let it just do a little bit of pointer math and
> construct a bytes object from the given area of memory -- after all,
> it must have a pointer to that memory area in order to do the matching
> in the first place (although I realize the code may be separated by a
> gulf of abstraction :-).
> 
> --Guido
> 
> On Mon, Jul 15, 2013 at 8:03 PM, Nick Coghlan <ncoghlan at gmail.com>
> wrote:
> > On 16 July 2013 12:20, Guido van Rossum <guido at python.org> wrote:
> >> On Mon, Jul 15, 2013 at 7:03 PM, Stephen J. Turnbull
> >> <stephen at xemacs.org> wrote:
> >>> Or is this something deeper, that a group *is* a new object in
> >>> principle?
> >>
> >> No, I just think of it as returning "a string" and I think it's
> >> most useful if that is always an immutable object, even if the
> >> target string is some other bytes buffer.
> >>
> >> FWIW, it feels as if the change in behavior is probably just due to
> >> how slices work.
> >
> > I took a look at the way the 2.7 re code works, and the change does
> > indeed appear to be due to the difference in the way slices work for
> > buffer and memoryview objects:
> >
> > Slicing a buffer creates an 8-bit string:
> >
> >>>> buffer(b"abc")[0:1]
> > 'a'
> >
> > Slicing a memoryview creates another memoryview:
> >
> >>>> memoryview(b"abc")[0:1]
> > <memory at 0x7f3320541b98>
> >
> > Unfortunately, memoryview doesn't currently allow subclasses, so it
> > isn't easy to create a derivative that coerces to bytes on
> > slicing :(
> >
> > Cheers,
> > Nick.
> >
> > --
> > Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> 
> 
> 




From solipsis at pitrou.net  Tue Jul 16 09:57:50 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 16 Jul 2013 09:57:50 +0200
Subject: [Python-Dev] Is it safe to call PyEval_EvalFrameEx() with an
	exception set?
References: <CAMpsgwbsDHwGhyAmaxwzhPzXtW8cQ6ifxHD2RSe2Swrpws7APA@mail.gmail.com>
Message-ID: <20130716095750.77bdf874@pitrou.net>

Le Tue, 16 Jul 2013 02:34:49 +0200,
Victor Stinner <victor.stinner at gmail.com> a ?crit :
> Hi,
> 
> I'm working on the issue #18408 (fix issues found by my pyfailmalloc
> tool). To analyze some bugs, I have to add debug traces in various
> functions to find which function returned NULL without setting an
> error, or the opposite: returned a valid object, but with an exception
> set.
> 
> I would like to add assertions in Python/ceval.c to detect such bugs
> earlier. The problem is that some functions rely on the ability to
> call PyEval_EvalFrameEx() with an exception set. Is it expected?
> Should it be avoided?

Well, if your frame is a generator frame, you must be able to throw()
into it, i.e. resume it with an exception set.

The difficulty here is that there are different meanings for "an
exception is set":
- PyErr_Occurred() is true
- the f_exc_type field and friends are set
- possibly another one that I'm forgetting about

Regards

Antoine.



From tjreedy at udel.edu  Tue Jul 16 10:41:55 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 16 Jul 2013 04:41:55 -0400
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <CADiSq7ck152x2_dPJ8TET6mJbk3maO9-q2FU5rt6pAaA_G0+fQ@mail.gmail.com>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
	<ks08fk$vnu$1@ger.gmane.org>
	<CADiSq7fAS+-keOjBmOEs-UvV=bV4qB9=ikiN5dkvRW7LSrge6g@mail.gmail.com>
	<20130715184039.60590e62@anarchist>
	<1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info>
	<1373941235.30382.62.camel@thinko>
	<CADiSq7cr8JrYJHtfxjgzdHjiJjC-ipcm98ejKTsTAHL6SPMPyg@mail.gmail.com>
	<1373943762.30382.88.camel@thinko>
	<CADiSq7ck152x2_dPJ8TET6mJbk3maO9-q2FU5rt6pAaA_G0+fQ@mail.gmail.com>
Message-ID: <ks310f$90g$1@ger.gmane.org>

On 7/15/2013 11:11 PM, Nick Coghlan wrote:

> I'll look into adding some stronger wording at the top making it clear
> that while PEP 8 is a useful starting point and a good default if a
> project doesn't have a defined style guide of it's own, it is *not*
> the be-all-and-end-all for Python style guides. Treating it as such as
> an abuse of the PEP, pure and simple.

How about (re)naming it
Style Guide for Python Standard Library Code

Without 'Standard Library' in the title, I can see how some people might 
think it meant to apply to all code. I do not think we need to worry 
about the above being too narrow.

-- 
Terry Jan Reedy


From tjreedy at udel.edu  Tue Jul 16 11:18:28 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 16 Jul 2013 05:18:28 -0400
Subject: [Python-Dev] Misc re.match() complaint
In-Reply-To: <CAP7+vJKdLzJ3BOgD9e=Nw8KT0v4apdgutu7nKLPAgtEGCapA5A@mail.gmail.com>
References: <CAP7+vJKdLzJ3BOgD9e=Nw8KT0v4apdgutu7nKLPAgtEGCapA5A@mail.gmail.com>
Message-ID: <ks3351$jn$1@ger.gmane.org>

On 7/15/2013 7:14 PM, Guido van Rossum wrote:
> In a discussion about mypy I discovered that the Python 3 version of
> the re module's Match object behaves subtly different from the Python
> 2 version when the target string (i.e. the haystack, not the needle)
> is a buffer object.
>
> In Python 2, the type of the return value of group() is always either
> a Unicode string or an 8-bit string, and the type is determined by
> looking at the target string -- if the target is unicode, group()
> returns a unicode string, otherwise, group() returns an 8-bit string.
> In particular, if the target is a buffer object, group() returns an
> 8-bit string. I think this is the appropriate behavior: otherwise
> using regular expression matching to extract a small substring from a
> large target string would unnecessarily keep the large target string
> alive as long as the substring is alive.
>
> But in Python 3, the behavior of group() has changed so that its
> return type always matches that of the target string. I think this is
> bad -- apart from the lifetime concern, it means that if your target
> happens to be a bytearray, the return value isn't even hashable!
>
> Does anyone remember whether this was a conscious decision? Is it too
> late to fix?

In both Python 2 and Python 3, the second sentence of the docs is "Both 
patterns and strings to be searched can be Unicode strings as well as 
8-bit strings." The Python 3 version goes on to say that patterns and 
targets must match. "However, Unicode strings and 8-bit strings cannot 
be mixed." I normally consider '8-bit string' to mean 'bytes'. It 
certainly meant that in Python 2. We use 'buffer object' or 'object 
satisfying the buffer protocol' to mean 'bytes, byte_arrays, or 
memoryviews'.

I wonder if the change was an artifact of changing the code to prohibit 
mixing Unicode and bytes.

Going on

"match.group([group1, ...])
     Returns one or more subgroups of the match. If there is a single 
argument, the result is a single string;"

In both 2.x and 3.x docs, I usually understand generic 'string' to mean 
'Unicode or bytes'. In any case, The sentence and a half from 'Returns' 
to 'string' is *exactly the same* as in the 2.x docs. As near as I could 
tell looking by the, the rest of the entry for match.group is unchanged 
from 2.x to 3.x. So it is easy to think that the behavior change is an 
unintended regression.

-- 
Terry Jan Reedy



From tjreedy at udel.edu  Tue Jul 16 11:23:54 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 16 Jul 2013 05:23:54 -0400
Subject: [Python-Dev] Misc re.match() complaint
In-Reply-To: <CAP7+vJ+WgZSed2zwiCgBpNDVcr1urvvctv8UK705dJ6iEBGQfg@mail.gmail.com>
References: <CAP7+vJKdLzJ3BOgD9e=Nw8KT0v4apdgutu7nKLPAgtEGCapA5A@mail.gmail.com>
	<CAGE7PNJs5qhUiWhv46HBzwpPHsk6Mt_N3+h3XSL2CoNOg7KrYw@mail.gmail.com>
	<51E48F8B.1090905@mrabarnett.plus.com>
	<CAP7+vJKRuKL6vsPRmvo-+9gPtB=usp57F31_1H--sVN0_qi6GQ@mail.gmail.com>
	<87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAP7+vJ+WgZSed2zwiCgBpNDVcr1urvvctv8UK705dJ6iEBGQfg@mail.gmail.com>
Message-ID: <ks33f6$jn$2@ger.gmane.org>

On 7/15/2013 10:20 PM, Guido van Rossum wrote:

>> Or is this something deeper, that a group *is* a new object in
>> principle?
>
> No, I just think of it as returning "a string"

That is exactly what the doc says it does. See my other post.

-- 
Terry Jan Reedy


From victor.stinner at gmail.com  Tue Jul 16 12:08:11 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 16 Jul 2013 12:08:11 +0200
Subject: [Python-Dev] Is it safe to call PyEval_EvalFrameEx() with an
 exception set?
In-Reply-To: <20130716095750.77bdf874@pitrou.net>
References: <CAMpsgwbsDHwGhyAmaxwzhPzXtW8cQ6ifxHD2RSe2Swrpws7APA@mail.gmail.com>
	<20130716095750.77bdf874@pitrou.net>
Message-ID: <CAMpsgwYeOawD54=VZS6eQwUZCpLL0=3Bi25_4YbCTrRgEBATEg@mail.gmail.com>

2013/7/16 Antoine Pitrou <solipsis at pitrou.net>:
> Le Tue, 16 Jul 2013 02:34:49 +0200,
> Victor Stinner <victor.stinner at gmail.com> a ?crit :
>> I would like to add assertions in Python/ceval.c to detect such bugs
>> earlier. The problem is that some functions rely on the ability to
>> call PyEval_EvalFrameEx() with an exception set. Is it expected?
>> Should it be avoided?
>
> Well, if your frame is a generator frame, you must be able to throw()
> into it, i.e. resume it with an exception set.
>
> The difficulty here is that there are different meanings for "an
> exception is set":
> - PyErr_Occurred() is true
> - the f_exc_type field and friends are set
> - possibly another one that I'm forgetting about

Ah yes, a generator can call PyEval_EvalFrameEx() with an exception
set (PyErr_Occurred() is non-NULL), but it sets throwflag to 1.

My question is when an exception is set (PyErr_Occurred() is non-NULL)
at the beginning of the bytecode evaluation loop. Just after the "goto
error;" in this extract of ceval.c:

    if (throwflag) /* support for generator.throw() */
        goto error;

    for (;;) { ... }

Victor

From shibturn at gmail.com  Tue Jul 16 12:28:28 2013
From: shibturn at gmail.com (Richard Oudkerk)
Date: Tue, 16 Jul 2013 11:28:28 +0100
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <CADiSq7fxi8rMbcdVN_SsUycjNWmB_61KQomcowV+m8m9+Qp79A@mail.gmail.com>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
	<ks08fk$vnu$1@ger.gmane.org>
	<CADiSq7fAS+-keOjBmOEs-UvV=bV4qB9=ikiN5dkvRW7LSrge6g@mail.gmail.com>
	<20130715184039.60590e62@anarchist>
	<1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info>
	<1373941235.30382.62.camel@thinko>
	<CADiSq7cr8JrYJHtfxjgzdHjiJjC-ipcm98ejKTsTAHL6SPMPyg@mail.gmail.com>
	<1373943762.30382.88.camel@thinko>
	<CADiSq7ck152x2_dPJ8TET6mJbk3maO9-q2FU5rt6pAaA_G0+fQ@mail.gmail.com>
	<1373944606.30382.92.camel@thinko>
	<CADiSq7fxi8rMbcdVN_SsUycjNWmB_61KQomcowV+m8m9+Qp79A@mail.gmail.com>
Message-ID: <ks3788$ere$1@ger.gmane.org>

On 16/07/2013 6:44am, Nick Coghlan wrote:
> Clarifying what constitutes an internal interface in a way that
> doesn't require renaming anything is a necessary prerequisite for
> bundling or bootstrapping the pip CLI in Python 3.4 (as pip exposes
> its internal implemetnation API as "import pip" rather than "import
> _pip" and renaming it would lead to a lot of pointless code churn).
> Without that concern, the topic never would have come up.

BTW, how does the use of __all__ effect things?  Somewhere I got the 
idea that if a module uses __all__ then anything not listed is internal. 
  I take it that is wrong?

-- 
Richard


From ncoghlan at gmail.com  Tue Jul 16 13:03:51 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 16 Jul 2013 21:03:51 +1000
Subject: [Python-Dev] Misc re.match() complaint
In-Reply-To: <ks3351$jn$1@ger.gmane.org>
References: <CAP7+vJKdLzJ3BOgD9e=Nw8KT0v4apdgutu7nKLPAgtEGCapA5A@mail.gmail.com>
	<ks3351$jn$1@ger.gmane.org>
Message-ID: <CADiSq7fGWHdTMMDd_0jGa6ZGHsuKsvBoDgSuYP5=ANSWHYb=1A@mail.gmail.com>

On 16 July 2013 19:18, Terry Reedy <tjreedy at udel.edu> wrote:
> I wonder if the change was an artifact of changing the code to prohibit
> mixing Unicode and bytes.

I'm pretty sure we the only thing we changed in 3.x is to migrate re
to the PEP 3118 buffer API, and the behavioural change Guido is seeing
is actually the one between the 2.x buffer (which returns 8-bit
strings when sliced) and other types (including memoryview) which
return instances of themselves.

Getting the old buffer behaviour in 3.x without an extra copy
operation should just be a matter of wrapping the input with
memoryview (to avoid copying the group elements in the match object)
and the output with bytes (to avoid keeping the entire original object
alive just to reference a few small pieces of it that were matched by
the regex):

>>> import re
>>> data = bytearray(b"aaabbbcccddd")
>>> re.match(b"(a*)b*c*(d*)", data).group(2)
bytearray(b'ddd')
>>> bytes(re.match(b"(a*)b*c*(d*)", memoryview(data)).group(2))
b'ddd'

Given that, I'm inclined to keep the existing behaviour on backwards
compatibility grounds. To make the above code work on both 2.x *and*
3.x without making an extra copy, it's possible to keep the bytes call
(it should be a no-op on 2.x) and dynamically switch the type used to
wrap the input between buffer in 2.x and memoryview in 3.x
(unfortunately, the 2.x memoryview doesn't work for this case, as the
2.x re API doesn't accept it as valid input).

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Tue Jul 16 13:09:23 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 16 Jul 2013 21:09:23 +1000
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <ks310f$90g$1@ger.gmane.org>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
	<ks08fk$vnu$1@ger.gmane.org>
	<CADiSq7fAS+-keOjBmOEs-UvV=bV4qB9=ikiN5dkvRW7LSrge6g@mail.gmail.com>
	<20130715184039.60590e62@anarchist>
	<1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info>
	<1373941235.30382.62.camel@thinko>
	<CADiSq7cr8JrYJHtfxjgzdHjiJjC-ipcm98ejKTsTAHL6SPMPyg@mail.gmail.com>
	<1373943762.30382.88.camel@thinko>
	<CADiSq7ck152x2_dPJ8TET6mJbk3maO9-q2FU5rt6pAaA_G0+fQ@mail.gmail.com>
	<ks310f$90g$1@ger.gmane.org>
Message-ID: <CADiSq7eRjyXK0VEHj7cBfpJj4Z4QiMxdX5tJr4nfbODatzGVXw@mail.gmail.com>

On 16 July 2013 18:41, Terry Reedy <tjreedy at udel.edu> wrote:
> On 7/15/2013 11:11 PM, Nick Coghlan wrote:
>
>> I'll look into adding some stronger wording at the top making it clear
>> that while PEP 8 is a useful starting point and a good default if a
>> project doesn't have a defined style guide of it's own, it is *not*
>> the be-all-and-end-all for Python style guides. Treating it as such as
>> an abuse of the PEP, pure and simple.
>
>
> How about (re)naming it
> Style Guide for Python Standard Library Code
>
> Without 'Standard Library' in the title, I can see how some people might
> think it meant to apply to all code. I do not think we need to worry about
> the above being too narrow.

I reread the whole thing today - it turns out we *have* tweaked it
over time to account for other people using it for their projects as
well. The one genuinely standard library specific element still in it
(the prohibition on the use of function annotations) is explicitly
qualified as only applying to the standard library and not anything
else. So I think we need to continue supporting that use case.

I did find it interesting that we *don't* explicitly advise against
the use of "import *" for anything other than optional accelerator
modules or republishing internal interfaces through a public API, even
though we advice against the practice in the tutorial. Perhaps another
oversight worth correcting?

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Tue Jul 16 13:10:50 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 16 Jul 2013 21:10:50 +1000
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <ks3788$ere$1@ger.gmane.org>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
	<ks08fk$vnu$1@ger.gmane.org>
	<CADiSq7fAS+-keOjBmOEs-UvV=bV4qB9=ikiN5dkvRW7LSrge6g@mail.gmail.com>
	<20130715184039.60590e62@anarchist>
	<1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info>
	<1373941235.30382.62.camel@thinko>
	<CADiSq7cr8JrYJHtfxjgzdHjiJjC-ipcm98ejKTsTAHL6SPMPyg@mail.gmail.com>
	<1373943762.30382.88.camel@thinko>
	<CADiSq7ck152x2_dPJ8TET6mJbk3maO9-q2FU5rt6pAaA_G0+fQ@mail.gmail.com>
	<1373944606.30382.92.camel@thinko>
	<CADiSq7fxi8rMbcdVN_SsUycjNWmB_61KQomcowV+m8m9+Qp79A@mail.gmail.com>
	<ks3788$ere$1@ger.gmane.org>
Message-ID: <CADiSq7f5h_4V=6aidcr4qo0d3V5N8NLcRc3cSDUFs6rmvuOMeg@mail.gmail.com>

On 16 July 2013 20:28, Richard Oudkerk <shibturn at gmail.com> wrote:
> On 16/07/2013 6:44am, Nick Coghlan wrote:
>>
>> Clarifying what constitutes an internal interface in a way that
>> doesn't require renaming anything is a necessary prerequisite for
>> bundling or bootstrapping the pip CLI in Python 3.4 (as pip exposes
>> its internal implemetnation API as "import pip" rather than "import
>> _pip" and renaming it would lead to a lot of pointless code churn).
>> Without that concern, the topic never would have come up.
>
>
> BTW, how does the use of __all__ effect things?  Somewhere I got the idea
> that if a module uses __all__ then anything not listed is internal.  I take
> it that is wrong?

Gah, you're right. I did consider that, then forgot to include it in
the proposed text.

Rather than throwing more versions of the text at the list, I'll
create a tracker issue and post a proposed patch for PEP 8.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Tue Jul 16 13:40:00 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 16 Jul 2013 21:40:00 +1000
Subject: [Python-Dev] Why does PEP 8 advise against explicit relative
	imports?
Message-ID: <CADiSq7cC=GYwkx+viaNvNB+wMS4uU-XdSqH5F9vp8SagKkaTeg@mail.gmail.com>

PEP 8 advises developers to use absolute imports rather than explicit
relative imports.

Why? Using absolute imports couple the internal implementation of a
package to its public name - you can't just change the top level
directory name any more, you have to go through and change all the
absolute imports as well. You also can't easily vendor a package that
uses absolute imports inside another project either, since all the
absolute imports will break.

What's the *concrete* benefit of using absolute imports that overcomes
these significant shortcomings in maintainability and composability?

Regards,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From thomas at python.org  Tue Jul 16 14:02:28 2013
From: thomas at python.org (Thomas Wouters)
Date: Tue, 16 Jul 2013 14:02:28 +0200
Subject: [Python-Dev] Why does PEP 8 advise against explicit relative
	imports?
In-Reply-To: <CADiSq7cC=GYwkx+viaNvNB+wMS4uU-XdSqH5F9vp8SagKkaTeg@mail.gmail.com>
References: <CADiSq7cC=GYwkx+viaNvNB+wMS4uU-XdSqH5F9vp8SagKkaTeg@mail.gmail.com>
Message-ID: <CAPdQG2rJMdC=QfOcf9gGdiFiLwEskf90v8HUBhEsR8y+W47JcQ@mail.gmail.com>

On Tue, Jul 16, 2013 at 1:40 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> PEP 8 advises developers to use absolute imports rather than explicit
> relative imports.
>
> Why? Using absolute imports couple the internal implementation of a
> package to its public name - you can't just change the top level
> directory name any more, you have to go through and change all the
> absolute imports as well. You also can't easily vendor a package that
> uses absolute imports inside another project either, since all the
> absolute imports will break.
>

The problem with relative imports (both explicit and implicit) is that it
makes it less obvious when you are importing a module under the wrong name.
If a package ends up in sys.path directly (for example, by executing
something that lives inside it directly) then an explicit relative import
directly in the package will fail, but an explicit relative import in a
sub-package won't, and you can end up with the subtly confusing mess of
multiple imports of the same .py file under different names.


> What's the *concrete* benefit of using absolute imports that overcomes
> these significant shortcomings in maintainability and composability?
>

In my experience changing the name of a package is problematic for more
reasons than just 'import'. We used to do the put-in-a-package thing for
third-party packages at Google, and we quit doing it because it was just
too painful: it's hard to find all the places that reference the package by
name (think of __import__, or mucking in sys.modules), and it's extra
problematic with packages that contain extension modules. You can't rename
a package, even for 'vendoring a package', without carefully checking
whether it'll work -- unless you don't care if it'll work, of course :) The
alternative is to include a package *without* changing its name (by adding
the right directory, without __init__.py, to sys.path) and that has worked
out a lot better for us.

Even so, there's no reason for *the standard library* to use explicit
relative imports, and that's what PEP 8 is supposed to cover, right? :)

-- 
Thomas Wouters <thomas at python.org>

Hi! I'm an email virus! Think twice before sending your email to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130716/aaad9e4a/attachment.html>

From martin at v.loewis.de  Tue Jul 16 14:17:39 2013
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 16 Jul 2013 14:17:39 +0200
Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL
In-Reply-To: <51DF469B.6010007@python.org>
References: <51DF469B.6010007@python.org>
Message-ID: <51E539E3.2020104@v.loewis.de>

Am 12.07.13 01:58, schrieb Christian Heimes:
> For Python 3.4 is going to be a very close call. According to PEP 429
> 3.4.0 final is scheduled for February 22, 2014. The extended support
> phase of Windows XP ends merely 45 days later on April 8, 2014. Do we
> really have to restrict ourselves to an API that is going to become
> deprecated 45 days after the estimated release of 3.4.0?

I suggest to stick to the words of the PEP, which means that XP needs
to be supported in 3.4. For 3.5, according to the PEP, XP support
*may* be dropped - it doesn't have to be dropped.

Notice that you cannot yet know for sure that XP support ends on April 8
- even though it's likely that it will. Microsoft might (and, in the
past did) extend an earlier end-of-support date (although they probably
won't do so for XP anymore).

The most significant reason for dropping XP support in Python will
likely be the desire to switch to new VC versions: Newer CRTs don't
support XP anymore.

Regards,
Martin


From ncoghlan at gmail.com  Tue Jul 16 14:24:17 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 16 Jul 2013 22:24:17 +1000
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <CADiSq7f5h_4V=6aidcr4qo0d3V5N8NLcRc3cSDUFs6rmvuOMeg@mail.gmail.com>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
	<ks08fk$vnu$1@ger.gmane.org>
	<CADiSq7fAS+-keOjBmOEs-UvV=bV4qB9=ikiN5dkvRW7LSrge6g@mail.gmail.com>
	<20130715184039.60590e62@anarchist>
	<1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info>
	<1373941235.30382.62.camel@thinko>
	<CADiSq7cr8JrYJHtfxjgzdHjiJjC-ipcm98ejKTsTAHL6SPMPyg@mail.gmail.com>
	<1373943762.30382.88.camel@thinko>
	<CADiSq7ck152x2_dPJ8TET6mJbk3maO9-q2FU5rt6pAaA_G0+fQ@mail.gmail.com>
	<1373944606.30382.92.camel@thinko>
	<CADiSq7fxi8rMbcdVN_SsUycjNWmB_61KQomcowV+m8m9+Qp79A@mail.gmail.com>
	<ks3788$ere$1@ger.gmane.org>
	<CADiSq7f5h_4V=6aidcr4qo0d3V5N8NLcRc3cSDUFs6rmvuOMeg@mail.gmail.com>
Message-ID: <CADiSq7eHTLmSDDm67TSh6+6FWy2aXmUzEKshNC5PX+946ZvqNw@mail.gmail.com>

On 16 July 2013 21:10, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 16 July 2013 20:28, Richard Oudkerk <shibturn at gmail.com> wrote:
>> On 16/07/2013 6:44am, Nick Coghlan wrote:
>>>
>>> Clarifying what constitutes an internal interface in a way that
>>> doesn't require renaming anything is a necessary prerequisite for
>>> bundling or bootstrapping the pip CLI in Python 3.4 (as pip exposes
>>> its internal implemetnation API as "import pip" rather than "import
>>> _pip" and renaming it would lead to a lot of pointless code churn).
>>> Without that concern, the topic never would have come up.
>>
>>
>> BTW, how does the use of __all__ effect things?  Somewhere I got the idea
>> that if a module uses __all__ then anything not listed is internal.  I take
>> it that is wrong?
>
> Gah, you're right. I did consider that, then forgot to include it in
> the proposed text.
>
> Rather than throwing more versions of the text at the list, I'll
> create a tracker issue and post a proposed patch for PEP 8.

Tracker issue: http://bugs.python.org/issue18472

When I went to write this in context, I realised it made more sense to
focus on defining what a *public* interface was, and how that differed
from an internal interface. WIth that approach, the natural flow of
the text was:

* the public interface is preferentially defined by the documentation
* __all__ is the main programmatic interface
* leading underscores are then mostly a helper that makes it easier to
remember which is which while actually *working* on the code (but
still a good practice)

I also spotted a few other oddities as I went through. The obviously
wrong ones I just fixed, but there were a couple relating to
exceptions that seem unnecessary or ill-advised given PEP 352 and
3151, plus the one about absolute vs explicit relative imports that I
started a separate thread for.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Tue Jul 16 14:27:03 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 16 Jul 2013 22:27:03 +1000
Subject: [Python-Dev] Why does PEP 8 advise against explicit relative
	imports?
In-Reply-To: <CAPdQG2rJMdC=QfOcf9gGdiFiLwEskf90v8HUBhEsR8y+W47JcQ@mail.gmail.com>
References: <CADiSq7cC=GYwkx+viaNvNB+wMS4uU-XdSqH5F9vp8SagKkaTeg@mail.gmail.com>
	<CAPdQG2rJMdC=QfOcf9gGdiFiLwEskf90v8HUBhEsR8y+W47JcQ@mail.gmail.com>
Message-ID: <CADiSq7f3fZLgEzmETHtQDTH=bCPQSigEQ2wB_Hy2-fO=vgemdQ@mail.gmail.com>

On 16 July 2013 22:02, Thomas Wouters <thomas at python.org> wrote:
> On Tue, Jul 16, 2013 at 1:40 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> PEP 8 advises developers to use absolute imports rather than explicit
>> relative imports.
>>
>> Why? Using absolute imports couple the internal implementation of a
>> package to its public name - you can't just change the top level
>> directory name any more, you have to go through and change all the
>> absolute imports as well. You also can't easily vendor a package that
>> uses absolute imports inside another project either, since all the
>> absolute imports will break.
>
>
> The problem with relative imports (both explicit and implicit) is that it
> makes it less obvious when you are importing a module under the wrong name.
> If a package ends up in sys.path directly (for example, by executing
> something that lives inside it directly) then an explicit relative import
> directly in the package will fail, but an explicit relative import in a
> sub-package won't, and you can end up with the subtly confusing mess of
> multiple imports of the same .py file under different names.
>
>>
>> What's the *concrete* benefit of using absolute imports that overcomes
>> these significant shortcomings in maintainability and composability?
>
> In my experience changing the name of a package is problematic for more
> reasons than just 'import'. We used to do the put-in-a-package thing for
> third-party packages at Google, and we quit doing it because it was just too
> painful: it's hard to find all the places that reference the package by name
> (think of __import__, or mucking in sys.modules), and it's extra problematic
> with packages that contain extension modules. You can't rename a package,
> even for 'vendoring a package', without carefully checking whether it'll
> work -- unless you don't care if it'll work, of course :) The alternative is
> to include a package *without* changing its name (by adding the right
> directory, without __init__.py, to sys.path) and that has worked out a lot
> better for us.

These are good reasons, but they're not the reasons the PEP currently
gives. "Better failure modes" is a good practical benefit :)

> Even so, there's no reason for *the standard library* to use explicit
> relative imports, and that's what PEP 8 is supposed to cover, right? :)

I realised in re-reading the whole thing that we actually admitted
defeat on that front back during the last function annotations
discussion :)

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Tue Jul 16 14:35:08 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 16 Jul 2013 22:35:08 +1000
Subject: [Python-Dev] Why does PEP 8 advise against explicit relative
	imports?
In-Reply-To: <CADiSq7f3fZLgEzmETHtQDTH=bCPQSigEQ2wB_Hy2-fO=vgemdQ@mail.gmail.com>
References: <CADiSq7cC=GYwkx+viaNvNB+wMS4uU-XdSqH5F9vp8SagKkaTeg@mail.gmail.com>
	<CAPdQG2rJMdC=QfOcf9gGdiFiLwEskf90v8HUBhEsR8y+W47JcQ@mail.gmail.com>
	<CADiSq7f3fZLgEzmETHtQDTH=bCPQSigEQ2wB_Hy2-fO=vgemdQ@mail.gmail.com>
Message-ID: <CADiSq7dpb3EmQrXRhn57d1iX0ER-J2s4KMv0JU8FGos3zJydtA@mail.gmail.com>

On 16 July 2013 22:27, Nick Coghlan <ncoghlan at gmail.com> wrote:
> These are good reasons, but they're not the reasons the PEP currently
> gives. "Better failure modes" is a good practical benefit :)

I have now included an updated rationale for this rule in the patch
attached to http://bugs.python.org/issue18472 (the recommendation to
use absolute imports remains unchanged).

Cheers,
Nick.


--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From mail at timgolden.me.uk  Tue Jul 16 14:42:10 2013
From: mail at timgolden.me.uk (Tim Golden)
Date: Tue, 16 Jul 2013 13:42:10 +0100
Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL
In-Reply-To: <51DF469B.6010007@python.org>
References: <51DF469B.6010007@python.org>
Message-ID: <51E53FA2.4030102@timgolden.me.uk>

On 12/07/2013 00:58, Christian Heimes wrote:
> Hi,
> 
> how do you feel about dropping Windows XP support for Python 3.4? It
> would enable us to use some features that are only available on Windows
> Vista and newer, for example http://bugs.python.org/issue6926 and
> http://bugs.python.org/issue1763 .
> 
> PEP 11 says:
>   A new feature release X.Y.0 will support all Windows releases
>   whose extended support phase is not yet expired.
> 
> For Python 3.4 is going to be a very close call. According to PEP 429
> 3.4.0 final is scheduled for February 22, 2014. The extended support
> phase of Windows XP ends merely 45 days later on April 8, 2014. Do we
> really have to restrict ourselves to an API that is going to become
> deprecated 45 days after the estimated release of 3.4.0?

I would like to continue support for WinXP. It's still widely, widely
used -- MS support notwithstanding. The situation might have been
different if Vista had been a viable corporate desktop, but I suspect
that many outfits have waited (as we did here) until Win7 before moving
forward. Win7 is now our default desktop for new machines, but we're
still running a slight majority of WinXP machines.

TJG

From steve at pearwood.info  Tue Jul 16 15:19:21 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 16 Jul 2013 23:19:21 +1000
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <ks3788$ere$1@ger.gmane.org>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
	<ks08fk$vnu$1@ger.gmane.org>
	<CADiSq7fAS+-keOjBmOEs-UvV=bV4qB9=ikiN5dkvRW7LSrge6g@mail.gmail.com>
	<20130715184039.60590e62@anarchist>
	<1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info>
	<1373941235.30382.62.camel@thinko>
	<CADiSq7cr8JrYJHtfxjgzdHjiJjC-ipcm98ejKTsTAHL6SPMPyg@mail.gmail.com>
	<1373943762.30382.88.camel@thinko>
	<CADiSq7ck152x2_dPJ8TET6mJbk3maO9-q2FU5rt6pAaA_G0+fQ@mail.gmail.com>
	<1373944606.30382.92.camel@thinko>
	<CADiSq7fxi8rMbcdVN_SsUycjNWmB_61KQomcowV+m8m9+Qp79A@mail.gmail.com>
	<ks3788$ere$1@ger.gmane.org>
Message-ID: <51E54859.4070901@pearwood.info>

On 16/07/13 20:28, Richard Oudkerk wrote:
> On 16/07/2013 6:44am, Nick Coghlan wrote:
>> Clarifying what constitutes an internal interface in a way that
>> doesn't require renaming anything is a necessary prerequisite for
>> bundling or bootstrapping the pip CLI in Python 3.4 (as pip exposes
>> its internal implemetnation API as "import pip" rather than "import
>> _pip" and renaming it would lead to a lot of pointless code churn).
>> Without that concern, the topic never would have come up.
>
> BTW, how does the use of __all__ effect things?  Somewhere I got the idea that if a module uses __all__ then anything not listed is internal.  I take it that is wrong?


That is not how I interpret __all__. In the absence of any explicit documentation, I interpret __all__ as nothing more than a list of names which wildcard imports will bring in, without necessarily meaning that other names are private. For example, I might have a module explicitly designed for wildcard imports at the interactive interpreter:

from module import *

brings in the functions which I expect will be useful interactively, not necessarily the entire public API.

For example, pkgutil includes classes with single-underscore methods, which I take as private. It also has a function simplegeneric, which is undocumented and not listed in __all__. In in the absence of even a comment saying "Don't use this", I take it as an oversight, not policy that simplegeneric is private.



-- 
Steven

From rdmurray at bitdance.com  Tue Jul 16 15:39:12 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Tue, 16 Jul 2013 09:39:12 -0400
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <51E54859.4070901@pearwood.info>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
	<ks08fk$vnu$1@ger.gmane.org>
	<CADiSq7fAS+-keOjBmOEs-UvV=bV4qB9=ikiN5dkvRW7LSrge6g@mail.gmail.com>
	<20130715184039.60590e62@anarchist>
	<1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info>
	<1373941235.30382.62.camel@thinko>
	<CADiSq7cr8JrYJHtfxjgzdHjiJjC-ipcm98ejKTsTAHL6SPMPyg@mail.gmail.com>
	<1373943762.30382.88.camel@thinko>
	<CADiSq7ck152x2_dPJ8TET6mJbk3maO9-q2FU5rt6pAaA_G0+fQ@mail.gmail.com>
	<1373944606.30382.92.camel@thinko>
	<CADiSq7fxi8rMbcdVN_SsUycjNWmB_61KQomcowV+m8m9+Qp79A@mail.gmail.com>
	<ks3788$ere$1@ger.gmane.org> <51E54859.4070901@pearwood.info>
Message-ID: <20130716133912.C532325007E@webabinitio.net>

On Tue, 16 Jul 2013 23:19:21 +1000, Steven D'Aprano <steve at pearwood.info> wrote:
> On 16/07/13 20:28, Richard Oudkerk wrote:
> > On 16/07/2013 6:44am, Nick Coghlan wrote:
> >> Clarifying what constitutes an internal interface in a way that
> >> doesn't require renaming anything is a necessary prerequisite for
> >> bundling or bootstrapping the pip CLI in Python 3.4 (as pip exposes
> >> its internal implemetnation API as "import pip" rather than "import
> >> _pip" and renaming it would lead to a lot of pointless code churn).
> >> Without that concern, the topic never would have come up.
> >
> > BTW, how does the use of __all__ effect things?  Somewhere I got the idea that if a module uses __all__ then anything not listed is internal.  I take it that is wrong?
> 
> 
> That is not how I interpret __all__. In the absence of any explicit documentation, I interpret __all__ as nothing more than a list of names which wildcard imports will bring in, without necessarily meaning that other names are private. For example, I might have a module explicitly designed for wildcard imports at the interactive interpreter:
> 
> from module import *
> 
> brings in the functions which I expect will be useful interactively, not necessarily the entire public API.
> 
> For example, pkgutil includes classes with single-underscore methods, which I take as private. It also has a function simplegeneric, which is undocumented and not listed in __all__. In in the absence of even a comment saying "Don't use this", I take it as an oversight, not policy that simplegeneric is private.

I think you'd be wrong about that, though.  simplegeneric should really be
treated as private.  I'm speaking here not about the general principle of
the thing, but about my understanding of simplegeneric's specific history.

--David

From ncoghlan at gmail.com  Tue Jul 16 15:48:06 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 16 Jul 2013 23:48:06 +1000
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <20130716133912.C532325007E@webabinitio.net>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
	<ks08fk$vnu$1@ger.gmane.org>
	<CADiSq7fAS+-keOjBmOEs-UvV=bV4qB9=ikiN5dkvRW7LSrge6g@mail.gmail.com>
	<20130715184039.60590e62@anarchist>
	<1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info>
	<1373941235.30382.62.camel@thinko>
	<CADiSq7cr8JrYJHtfxjgzdHjiJjC-ipcm98ejKTsTAHL6SPMPyg@mail.gmail.com>
	<1373943762.30382.88.camel@thinko>
	<CADiSq7ck152x2_dPJ8TET6mJbk3maO9-q2FU5rt6pAaA_G0+fQ@mail.gmail.com>
	<1373944606.30382.92.camel@thinko>
	<CADiSq7fxi8rMbcdVN_SsUycjNWmB_61KQomcowV+m8m9+Qp79A@mail.gmail.com>
	<ks3788$ere$1@ger.gmane.org> <51E54859.4070901@pearwood.info>
	<20130716133912.C532325007E@webabinitio.net>
Message-ID: <CADiSq7cUCf9KRrLGK1C-x8DWzj8+yHL30rWCRzm0KQarjwY_+A@mail.gmail.com>

On 16 July 2013 23:39, R. David Murray <rdmurray at bitdance.com> wrote:
> On Tue, 16 Jul 2013 23:19:21 +1000, Steven D'Aprano <steve at pearwood.info> wrote:
>> On 16/07/13 20:28, Richard Oudkerk wrote:
>> > On 16/07/2013 6:44am, Nick Coghlan wrote:
>> >> Clarifying what constitutes an internal interface in a way that
>> >> doesn't require renaming anything is a necessary prerequisite for
>> >> bundling or bootstrapping the pip CLI in Python 3.4 (as pip exposes
>> >> its internal implemetnation API as "import pip" rather than "import
>> >> _pip" and renaming it would lead to a lot of pointless code churn).
>> >> Without that concern, the topic never would have come up.
>> >
>> > BTW, how does the use of __all__ effect things?  Somewhere I got the idea that if a module uses __all__ then anything not listed is internal.  I take it that is wrong?
>>
>>
>> That is not how I interpret __all__. In the absence of any explicit documentation, I interpret __all__ as nothing more than a list of names which wildcard imports will bring in, without necessarily meaning that other names are private. For example, I might have a module explicitly designed for wildcard imports at the interactive interpreter:
>>
>> from module import *
>>
>> brings in the functions which I expect will be useful interactively, not necessarily the entire public API.
>>
>> For example, pkgutil includes classes with single-underscore methods, which I take as private. It also has a function simplegeneric, which is undocumented and not listed in __all__. In in the absence of even a comment saying "Don't use this", I take it as an oversight, not policy that simplegeneric is private.
>
> I think you'd be wrong about that, though.  simplegeneric should really be
> treated as private.  I'm speaking here not about the general principle of
> the thing, but about my understanding of simplegeneric's specific history.

And, indeed, you're correct (the issue that eventually became the
functools.singledispatch PEP started life with a title like "move
simplegeneric to functools and make it public").

For the general case, the patch I posted to the issue tracker sets the
precedence as docs -> __all__ -> leading underscore. If a module has
public APIs that aren't in __all__, it should cover them in the docs,
otherwise people should assume they're private. It's OK if there are
exceptions to that general rule - there's a reason PEP 8 has the great
big qualifier pointing out that it isn't universally applicable (even
if we might sometimes wish otherwise).

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From barry at python.org  Tue Jul 16 16:48:55 2013
From: barry at python.org (Barry Warsaw)
Date: Tue, 16 Jul 2013 10:48:55 -0400
Subject: [Python-Dev] Why does PEP 8 advise against explicit relative
 imports?
In-Reply-To: <CAPdQG2rJMdC=QfOcf9gGdiFiLwEskf90v8HUBhEsR8y+W47JcQ@mail.gmail.com>
References: <CADiSq7cC=GYwkx+viaNvNB+wMS4uU-XdSqH5F9vp8SagKkaTeg@mail.gmail.com>
	<CAPdQG2rJMdC=QfOcf9gGdiFiLwEskf90v8HUBhEsR8y+W47JcQ@mail.gmail.com>
Message-ID: <20130716104855.4c2cbecb@anarchist>

On Jul 16, 2013, at 02:02 PM, Thomas Wouters wrote:

>In my experience changing the name of a package is problematic for more
>reasons than just 'import'. We used to do the put-in-a-package thing for
>third-party packages at Google, and we quit doing it because it was just
>too painful: it's hard to find all the places that reference the package by
>name (think of __import__, or mucking in sys.modules), and it's extra
>problematic with packages that contain extension modules. You can't rename
>a package, even for 'vendoring a package', without carefully checking
>whether it'll work -- unless you don't care if it'll work, of course :) The
>alternative is to include a package *without* changing its name (by adding
>the right directory, without __init__.py, to sys.path) and that has worked
>out a lot better for us.

Vendoring also gives no end of headaches to distro packagers because we have
to un-vendor that stuff, which can lead to a mess of patches on top of
upstream.

>Even so, there's no reason for *the standard library* to use explicit
>relative imports, and that's what PEP 8 is supposed to cover, right? :)

Right!  It doesn't bother me at all that this particular recommendation may be
followed by third parties too. :)

-Barry

From barry at python.org  Tue Jul 16 17:14:03 2013
From: barry at python.org (Barry Warsaw)
Date: Tue, 16 Jul 2013 11:14:03 -0400
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
 underscores
In-Reply-To: <1373943762.30382.88.camel@thinko>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
	<ks08fk$vnu$1@ger.gmane.org>
	<CADiSq7fAS+-keOjBmOEs-UvV=bV4qB9=ikiN5dkvRW7LSrge6g@mail.gmail.com>
	<20130715184039.60590e62@anarchist>
	<1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info>
	<1373941235.30382.62.camel@thinko>
	<CADiSq7cr8JrYJHtfxjgzdHjiJjC-ipcm98ejKTsTAHL6SPMPyg@mail.gmail.com>
	<1373943762.30382.88.camel@thinko>
Message-ID: <20130716111403.0afd68ac@anarchist>

On Jul 15, 2013, at 11:02 PM, Chris McDonough wrote:

>Given that misunderstanding, is there a way to divorce stdlib-centric
>guidelines like the one being discussed now from "PEP8"-ness?  I don't
>think any amount of marketing effort or reasoned explanation is going to
>separate "PEP8" from "correct code" in people's minds at this point.

The standard seems to be "if the pep8 program complains, I forward that
upstream", so it's probably most effective to discuss with them.  OTOH, we've
had to clarify PEP 8 in order to get the pep8 developers to adjust their tool,
most recently e.g. the closing brace/bracket/paren for multiline collections.

-Barry

From barry at python.org  Tue Jul 16 17:19:17 2013
From: barry at python.org (Barry Warsaw)
Date: Tue, 16 Jul 2013 11:19:17 -0400
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
 underscores
In-Reply-To: <ks3788$ere$1@ger.gmane.org>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
	<ks08fk$vnu$1@ger.gmane.org>
	<CADiSq7fAS+-keOjBmOEs-UvV=bV4qB9=ikiN5dkvRW7LSrge6g@mail.gmail.com>
	<20130715184039.60590e62@anarchist>
	<1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info>
	<1373941235.30382.62.camel@thinko>
	<CADiSq7cr8JrYJHtfxjgzdHjiJjC-ipcm98ejKTsTAHL6SPMPyg@mail.gmail.com>
	<1373943762.30382.88.camel@thinko>
	<CADiSq7ck152x2_dPJ8TET6mJbk3maO9-q2FU5rt6pAaA_G0+fQ@mail.gmail.com>
	<1373944606.30382.92.camel@thinko>
	<CADiSq7fxi8rMbcdVN_SsUycjNWmB_61KQomcowV+m8m9+Qp79A@mail.gmail.com>
	<ks3788$ere$1@ger.gmane.org>
Message-ID: <20130716111917.654718b0@anarchist>

On Jul 16, 2013, at 11:28 AM, Richard Oudkerk wrote:

>BTW, how does the use of __all__ effect things?  Somewhere I got the idea
>that if a module uses __all__ then anything not listed is internal.  I take
>it that is wrong?

Purely technically, __all__ is there to affect how from-import-* works.  I
personally think it's a good idea to include all your public names, and none
of your non-public names, in __all__, but it's not always easy to keep
up-to-date.  pyflakes has the nice benefit of complaining when something is
named in __all__ that doesn't exist in the module, but that's only one part of
the keeping-things-up-to-date problem.

-Barry

From barry at python.org  Tue Jul 16 17:21:14 2013
From: barry at python.org (Barry Warsaw)
Date: Tue, 16 Jul 2013 11:21:14 -0400
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
 underscores
In-Reply-To: <CADiSq7cUCf9KRrLGK1C-x8DWzj8+yHL30rWCRzm0KQarjwY_+A@mail.gmail.com>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
	<ks08fk$vnu$1@ger.gmane.org>
	<CADiSq7fAS+-keOjBmOEs-UvV=bV4qB9=ikiN5dkvRW7LSrge6g@mail.gmail.com>
	<20130715184039.60590e62@anarchist>
	<1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info>
	<1373941235.30382.62.camel@thinko>
	<CADiSq7cr8JrYJHtfxjgzdHjiJjC-ipcm98ejKTsTAHL6SPMPyg@mail.gmail.com>
	<1373943762.30382.88.camel@thinko>
	<CADiSq7ck152x2_dPJ8TET6mJbk3maO9-q2FU5rt6pAaA_G0+fQ@mail.gmail.com>
	<1373944606.30382.92.camel@thinko>
	<CADiSq7fxi8rMbcdVN_SsUycjNWmB_61KQomcowV+m8m9+Qp79A@mail.gmail.com>
	<ks3788$ere$1@ger.gmane.org> <51E54859.4070901@pearwood.info>
	<20130716133912.C532325007E@webabinitio.net>
	<CADiSq7cUCf9KRrLGK1C-x8DWzj8+yHL30rWCRzm0KQarjwY_+A@mail.gmail.com>
Message-ID: <20130716112114.3f04711b@anarchist>

On Jul 16, 2013, at 11:48 PM, Nick Coghlan wrote:

>For the general case, the patch I posted to the issue tracker sets the
>precedence as docs -> __all__ -> leading underscore.

+1

-Barry

From barry at python.org  Tue Jul 16 17:26:55 2013
From: barry at python.org (Barry Warsaw)
Date: Tue, 16 Jul 2013 11:26:55 -0400
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
 underscores
In-Reply-To: <CADiSq7e1SFWMY-YG6FJMq-VG1M2e-Gv1RmPEySbcRKVqtUXwNA@mail.gmail.com>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
	<ks08fk$vnu$1@ger.gmane.org>
	<CADiSq7fAS+-keOjBmOEs-UvV=bV4qB9=ikiN5dkvRW7LSrge6g@mail.gmail.com>
	<20130715184039.60590e62@anarchist>
	<1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info>
	<1373941235.30382.62.camel@thinko>
	<CADiSq7cr8JrYJHtfxjgzdHjiJjC-ipcm98ejKTsTAHL6SPMPyg@mail.gmail.com>
	<1373943762.30382.88.camel@thinko>
	<CADiSq7ck152x2_dPJ8TET6mJbk3maO9-q2FU5rt6pAaA_G0+fQ@mail.gmail.com>
	<1373944606.30382.92.camel@thinko>
	<CADiSq7fxi8rMbcdVN_SsUycjNWmB_61KQomcowV+m8m9+Qp79A@mail.gmail.com>
	<1373954482.30382.111.camel@thinko>
	<CADiSq7e1SFWMY-YG6FJMq-VG1M2e-Gv1RmPEySbcRKVqtUXwNA@mail.gmail.com>
Message-ID: <20130716112655.42281637@anarchist>

On Jul 16, 2013, at 04:10 PM, Nick Coghlan wrote:

>Yeah, we should probably be a bit more vocal in pointing out that PEP
>8 is something to be applied judiciously, *not* treated as "sacred
>writ, never to be questioned". It's very tempting to just sigh and
>ignore it, instead of explicitly pointing out the problems with taking
>it too literally for people's personal projects (or, worse, using it
>to abuse developers of projects that long predate the recommendations
>in more recent versions) :P

In my experience, most PEP 8 violation bugs that I see for various upstreams
(mine and others) comes from running the pep8 tool over the code.  A lot of
upstreams do this as part of their normal QA (e.g. you can set up unittests
that run pep8, pyflakes, and other linters).  Those will generally be fine
because they're already committed to conformance to pep8's notion of what PEP
8 says <wink>[1].

In other cases, Someone Out There runs pep8 over your code, and files bugs on
all the complaints that get produced.  That can definitely lead to churn and
waste, especially on big, established code bases.

-Barry

[1] which I do not always agree with anyway, as evidenced by the recent
discussion on closing paren indentation for multi-line collections.

From guido at python.org  Tue Jul 16 19:21:36 2013
From: guido at python.org (Guido van Rossum)
Date: Tue, 16 Jul 2013 10:21:36 -0700
Subject: [Python-Dev] Misc re.match() complaint
In-Reply-To: <20130716095551.5aca0285@pitrou.net>
References: <CAP7+vJKdLzJ3BOgD9e=Nw8KT0v4apdgutu7nKLPAgtEGCapA5A@mail.gmail.com>
	<CAGE7PNJs5qhUiWhv46HBzwpPHsk6Mt_N3+h3XSL2CoNOg7KrYw@mail.gmail.com>
	<51E48F8B.1090905@mrabarnett.plus.com>
	<CAP7+vJKRuKL6vsPRmvo-+9gPtB=usp57F31_1H--sVN0_qi6GQ@mail.gmail.com>
	<87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAP7+vJ+WgZSed2zwiCgBpNDVcr1urvvctv8UK705dJ6iEBGQfg@mail.gmail.com>
	<CADiSq7dTdxBbEqtsV-fdWab5O90JdvnEZRLKXZY9tO86iArPSA@mail.gmail.com>
	<CAP7+vJ+WCtJMjGX480QYB4-LJ2TtQ4jS8ee=H7SP4VsyuzvCpg@mail.gmail.com>
	<20130716095551.5aca0285@pitrou.net>
Message-ID: <CAP7+vJKEAy+Xz7g2xz2SuJH52a9HtQdbMbL5HuH_8dTi_g-aAA@mail.gmail.com>

On Tue, Jul 16, 2013 at 12:55 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Is there a strong enough use case to change it? I can't say the current
> behaviour seems very useful either, but some people may depend on it.

This is the crucial question. I personally see the current behavior as
an artifact of the (lack of) design process, not as a conscious
decision. Given that we also have m.string, m.start(grp) and
m.end(grp), those who need something matching the original type (or
even something that is known to be a reference into the original
object) can use that API; for most use cases, all you care about is is
the selected group as a string, and it is more useful if that is
always an immutable string (bytes or str).

The situation is most egregious if the target string is a bytearray,
where there is currently no way to get the result as an immutable
bytes object without an extra copy. (There's no API that lets you
create a bytes object directly from a slice of a bytearray.)

In terms of backwards compatibility, I wouldn't want to do this in a
bugfix release, but for a feature release I think it's fine -- the
number of applications that could be bitten by this must be extremely
small (and the work-around is backward-compatible: just use
m.string[m.start() : m.stop()]).

> I already find it a bit weird that you're passing a bytearray or
> memoryview to re.match(), to be honest :-)

Yes, this is somewhat of an odd corner, but actually most built-in
APIs taking bytes also take anything else that can be coerced to bytes
(io.open() seems to be the exception, and it feels like an accident --
os.open() *does* accept bytearray and friends). This is quite useful
for code that interacts with C code or system calls -- often you have
a large buffer shared between C and Python code for efficiency, and
being able to do pretty much anything to the buffer that you can do to
a bytes object (apart from using it as a dict key) helps a lot.

-- 
--Guido van Rossum (python.org/~guido)

From tjreedy at udel.edu  Tue Jul 16 23:27:37 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 16 Jul 2013 17:27:37 -0400
Subject: [Python-Dev] Why does PEP 8 advise against explicit relative
	imports?
In-Reply-To: <CADiSq7cC=GYwkx+viaNvNB+wMS4uU-XdSqH5F9vp8SagKkaTeg@mail.gmail.com>
References: <CADiSq7cC=GYwkx+viaNvNB+wMS4uU-XdSqH5F9vp8SagKkaTeg@mail.gmail.com>
Message-ID: <ks4ds6$jg4$1@ger.gmane.org>

On 7/16/2013 7:40 AM, Nick Coghlan wrote:
> PEP 8 advises developers to use absolute imports rather than explicit
> relative imports.
>
> Why? Using absolute imports couple the internal implementation of a
> package to its public name - you can't just change the top level
> directory name any more, you have to go through and change all the
> absolute imports as well. You also can't easily vendor a package that
> uses absolute imports inside another project either, since all the
> absolute imports will break.
>
> What's the *concrete* benefit of using absolute imports that overcomes
> these significant shortcomings in maintainability and composability?

 From my viewpoint, absolute imports always work, on all python 
versions, whereas people have reported various problems with relative 
imports on python-list. At least some of the complaints were valid, at 
least at the time, but I have not paid too much attention.

Even if idlelib files used relative imports, 'idlelib' cannot change and 
I see no reason to ever change 'idle_test'. I would like to change 
camelCaseFileNames.py, but if ever done, that would affect both types of 
imports.

-- 
Terry Jan Reedy


From tjreedy at udel.edu  Tue Jul 16 23:46:34 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 16 Jul 2013 17:46:34 -0400
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <20130716133912.C532325007E@webabinitio.net>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
	<ks08fk$vnu$1@ger.gmane.org>
	<CADiSq7fAS+-keOjBmOEs-UvV=bV4qB9=ikiN5dkvRW7LSrge6g@mail.gmail.com>
	<20130715184039.60590e62@anarchist>
	<1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info>
	<1373941235.30382.62.camel@thinko>
	<CADiSq7cr8JrYJHtfxjgzdHjiJjC-ipcm98ejKTsTAHL6SPMPyg@mail.gmail.com>
	<1373943762.30382.88.camel@thinko>
	<CADiSq7ck152x2_dPJ8TET6mJbk3maO9-q2FU5rt6pAaA_G0+fQ@mail.gmail.com>
	<1373944606.30382.92.camel@thinko>
	<CADiSq7fxi8rMbcdVN_SsUycjNWmB_61KQomcowV+m8m9+Qp79A@mail.gmail.com>
	<ks3788$ere$1@ger.gmane.org> <51E54859.4070901@pearwood.info>
	<20130716133912.C532325007E@webabinitio.net>
Message-ID: <ks4evn$urm$1@ger.gmane.org>

On 7/16/2013 9:39 AM, R. David Murray wrote:
> On Tue, 16 Jul 2013 23:19:21 +1000, Steven D'Aprano <steve at pearwood.info> wrote:

>> For example, pkgutil includes classes with single-underscore methods, which I take as private. It also has a function simplegeneric, which is undocumented and not listed in __all__. In in the absence of even a comment saying "Don't use this", I take it as an oversight, not policy that simplegeneric is private.
>
> I think you'd be wrong about that, though.  simplegeneric should really be
> treated as private.  I'm speaking here not about the general principle of
> the thing, but about my understanding of simplegeneric's specific history.

I think Steven (valid) point is "Why not, then, say it is internal 
either in docs or name?"-- which in this case would be in the docs.


-- 
Terry Jan Reedy


From storchaka at gmail.com  Tue Jul 16 23:56:00 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Wed, 17 Jul 2013 00:56:00 +0300
Subject: [Python-Dev] cpython: Issue #18408: Fix fileio_read() on
 _PyBytes_Resize() failure
In-Reply-To: <3bvvP70WNXz7Ljg@mail.python.org>
References: <3bvvP70WNXz7Ljg@mail.python.org>
Message-ID: <ks4fh9$3lv$1@ger.gmane.org>

17.07.13 00:09, victor.stinner ???????(??):
> http://hg.python.org/cpython/rev/533eb9ab895a
> changeset:   84669:533eb9ab895a
> user:        Victor Stinner <victor.stinner at gmail.com>
> date:        Tue Jul 16 21:36:02 2013 +0200
> summary:
>    Issue #18408: Fix fileio_read() on _PyBytes_Resize() failure
>
> bytes is NULL on _PyBytes_Resize() failure
>
> files:
>    Modules/_io/fileio.c |  2 +-
>    1 files changed, 1 insertions(+), 1 deletions(-)
>
>
> diff --git a/Modules/_io/fileio.c b/Modules/_io/fileio.c
> --- a/Modules/_io/fileio.c
> +++ b/Modules/_io/fileio.c
> @@ -739,7 +739,7 @@
>
>       if (n != size) {
>           if (_PyBytes_Resize(&bytes, n) < 0) {
> -            Py_DECREF(bytes);
> +            Py_CLEAR(bytes);
>               return NULL;
>           }
>       }

Why not Py_DECREF?



From victor.stinner at gmail.com  Wed Jul 17 00:03:14 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 17 Jul 2013 00:03:14 +0200
Subject: [Python-Dev] cpython: Issue #18408: Fix fileio_read() on
 _PyBytes_Resize() failure
In-Reply-To: <ks4fh9$3lv$1@ger.gmane.org>
References: <3bvvP70WNXz7Ljg@mail.python.org> <ks4fh9$3lv$1@ger.gmane.org>
Message-ID: <CAMpsgwa46sUPMFZYacwjSGDTuU5LYoHXSfcoirm5gZUqS0Vd3A@mail.gmail.com>

2013/7/16 Serhiy Storchaka <storchaka at gmail.com>:
>> http://hg.python.org/cpython/rev/533eb9ab895a
>> summary:
>>    Issue #18408: Fix fileio_read() on _PyBytes_Resize() failure
>>
>> bytes is NULL on _PyBytes_Resize() failure
>
> Why not Py_DECREF?

Because Py_DECREF(NULL) does crash.

Victor

From storchaka at gmail.com  Wed Jul 17 00:25:15 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Wed, 17 Jul 2013 01:25:15 +0300
Subject: [Python-Dev] cpython: Issue #18408: Fix fileio_read() on
 _PyBytes_Resize() failure
In-Reply-To: <CAMpsgwa46sUPMFZYacwjSGDTuU5LYoHXSfcoirm5gZUqS0Vd3A@mail.gmail.com>
References: <3bvvP70WNXz7Ljg@mail.python.org> <ks4fh9$3lv$1@ger.gmane.org>
	<CAMpsgwa46sUPMFZYacwjSGDTuU5LYoHXSfcoirm5gZUqS0Vd3A@mail.gmail.com>
Message-ID: <ks4h9l$m2r$1@ger.gmane.org>

17.07.13 01:03, Victor Stinner ???????(??):
> 2013/7/16 Serhiy Storchaka <storchaka at gmail.com>:
>>> http://hg.python.org/cpython/rev/533eb9ab895a
>>> summary:
>>>     Issue #18408: Fix fileio_read() on _PyBytes_Resize() failure
>>>
>>> bytes is NULL on _PyBytes_Resize() failure
>>
>> Why not Py_DECREF?
>
> Because Py_DECREF(NULL) does crash.

Oh, I meaned Py_XDECREF.


From christian at python.org  Wed Jul 17 00:29:05 2013
From: christian at python.org (Christian Heimes)
Date: Wed, 17 Jul 2013 00:29:05 +0200
Subject: [Python-Dev] cpython: Issue #18408: Fix fileio_read() on
 _PyBytes_Resize() failure
In-Reply-To: <CAMpsgwa46sUPMFZYacwjSGDTuU5LYoHXSfcoirm5gZUqS0Vd3A@mail.gmail.com>
References: <3bvvP70WNXz7Ljg@mail.python.org> <ks4fh9$3lv$1@ger.gmane.org>
	<CAMpsgwa46sUPMFZYacwjSGDTuU5LYoHXSfcoirm5gZUqS0Vd3A@mail.gmail.com>
Message-ID: <ks4hfa$ocr$1@ger.gmane.org>

Am 17.07.2013 00:03, schrieb Victor Stinner:
> 2013/7/16 Serhiy Storchaka <storchaka at gmail.com>:
>>> http://hg.python.org/cpython/rev/533eb9ab895a
>>> summary:
>>>    Issue #18408: Fix fileio_read() on _PyBytes_Resize() failure
>>>
>>> bytes is NULL on _PyBytes_Resize() failure
>>
>> Why not Py_DECREF?
> 
> Because Py_DECREF(NULL) does crash.

Why not Py_XDECREF() then?



From victor.stinner at gmail.com  Wed Jul 17 00:35:17 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 17 Jul 2013 00:35:17 +0200
Subject: [Python-Dev] cpython: Issue #18408: Fix fileio_read() on
 _PyBytes_Resize() failure
In-Reply-To: <ks4h9l$m2r$1@ger.gmane.org>
References: <3bvvP70WNXz7Ljg@mail.python.org> <ks4fh9$3lv$1@ger.gmane.org>
	<CAMpsgwa46sUPMFZYacwjSGDTuU5LYoHXSfcoirm5gZUqS0Vd3A@mail.gmail.com>
	<ks4h9l$m2r$1@ger.gmane.org>
Message-ID: <CAMpsgwb-yOcjZGcWdGRe9oEv4tSBUUcaOU5kM33RNPnTbXDk=w@mail.gmail.com>

2013/7/17 Serhiy Storchaka <storchaka at gmail.com>:
> Oh, I meaned Py_XDECREF.

Ah ok :-) Well, for this specific code, it can probably be replaced with:

         if (_PyBytes_Resize(&bytes, n) < 0)
             return NULL;

I'm not sure that _PyBytes_Resize() *always* decref bytes and then set
bytes to NULL. I was too lazy to check this.

If someone wants to audit the code of _PyBytes_Resize(), there are
many other places where Py_XDECREF / Py_CLEAR can be removed on
_PyBytes_Resize() failure.

And I'm quite sure that there are still places where Py_DECREF() is
still used on _PyBytes_Resize() failure ;-)

Victor

From victor.stinner at gmail.com  Wed Jul 17 01:08:31 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 17 Jul 2013 01:08:31 +0200
Subject: [Python-Dev] Is it safe to call PyEval_EvalFrameEx() with an
	exception set?
In-Reply-To: <CAMpsgwbsDHwGhyAmaxwzhPzXtW8cQ6ifxHD2RSe2Swrpws7APA@mail.gmail.com>
References: <CAMpsgwbsDHwGhyAmaxwzhPzXtW8cQ6ifxHD2RSe2Swrpws7APA@mail.gmail.com>
Message-ID: <CAMpsgwZyFmO47NW+eG2Hwni-XC2SF+7Es7eboVQJrey9qsTMyA@mail.gmail.com>

2013/7/16 Victor Stinner <victor.stinner at gmail.com>:
> I would like to add assertions in Python/ceval.c to detect such bugs
> earlier. The problem is that some functions rely on the ability to
> call PyEval_EvalFrameEx() with an exception set. Is it expected?
> Should it be avoided? The current exception can be replaced with a new
> exception.

Another example. Py_ReprLeave() function can be called while an
exception is set. I'm not sure that it is "safe" to call new code
while an exception is set.

The function can raise a new exception. For example, PyList_SetSlice()
can fail because of a MemoryError.

I modified Py_ReprLeave() to save and then restore the current exception:
http://hg.python.org/cpython/rev/28ff7ac91477

If Py_ReprLeave() raises a new exception, it is clears (replaced by
the previous exception context).

Victor

From tseaver at palladion.com  Wed Jul 17 01:34:03 2013
From: tseaver at palladion.com (Tres Seaver)
Date: Tue, 16 Jul 2013 19:34:03 -0400
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <CADiSq7eRjyXK0VEHj7cBfpJj4Z4QiMxdX5tJr4nfbODatzGVXw@mail.gmail.com>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
	<ks08fk$vnu$1@ger.gmane.org>
	<CADiSq7fAS+-keOjBmOEs-UvV=bV4qB9=ikiN5dkvRW7LSrge6g@mail.gmail.com>
	<20130715184039.60590e62@anarchist>
	<1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info>
	<1373941235.30382.62.camel@thinko>
	<CADiSq7cr8JrYJHtfxjgzdHjiJjC-ipcm98ejKTsTAHL6SPMPyg@mail.gmail.com>
	<1373943762.30382.88.camel@thinko>
	<CADiSq7ck152x2_dPJ8TET6mJbk3maO9-q2FU5rt6pAaA_G0+fQ@mail.gmail.com>
	<ks310f$90g$1@ger.gmane.org>
	<CADiSq7eRjyXK0VEHj7cBfpJj4Z4QiMxdX5tJr4nfbODatzGVXw@mail.gmail.com>
Message-ID: <ks4l94$fqf$2@ger.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 07/16/2013 07:09 AM, Nick Coghlan wrote:

> I did find it interesting that we *don't* explicitly advise against 
> the use of "import *" for anything other than optional accelerator 
> modules or republishing internal interfaces through a public API,
> even though we advice against the practice in the tutorial. Perhaps
> another oversight worth correcting?

+1.  'from foo import *' (from any source other than another module in
the same package) is a code stench far worse than anything else PEP8
proscribes.


Tres.
- -- 
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlHl2GsACgkQ+gerLs4ltQ6+1QCgmu5k6p5otzPxvzGh5mA1Ch7t
2f0AoK2a0/m144bnIwBFLLuY9l2bdWMN
=878p
-----END PGP SIGNATURE-----


From barry at python.org  Wed Jul 17 03:07:33 2013
From: barry at python.org (Barry Warsaw)
Date: Tue, 16 Jul 2013 21:07:33 -0400
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
 underscores
In-Reply-To: <ks4l94$fqf$2@ger.gmane.org>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
	<ks08fk$vnu$1@ger.gmane.org>
	<CADiSq7fAS+-keOjBmOEs-UvV=bV4qB9=ikiN5dkvRW7LSrge6g@mail.gmail.com>
	<20130715184039.60590e62@anarchist>
	<1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info>
	<1373941235.30382.62.camel@thinko>
	<CADiSq7cr8JrYJHtfxjgzdHjiJjC-ipcm98ejKTsTAHL6SPMPyg@mail.gmail.com>
	<1373943762.30382.88.camel@thinko>
	<CADiSq7ck152x2_dPJ8TET6mJbk3maO9-q2FU5rt6pAaA_G0+fQ@mail.gmail.com>
	<ks310f$90g$1@ger.gmane.org>
	<CADiSq7eRjyXK0VEHj7cBfpJj4Z4QiMxdX5tJr4nfbODatzGVXw@mail.gmail.com>
	<ks4l94$fqf$2@ger.gmane.org>
Message-ID: <20130716210733.2ccd5192@limelight.wooz.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Jul 16, 2013, at 07:34 PM, Tres Seaver wrote:

>On 07/16/2013 07:09 AM, Nick Coghlan wrote:
>
>> I did find it interesting that we *don't* explicitly advise against 
>> the use of "import *" for anything other than optional accelerator 
>> modules or republishing internal interfaces through a public API,
>> even though we advice against the practice in the tutorial. Perhaps
>> another oversight worth correcting?
>
>+1.  'from foo import *' (from any source other than another module in
>the same package) is a code stench far worse than anything else PEP8
>proscribes.

Maybe we should disable it for anything that isn't at the interactive prompt
<0.5 wink>.

- -Barry
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.20 (GNU/Linux)

iQIcBAEBCAAGBQJR5e5VAAoJEBJutWOnSwa/SK4P/22KvWNRQ7sjDWJ+jebgpXyD
mkUeO4r905uqp5Bvedlyj4GoFhdFPQoeQEGgQeNwiMkFxCGJDKiY1XoJAgqj2zOE
vUgoyLE5YSAnz8xG/Ib3ZZPXhDJY14gu/czbX/cc83Q+cQJd0+RpnhUt5jQkxAYR
k6xXF7hwTHaTYlqzdTDQUtimg21AUb0lgI5XqCMGL4aac4H69JiN+3ydI3BQxqxY
MXe4h4rzDxnaBnekHYCouM1jsN6CsLeIHafJR8W4Zgm9vuxZYIha+kAQd+tILFXG
d/0hpvLIGP+ubNPIvS1Ay+aYsH7PiL4keCoP0pi+llMTQv+W5wDeCumLpvXjaEFj
46c7HaRaxbBqpactsG98rZOtmBwSgEwzzPSQak+sWHDCX4RFOS3qDh3WhtY0Lw7n
NdEkJ0f+6bEt2ot+emDpth/SAxgOEUjNOhZ8/glYtvZcz2wBWRGkCAIEwSJtP1VZ
YgRHFsoUUbS6KoI9cZWhwynmIVQf4Wn9a2Zx5/YUbHIM2mY1s60MxcOn3KgPpEU4
Zjp/gyPHsoIORrZU2Q2pwzdWEmSzorZBP/NqRqlhwBTexKdfqyRFztsFMVcnLPDv
Cl0t04BGXH8/0fZ+peEOccqnuPsHumr9VqmhMCcLChRPRQyBD9ETSPVwHhprdtka
qkLcyWl/M3ijyuAkdYbl
=Q+Du
-----END PGP SIGNATURE-----

From steve at pearwood.info  Wed Jul 17 03:55:42 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 17 Jul 2013 11:55:42 +1000
Subject: [Python-Dev] Why does PEP 8 advise against explicit relative
 imports?
In-Reply-To: <CAPdQG2rJMdC=QfOcf9gGdiFiLwEskf90v8HUBhEsR8y+W47JcQ@mail.gmail.com>
References: <CADiSq7cC=GYwkx+viaNvNB+wMS4uU-XdSqH5F9vp8SagKkaTeg@mail.gmail.com>
	<CAPdQG2rJMdC=QfOcf9gGdiFiLwEskf90v8HUBhEsR8y+W47JcQ@mail.gmail.com>
Message-ID: <51E5F99E.4080301@pearwood.info>

On 16/07/13 22:02, Thomas Wouters wrote:
> On Tue, Jul 16, 2013 at 1:40 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>> PEP 8 advises developers to use absolute imports rather than explicit
>> relative imports.
>>
>> Why? Using absolute imports couple the internal implementation of a
>> package to its public name - you can't just change the top level
>> directory name any more, you have to go through and change all the
>> absolute imports as well. You also can't easily vendor a package that
>> uses absolute imports inside another project either, since all the
>> absolute imports will break.
>>
>
> The problem with relative imports (both explicit and implicit) is that it
> makes it less obvious when you are importing a module under the wrong name.
> If a package ends up in sys.path directly (for example, by executing
> something that lives inside it directly) then an explicit relative import
> directly in the package will fail, but an explicit relative import in a
> sub-package won't, and you can end up with the subtly confusing mess of
> multiple imports of the same .py file under different names.

I don't understand this objection. What do you mean by "importing a module under the wrong name"?

As I see it, if relative imports are a bad idea for the std lib, they're a bad idea for everyone. (What am I missing?) So why do we have relative imports? Was it a mistake? I don't think so.

It seems to me that you're giving some fairly obscure edge cases, if I've understood you correctly. If adding a package directory to sys.path breaks relative imports, isn't that an argument against adding the package directory to sys.path rather than relative imports?


>> What's the *concrete* benefit of using absolute imports that overcomes
>> these significant shortcomings in maintainability and composability?
>
> In my experience changing the name of a package is problematic for more
> reasons than just 'import'. We used to do the put-in-a-package thing for
> third-party packages at Google, and we quit doing it because it was just
> too painful: it's hard to find all the places that reference the package by
> name (think of __import__, or mucking in sys.modules),

Again, these are some fairly odd edge cases. I don't mean to imply that they never occur in practice, obviously they do, but perhaps you're overly sensitive to them. It seems strange to me that PEP 8 should recommend against using relative imports for the 49 packages that would benefit from it just to avoid problems with the 1 that does weird things with __import__ or manually mucks about with sys.modules.

Have I missed something?


> Even so, there's no reason for *the standard library* to use explicit
> relative imports, and that's what PEP 8 is supposed to cover, right? :)

If a third-party package uses relative imports, are you suggesting that those rel imports should be turned into absolute imports as a precondition of it being moved into the std lib?



-- 
Steven

From guido at python.org  Wed Jul 17 04:15:45 2013
From: guido at python.org (Guido van Rossum)
Date: Tue, 16 Jul 2013 19:15:45 -0700
Subject: [Python-Dev] Why does PEP 8 advise against explicit relative
	imports?
In-Reply-To: <51E5F99E.4080301@pearwood.info>
References: <CADiSq7cC=GYwkx+viaNvNB+wMS4uU-XdSqH5F9vp8SagKkaTeg@mail.gmail.com>
	<CAPdQG2rJMdC=QfOcf9gGdiFiLwEskf90v8HUBhEsR8y+W47JcQ@mail.gmail.com>
	<51E5F99E.4080301@pearwood.info>
Message-ID: <CAP7+vJJE0McMuLV+E8LPfyOzYukKR04j=nzGbY0uMcs46y8p1Q@mail.gmail.com>

I have to say that a long time ago, when you couldn't tell a relative
import from a top-level import, the ban on relative imports made more
sense.

-- 
--Guido van Rossum (python.org/~guido)

From ncoghlan at gmail.com  Wed Jul 17 04:38:54 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 17 Jul 2013 12:38:54 +1000
Subject: [Python-Dev] Why does PEP 8 advise against explicit relative
	imports?
In-Reply-To: <51E5F99E.4080301@pearwood.info>
References: <CADiSq7cC=GYwkx+viaNvNB+wMS4uU-XdSqH5F9vp8SagKkaTeg@mail.gmail.com>
	<CAPdQG2rJMdC=QfOcf9gGdiFiLwEskf90v8HUBhEsR8y+W47JcQ@mail.gmail.com>
	<51E5F99E.4080301@pearwood.info>
Message-ID: <CADiSq7ftby-gzzpMBzHfZ80+WqHrK7Ko86jVDSzpsyGuFQuhtw@mail.gmail.com>

On 17 July 2013 11:55, Steven D'Aprano <steve at pearwood.info> wrote:
> On 16/07/13 22:02, Thomas Wouters wrote:
> As I see it, if relative imports are a bad idea for the std lib, they're a
> bad idea for everyone. (What am I missing?) So why do we have relative
> imports? Was it a mistake? I don't think so.
>
> It seems to me that you're giving some fairly obscure edge cases, if I've
> understood you correctly. If adding a package directory to sys.path breaks
> relative imports, isn't that an argument against adding the package
> directory to sys.path rather than relative imports?

What Thomas is saying is that, given the current state of the import
system, using absolute imports usually give you better behaviour (or
at least better error messages) than explicit relative imports do when
things go wrong. At present, there are still too many ways to get your
import configuration into a bad state (most notably running a module
inside a package by filepath rather than the -m switch), and if that
happens, using absolute imports is likely to make it a bit easier to
find your way out again.

I actually agree with that rationale - while I prefer explicit
relative imports myself, I'm also more experienced than most when it
comes to identifying and resolving import configuration problems (see
http://python-notes.curiousefficiency.org/en/latest/python_concepts/import_traps.html)

So, for now, avoiding even explicit relative imports is still a good
default position. People who feel qualified to reject PEP 8's advice
on this topic should also know enough to deal with the potential
consequences. I'll be proposing a rewording of this admonition to give
a better rationale, but I won't be proposing that the recommendation
be changed :)

Once we get sys.path initialisation to a better place (amongst other
things), then it will be time to revisit the recommendation itself.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From guido at python.org  Wed Jul 17 05:48:03 2013
From: guido at python.org (Guido van Rossum)
Date: Tue, 16 Jul 2013 20:48:03 -0700
Subject: [Python-Dev] Why does PEP 8 advise against explicit relative
	imports?
In-Reply-To: <CADiSq7ftby-gzzpMBzHfZ80+WqHrK7Ko86jVDSzpsyGuFQuhtw@mail.gmail.com>
References: <CADiSq7cC=GYwkx+viaNvNB+wMS4uU-XdSqH5F9vp8SagKkaTeg@mail.gmail.com>
	<CAPdQG2rJMdC=QfOcf9gGdiFiLwEskf90v8HUBhEsR8y+W47JcQ@mail.gmail.com>
	<51E5F99E.4080301@pearwood.info>
	<CADiSq7ftby-gzzpMBzHfZ80+WqHrK7Ko86jVDSzpsyGuFQuhtw@mail.gmail.com>
Message-ID: <CAP7+vJ+3AuxMLWdZGLxon+Y0fOEwXrc98cUX86vBs-E4nFQqDw@mail.gmail.com>

If sounds like the main problem with relative imports (even explicit)
is caused by ignoring the package structure and running modules inside
a package as a script, without using -m.

Maybe we should update the PEP to make this abundantly clear?

I note that an additional problem with informational PEPs like PEP 8
is that they aren't tied to a particular Python version, so even if we
solved the above issue perfectly in Python 3.4 there'd still be tons
of people applying PEP 8 to older code who would benefit from the
warning against relative imports. So maybe we should also add that to
the PEP.

-- 
--Guido van Rossum (python.org/~guido)

From stephen at xemacs.org  Wed Jul 17 06:15:16 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 17 Jul 2013 13:15:16 +0900
Subject: [Python-Dev] Misc re.match() complaint
In-Reply-To: <ks33f6$jn$2@ger.gmane.org>
References: <CAP7+vJKdLzJ3BOgD9e=Nw8KT0v4apdgutu7nKLPAgtEGCapA5A@mail.gmail.com>
	<CAGE7PNJs5qhUiWhv46HBzwpPHsk6Mt_N3+h3XSL2CoNOg7KrYw@mail.gmail.com>
	<51E48F8B.1090905@mrabarnett.plus.com>
	<CAP7+vJKRuKL6vsPRmvo-+9gPtB=usp57F31_1H--sVN0_qi6GQ@mail.gmail.com>
	<87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAP7+vJ+WgZSed2zwiCgBpNDVcr1urvvctv8UK705dJ6iEBGQfg@mail.gmail.com>
	<ks33f6$jn$2@ger.gmane.org>
Message-ID: <87li55styj.fsf@uwakimon.sk.tsukuba.ac.jp>

Terry Reedy writes:
 > On 7/15/2013 10:20 PM, Guido van Rossum wrote:
 > 
 > >> Or is this something deeper, that a group *is* a new object in
 > >> principle?
 > >
 > > No, I just think of it as returning "a string"
 > 
 > That is exactly what the doc says it does. See my other post.

The problem is that IIUC '"a string"' is intentionally *not* referring
to the usual "str or bytes objects" (at least that's one of the
standard uses for scare quotes, to indicate an unusual usage).  Either
the docstring is using "string" in a similarly ambiguous way, or else
it's incorrect under the interpretation that buffer objects are *not*
"strings", so they should be inadmissible as targets.  Something
should be fixed, and I suppose it should be the return type of group().

BTW, I suggest that Terry's usage of "string" (to mean "str or bytes"
in 3.x, "unicode or str" in 2.x) be adopted, and Guido's "stringish"
be given expanded meaning, including buffer objects.  Then we can say
informally that in searching and matching a target is a stringish, the
pattern is a stringish (?) or compiled re, but the group method
returns a string.

Steve

From ronaldoussoren at mac.com  Wed Jul 17 07:58:03 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Wed, 17 Jul 2013 07:58:03 +0200
Subject: [Python-Dev] Why does PEP 8 advise against explicit relative
	imports?
In-Reply-To: <CAPdQG2rJMdC=QfOcf9gGdiFiLwEskf90v8HUBhEsR8y+W47JcQ@mail.gmail.com>
References: <CADiSq7cC=GYwkx+viaNvNB+wMS4uU-XdSqH5F9vp8SagKkaTeg@mail.gmail.com>
	<CAPdQG2rJMdC=QfOcf9gGdiFiLwEskf90v8HUBhEsR8y+W47JcQ@mail.gmail.com>
Message-ID: <741654DE-CA5C-42D7-AB3A-D950EAEB28BC@mac.com>


On 16 Jul, 2013, at 14:02, Thomas Wouters <thomas at python.org> wrote:

> On Tue, Jul 16, 2013 at 1:40 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
>> PEP 8 advises developers to use absolute imports rather than explicit
>> relative imports.
>> 
>> Why? Using absolute imports couple the internal implementation of a
>> package to its public name - you can't just change the top level
>> directory name any more, you have to go through and change all the
>> absolute imports as well. You also can't easily vendor a package that
>> uses absolute imports inside another project either, since all the
>> absolute imports will break.
>> 
> 
> The problem with relative imports (both explicit and implicit) is that it
> makes it less obvious when you are importing a module under the wrong name.
> If a package ends up in sys.path directly (for example, by executing
> something that lives inside it directly) then an explicit relative import
> directly in the package will fail, but an explicit relative import in a
> sub-package won't, and you can end up with the subtly confusing mess of
> multiple imports of the same .py file under different names.

That's only a problem for subpackages (that is, for example when
distutils.commands ends up being importable as commands), because explicit
relative imports (``from .other import attr``) don't work in toplevel modules
or scripts because the leading dot means "the current package".

I tend to use explicit relative imports in my code when that increases
readability by reducing unnecessary duplication. That is, readability
tends to suffer when you have a number of lines like with
"from package.module1 import name".

Ronald

From tjreedy at udel.edu  Wed Jul 17 11:05:50 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 17 Jul 2013 05:05:50 -0400
Subject: [Python-Dev] Misc re.match() complaint
In-Reply-To: <87li55styj.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <CAP7+vJKdLzJ3BOgD9e=Nw8KT0v4apdgutu7nKLPAgtEGCapA5A@mail.gmail.com>
	<CAGE7PNJs5qhUiWhv46HBzwpPHsk6Mt_N3+h3XSL2CoNOg7KrYw@mail.gmail.com>
	<51E48F8B.1090905@mrabarnett.plus.com>
	<CAP7+vJKRuKL6vsPRmvo-+9gPtB=usp57F31_1H--sVN0_qi6GQ@mail.gmail.com>
	<87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAP7+vJ+WgZSed2zwiCgBpNDVcr1urvvctv8UK705dJ6iEBGQfg@mail.gmail.com>
	<ks33f6$jn$2@ger.gmane.org>
	<87li55styj.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <ks5mpc$8sh$1@ger.gmane.org>

On 7/17/2013 12:15 AM, Stephen J. Turnbull wrote:
> Terry Reedy writes:
>   > On 7/15/2013 10:20 PM, Guido van Rossum wrote:
>   >
>   > >> Or is this something deeper, that a group *is* a new object in
>   > >> principle?
>   > >
>   > > No, I just think of it as returning "a string"
>   >
>   > That is exactly what the doc says it does. See my other post.
>
> The problem is that IIUC '"a string"' is intentionally *not* referring
> to the usual "str or bytes objects" (at least that's one of the
> standard uses for scare quotes, to indicate an unusual usage).

There are no 'scare quotes' in the doc. I put quote marks on things to 
indicated that I was quoting. I do not know how Guido regarded his marks.

 > Either
> the docstring is using "string" in a similarly ambiguous way, or else
> it's incorrect under the interpretation that buffer objects are *not*
> "strings", so they should be inadmissible as targets.

Saying that input arguments can be "Unicode strings as well as 8-bit 
strings' (the wording is from 2.x, carried over to 3.x) does not 
necessary exclude other inputs. CPython is somethimes more more 
permissive than the doc requires. If the doc said str, bytes, butearray, 
or memoryview, then other implementations would have to do the same to 
be conforming. I do not know if that is intended or not.

The question is whether CPython should be just as permissive as to the 
output types of .group(). (And what, if any requirement should be 
imposed on other implementations.)

 > Something
> should be fixed, and I suppose it should be the return type of group().
>
> BTW, I suggest that Terry's usage of "string" (to mean "str or bytes"
> in 3.x, "unicode or str" in 2.x) be adopted, and Guido's "stringish"

This word is an adjective, not a noun.

> be given expanded meaning, including buffer objects.  Then we can say
> informally that in searching and matching a target is a stringish, the
> pattern is a stringish (?) or compiled re, but the group method
> returns a string.

Guido's idea to fix (tighten up) the output in 3.4 is fine with me.

-- 
Terry Jan Reedy


From stephen at xemacs.org  Wed Jul 17 13:15:38 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 17 Jul 2013 20:15:38 +0900
Subject: [Python-Dev] Misc re.match() complaint
In-Reply-To: <ks5mpc$8sh$1@ger.gmane.org>
References: <CAP7+vJKdLzJ3BOgD9e=Nw8KT0v4apdgutu7nKLPAgtEGCapA5A@mail.gmail.com>
	<CAGE7PNJs5qhUiWhv46HBzwpPHsk6Mt_N3+h3XSL2CoNOg7KrYw@mail.gmail.com>
	<51E48F8B.1090905@mrabarnett.plus.com>
	<CAP7+vJKRuKL6vsPRmvo-+9gPtB=usp57F31_1H--sVN0_qi6GQ@mail.gmail.com>
	<87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAP7+vJ+WgZSed2zwiCgBpNDVcr1urvvctv8UK705dJ6iEBGQfg@mail.gmail.com>
	<ks33f6$jn$2@ger.gmane.org>
	<87li55styj.fsf@uwakimon.sk.tsukuba.ac.jp>
	<ks5mpc$8sh$1@ger.gmane.org>
Message-ID: <87ehaxsahx.fsf@uwakimon.sk.tsukuba.ac.jp>

Terry Reedy writes:

 > > "stringish"
 > 
 > This word is an adjective, not a noun.

Ah, a strict grammarian.  That trumps any cards I could possibly play.





From victor.stinner at gmail.com  Wed Jul 17 13:16:27 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 17 Jul 2013 13:16:27 +0200
Subject: [Python-Dev] PEP 446: Add new parameters to configure the
 inherance of files and for non-blocking sockets
In-Reply-To: <CAH_1eM19BzWhgy=g8XtRaJdT0DXckxYHFdY=8pvPhbYHmgzzSg@mail.gmail.com>
References: <CAH_1eM1RCDSHeaaBQT62Cjf2V9HAjSUJZS3zvZoidJ9Pyfk2FQ@mail.gmail.com>
	<20130707003056.GA55329@cskk.homeip.net>
	<CAH_1eM19BzWhgy=g8XtRaJdT0DXckxYHFdY=8pvPhbYHmgzzSg@mail.gmail.com>
Message-ID: <CAMpsgwY9Lny5bTOLJY9OHNBX3xA-Pa4deszQQMd84Ka8Pr+xQA@mail.gmail.com>

2013/7/7 Charles-Fran?ois Natali <cf.natali at gmail.com>:
> Well, it complicates the signature and implementation.
> If we go the same way, why stop there and not expose O_DSYNC, O_SYNC,
> O_DIRECT...

I added this counter-argument to the PEP.

> If you want precise control over the open() sementics, os.open() is
> the way to go (that's also the rationale behind io.open() opener
> argument, see http://bugs.python.org/issue12105)

Right.

Victor

From victor.stinner at gmail.com  Wed Jul 17 13:31:14 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 17 Jul 2013 13:31:14 +0200
Subject: [Python-Dev] PEP 446: Add new parameters to configure the
 inherance of files and for non-blocking sockets
In-Reply-To: <20130707002912.GA49468@cskk.homeip.net>
References: <CAMpsgwYN3j_3Vih7681qChb=jOwdP5zBQ=T-_fYDNrj9Xij=Ag@mail.gmail.com>
	<20130707002912.GA49468@cskk.homeip.net>
Message-ID: <CAMpsgwbsBCcXv7nqhGZZKdt7pd+Mc1vRkxn48+QW+Yzus6P6YA@mail.gmail.com>

2013/7/7 Cameron Simpson <cs at zip.com.au>:
> On 06Jul2013 14:43, Victor Stinner <victor.stinner at gmail.com> wrote:
> | 2013/7/6 Cameron Simpson <cs at zip.com.au>:
> | > Yes. Please forget I mentioned fork(); it is only relevant if you
> | > were offering some facility to undo the addition of cloexec to a
> | > Popen passed file descriptor. Which you are not.
> |
> | Oh... gotcha. I now understood your concern.
> |
> | There is a little "trick" here: at fork, file descriptors are
> | duplicated in the child process and almost all properties (open state,
> | flags, offset, etc.) are shared. There is one little exception: file
> | attributes are not shared, and there is only one file attribute:
> | O_CLOEXEC. Setting O_CLOEXEC in a child process does not affect the
> | flag in the parent process ;-) I will add a unit test to check this.
>
> That's news to me. Interesting.
>
> I can't find UNIX doco to support this, though I haven't finished
> looking. None of the POSIX, Linux or MacOSX fork() manual pages
> seem to describe this.
>
> Can you point me at doco for this? I thought file descriptors were
> a direct index into a per-process table of file handle structures,
> and did not think the latter got part copies, part shared over a
> fork.

Sorry, I found this fact my mistake, not in the doc :-) You can verify
this behaviour using the follownig script:
hg.python.org/peps/file/tip/pep-0466/test_cloexec.py

Output on Linux:

initial state: fd=3, cloexec=False
child process after fork, set cloexec: cloexec=True
child process after exec: fd=3, cloexec=<invalid file descriptor>
parent process after fork: cloexec=False
parent process after exec: fd=3, cloexec=False

"parent process after fork: cloexec=False" means that the flag was not
modified in the parent process.

> Also, is O_CLOEXEC really the only file attribute? So O_NONBLOCK
> is a flag and not an attribute (guessing the terminology distinguishes
> these)? Now you mention it, ISTR noticing that there were distinct
> ioctls to adjust flags and something else.

See the manual page:
http://linux.die.net/man/2/fcntl

There are "File descriptor flags " and "File status flags", it looks
like they are inherited/shared differently between the parent and the
child process.

> Your change says:
>
>   On Windows, the close-on-exec flag is ``HANDLE_FLAG_INHERIT``.
>
> I feel it would be clearer to say:
>
>   On Windows, the close-on-exec flag is the inverse of ``HANDLE_FLAG_INHERIT``.

Ok, fixed in the PEP.

Victor

From victor.stinner at gmail.com  Wed Jul 17 13:31:40 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 17 Jul 2013 13:31:40 +0200
Subject: [Python-Dev] PEP 446: Add new parameters to configure the
 inherance of files and for non-blocking sockets
In-Reply-To: <20130713025702.GA23103@cskk.homeip.net>
References: <CAMpsgwYzTCbNTw_AT3RY2qWxWbf48ZG1F14+1EemaX=9i6gABQ@mail.gmail.com>
	<20130713025702.GA23103@cskk.homeip.net>
Message-ID: <CAMpsgwZuXvwbaxa8z8TVAh9qwRvWYjaZGc-2Guge7k06RwLpkA@mail.gmail.com>

2013/7/13 Cameron Simpson <cs at zip.com.au>:
> This sentence:
>
>   The flag is cleared in the child process before executing the
>   program, the change does not change the flag in the parent process.
>
> needs a semicolon, not a comma.

Fixed.

Victor

From steve at pearwood.info  Wed Jul 17 13:50:42 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 17 Jul 2013 21:50:42 +1000
Subject: [Python-Dev] String terminology [was Re: Misc re.match() complaint]
In-Reply-To: <ks5mpc$8sh$1@ger.gmane.org>
References: <CAP7+vJKdLzJ3BOgD9e=Nw8KT0v4apdgutu7nKLPAgtEGCapA5A@mail.gmail.com>
	<CAGE7PNJs5qhUiWhv46HBzwpPHsk6Mt_N3+h3XSL2CoNOg7KrYw@mail.gmail.com>
	<51E48F8B.1090905@mrabarnett.plus.com>
	<CAP7+vJKRuKL6vsPRmvo-+9gPtB=usp57F31_1H--sVN0_qi6GQ@mail.gmail.com>
	<87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAP7+vJ+WgZSed2zwiCgBpNDVcr1urvvctv8UK705dJ6iEBGQfg@mail.gmail.com>
	<ks33f6$jn$2@ger.gmane.org>
	<87li55styj.fsf@uwakimon.sk.tsukuba.ac.jp>
	<ks5mpc$8sh$1@ger.gmane.org>
Message-ID: <51E68512.1000506@pearwood.info>

On 17/07/13 19:05, Terry Reedy wrote:

> Saying that input arguments can be "Unicode strings as well as 8-bit strings' (the wording is from 2.x, carried over to 3.x) does not necessary exclude other inputs.

"8-bit strings" seems somewhat ambiguous to me. In UTF-8, many Unicode strings are 8-bit, as they can be with Python 3.3's flexible string format. I prefer to stick to Unicode or text string, versus byte string.

Pedants who point out that "byte" does not necessarily mean 8-bits, and therefore we should talk about octets, will be slapped with a large halibut :-)


-- 
Steven







From storchaka at gmail.com  Wed Jul 17 15:12:34 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Wed, 17 Jul 2013 16:12:34 +0300
Subject: [Python-Dev] Misc re.match() complaint
In-Reply-To: <CAP7+vJKEAy+Xz7g2xz2SuJH52a9HtQdbMbL5HuH_8dTi_g-aAA@mail.gmail.com>
References: <CAP7+vJKdLzJ3BOgD9e=Nw8KT0v4apdgutu7nKLPAgtEGCapA5A@mail.gmail.com>
	<CAGE7PNJs5qhUiWhv46HBzwpPHsk6Mt_N3+h3XSL2CoNOg7KrYw@mail.gmail.com>
	<51E48F8B.1090905@mrabarnett.plus.com>
	<CAP7+vJKRuKL6vsPRmvo-+9gPtB=usp57F31_1H--sVN0_qi6GQ@mail.gmail.com>
	<87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAP7+vJ+WgZSed2zwiCgBpNDVcr1urvvctv8UK705dJ6iEBGQfg@mail.gmail.com>
	<CADiSq7dTdxBbEqtsV-fdWab5O90JdvnEZRLKXZY9tO86iArPSA@mail.gmail.com>
	<CAP7+vJ+WCtJMjGX480QYB4-LJ2TtQ4jS8ee=H7SP4VsyuzvCpg@mail.gmail.com>
	<20130716095551.5aca0285@pitrou.net>
	<CAP7+vJKEAy+Xz7g2xz2SuJH52a9HtQdbMbL5HuH_8dTi_g-aAA@mail.gmail.com>
Message-ID: <ks657s$knl$1@ger.gmane.org>

16.07.13 20:21, Guido van Rossum ???????(??):
> The situation is most egregious if the target string is a bytearray,
> where there is currently no way to get the result as an immutable
> bytes object without an extra copy. (There's no API that lets you
> create a bytes object directly from a slice of a bytearray.)

m = memoryview(data)
if m:
    return m.cast('B')[low:high].tobytes()
else:
    # cast() doesn't work for empty memoryview
    return b''



From brett at python.org  Wed Jul 17 15:35:59 2013
From: brett at python.org (Brett Cannon)
Date: Wed, 17 Jul 2013 09:35:59 -0400
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <20130716210733.2ccd5192@limelight.wooz.org>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
	<ks08fk$vnu$1@ger.gmane.org>
	<CADiSq7fAS+-keOjBmOEs-UvV=bV4qB9=ikiN5dkvRW7LSrge6g@mail.gmail.com>
	<20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko>
	<51E4A0EF.8070205@pearwood.info> <1373941235.30382.62.camel@thinko>
	<CADiSq7cr8JrYJHtfxjgzdHjiJjC-ipcm98ejKTsTAHL6SPMPyg@mail.gmail.com>
	<1373943762.30382.88.camel@thinko>
	<CADiSq7ck152x2_dPJ8TET6mJbk3maO9-q2FU5rt6pAaA_G0+fQ@mail.gmail.com>
	<ks310f$90g$1@ger.gmane.org>
	<CADiSq7eRjyXK0VEHj7cBfpJj4Z4QiMxdX5tJr4nfbODatzGVXw@mail.gmail.com>
	<ks4l94$fqf$2@ger.gmane.org>
	<20130716210733.2ccd5192@limelight.wooz.org>
Message-ID: <CAP1=2W5UNzbH=-nZBqaP6+v=VCuR0qb2a5Gf=TbHjiuFGCGFkw@mail.gmail.com>

On Tue, Jul 16, 2013 at 9:07 PM, Barry Warsaw <barry at python.org> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On Jul 16, 2013, at 07:34 PM, Tres Seaver wrote:
>
> >On 07/16/2013 07:09 AM, Nick Coghlan wrote:
> >
> >> I did find it interesting that we *don't* explicitly advise against
> >> the use of "import *" for anything other than optional accelerator
> >> modules or republishing internal interfaces through a public API,
> >> even though we advice against the practice in the tutorial. Perhaps
> >> another oversight worth correcting?
> >
> >+1.  'from foo import *' (from any source other than another module in
> >the same package) is a code stench far worse than anything else PEP8
> >proscribes.
>
> Maybe we should disable it for anything that isn't at the interactive
> prompt
> <0.5 wink>.
>

Or how about dropping the whole ``from ... import ...`` format entirely?
<0.5 wink; seriously, it's implementation is a pain and it has wonky
semantics>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130717/4612d998/attachment.html>

From python at mrabarnett.plus.com  Wed Jul 17 15:56:44 2013
From: python at mrabarnett.plus.com (MRAB)
Date: Wed, 17 Jul 2013 14:56:44 +0100
Subject: [Python-Dev] Misc re.match() complaint
In-Reply-To: <87li55styj.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <CAP7+vJKdLzJ3BOgD9e=Nw8KT0v4apdgutu7nKLPAgtEGCapA5A@mail.gmail.com>
	<CAGE7PNJs5qhUiWhv46HBzwpPHsk6Mt_N3+h3XSL2CoNOg7KrYw@mail.gmail.com>
	<51E48F8B.1090905@mrabarnett.plus.com>
	<CAP7+vJKRuKL6vsPRmvo-+9gPtB=usp57F31_1H--sVN0_qi6GQ@mail.gmail.com>
	<87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAP7+vJ+WgZSed2zwiCgBpNDVcr1urvvctv8UK705dJ6iEBGQfg@mail.gmail.com>
	<ks33f6$jn$2@ger.gmane.org>
	<87li55styj.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <51E6A29C.1070001@mrabarnett.plus.com>

On 17/07/2013 05:15, Stephen J. Turnbull wrote:
> Terry Reedy writes:
>   > On 7/15/2013 10:20 PM, Guido van Rossum wrote:
>   >
>   > >> Or is this something deeper, that a group *is* a new object in
>   > >> principle?
>   > >
>   > > No, I just think of it as returning "a string"
>   >
>   > That is exactly what the doc says it does. See my other post.
>
> The problem is that IIUC '"a string"' is intentionally *not* referring
> to the usual "str or bytes objects" (at least that's one of the
> standard uses for scare quotes, to indicate an unusual usage).  Either
> the docstring is using "string" in a similarly ambiguous way, or else
> it's incorrect under the interpretation that buffer objects are *not*
> "strings", so they should be inadmissible as targets.  Something
> should be fixed, and I suppose it should be the return type of group().
>
> BTW, I suggest that Terry's usage of "string" (to mean "str or bytes"
> in 3.x, "unicode or str" in 2.x) be adopted, and Guido's "stringish"
> be given expanded meaning, including buffer objects.  Then we can say
> informally that in searching and matching a target is a stringish, the
> pattern is a stringish (?) or compiled re, but the group method
> returns a string.
>
Instead of "stringish", how about "stringoid"? To me, "stringish" is an
adjective, but "stringoid" can be a noun or an adjective.

According to http://dictionary.reference.com:

"""
-oid

?suffix forming adjectives, ?suffix forming nouns
	indicating likeness, resemblance, or similarity: anthropoid
"""


From barry at python.org  Wed Jul 17 15:56:09 2013
From: barry at python.org (Barry Warsaw)
Date: Wed, 17 Jul 2013 09:56:09 -0400
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
 underscores
In-Reply-To: <CAP1=2W5UNzbH=-nZBqaP6+v=VCuR0qb2a5Gf=TbHjiuFGCGFkw@mail.gmail.com>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
	<ks08fk$vnu$1@ger.gmane.org>
	<CADiSq7fAS+-keOjBmOEs-UvV=bV4qB9=ikiN5dkvRW7LSrge6g@mail.gmail.com>
	<20130715184039.60590e62@anarchist>
	<1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info>
	<1373941235.30382.62.camel@thinko>
	<CADiSq7cr8JrYJHtfxjgzdHjiJjC-ipcm98ejKTsTAHL6SPMPyg@mail.gmail.com>
	<1373943762.30382.88.camel@thinko>
	<CADiSq7ck152x2_dPJ8TET6mJbk3maO9-q2FU5rt6pAaA_G0+fQ@mail.gmail.com>
	<ks310f$90g$1@ger.gmane.org>
	<CADiSq7eRjyXK0VEHj7cBfpJj4Z4QiMxdX5tJr4nfbODatzGVXw@mail.gmail.com>
	<ks4l94$fqf$2@ger.gmane.org>
	<20130716210733.2ccd5192@limelight.wooz.org>
	<CAP1=2W5UNzbH=-nZBqaP6+v=VCuR0qb2a5Gf=TbHjiuFGCGFkw@mail.gmail.com>
Message-ID: <20130717095609.6824d790@anarchist>

On Jul 17, 2013, at 09:35 AM, Brett Cannon wrote:

>Or how about dropping the whole ``from ... import ...`` format entirely?
><0.5 wink; seriously, it's implementation is a pain and it has wonky
>semantics>

Yes, let's break *everything* :)

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130717/d3ada834/attachment.pgp>

From guido at python.org  Wed Jul 17 16:55:13 2013
From: guido at python.org (Guido van Rossum)
Date: Wed, 17 Jul 2013 07:55:13 -0700
Subject: [Python-Dev] String terminology [was Re: Misc re.match()
	complaint]
In-Reply-To: <51E68512.1000506@pearwood.info>
References: <CAP7+vJKdLzJ3BOgD9e=Nw8KT0v4apdgutu7nKLPAgtEGCapA5A@mail.gmail.com>
	<CAGE7PNJs5qhUiWhv46HBzwpPHsk6Mt_N3+h3XSL2CoNOg7KrYw@mail.gmail.com>
	<51E48F8B.1090905@mrabarnett.plus.com>
	<CAP7+vJKRuKL6vsPRmvo-+9gPtB=usp57F31_1H--sVN0_qi6GQ@mail.gmail.com>
	<87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAP7+vJ+WgZSed2zwiCgBpNDVcr1urvvctv8UK705dJ6iEBGQfg@mail.gmail.com>
	<ks33f6$jn$2@ger.gmane.org> <87li55styj.fsf@uwakimon.sk.tsukuba.ac.jp>
	<ks5mpc$8sh$1@ger.gmane.org> <51E68512.1000506@pearwood.info>
Message-ID: <CAP7+vJL5Byz8LgMUS-OJ3qXcq09dqA-runnVphuJpFxy5vXT8A@mail.gmail.com>

When precision is needed I say things like 'a str object' or 'a bytes
object'. There is no shame in a bit of verbosity around such issues,
especially in the reference docs (tutorials are a different issue).

On Wed, Jul 17, 2013 at 4:50 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> On 17/07/13 19:05, Terry Reedy wrote:
>
>> Saying that input arguments can be "Unicode strings as well as 8-bit
>> strings' (the wording is from 2.x, carried over to 3.x) does not necessary
>> exclude other inputs.
>
>
> "8-bit strings" seems somewhat ambiguous to me. In UTF-8, many Unicode
> strings are 8-bit, as they can be with Python 3.3's flexible string format.
> I prefer to stick to Unicode or text string, versus byte string.
>
> Pedants who point out that "byte" does not necessarily mean 8-bits, and
> therefore we should talk about octets, will be slapped with a large halibut
> :-)
>
>
> --
> Steven
>
>
>
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)

From rdmurray at bitdance.com  Wed Jul 17 18:20:34 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Wed, 17 Jul 2013 12:20:34 -0400
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <ks4evn$urm$1@ger.gmane.org>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
	<ks08fk$vnu$1@ger.gmane.org>
	<CADiSq7fAS+-keOjBmOEs-UvV=bV4qB9=ikiN5dkvRW7LSrge6g@mail.gmail.com>
	<20130715184039.60590e62@anarchist>
	<1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info>
	<1373941235.30382.62.camel@thinko>
	<CADiSq7cr8JrYJHtfxjgzdHjiJjC-ipcm98ejKTsTAHL6SPMPyg@mail.gmail.com>
	<1373943762.30382.88.camel@thinko>
	<CADiSq7ck152x2_dPJ8TET6mJbk3maO9-q2FU5rt6pAaA_G0+fQ@mail.gmail.com>
	<1373944606.30382.92.camel@thinko>
	<CADiSq7fxi8rMbcdVN_SsUycjNWmB_61KQomcowV+m8m9+Qp79A@mail.gmail.com>
	<ks3788$ere$1@ger.gmane.org> <51E54859.4070901@pearwood.info>
	<20130716133912.C532325007E@webabinitio.net>
	<ks4evn$urm$1@ger.gmane.org>
Message-ID: <20130717162034.601F325014C@webabinitio.net>

On Tue, 16 Jul 2013 17:46:34 -0400, Terry Reedy <tjreedy at udel.edu> wrote:
> On 7/16/2013 9:39 AM, R. David Murray wrote:
> > On Tue, 16 Jul 2013 23:19:21 +1000, Steven D'Aprano <steve at pearwood.info> wrote:
> 
> >> For example, pkgutil includes classes with single-underscore methods, which I take as private. It also has a function simplegeneric, which is undocumented and not listed in __all__. In in the absence of even a comment saying "Don't use this", I take it as an oversight, not policy that simplegeneric is private.
> >
> > I think you'd be wrong about that, though.  simplegeneric should really be
> > treated as private.  I'm speaking here not about the general principle of
> > the thing, but about my understanding of simplegeneric's specific history.
> 
> I think Steven (valid) point is "Why not, then, say it is internal 
> either in docs or name?"-- which in this case would be in the docs.

I don't think that's what he was saying; but yes, we should do that :)

--David

From dholth at gmail.com  Wed Jul 17 19:47:55 2013
From: dholth at gmail.com (Daniel Holth)
Date: Wed, 17 Jul 2013 13:47:55 -0400
Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading
	underscores
In-Reply-To: <20130717162034.601F325014C@webabinitio.net>
References: <CADiSq7fcw48f=nvMqRHBK3TLvo74PjY-3rQx3ohZ30OP86m7fg@mail.gmail.com>
	<CADiSq7ehq8ni45rq2K6zjUZ0qEUs4q3HSYhUzBT22D73TOituQ@mail.gmail.com>
	<CAGE7PNK+oNrzqfEJ9HeLpdFozT5mG=e_P+0X1SVNgAjYt2Oecg@mail.gmail.com>
	<CADiSq7cKZoZtOk+7tyeamXLNnWcyQ8kPKL5jvpyu857DFNm0NA@mail.gmail.com>
	<ks08fk$vnu$1@ger.gmane.org>
	<CADiSq7fAS+-keOjBmOEs-UvV=bV4qB9=ikiN5dkvRW7LSrge6g@mail.gmail.com>
	<20130715184039.60590e62@anarchist>
	<1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info>
	<1373941235.30382.62.camel@thinko>
	<CADiSq7cr8JrYJHtfxjgzdHjiJjC-ipcm98ejKTsTAHL6SPMPyg@mail.gmail.com>
	<1373943762.30382.88.camel@thinko>
	<CADiSq7ck152x2_dPJ8TET6mJbk3maO9-q2FU5rt6pAaA_G0+fQ@mail.gmail.com>
	<1373944606.30382.92.camel@thinko>
	<CADiSq7fxi8rMbcdVN_SsUycjNWmB_61KQomcowV+m8m9+Qp79A@mail.gmail.com>
	<ks3788$ere$1@ger.gmane.org> <51E54859.4070901@pearwood.info>
	<20130716133912.C532325007E@webabinitio.net>
	<ks4evn$urm$1@ger.gmane.org>
	<20130717162034.601F325014C@webabinitio.net>
Message-ID: <CAG8k2+6znjfJGn9DuFD668qrCQ4Wiivs7S+gGiQ1u-rrXEqXzg@mail.gmail.com>

https://pypi.python.org/pypi/apipkg provides a much more effective way
to denote API than an _

On Wed, Jul 17, 2013 at 12:20 PM, R. David Murray <rdmurray at bitdance.com> wrote:
> On Tue, 16 Jul 2013 17:46:34 -0400, Terry Reedy <tjreedy at udel.edu> wrote:
>> On 7/16/2013 9:39 AM, R. David Murray wrote:
>> > On Tue, 16 Jul 2013 23:19:21 +1000, Steven D'Aprano <steve at pearwood.info> wrote:
>>
>> >> For example, pkgutil includes classes with single-underscore methods, which I take as private. It also has a function simplegeneric, which is undocumented and not listed in __all__. In in the absence of even a comment saying "Don't use this", I take it as an oversight, not policy that simplegeneric is private.
>> >
>> > I think you'd be wrong about that, though.  simplegeneric should really be
>> > treated as private.  I'm speaking here not about the general principle of
>> > the thing, but about my understanding of simplegeneric's specific history.
>>
>> I think Steven (valid) point is "Why not, then, say it is internal
>> either in docs or name?"-- which in this case would be in the docs.
>
> I don't think that's what he was saying; but yes, we should do that :)
>
> --David
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com

From ezio.melotti at gmail.com  Thu Jul 18 15:15:44 2013
From: ezio.melotti at gmail.com (Ezio Melotti)
Date: Thu, 18 Jul 2013 15:15:44 +0200
Subject: [Python-Dev] Misc re.match() complaint
In-Reply-To: <87li55styj.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <CAP7+vJKdLzJ3BOgD9e=Nw8KT0v4apdgutu7nKLPAgtEGCapA5A@mail.gmail.com>
	<CAGE7PNJs5qhUiWhv46HBzwpPHsk6Mt_N3+h3XSL2CoNOg7KrYw@mail.gmail.com>
	<51E48F8B.1090905@mrabarnett.plus.com>
	<CAP7+vJKRuKL6vsPRmvo-+9gPtB=usp57F31_1H--sVN0_qi6GQ@mail.gmail.com>
	<87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAP7+vJ+WgZSed2zwiCgBpNDVcr1urvvctv8UK705dJ6iEBGQfg@mail.gmail.com>
	<ks33f6$jn$2@ger.gmane.org>
	<87li55styj.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <CACBhJdFqWJRa9R6sCW9K60vs4dzUco8mD8=Rd8FBn8u56f8SPw@mail.gmail.com>

Hi,

On Wed, Jul 17, 2013 at 6:15 AM, Stephen J. Turnbull <stephen at xemacs.org> wrote:
>
> BTW, I suggest that Terry's usage of "string" (to mean "str or bytes"
> in 3.x, "unicode or str" in 2.x) be adopted, and Guido's "stringish"
> be given expanded meaning, including buffer objects.

"string" means "str", "bytes" means "bytes", "bytes-like object" means
"any object that supports the buffer protocol" [0] (including bytes).
"string and bytes-like object" includes all of them.
I don't think we need to introduce new terms.

Best Regards,
Ezio Melotti

[0]: http://docs.python.org/3/glossary.html#term-bytes-like-object

>  Then we can say
> informally that in searching and matching a target is a stringish, the
> pattern is a stringish (?) or compiled re, but the group method
> returns a string.
>
> Steve

From guido at python.org  Thu Jul 18 17:02:27 2013
From: guido at python.org (Guido van Rossum)
Date: Thu, 18 Jul 2013 08:02:27 -0700
Subject: [Python-Dev] Misc re.match() complaint
In-Reply-To: <CACBhJdFqWJRa9R6sCW9K60vs4dzUco8mD8=Rd8FBn8u56f8SPw@mail.gmail.com>
References: <CAP7+vJKdLzJ3BOgD9e=Nw8KT0v4apdgutu7nKLPAgtEGCapA5A@mail.gmail.com>
	<CAGE7PNJs5qhUiWhv46HBzwpPHsk6Mt_N3+h3XSL2CoNOg7KrYw@mail.gmail.com>
	<51E48F8B.1090905@mrabarnett.plus.com>
	<CAP7+vJKRuKL6vsPRmvo-+9gPtB=usp57F31_1H--sVN0_qi6GQ@mail.gmail.com>
	<87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAP7+vJ+WgZSed2zwiCgBpNDVcr1urvvctv8UK705dJ6iEBGQfg@mail.gmail.com>
	<ks33f6$jn$2@ger.gmane.org> <87li55styj.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CACBhJdFqWJRa9R6sCW9K60vs4dzUco8mD8=Rd8FBn8u56f8SPw@mail.gmail.com>
Message-ID: <CAP7+vJ+Aac1Gv+Y9aYNWk6oZgYRW=ixrMYTCJ0ebnqP1KWHmew@mail.gmail.com>

On Thu, Jul 18, 2013 at 6:15 AM, Ezio Melotti <ezio.melotti at gmail.com> wrote:
> I don't think we need to introduce new terms.
+1

-- 
--Guido van Rossum (python.org/~guido)

From storchaka at gmail.com  Thu Jul 18 19:38:32 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Thu, 18 Jul 2013 20:38:32 +0300
Subject: [Python-Dev] Dash
Message-ID: <ks996h$p10$1@ger.gmane.org>

What type of dash is preferable in the documentation? The en dash (?) or 
the em dash (?)?


From techtonik at gmail.com  Thu Jul 18 19:43:07 2013
From: techtonik at gmail.com (anatoly techtonik)
Date: Thu, 18 Jul 2013 20:43:07 +0300
Subject: [Python-Dev] Fwd: Issue 1229 in nativeclient: Get Python to work
	under Native Client
In-Reply-To: <8-11067354921584041717-18302888370781815686-nativeclient=googlecode.com@googlecode.com>
References: <0-11067354921584041717-18302888370781815686-nativeclient=googlecode.com@googlecode.com>
	<7-11067354921584041717-18302888370781815686-nativeclient=googlecode.com@googlecode.com>
	<8-11067354921584041717-18302888370781815686-nativeclient=googlecode.com@googlecode.com>
Message-ID: <CAPkN8xK4uLhTJeSVMAOLyqTtXVMJLV-2K9+=a4Gc6uBDdu4kSg@mail.gmail.com>

Does Python build system support cross-compiling? NaCl projects seems to
have problem with that. Just thought you might be interested to know about
it.



---------- Forwarded message ----------
From: <nativeclient at googlecode.com>
Date: Tue, Jul 16, 2013 at 9:24 PM
Subject: Re: Issue 1229 in nativeclient: Get Python to work under Native
Client
To: techtonik at gmail.com


Updates:
        Status: Fixed

Comment #8 on issue 1229 by mseaborn at chromium.org: Get Python to work under
Native Client
http://code.google.com/p/**nativeclient/issues/detail?id=**1229<http://code.google.com/p/nativeclient/issues/detail?id=1229>

This is done in naclports.  Sam Clegg added initial support for building
against nacl-glibc in https://code.google.com/p/**
naclports/source/detail?r=802<https://code.google.com/p/naclports/source/detail?r=802>
.

Sam found a way to work around the Python build's lack of support for
cross-compiling.  The workaround is to build a native "pgen" first and
inject it into the later cross-build.

Building Python with nacl-newlib and PNaCl is a work in progress:
https://codereview.chromium.**org/18112024/<https://codereview.chromium.org/18112024/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130718/bba1c711/attachment.html>

From tjreedy at udel.edu  Thu Jul 18 18:44:39 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 18 Jul 2013 12:44:39 -0400
Subject: [Python-Dev] Misc re.match() complaint
In-Reply-To: <CACBhJdFqWJRa9R6sCW9K60vs4dzUco8mD8=Rd8FBn8u56f8SPw@mail.gmail.com>
References: <CAP7+vJKdLzJ3BOgD9e=Nw8KT0v4apdgutu7nKLPAgtEGCapA5A@mail.gmail.com>
	<CAGE7PNJs5qhUiWhv46HBzwpPHsk6Mt_N3+h3XSL2CoNOg7KrYw@mail.gmail.com>
	<51E48F8B.1090905@mrabarnett.plus.com>
	<CAP7+vJKRuKL6vsPRmvo-+9gPtB=usp57F31_1H--sVN0_qi6GQ@mail.gmail.com>
	<87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAP7+vJ+WgZSed2zwiCgBpNDVcr1urvvctv8UK705dJ6iEBGQfg@mail.gmail.com>
	<ks33f6$jn$2@ger.gmane.org> <87li55styj.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CACBhJdFqWJRa9R6sCW9K60vs4dzUco8mD8=Rd8FBn8u56f8SPw@mail.gmail.com>
Message-ID: <51E81B77.6020209@udel.edu>


On 7/18/2013 9:15 AM, Ezio Melotti wrote:

In 3.x

> "string" means "str", "bytes" means "bytes", "bytes-like object" means
> "any object that supports the buffer protocol" [0] (including bytes).
> "string and bytes-like object" includes all of them.
> I don't think we need to introduce new terms.

I agree. We just need to use them consistently, and update docs carried 
over without change from 2.x (like re doc), where 'string' meant 
'unicode or str (bytes)' or even 'unicode and bytes-like'.

From guido at python.org  Thu Jul 18 19:48:37 2013
From: guido at python.org (Guido van Rossum)
Date: Thu, 18 Jul 2013 10:48:37 -0700
Subject: [Python-Dev] Dash
In-Reply-To: <ks996h$p10$1@ger.gmane.org>
References: <ks996h$p10$1@ger.gmane.org>
Message-ID: <CAP7+vJJAiLM7n88LywEk-Z9L7jv4nEJFvhwkR3E8-qkTnZbasg@mail.gmail.com>

I believe there are only a few places where en-dashes should be used,
for most things you should use either em-dash or hyphen. Consult your
trusted typography source (for US English, please, punctuation
preferences vary by locale). E.g. Google for "em dash en dash".

On Thu, Jul 18, 2013 at 10:38 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> What type of dash is preferable in the documentation? The en dash (?) or the
> em dash (?)?

-- 
--Guido van Rossum (python.org/~guido)

From storchaka at gmail.com  Thu Jul 18 20:46:09 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Thu, 18 Jul 2013 21:46:09 +0300
Subject: [Python-Dev] Dash
In-Reply-To: <CAP7+vJJAiLM7n88LywEk-Z9L7jv4nEJFvhwkR3E8-qkTnZbasg@mail.gmail.com>
References: <ks996h$p10$1@ger.gmane.org>
	<CAP7+vJJAiLM7n88LywEk-Z9L7jv4nEJFvhwkR3E8-qkTnZbasg@mail.gmail.com>
Message-ID: <ks9d5b$9sc$1@ger.gmane.org>

18.07.13 20:48, Guido van Rossum ???????(??):
> I believe there are only a few places where en-dashes should be used,
> for most things you should use either em-dash or hyphen. Consult your
> trusted typography source (for US English, please, punctuation
> preferences vary by locale). E.g. Google for "em dash en dash".

Currently Python documentation in most cases uses en-dashes. Should we 
replace them to em-dashes? Should we remove spaces around dashes?

Or we should replace a half-dozen of em-dashes found in Python 
documentation to en-dashes?

I believe all hypens used in place of dash should be replaced to dash 
(but to en- or em- dash?) in any case.



From brian at python.org  Thu Jul 18 20:54:50 2013
From: brian at python.org (Brian Curtin)
Date: Thu, 18 Jul 2013 13:54:50 -0500
Subject: [Python-Dev] Dash
In-Reply-To: <ks9d5b$9sc$1@ger.gmane.org>
References: <ks996h$p10$1@ger.gmane.org>
	<CAP7+vJJAiLM7n88LywEk-Z9L7jv4nEJFvhwkR3E8-qkTnZbasg@mail.gmail.com>
	<ks9d5b$9sc$1@ger.gmane.org>
Message-ID: <CAD+XWwreHB3v_kvbZ0tJsqxhydqkwZ0K9UGkrRsAJL0+nmX6mA@mail.gmail.com>

On Jul 18, 2013 1:46 PM, "Serhiy Storchaka" <storchaka at gmail.com> wrote:
>
> 18.07.13 20:48, Guido van Rossum ???????(??):
>
>> I believe there are only a few places where en-dashes should be used,
>> for most things you should use either em-dash or hyphen. Consult your
>> trusted typography source (for US English, please, punctuation
>> preferences vary by locale). E.g. Google for "em dash en dash".
>
>
> Currently Python documentation in most cases uses en-dashes. Should we
replace them to em-dashes? Should we remove spaces around dashes?
>
> Or we should replace a half-dozen of em-dashes found in Python
documentation to en-dashes?
>
> I believe all hypens used in place of dash should be replaced to dash
(but to en- or em- dash?) in any case.

Besides visual consistency in a couple of places, is there a reason to care
enough to make a wholesale change?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130718/bbe6e397/attachment.html>

From guido at python.org  Thu Jul 18 20:55:29 2013
From: guido at python.org (Guido van Rossum)
Date: Thu, 18 Jul 2013 11:55:29 -0700
Subject: [Python-Dev] Dash
In-Reply-To: <ks9d5b$9sc$1@ger.gmane.org>
References: <ks996h$p10$1@ger.gmane.org>
	<CAP7+vJJAiLM7n88LywEk-Z9L7jv4nEJFvhwkR3E8-qkTnZbasg@mail.gmail.com>
	<ks9d5b$9sc$1@ger.gmane.org>
Message-ID: <CAP7+vJJYN6DY0=zcDNZUgUfYF2T_SoC57fZCTVzpWtBk08W1+Q@mail.gmail.com>

On Thu, Jul 18, 2013 at 11:46 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> 18.07.13 20:48, Guido van Rossum ???????(??):
>
>> I believe there are only a few places where en-dashes should be used,
>> for most things you should use either em-dash or hyphen. Consult your
>> trusted typography source (for US English, please, punctuation
>> preferences vary by locale). E.g. Google for "em dash en dash".
>
>
> Currently Python documentation in most cases uses en-dashes. Should we
> replace them to em-dashes? Should we remove spaces around dashes?

This may well be a relic from times when in typewritten text there
were really only two types of dashes: a short one, typed as a hyphen,
used for both hyphen and en-dash (and minus sign), and a long one,
typed as two hyphens, used for em-dash.

I suspect at some point the conversion to HTML or typeset text changed
so that two hyphens mean en-dash and three mean em-dash, and few
people noticed (and nobody really wants to type three hyphens except
for a handful of typographical nuts).

> Or we should replace a half-dozen of em-dashes found in Python documentation
> to en-dashes?

If my theory is right that makes sense. Especially if it's only a half-dozen.

> I believe all hypens used in place of dash should be replaced to dash (but
> to en- or em- dash?) in any case.

But I wouldn't go change hundreds of hyphens -- chances are that you
get bored reviewing the scripted automation you're using and break a
few places. I'd say there are more important things to fix in the docs
(like writing better overviews for many modules and classes). If we
have a writing guide for the Python docs it would be useful to add
something about dash types though. And if we don't have a writing
guide, perhaps it makes sense to start one?

-- 
--Guido van Rossum (python.org/~guido)

From shibturn at gmail.com  Thu Jul 18 21:16:01 2013
From: shibturn at gmail.com (Richard Oudkerk)
Date: Thu, 18 Jul 2013 20:16:01 +0100
Subject: [Python-Dev] Dash
In-Reply-To: <CAP7+vJJYN6DY0=zcDNZUgUfYF2T_SoC57fZCTVzpWtBk08W1+Q@mail.gmail.com>
References: <ks996h$p10$1@ger.gmane.org>
	<CAP7+vJJAiLM7n88LywEk-Z9L7jv4nEJFvhwkR3E8-qkTnZbasg@mail.gmail.com>
	<ks9d5b$9sc$1@ger.gmane.org>
	<CAP7+vJJYN6DY0=zcDNZUgUfYF2T_SoC57fZCTVzpWtBk08W1+Q@mail.gmail.com>
Message-ID: <ks9eth$to0$1@ger.gmane.org>

On 18/07/2013 7:55pm, Guido van Rossum wrote:
> This may well be a relic from times when in typewritten text there
> were really only two types of dashes: a short one, typed as a hyphen,
> used for both hyphen and en-dash (and minus sign), and a long one,
> typed as two hyphens, used for em-dash.
>
> I suspect at some point the conversion to HTML or typeset text changed
> so that two hyphens mean en-dash and three mean em-dash, and few
> people noticed (and nobody really wants to type three hyphens except
> for a handful of typographical nuts).

Weren't the docs originally done LaTeX?  In LaTeX "--" is also an 
en-dash and "---" is an em-dash.

-- 
Richard


From guido at python.org  Thu Jul 18 21:50:27 2013
From: guido at python.org (Guido van Rossum)
Date: Thu, 18 Jul 2013 12:50:27 -0700
Subject: [Python-Dev] Dash
In-Reply-To: <ks9eth$to0$1@ger.gmane.org>
References: <ks996h$p10$1@ger.gmane.org>
	<CAP7+vJJAiLM7n88LywEk-Z9L7jv4nEJFvhwkR3E8-qkTnZbasg@mail.gmail.com>
	<ks9d5b$9sc$1@ger.gmane.org>
	<CAP7+vJJYN6DY0=zcDNZUgUfYF2T_SoC57fZCTVzpWtBk08W1+Q@mail.gmail.com>
	<ks9eth$to0$1@ger.gmane.org>
Message-ID: <CAP7+vJJ5t-D_S=F72LTihUjdRtnRsDwUug4gdKApdWSMFNW0EQ@mail.gmail.com>

On Thu, Jul 18, 2013 at 12:16 PM, Richard Oudkerk <shibturn at gmail.com> wrote:
> On 18/07/2013 7:55pm, Guido van Rossum wrote:
>>
>> This may well be a relic from times when in typewritten text there
>> were really only two types of dashes: a short one, typed as a hyphen,
>> used for both hyphen and en-dash (and minus sign), and a long one,
>> typed as two hyphens, used for em-dash.
>>
>> I suspect at some point the conversion to HTML or typeset text changed
>> so that two hyphens mean en-dash and three mean em-dash, and few
>> people noticed (and nobody really wants to type three hyphens except
>> for a handful of typographical nuts).
>
>
> Weren't the docs originally done LaTeX?  In LaTeX "--" is also an en-dash
> and "---" is an em-dash.

Right -- but I've been typing two hyphens to mean an em-dash all my life. :-)

-- 
--Guido van Rossum (python.org/~guido)

From ezio.melotti at gmail.com  Thu Jul 18 23:49:18 2013
From: ezio.melotti at gmail.com (Ezio Melotti)
Date: Thu, 18 Jul 2013 23:49:18 +0200
Subject: [Python-Dev] Dash
In-Reply-To: <ks996h$p10$1@ger.gmane.org>
References: <ks996h$p10$1@ger.gmane.org>
Message-ID: <CACBhJdG5z-mJ3jvx3rjHc6a2vB4jYeiiu9mQAwoVD5to7Ha0Zw@mail.gmail.com>

Hi,

On Thu, Jul 18, 2013 at 7:38 PM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> What type of dash is preferable in the documentation? The en dash (?) or the
> em dash (?)?
>

Both should be used where appropriate [0].  The em dash is more
common, however "--" (commonly used to indicate the em dash in e.g.
emails) gets converted to an en dash by Sphinx [1].  I noticed this a
while ago, and I started using "---" in the documentation whenever I
wanted an em dash.
If this is not documented it should be added to the "documenting" page
of the devguide, so that people start using the right ones and convert
the wrong ones when they come across them.

Best Regards,
Ezio Melotti

[0]: http://en.wikipedia.org/wiki/Dash
[1]: https://bitbucket.org/birkenfeld/sphinx/src/default/sphinx/util/smartypants.py#cl-261

From steve at pearwood.info  Fri Jul 19 06:51:39 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 19 Jul 2013 14:51:39 +1000
Subject: [Python-Dev] Dash
In-Reply-To: <ks9d5b$9sc$1@ger.gmane.org>
References: <ks996h$p10$1@ger.gmane.org>
	<CAP7+vJJAiLM7n88LywEk-Z9L7jv4nEJFvhwkR3E8-qkTnZbasg@mail.gmail.com>
	<ks9d5b$9sc$1@ger.gmane.org>
Message-ID: <51E8C5DB.9010701@pearwood.info>

On 19/07/13 04:46, Serhiy Storchaka wrote:
> 18.07.13 20:48, Guido van Rossum ???????(??):
>> I believe there are only a few places where en-dashes should be used,
>> for most things you should use either em-dash or hyphen. Consult your
>> trusted typography source (for US English, please, punctuation
>> preferences vary by locale). E.g. Google for "em dash en dash".
>
> Currently Python documentation in most cases uses en-dashes. Should we replace them to em-dashes? Should we remove spaces around dashes?
>
> Or we should replace a half-dozen of em-dashes found in Python documentation to en-dashes?
>
> I believe all hypens used in place of dash should be replaced to dash (but to en- or em- dash?) in any case.

It depends on the context, and I don't believe you could completely automate the process (at least not without using something that understands natural language, like NLTK, and probably not even then). I think it will require a human reader to review them, like any other style and grammar edit.

Wikipedia has a good overview which mostly agrees with my typesetting and style books:

https://en.wikipedia.org/wiki/Dash

Hyphens are commonly used for compound words, although in practice hyphenated words gradually lose the hyphen. E.g. we usually write "inbox" rather than "in-box", although we still write "in-tray". Hyphens are also used at the end of the line to break a word to the next line.

En-dashes are used for durations and ranges (sometimes with a thin space on either side, otherwise a regular space can be used). E.g. "October?December".

En-dash is also used when making a compound word from words which themselves are compound words, e.g. "The pre?World War II economy" joins "pre-" with "World War II", not just "World".

Em-dashes are used for parenthetical asides, or to indicate a break in speech or thought. They are often used when a comma is too weak and a period is too strong?a bit like a colon.

Different sources give different recommendations regarding spaces around dashes. The Chicago Manual of Style agrees with most British sources that em-dashes should never have spaces around them, but the New York Times style guide sets hair-spaces around them. Unusually for me, I tend to agree with the NY Times on this one. A regular space is usually too wide.

Many of these conventions are style conventions, rather than strictly grammatical. For example, although English grammar says we can use an en-dash to make ranges of numbers, the SI standard recommends against expressions like "10?50 volts" since it can be mistaken for subtraction, and recommends "10 to 50 volts".

Optimistically, I think it would probably be safe[1] to replace " -- " or " --- " in text with "\N{THIN SPACE}\N{EM DASH}\N{THIN SPACE}" (or \N{HAIR SPACE} if you prefer) without human review, but for any other changes, I wouldn't even try to automate it.






[1] Famous last words.


-- 
Steven

From ben+python at benfinney.id.au  Fri Jul 19 09:25:39 2013
From: ben+python at benfinney.id.au (Ben Finney)
Date: Fri, 19 Jul 2013 17:25:39 +1000
Subject: [Python-Dev] Dash
References: <ks996h$p10$1@ger.gmane.org>
Message-ID: <7wmwpjqado.fsf@benfinney.id.au>

Serhiy Storchaka <storchaka at gmail.com> writes:

> What type of dash is preferable in the documentation? The en dash (?)
> or the em dash (?)?

They have different purposes; use whichever is appropriate for the
context.

This isn't an official Python developer position. But all of ?-?, ???,
???, ???, etc. have distinct and common meanings in English text. So
it's a mistaken idea to think of ?which dash is preferable? since they
are *all* preferable for their distinct uses.

-- 
 \                ?Science doesn't work by vote and it doesn't work by |
  `\        authority.? ?Richard Dawkins, _Big Mistake_ (The Guardian, |
_o__)                                                      2006-12-27) |
Ben Finney


From ncoghlan at gmail.com  Fri Jul 19 10:38:05 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 19 Jul 2013 18:38:05 +1000
Subject: [Python-Dev] Dash
In-Reply-To: <7wmwpjqado.fsf@benfinney.id.au>
References: <ks996h$p10$1@ger.gmane.org>
	<7wmwpjqado.fsf@benfinney.id.au>
Message-ID: <CADiSq7d0gm3kyRzp-Q40qG9Z24CcxqA9=_sCZmBtJhVqCwhXQg@mail.gmail.com>

On 19 July 2013 17:25, Ben Finney <ben+python at benfinney.id.au> wrote:
> Serhiy Storchaka <storchaka at gmail.com> writes:
>
>> What type of dash is preferable in the documentation? The en dash (-)
>> or the em dash (--)?
>
> They have different purposes; use whichever is appropriate for the
> context.
>
> This isn't an official Python developer position. But all of '-', '-',
> '--', '-', etc. have distinct and common meanings in English text. So
> it's a mistaken idea to think of "which dash is preferable" since they
> are *all* preferable for their distinct uses.

I don't know about "common". I had no idea there were 3 dash types
until a couple of weeks ago. I thought there were only two
(short/hyphen/minus-sign and long/em-dash). I still don't really know
what the difference is between the two long ones (en-dash vs em-dash).

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From solipsis at pitrou.net  Fri Jul 19 10:46:20 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 19 Jul 2013 10:46:20 +0200
Subject: [Python-Dev] Dash
References: <ks996h$p10$1@ger.gmane.org> <7wmwpjqado.fsf@benfinney.id.au>
	<CADiSq7d0gm3kyRzp-Q40qG9Z24CcxqA9=_sCZmBtJhVqCwhXQg@mail.gmail.com>
Message-ID: <20130719104620.3c5ce104@pitrou.net>

Le Fri, 19 Jul 2013 18:38:05 +1000,
Nick Coghlan <ncoghlan at gmail.com> a ?crit :
> On 19 July 2013 17:25, Ben Finney <ben+python at benfinney.id.au> wrote:
> > Serhiy Storchaka <storchaka at gmail.com> writes:
> >
> >> What type of dash is preferable in the documentation? The en dash
> >> (-) or the em dash (--)?
> >
> > They have different purposes; use whichever is appropriate for the
> > context.
> >
> > This isn't an official Python developer position. But all of '-',
> > '-', '--', '-', etc. have distinct and common meanings in English
> > text. So it's a mistaken idea to think of "which dash is
> > preferable" since they are *all* preferable for their distinct uses.
> 
> I don't know about "common". I had no idea there were 3 dash types
> until a couple of weeks ago. I thought there were only two
> (short/hyphen/minus-sign and long/em-dash). I still don't really know
> what the difference is between the two long ones (en-dash vs em-dash).

But there aren't 3 dash types. There are at least 5 of them!
https://en.wikipedia.org/wiki/Dash#Common_dashes

(not to mention that Twisted has/had its own dash, IIRC)

cheers

Antoine.



From steve at pearwood.info  Fri Jul 19 11:53:10 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 19 Jul 2013 19:53:10 +1000
Subject: [Python-Dev] Dash
In-Reply-To: <CADiSq7d0gm3kyRzp-Q40qG9Z24CcxqA9=_sCZmBtJhVqCwhXQg@mail.gmail.com>
References: <ks996h$p10$1@ger.gmane.org> <7wmwpjqado.fsf@benfinney.id.au>
	<CADiSq7d0gm3kyRzp-Q40qG9Z24CcxqA9=_sCZmBtJhVqCwhXQg@mail.gmail.com>
Message-ID: <51E90C86.90205@pearwood.info>

On 19/07/13 18:38, Nick Coghlan wrote:

> I don't know about "common". I had no idea there were 3 dash types

*Way* more than three.

hyphen
minus sign (not the same as a hyphen!)
en-dash
em-dash
two-em-dash
three-em-dash
figure dash
horizontal bar
swung dash

plus another half-dozen or so non-English hyphens, all of which have different typographic (and sometimes grammatical) purposes. Wikipedia has a good description:

https://en.wikipedia.org/wiki/Dash

but a brief summary is:

? hyphen - is the smallest and is used for compound words like "out-tray";
? minus sign ? is normally raised compared to the others;
? en-dash ? should be the width of "n" and is used for ranges, like Mon?Fri;
? em-dash ? should be the width of "m" and is used for a break stronger than a comma but weaker than a full stop;
? horizontal bar ? should be longer than the em-dash and is used for quotations in some European languages;
? swung dash ? is normally found in dictionaries (paper, not Python) to stand in for the word being defined.


But of course different fonts follow these rules to a greater or lesser degree.



-- 
Steven

From storchaka at gmail.com  Fri Jul 19 11:55:01 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Fri, 19 Jul 2013 12:55:01 +0300
Subject: [Python-Dev] Dash
In-Reply-To: <CAD+XWwreHB3v_kvbZ0tJsqxhydqkwZ0K9UGkrRsAJL0+nmX6mA@mail.gmail.com>
References: <ks996h$p10$1@ger.gmane.org>
	<CAP7+vJJAiLM7n88LywEk-Z9L7jv4nEJFvhwkR3E8-qkTnZbasg@mail.gmail.com>
	<ks9d5b$9sc$1@ger.gmane.org>
	<CAD+XWwreHB3v_kvbZ0tJsqxhydqkwZ0K9UGkrRsAJL0+nmX6mA@mail.gmail.com>
Message-ID: <ksb2dd$n6l$1@ger.gmane.org>

18.07.13 21:54, Brian Curtin ???????(??):
> Besides visual consistency in a couple of places, is there a reason to
> care enough to make a wholesale change?

Single hyphen instead of a dash just looks too ugly to me.

Trying to fix this I noticed that the documentation is inconsistent 
regarding the dash. Sometimes en and em dashes meet in the same line and 
in the same sentence. I.e.:

"""
The :mod:`weakref` module also allows creating proxy objects which 
behave like
weak references --- an object referenced only by proxy objects is 
deallocated --
but instead of requiring an explicit call to retrieve the object, the proxy
"""



From storchaka at gmail.com  Fri Jul 19 12:03:59 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Fri, 19 Jul 2013 13:03:59 +0300
Subject: [Python-Dev] Dash
In-Reply-To: <CAP7+vJJYN6DY0=zcDNZUgUfYF2T_SoC57fZCTVzpWtBk08W1+Q@mail.gmail.com>
References: <ks996h$p10$1@ger.gmane.org>
	<CAP7+vJJAiLM7n88LywEk-Z9L7jv4nEJFvhwkR3E8-qkTnZbasg@mail.gmail.com>
	<ks9d5b$9sc$1@ger.gmane.org>
	<CAP7+vJJYN6DY0=zcDNZUgUfYF2T_SoC57fZCTVzpWtBk08W1+Q@mail.gmail.com>
Message-ID: <ksb2u7$s7u$1@ger.gmane.org>

18.07.13 21:55, Guido van Rossum ???????(??):
> On Thu, Jul 18, 2013 at 11:46 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:
>> Or we should replace a half-dozen of em-dashes found in Python documentation
>> to en-dashes?
>
> If my theory is right that makes sense. Especially if it's only a half-dozen.

I'm sorry, I were wrong. Actually em-dash used much more then en-dash in 
Python documentation.



From storchaka at gmail.com  Fri Jul 19 12:16:45 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Fri, 19 Jul 2013 13:16:45 +0300
Subject: [Python-Dev] Dash
In-Reply-To: <CACBhJdG5z-mJ3jvx3rjHc6a2vB4jYeiiu9mQAwoVD5to7Ha0Zw@mail.gmail.com>
References: <ks996h$p10$1@ger.gmane.org>
	<CACBhJdG5z-mJ3jvx3rjHc6a2vB4jYeiiu9mQAwoVD5to7Ha0Zw@mail.gmail.com>
Message-ID: <ksb3m5$4q4$1@ger.gmane.org>

19.07.13 00:49, Ezio Melotti ???????(??):
> On Thu, Jul 18, 2013 at 7:38 PM, Serhiy Storchaka <storchaka at gmail.com> wrote:
>> What type of dash is preferable in the documentation? The en dash (?) or the
>> em dash (?)?
>>
>
> Both should be used where appropriate [0].

Of course I looked in Wikipedia before asking on this list. And I were 
surprised that en-dash allowed in same places as em-dashes:

"""
Either version may be used to denote a break in a sentence or to set off 
parenthetical statements, although writers are generally cautioned to 
use a single form consistently within their work. In this function, en 
dashes are used with spaces and em dashes are used without them:[1]

[Em dash:] In matters of grave importance, style?not sincerity?is the 
vital thing.

[En dash:] In matters of grave importance, style ? not sincerity ? is 
the vital thing.
"""

I'm asking only about this case, when the dash is used to denote a break 
in a sentence or to set off parenthetical statements. Currently Python 
documentation uses fourth variants:

1. A single hyphen "-" surrounded with spaces. "In matters of grave 
importance, style - not sincerity - is the vital thing."
I think this case should be converted to one of followed, but to what of 
them?

2. A double hyphen "--" (converted to en-dash) surrounded with spaces. 
"In matters of grave importance, style -- not sincerity -- is the vital 
thing."

3. A triple hyphen "---" (converted to em-dash) surrounded with spaces. 
"In matters of grave importance, style --- not sincerity --- is the 
vital thing."

4. A triple hyphen "---" (converted to em-dash) which isn't surrounded 
with spaces. "In matters of grave importance, style---not sincerity---is 
the vital thing."

>  The em dash is more
> common, however "--" (commonly used to indicate the em dash in e.g.
> emails) gets converted to an en dash by Sphinx [1].  I noticed this a
> while ago, and I started using "---" in the documentation whenever I
> wanted an em dash.

This is common in TeX.



From storchaka at gmail.com  Fri Jul 19 12:20:30 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Fri, 19 Jul 2013 13:20:30 +0300
Subject: [Python-Dev] Dash
In-Reply-To: <7wmwpjqado.fsf@benfinney.id.au>
References: <ks996h$p10$1@ger.gmane.org> <7wmwpjqado.fsf@benfinney.id.au>
Message-ID: <ksb3t7$7h3$1@ger.gmane.org>

19.07.13 10:25, Ben Finney ???????(??):
> Serhiy Storchaka <storchaka at gmail.com> writes:
>> What type of dash is preferable in the documentation? The en dash (?)
>> or the em dash (?)?
>
> They have different purposes; use whichever is appropriate for the
> context.

I mean only a context where a dash is used to denote a break in a 
sentence or to set off parenthetical statements. Currently Python 
documentation uses four different variants (see my answer to Ezio).



From storchaka at gmail.com  Fri Jul 19 12:32:34 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Fri, 19 Jul 2013 13:32:34 +0300
Subject: [Python-Dev] Dash
In-Reply-To: <51E8C5DB.9010701@pearwood.info>
References: <ks996h$p10$1@ger.gmane.org>
	<CAP7+vJJAiLM7n88LywEk-Z9L7jv4nEJFvhwkR3E8-qkTnZbasg@mail.gmail.com>
	<ks9d5b$9sc$1@ger.gmane.org> <51E8C5DB.9010701@pearwood.info>
Message-ID: <ksb4jq$eta$1@ger.gmane.org>

19.07.13 07:51, Steven D'Aprano ???????(??):
> Optimistically, I think it would probably be safe[1] to replace " -- "
> or " --- " in text with "\N{THIN SPACE}\N{EM DASH}\N{THIN SPACE}" (or
> \N{HAIR SPACE} if you prefer) without human review, but for any other
> changes, I wouldn't even try to automate it.

No, it is not safe. " -- " used in code examples in which it doesn't 
converted to en-dash. Also we should correct tables and underlines in 
titles.

I have a manually written and multiple times checked patch (modifies 
about 640 lines) which safely replaces " -- " with " --- " but I doubt 
what actually should be a preferable dash.


From status at bugs.python.org  Fri Jul 19 18:07:37 2013
From: status at bugs.python.org (Python tracker)
Date: Fri, 19 Jul 2013 18:07:37 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20130719160737.0EE1A56A99@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2013-07-12 - 2013-07-19)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    4092 (+25)
  closed 26185 (+45)
  total  30277 (+70)

Open issues with patches: 1846 


Issues opened (46)
==================

#18436: Add mapping of symbol to function to operator module
http://bugs.python.org/issue18436  opened by zach.ware

#18438: Obsolete url in comment inside decimal module
http://bugs.python.org/issue18438  opened by vajrasky

#18439: Patchcheck for ACKS, NEWS does not work on Windows.
http://bugs.python.org/issue18439  opened by terry.reedy

#18441: Idle: Make test.support.requires('gui') skip when it should.
http://bugs.python.org/issue18441  opened by terry.reedy

#18443: Misc/Readme still documents TextMate
http://bugs.python.org/issue18443  opened by iwontbecreative

#18444: IDLE: Mac OS X pressing ctrl-A in Python shell moved cursor be
http://bugs.python.org/issue18444  opened by Todd.Rovito

#18445: Tools/Script/Readme is outdated
http://bugs.python.org/issue18445  opened by iwontbecreative

#18451: Omit test files in devinabox coverage run
http://bugs.python.org/issue18451  opened by brett.cannon

#18453: There are unused variables inside DateTimeTestCase class in te
http://bugs.python.org/issue18453  opened by vajrasky

#18454: distutils crashes when uploading to PyPI having only the usern
http://bugs.python.org/issue18454  opened by jgehrcke

#18456: Documentation for PyDict_Update is incorrect
http://bugs.python.org/issue18456  opened by ncoghlan

#18458: libedit history offset workaround
http://bugs.python.org/issue18458  opened by ronaldoussoren

#18459: readline: libedit support on non-apple platforms
http://bugs.python.org/issue18459  opened by ronaldoussoren

#18460: .chm documentation files advertised in download.html but not t
http://bugs.python.org/issue18460  opened by iwontbecreative

#18461: X Error in tkinter
http://bugs.python.org/issue18461  opened by serhiy.storchaka

#18465: There are unused variables and unused import in Lib/test/test_
http://bugs.python.org/issue18465  opened by vajrasky

#18466: Spelling mistakes in various code comments.
http://bugs.python.org/issue18466  opened by iwontbecreative

#18467: argparse - problematic 'default' behavior
http://bugs.python.org/issue18467  opened by matej.tyc

#18468: re.group() should never return a bytearray
http://bugs.python.org/issue18468  opened by gvanrossum

#18472: PEP 8 updates - internal interfaces and import *
http://bugs.python.org/issue18472  opened by ncoghlan

#18473: some objects pickled by Python 3.x are not unpicklable in Pyth
http://bugs.python.org/issue18473  opened by kmike

#18476: No way to pass custom arguments for loggers and formatters.
http://bugs.python.org/issue18476  opened by piotr.dobrogost

#18477: fix_import (2to3 fixer) is not case-sensitive
http://bugs.python.org/issue18477  opened by Valentin.Lorentz

#18478: Class bodies: when does a name become local?
http://bugs.python.org/issue18478  opened by terry.reedy

#18481: lcov report
http://bugs.python.org/issue18481  opened by christian.heimes

#18483: Add hour-24 type of time to test_http2time_formats in test_htt
http://bugs.python.org/issue18483  opened by vajrasky

#18484: No unit test for iso2time function from http.cookiejar module
http://bugs.python.org/issue18484  opened by vajrasky

#18485: mingw: configure for shared build
http://bugs.python.org/issue18485  opened by rpetrov

#18486: mingw: dynamic loading support
http://bugs.python.org/issue18486  opened by rpetrov

#18487: mingw implement exec prefix
http://bugs.python.org/issue18487  opened by rpetrov

#18488: sqlite: finalize() method of user function may be called with 
http://bugs.python.org/issue18488  opened by haypo

#18489: IDLE Unit test for SearchEngine.py
http://bugs.python.org/issue18489  opened by philwebster

#18491: Add "exe wrapper" functionality to Windows launcher
http://bugs.python.org/issue18491  opened by pmoore

#18492: Add test.support.regrtest_run flag, simplify support.requires
http://bugs.python.org/issue18492  opened by zach.ware

#18493: make profile-opt fails with pre-existing python2.7 in path
http://bugs.python.org/issue18493  opened by Claudio.Freire

#18494: PyType_GenericSet/GetDict functions misnamed in docs?
http://bugs.python.org/issue18494  opened by Wolf.Ihlenfeldt

#18495: mingw: ignore main program for frozen scripts
http://bugs.python.org/issue18495  opened by rpetrov

#18496: mingw: setup exclude termios module
http://bugs.python.org/issue18496  opened by rpetrov

#18497: mingw: setup _multiprocessing module
http://bugs.python.org/issue18497  opened by rpetrov

#18498: mingw: setup select module
http://bugs.python.org/issue18498  opened by rpetrov

#18499: mingw: setup _ctypes module with system libffi
http://bugs.python.org/issue18499  opened by rpetrov

#18500: mingw: defect winsock2 and setup _socket module
http://bugs.python.org/issue18500  opened by rpetrov

#18501: _elementtree.c calls Python callbacks while a Python exception
http://bugs.python.org/issue18501  opened by haypo

#18502: CDLL does not use same paths as util.find_library
http://bugs.python.org/issue18502  opened by shjohnson.pi

#18503: No assertion in test_del_param_on_nonexistent_header function
http://bugs.python.org/issue18503  opened by vajrasky

#18504: IDLE:Improvements- Improving Mock_Text
http://bugs.python.org/issue18504  opened by JayKrish



Most recent 15 issues with no replies (15)
==========================================

#18504: IDLE:Improvements- Improving Mock_Text
http://bugs.python.org/issue18504

#18502: CDLL does not use same paths as util.find_library
http://bugs.python.org/issue18502

#18500: mingw: defect winsock2 and setup _socket module
http://bugs.python.org/issue18500

#18499: mingw: setup _ctypes module with system libffi
http://bugs.python.org/issue18499

#18498: mingw: setup select module
http://bugs.python.org/issue18498

#18497: mingw: setup _multiprocessing module
http://bugs.python.org/issue18497

#18496: mingw: setup exclude termios module
http://bugs.python.org/issue18496

#18495: mingw: ignore main program for frozen scripts
http://bugs.python.org/issue18495

#18494: PyType_GenericSet/GetDict functions misnamed in docs?
http://bugs.python.org/issue18494

#18493: make profile-opt fails with pre-existing python2.7 in path
http://bugs.python.org/issue18493

#18489: IDLE Unit test for SearchEngine.py
http://bugs.python.org/issue18489

#18487: mingw implement exec prefix
http://bugs.python.org/issue18487

#18486: mingw: dynamic loading support
http://bugs.python.org/issue18486

#18485: mingw: configure for shared build
http://bugs.python.org/issue18485

#18484: No unit test for iso2time function from http.cookiejar module
http://bugs.python.org/issue18484



Most recent 15 issues waiting for review (15)
=============================================

#18500: mingw: defect winsock2 and setup _socket module
http://bugs.python.org/issue18500

#18499: mingw: setup _ctypes module with system libffi
http://bugs.python.org/issue18499

#18498: mingw: setup select module
http://bugs.python.org/issue18498

#18497: mingw: setup _multiprocessing module
http://bugs.python.org/issue18497

#18496: mingw: setup exclude termios module
http://bugs.python.org/issue18496

#18495: mingw: ignore main program for frozen scripts
http://bugs.python.org/issue18495

#18492: Add test.support.regrtest_run flag, simplify support.requires
http://bugs.python.org/issue18492

#18491: Add "exe wrapper" functionality to Windows launcher
http://bugs.python.org/issue18491

#18489: IDLE Unit test for SearchEngine.py
http://bugs.python.org/issue18489

#18487: mingw implement exec prefix
http://bugs.python.org/issue18487

#18486: mingw: dynamic loading support
http://bugs.python.org/issue18486

#18485: mingw: configure for shared build
http://bugs.python.org/issue18485

#18481: lcov report
http://bugs.python.org/issue18481

#18477: fix_import (2to3 fixer) is not case-sensitive
http://bugs.python.org/issue18477

#18472: PEP 8 updates - internal interfaces and import *
http://bugs.python.org/issue18472



Top 10 most discussed issues (10)
=================================

#18408: Fixes crashes found by pyfailmalloc
http://bugs.python.org/issue18408  34 msgs

#18373: implement sys.get/setbyteswarningflag()
http://bugs.python.org/issue18373   9 msgs

#18491: Add "exe wrapper" functionality to Windows launcher
http://bugs.python.org/issue18491   9 msgs

#11874: argparse assertion failure with brackets in metavars
http://bugs.python.org/issue11874   8 msgs

#17214: http.client.HTTPConnection.putrequest encode  error
http://bugs.python.org/issue17214   7 msgs

#17933: test_ftp failure / ftplib error formatting issue
http://bugs.python.org/issue17933   7 msgs

#18258: Fix test discovery for test_codecmaps*.py
http://bugs.python.org/issue18258   7 msgs

#18444: IDLE: Mac OS X pressing ctrl-A in Python shell moved cursor be
http://bugs.python.org/issue18444   7 msgs

#18305: [patch] Fast sum() for non-numbers
http://bugs.python.org/issue18305   6 msgs

#18387: Add 'symbols' link to pydoc's html menu bar.
http://bugs.python.org/issue18387   6 msgs



Issues closed (42)
==================

#17482: functools.update_wrapper mishandles __wrapped__
http://bugs.python.org/issue17482  closed by python-dev

#17767: Fix test discovery for test_locale.py
http://bugs.python.org/issue17767  closed by serhiy.storchaka

#17778: Fix test discovery for test_multiprocessing.py
http://bugs.python.org/issue17778  closed by sbt

#18021: Update broken link to Apple Publication Style Guide
http://bugs.python.org/issue18021  closed by ezio.melotti

#18042: Provide enum.unique class decorator
http://bugs.python.org/issue18042  closed by python-dev

#18266: Fix test discovery for test_largefile.py
http://bugs.python.org/issue18266  closed by serhiy.storchaka

#18279: IDLE Unit test for RstripExtension.py
http://bugs.python.org/issue18279  closed by terry.reedy

#18336: codecs: Link to readline module (history) instead of fd.readli
http://bugs.python.org/issue18336  closed by serhiy.storchaka

#18344: _bufferedreader_read_all() may leak reference to data
http://bugs.python.org/issue18344  closed by sbt

#18365: Idle: mock Text class and test thereof
http://bugs.python.org/issue18365  closed by terry.reedy

#18366: Look into using setuptools 0.8 with devinabox
http://bugs.python.org/issue18366  closed by brett.cannon

#18393: Gestalt() is deprecated on OSX 10.8, remove support?
http://bugs.python.org/issue18393  closed by ronaldoussoren

#18428: IDLE startup error
http://bugs.python.org/issue18428  closed by ned.deily

#18429: IDLE: Format Paragraph doesn't function with comment blocks
http://bugs.python.org/issue18429  closed by Todd.Rovito

#18431: Encoded words in structured headers are not decoded by the new
http://bugs.python.org/issue18431  closed by r.david.murray

#18432: sched modules queue property should return a list, not an iter
http://bugs.python.org/issue18432  closed by rhettinger

#18433: Clarify venv docs about which methods check for pre-existing p
http://bugs.python.org/issue18433  closed by python-dev

#18434: Update venv example to use setuptools
http://bugs.python.org/issue18434  closed by python-dev

#18435: Move venv from a custom Context class to types.SimpleNamespace
http://bugs.python.org/issue18435  closed by python-dev

#18437: Typo in comment inside class Charset about output_charset
http://bugs.python.org/issue18437  closed by r.david.murray

#18440: hash() and __hash__() do not document their size constraints
http://bugs.python.org/issue18440  closed by barry

#18442: Typo in developers.rst line 27
http://bugs.python.org/issue18442  closed by ned.deily

#18446: Dead link in documenting.rst
http://bugs.python.org/issue18446  closed by ezio.melotti

#18447: Pydoc crashes on large contents
http://bugs.python.org/issue18447  closed by ned.deily

#18448: Tools/demo/eiffel.py fails to run tests.
http://bugs.python.org/issue18448  closed by serhiy.storchaka

#18450: Typos in developers.rst
http://bugs.python.org/issue18450  closed by python-dev

#18452: Typo in Doc/howto/argparse.rst
http://bugs.python.org/issue18452  closed by python-dev

#18455: Multiprocessing connection SocketClient retries connection on 
http://bugs.python.org/issue18455  closed by sbt

#18457: Fix and cleanup the ss1.py demo
http://bugs.python.org/issue18457  closed by serhiy.storchaka

#18462: Download > Other documentation page contains a link to a frenc
http://bugs.python.org/issue18462  closed by ned.deily

#18464: Typo in the name of function in test__encoded_words.py
http://bugs.python.org/issue18464  closed by python-dev

#18469: PyStructSequence does not handle exceptions correctly
http://bugs.python.org/issue18469  closed by python-dev

#18470: tokenizer.c does not handle new_string() failure
http://bugs.python.org/issue18470  closed by python-dev

#18471: Typo in heapq documentation
http://bugs.python.org/issue18471  closed by python-dev

#18474: Lambda assigned to object does not automatically get self
http://bugs.python.org/issue18474  closed by amaury.forgeotdarc

#18475: No unittest.main in Lib/test/test_email/test_inversion.py
http://bugs.python.org/issue18475  closed by python-dev

#18479: Improvements to Powershell activate script for venv
http://bugs.python.org/issue18479  closed by python-dev

#18480: _elementtree: missing PyType_Ready call
http://bugs.python.org/issue18480  closed by ronaldoussoren

#18482: dis.dis fails on one letter strings.
http://bugs.python.org/issue18482  closed by r.david.murray

#18490: Error embedding datetime in C++
http://bugs.python.org/issue18490  closed by christian.heimes

#18449: Demo 'ss1.py' crash in python3
http://bugs.python.org/issue18449  closed by serhiy.storchaka

#18463: Download > Other documentation page outdated/dead links
http://bugs.python.org/issue18463  closed by ned.deily

From ben+python at benfinney.id.au  Fri Jul 19 21:26:30 2013
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sat, 20 Jul 2013 05:26:30 +1000
Subject: [Python-Dev] Dash
References: <ks996h$p10$1@ger.gmane.org> <7wmwpjqado.fsf@benfinney.id.au>
	<ksb3t7$7h3$1@ger.gmane.org>
Message-ID: <7wwqom4ahl.fsf@benfinney.id.au>

Serhiy Storchaka <storchaka at gmail.com> writes:

> 19.07.13 10:25, Ben Finney ???????(??):
> > Serhiy Storchaka <storchaka at gmail.com> writes:
> >> What type of dash is preferable in the documentation? The en dash
> >> (?) or the em dash (?)?
> >
> > They have different purposes; use whichever is appropriate for the
> > context.
>
> I mean only a context where a dash is used to denote a break in a
> sentence or to set off parenthetical statements.

Yes, and both of those are used for different purposes, so ?what type is
preferable (for all the purposes you mention)? is a poor way of looking
at it, IMO.

Local style guides have rules about when and how to use the different
dashes for specific contexts.

(For myself, I prefer parenthetic dashes to be en dashes ???, and a
break in a sentence to be marked by an em dash ???. But I also prefer to
surround the dash with space for the latter purpose, so my preference
isn't universal.)

Wikipedia's article (already referred) discusses the matter fairly well
<URL:https://en.wikipedia.org/wiki/Dashes>.

> Currently Python documentation uses four different variants (see my
> answer to Ezio).

I'd recommend that the Python documentation BDFL should choose and
dictate an existing well-groomed style guide for Python documentation,
preferably one which explicitly talks about when to use each of the
commonly-used dashes.

-- 
 \        ?If you have the facts on your side, pound the facts. If you |
  `\     have the law on your side, pound the law. If you have neither |
_o__)                       on your side, pound the table.? ?anonymous |
Ben Finney


From ben+python at benfinney.id.au  Fri Jul 19 21:29:13 2013
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sat, 20 Jul 2013 05:29:13 +1000
Subject: [Python-Dev] Dash
References: <ks996h$p10$1@ger.gmane.org>
	<CAP7+vJJAiLM7n88LywEk-Z9L7jv4nEJFvhwkR3E8-qkTnZbasg@mail.gmail.com>
	<ks9d5b$9sc$1@ger.gmane.org>
	<CAP7+vJJYN6DY0=zcDNZUgUfYF2T_SoC57fZCTVzpWtBk08W1+Q@mail.gmail.com>
	<ksb2u7$s7u$1@ger.gmane.org>
Message-ID: <7wsiza4ad2.fsf@benfinney.id.au>

Serhiy Storchaka <storchaka at gmail.com> writes:

> I'm sorry, I were wrong. Actually em-dash used much more then en-dash
> in Python documentation.

But, as you've also discovered, many authors will type two hyphens ?--?
when they want an em dash ???, yet this is rendered to an en dash ???.
So the prevalence of the latter in the documentation probably does not
reflect the preferences of the authors.

-- 
 \         ?Nature hath given men one tongue but two ears, that we may |
  `\          hear from others twice as much as we speak.? ?Epictetus, |
_o__)                                                      _Fragments_ |
Ben Finney


From ben+python at benfinney.id.au  Fri Jul 19 21:32:38 2013
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sat, 20 Jul 2013 05:32:38 +1000
Subject: [Python-Dev] Dash
References: <ks996h$p10$1@ger.gmane.org>
	<CACBhJdG5z-mJ3jvx3rjHc6a2vB4jYeiiu9mQAwoVD5to7Ha0Zw@mail.gmail.com>
	<ksb3m5$4q4$1@ger.gmane.org>
Message-ID: <7wob9y4a7d.fsf@benfinney.id.au>

Serhiy Storchaka <storchaka at gmail.com> writes:

> I'm asking only about this case, when the dash is used to denote a
> break in a sentence or to set off parenthetical statements.

That's two separate cases:

* denote a break in a sentence
* set off parenthetical statements

In my experience, an em dash is commonly used for the former, and en
dashes commonly used for the latter. Using the same dash for both is
unusual (and IMO needlessly ambiguous to the reader).

> Currently Python documentation uses fourth variants:

All your examples are only for parenthetical statements. Can you find
examples of the former, where a break (not parenthetical) in the
sentence is intended?

-- 
 \     ?When I was a kid I used to pray every night for a new bicycle. |
  `\    Then I realised that the Lord doesn't work that way so I stole |
_o__)                   one and asked Him to forgive me.? ?Emo Philips |
Ben Finney


From ethan at stoneleaf.us  Sat Jul 20 01:47:36 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 19 Jul 2013 16:47:36 -0700
Subject: [Python-Dev] dict __contains__ raises TypeError on unhashable input
Message-ID: <51E9D018.4030806@stoneleaf.us>

While working on issue #18508 I stumbled across this:

Traceback (most recent call last):
...
   File "/usr/local/lib/python3.4/enum.py", line 417, in __new__
     if value in cls._value2member_map:
TypeError: unhashable type: 'list'

I'll wrap it in a try-except block, but I must admit I was surprised the answer wasn't False.  After all, if the input 
is unhashable then obviously it's not in the dict; furthermore, if I were to compare the number 5 with a set() I would 
get False, not a TypeMismatch error, and dict keys are basically done by equality, the hash is just (?) a speed-up.

--
~Ethan~

From ethan at stoneleaf.us  Sat Jul 20 03:30:23 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 19 Jul 2013 18:30:23 -0700
Subject: [Python-Dev] dict __contains__ raises TypeError on unhashable
 input
In-Reply-To: <51E9D018.4030806@stoneleaf.us>
References: <51E9D018.4030806@stoneleaf.us>
Message-ID: <51E9E82F.2060804@stoneleaf.us>

http://bugs.python.org/issue18510

Commenting further:

     some_key in dict

is conceptually the same as

     some_key in dict.keys()

which /would/ return False for an unhashable key -- at least it did in 2.x; for 3.x you have to say

     some_key in list(dict.keys())

which seems like a step backwards.

Is it worth changing __contains__ and keys() to be in line with equality?

--
~Ethan~

From benhoyt at gmail.com  Sat Jul 20 05:11:44 2013
From: benhoyt at gmail.com (Ben Hoyt)
Date: Sat, 20 Jul 2013 15:11:44 +1200
Subject: [Python-Dev] dict __contains__ raises TypeError on unhashable
	input
In-Reply-To: <51E9E82F.2060804@stoneleaf.us>
References: <51E9D018.4030806@stoneleaf.us>
	<51E9E82F.2060804@stoneleaf.us>
Message-ID: <CAL9jXCEUGRdkSjrAYUrU3qYUzr5i=5z+-ZeAOA3i4WqeL6X4DA@mail.gmail.com>

"unhashable key" doesn't make sense to me. If it's a "key", it has to be
hashable, and I'd think it'd often be a sign of a bug if you were trying to
do "unhashable_value in mapping", so to me it seems good that it raises a
TypeError. It totally depends on what's on the RHS of the in -- for
instance, you get a TypeError if you do:

42 in 'xyz'

Whereas by your reasoning, it should return False, because the integer 42
can never be "in" that string. But really you want the TypeError, because
that's a bug waiting to happen.

This is something of an aside, but I like the fact that Python is
(relatively) strongly typed, and I think this is an example of that. Just
the other day I was surprised to learn that in both Java and C# you can do:

"foo" + 3 + 4

And get "foo34" without any compile-time or run-time error (in a statically
typed language like Java or C# I would have expected a compile-time error).
A friend had written code equivalent to that, except with the 3 and 4 being
integer variables. He was expecting to see "foo7" because that was his
intent, but then realized what was going on -- numbers are automatically
converted to strings in this context, and the + operators are applied
left-to-right.

Whereas if you try that in Python, bomp -- TypeError: cannot concatenate
'str' and 'int' objects!

-Ben


On Sat, Jul 20, 2013 at 1:30 PM, Ethan Furman <ethan at stoneleaf.us> wrote:

> http://bugs.python.org/**issue18510 <http://bugs.python.org/issue18510>
>
> Commenting further:
>
>     some_key in dict
>
> is conceptually the same as
>
>     some_key in dict.keys()
>
> which /would/ return False for an unhashable key -- at least it did in
> 2.x; for 3.x you have to say
>
>     some_key in list(dict.keys())
>
> which seems like a step backwards.
>
> Is it worth changing __contains__ and keys() to be in line with equality?
>
>
> --
> ~Ethan~
> ______________________________**_________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/**mailman/listinfo/python-dev<http://mail.python.org/mailman/listinfo/python-dev>
> Unsubscribe: http://mail.python.org/**mailman/options/python-dev/**
> benhoyt%40gmail.com<http://mail.python.org/mailman/options/python-dev/benhoyt%40gmail.com>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130720/cb889898/attachment-0001.html>

From benjamin at python.org  Sat Jul 20 09:13:47 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Sat, 20 Jul 2013 00:13:47 -0700
Subject: [Python-Dev] dict __contains__ raises TypeError on unhashable
	input
In-Reply-To: <51E9D018.4030806@stoneleaf.us>
References: <51E9D018.4030806@stoneleaf.us>
Message-ID: <CAPZV6o9AqZtBeXDRLPoLzzBeCrxxmyxsU63iu7b+NuV-5gm-Vg@mail.gmail.com>

2013/7/19 Ethan Furman <ethan at stoneleaf.us>:
> While working on issue #18508 I stumbled across this:
>
> Traceback (most recent call last):
> ...
>   File "/usr/local/lib/python3.4/enum.py", line 417, in __new__
>     if value in cls._value2member_map:
> TypeError: unhashable type: 'list'
>
> I'll wrap it in a try-except block, but I must admit I was surprised the
> answer wasn't False.  After all, if the input is unhashable then obviously
> it's not in the dict; furthermore, if I were to compare the number 5 with a
> set() I would get False, not a TypeMismatch error, and dict keys are
> basically done by equality, the hash is just (?) a speed-up.

I'm not exactly sure what the last part of that sentence means.

Anyway, it's well established that operations on a key in a dict are
going to involve looking up the key, and thus hashing it. You wouldn't
expect, {}.get(unhashable, None) not to raise, right?



--
Regards,
Benjamin

From ncoghlan at gmail.com  Sat Jul 20 09:39:29 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 20 Jul 2013 17:39:29 +1000
Subject: [Python-Dev] dict __contains__ raises TypeError on unhashable
	input
In-Reply-To: <51E9D018.4030806@stoneleaf.us>
References: <51E9D018.4030806@stoneleaf.us>
Message-ID: <CADiSq7cpfpcA156f4Hio9WgC14nLX1KzNZ8p=PfHXD5=yuamSg@mail.gmail.com>

On 20 July 2013 09:47, Ethan Furman <ethan at stoneleaf.us> wrote:
> While working on issue #18508 I stumbled across this:
>
> Traceback (most recent call last):
> ...
>   File "/usr/local/lib/python3.4/enum.py", line 417, in __new__
>     if value in cls._value2member_map:
> TypeError: unhashable type: 'list'
>
> I'll wrap it in a try-except block, but I must admit I was surprised the
> answer wasn't False.  After all, if the input is unhashable then obviously
> it's not in the dict; furthermore, if I were to compare the number 5 with a
> set() I would get False, not a TypeMismatch error, and dict keys are
> basically done by equality, the hash is just (?) a speed-up.

"in" is controlled entirely by the container. Both set() and dict()
consider passing them an unhashable object to be a categorical type
error:

>>> [] in set()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'list'
>>> [] in {}
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'list'

Equality is different, because people do equality comparisons between
unlike types all the time, so we try *very* hard to ensure "I do not
understand this type" is reported as False rather than "I don't know"
(as indicated by an exception). It's basically the only place where we
do that - pretty much everywhere else, we err on the side of "I do not
understand what this even means" translating to an exception (Python 3
took another big step in that direction by making sure ordered
comparisons between unlike types threw TypeError by default, instead
of returning a dubious answer).

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From solipsis at pitrou.net  Sat Jul 20 12:21:41 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 20 Jul 2013 12:21:41 +0200
Subject: [Python-Dev] dict __contains__ raises TypeError on unhashable
	input
References: <51E9D018.4030806@stoneleaf.us>
Message-ID: <20130720122141.148c8df8@fsol>

On Fri, 19 Jul 2013 16:47:36 -0700
Ethan Furman <ethan at stoneleaf.us> wrote:
> While working on issue #18508 I stumbled across this:
> 
> Traceback (most recent call last):
> ...
>    File "/usr/local/lib/python3.4/enum.py", line 417, in __new__
>      if value in cls._value2member_map:
> TypeError: unhashable type: 'list'
> 
> I'll wrap it in a try-except block, but I must admit I was surprised the answer wasn't False.  After all, if the input 
> is unhashable then obviously it's not in the dict; furthermore, if I were to compare the number 5 with a set() I would 
> get False, not a TypeMismatch error, and dict keys are basically done by equality, the hash is just (?) a speed-up.

Testing containment of an unhashable key in a dict is a programming
error, so I'd prefer that we keep the TypeError here.

Same reasoning as for:

>>> 5 in "abc"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'in <string>' requires string as left operand, not int

Sure, 5 technically isn't in "abc" and therefore the "in" operator
could return False, but it's better to highlight the programming error.

Regards

Antoine.



From ronaldoussoren at mac.com  Sat Jul 20 12:21:53 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Sat, 20 Jul 2013 12:21:53 +0200
Subject: [Python-Dev] dict __contains__ raises TypeError on unhashable
	input
In-Reply-To: <51E9D018.4030806@stoneleaf.us>
References: <51E9D018.4030806@stoneleaf.us>
Message-ID: <94771240-354C-4240-BFCF-BDFCE6FE2E9C@mac.com>


On 20 Jul, 2013, at 1:47, Ethan Furman <ethan at stoneleaf.us> wrote:

> While working on issue #18508 I stumbled across this:
> 
> Traceback (most recent call last):
> ...
>  File "/usr/local/lib/python3.4/enum.py", line 417, in __new__
>    if value in cls._value2member_map:
> TypeError: unhashable type: 'list'
> 
> I'll wrap it in a try-except block, but I must admit I was surprised the answer wasn't False.  After all, if the input is unhashable then obviously it's not in the dict; furthermore, if I were to compare the number 5 with a set() I would get False, not a TypeMismatch error, and dict keys are basically done by equality, the hash is just (?) a speed-up.

Not quite, there are some objects that compare equal without both of them being hashable:

>>> frozenset([1,2]) == set([1,2])
True
>>> dct = { frozenset([1,2]): 1 }
>>> frozenset([1,2]) in dct
True
>>> set([1,2]) in dct
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'set'

It would be strange if the last test would return False instead of raising an error.

Ronald


> 
> --
> ~Ethan~
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com


From storchaka at gmail.com  Sat Jul 20 14:18:02 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 20 Jul 2013 15:18:02 +0300
Subject: [Python-Dev] Dash
In-Reply-To: <7wob9y4a7d.fsf@benfinney.id.au>
References: <ks996h$p10$1@ger.gmane.org>
	<CACBhJdG5z-mJ3jvx3rjHc6a2vB4jYeiiu9mQAwoVD5to7Ha0Zw@mail.gmail.com>
	<ksb3m5$4q4$1@ger.gmane.org> <7wob9y4a7d.fsf@benfinney.id.au>
Message-ID: <ksdv5p$nja$1@ger.gmane.org>

19.07.13 22:32, Ben Finney ???????(??):
> Serhiy Storchaka <storchaka at gmail.com> writes:
>> I'm asking only about this case, when the dash is used to denote a
>> break in a sentence or to set off parenthetical statements.
>
> That's two separate cases:
>
> * denote a break in a sentence
> * set off parenthetical statements

In Wikipedia they considered as one case.

> In my experience, an em dash is commonly used for the former, and en
> dashes commonly used for the latter. Using the same dash for both is
> unusual (and IMO needlessly ambiguous to the reader).

This is definitely should be described in the official guide. I never 
heard about this.

>> Currently Python documentation uses fourth variants:
>
> All your examples are only for parenthetical statements. Can you find
> examples of the former, where a break (not parenthetical) in the
> sentence is intended?

(2) and (4) are from Wikipedia (which consider it as one case). Here are 
real random examples from Python documentation:

1. "Common installation tools such as ``Distribute`` and ``pip`` work as 
expected with venvs - i.e. when a venv is active, they install Python 
packages into the venv without needing to be told to do so explicitly."

2. "Enter *string* in the table of "interned" strings and return the 
interned string which is *string* itself or a copy."

3. "The :class:`set` type is mutable --- the contents can be changed 
using methods like :meth:`add` and :meth:`remove`."

4. "Three locking mechanisms are used---dot locking and, if available, 
the :c:func:`flock` and :c:func:`lockf` system calls."



From storchaka at gmail.com  Sat Jul 20 14:23:46 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 20 Jul 2013 15:23:46 +0300
Subject: [Python-Dev] cpython: Use strncat() instead of strcat() to
	silence some warnings.
In-Reply-To: <3by7Hm4W8dz7LjM@mail.python.org>
References: <3by7Hm4W8dz7LjM@mail.python.org>
Message-ID: <ksdvgc$qt5$1@ger.gmane.org>

20.07.13 15:12, christian.heimes ???????(??):
> http://hg.python.org/cpython/rev/c92f4172d122
> changeset:   84723:c92f4172d122
> user:        Christian Heimes <christian at cheimes.de>
> date:        Sat Jul 20 14:11:28 2013 +0200
> summary:
>    Use strncat() instead of strcat() to silence some warnings.
> CID 486616, CID 486617, CID 486615
>
> files:
>    Modules/ossaudiodev.c |  6 +++---
>    1 files changed, 3 insertions(+), 3 deletions(-)
>
>
> diff --git a/Modules/ossaudiodev.c b/Modules/ossaudiodev.c
> --- a/Modules/ossaudiodev.c
> +++ b/Modules/ossaudiodev.c
> @@ -245,7 +245,7 @@
>       int arg;
>
>       assert(strlen(fname) <= 30);
> -    strcat(argfmt, fname);
> +    strncat(argfmt, fname, 30);
>       if (!PyArg_ParseTuple(args, argfmt, &arg))
>           return NULL;
>
> @@ -270,7 +270,7 @@
>       int arg = 0;
>
>       assert(strlen(fname) <= 30);
> -    strcat(argfmt, fname);
> +    strncat(argfmt, fname, 30);
>       if (!PyArg_ParseTuple(args, argfmt, &arg))
>           return NULL;
>
> @@ -290,7 +290,7 @@
>       int rv;
>
>       assert(strlen(fname) <= 30);
> -    strcat(argfmt, fname);
> +    strncat(argfmt, fname, 30);
>       if (!PyArg_ParseTuple(args, argfmt))
>           return NULL;

This will wrong when strlen(fname) is 30. strncat() will copy only 30 
bytes, without terminal NUL.



From solipsis at pitrou.net  Sat Jul 20 14:36:04 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 20 Jul 2013 14:36:04 +0200
Subject: [Python-Dev] cpython: Use strncat() instead of strcat() to
 silence some warnings.
References: <3by7Hm4W8dz7LjM@mail.python.org>
	<ksdvgc$qt5$1@ger.gmane.org>
Message-ID: <20130720143604.596881f0@fsol>

On Sat, 20 Jul 2013 15:23:46 +0300
Serhiy Storchaka <storchaka at gmail.com> wrote:
> 20.07.13 15:12, christian.heimes ???????(??):
> > http://hg.python.org/cpython/rev/c92f4172d122
> > changeset:   84723:c92f4172d122
> > user:        Christian Heimes <christian at cheimes.de>
> > date:        Sat Jul 20 14:11:28 2013 +0200
> > summary:
> >    Use strncat() instead of strcat() to silence some warnings.
> > CID 486616, CID 486617, CID 486615
> >
[...]
> 
> This will wrong when strlen(fname) is 30. strncat() will copy only 30 
> bytes, without terminal NUL.

So, for the record, this is roughly how Rasmus Lerdorf introduced a
security hole in PHP 5.3.7:


"For people asking me out-of-band what the screw-up was, it was pretty
simple. I changed this code:

memcpy(passwd, MD5_MAGIC, MD5_MAGIC_LEN);
strlcpy(passwd + MD5_MAGIC_LEN, sp, sl + 1);
strcat(passwd, "$");

to:

memcpy(passwd, MD5_MAGIC, MD5_MAGIC_LEN);
strlcpy(passwd + MD5_MAGIC_LEN, sp, sl + 1);
strlcat(passwd, "$", 1);

**because the Coverity static analyzer warned about using strcat**
[emphasis mine] and we generally try to avoid naked strcat/strcpy in the
codebase even though in this case it is safe to do."

https://plus.google.com/113641248237520845183/posts/g68d9RvRA1i

Regards

Antoine.



From storchaka at gmail.com  Sat Jul 20 14:48:09 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 20 Jul 2013 15:48:09 +0300
Subject: [Python-Dev] cpython: Use strncat() instead of strcat() to
	silence some warnings.
In-Reply-To: <20130720143604.596881f0@fsol>
References: <3by7Hm4W8dz7LjM@mail.python.org> <ksdvgc$qt5$1@ger.gmane.org>
	<20130720143604.596881f0@fsol>
Message-ID: <kse0u3$7ff$1@ger.gmane.org>

20.07.13 15:36, Antoine Pitrou ???????(??):
> On Sat, 20 Jul 2013 15:23:46 +0300
> Serhiy Storchaka <storchaka at gmail.com> wrote:
>> 20.07.13 15:12, christian.heimes ???????(??):
>>> http://hg.python.org/cpython/rev/c92f4172d122
>>> changeset:   84723:c92f4172d122
>>> user:        Christian Heimes <christian at cheimes.de>
>>> date:        Sat Jul 20 14:11:28 2013 +0200
>>> summary:
>>>     Use strncat() instead of strcat() to silence some warnings.
>>> CID 486616, CID 486617, CID 486615
>>>
> [...]
>>
>> This will wrong when strlen(fname) is 30. strncat() will copy only 30
>> bytes, without terminal NUL.
>
> So, for the record, this is roughly how Rasmus Lerdorf introduced a
> security hole in PHP 5.3.7:
>
>
> "For people asking me out-of-band what the screw-up was, it was pretty
> simple. I changed this code:
>
> memcpy(passwd, MD5_MAGIC, MD5_MAGIC_LEN);
> strlcpy(passwd + MD5_MAGIC_LEN, sp, sl + 1);
> strcat(passwd, "$");
>
> to:
>
> memcpy(passwd, MD5_MAGIC, MD5_MAGIC_LEN);
> strlcpy(passwd + MD5_MAGIC_LEN, sp, sl + 1);
> strlcat(passwd, "$", 1);
>
> **because the Coverity static analyzer warned about using strcat**
> [emphasis mine] and we generally try to avoid naked strcat/strcpy in the
> codebase even though in this case it is safe to do."
>
> https://plus.google.com/113641248237520845183/posts/g68d9RvRA1i

strlcat != strncat. strlcat(dst, src, 1) actually do nothing.


From solipsis at pitrou.net  Sat Jul 20 15:00:49 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 20 Jul 2013 15:00:49 +0200
Subject: [Python-Dev] cpython: Use strncat() instead of strcat() to
 silence some warnings.
References: <3by7Hm4W8dz7LjM@mail.python.org> <ksdvgc$qt5$1@ger.gmane.org>
	<20130720143604.596881f0@fsol> <kse0u3$7ff$1@ger.gmane.org>
Message-ID: <20130720150049.5403ad1f@fsol>

On Sat, 20 Jul 2013 15:48:09 +0300
Serhiy Storchaka <storchaka at gmail.com> wrote:

> 20.07.13 15:36, Antoine Pitrou ???????(??):
> > On Sat, 20 Jul 2013 15:23:46 +0300
> > Serhiy Storchaka <storchaka at gmail.com> wrote:
> >> 20.07.13 15:12, christian.heimes ???????(??):
> >>> http://hg.python.org/cpython/rev/c92f4172d122
> >>> changeset:   84723:c92f4172d122
> >>> user:        Christian Heimes <christian at cheimes.de>
> >>> date:        Sat Jul 20 14:11:28 2013 +0200
> >>> summary:
> >>>     Use strncat() instead of strcat() to silence some warnings.
> >>> CID 486616, CID 486617, CID 486615
> >>>
> > [...]
> >>
> >> This will wrong when strlen(fname) is 30. strncat() will copy only 30
> >> bytes, without terminal NUL.
> >
> > So, for the record, this is roughly how Rasmus Lerdorf introduced a
> > security hole in PHP 5.3.7:
> >
> >
> > "For people asking me out-of-band what the screw-up was, it was pretty
> > simple. I changed this code:
> >
> > memcpy(passwd, MD5_MAGIC, MD5_MAGIC_LEN);
> > strlcpy(passwd + MD5_MAGIC_LEN, sp, sl + 1);
> > strcat(passwd, "$");
> >
> > to:
> >
> > memcpy(passwd, MD5_MAGIC, MD5_MAGIC_LEN);
> > strlcpy(passwd + MD5_MAGIC_LEN, sp, sl + 1);
> > strlcat(passwd, "$", 1);
> >
> > **because the Coverity static analyzer warned about using strcat**
> > [emphasis mine] and we generally try to avoid naked strcat/strcpy in the
> > codebase even though in this case it is safe to do."
> >
> > https://plus.google.com/113641248237520845183/posts/g68d9RvRA1i
> 
> strlcat != strncat. strlcat(dst, src, 1) actually do nothing.

This is true. But the trigger in the "fix" was the same (trying to
suppress a Coverity warning about strcat).

Regards

Antoine.



From christian at python.org  Sat Jul 20 15:27:01 2013
From: christian at python.org (Christian Heimes)
Date: Sat, 20 Jul 2013 15:27:01 +0200
Subject: [Python-Dev] cpython: Use strncat() instead of strcat() to
	silence some warnings.
In-Reply-To: <ksdvgc$qt5$1@ger.gmane.org>
References: <3by7Hm4W8dz7LjM@mail.python.org> <ksdvgc$qt5$1@ger.gmane.org>
Message-ID: <kse36u$r8v$1@ger.gmane.org>

> This will wrong when strlen(fname) is 30. strncat() will copy only 30
> bytes, without terminal NUL.

That's not how strncat() works. strncat(dest, src, n) writes n+1 chars
to the end of dest: n chars from src and +1 for the final NUL char. For
this reason dest must be large enough to hold strlen(dest) + n + 1 chars.



From christian at python.org  Sat Jul 20 15:29:45 2013
From: christian at python.org (Christian Heimes)
Date: Sat, 20 Jul 2013 15:29:45 +0200
Subject: [Python-Dev] cpython: Use strncat() instead of strcat() to
	silence some warnings.
In-Reply-To: <ksdvgc$qt5$1@ger.gmane.org>
References: <3by7Hm4W8dz7LjM@mail.python.org> <ksdvgc$qt5$1@ger.gmane.org>
Message-ID: <kse3c2$r8v$2@ger.gmane.org>

Am 20.07.2013 14:23, schrieb Serhiy Storchaka:
> This will wrong when strlen(fname) is 30. strncat() will copy only 30
> bytes, without terminal NUL.

http://linux.die.net/man/3/strncat

The strncat() function is similar, except that

* it will use at most n bytes from src; and

* src does not need to be null-terminated if it contains n or more bytes.

If src contains n or more bytes, strncat() writes n+1 bytes to dest (n
from src plus the terminating null byte). Therefore, the size of dest
must be at least strlen(dest)+n+1.


From storchaka at gmail.com  Sat Jul 20 15:56:25 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 20 Jul 2013 16:56:25 +0300
Subject: [Python-Dev] cpython: Use strncat() instead of strcat() to
	silence some warnings.
In-Reply-To: <kse36u$r8v$1@ger.gmane.org>
References: <3by7Hm4W8dz7LjM@mail.python.org> <ksdvgc$qt5$1@ger.gmane.org>
	<kse36u$r8v$1@ger.gmane.org>
Message-ID: <kse4u3$c3i$1@ger.gmane.org>

20.07.13 16:27, Christian Heimes ???????(??):
>> This will wrong when strlen(fname) is 30. strncat() will copy only 30
>> bytes, without terminal NUL.
>
> That's not how strncat() works. strncat(dest, src, n) writes n+1 chars
> to the end of dest: n chars from src and +1 for the final NUL char. For
> this reason dest must be large enough to hold strlen(dest) + n + 1 chars.

Oh, true. strncat() always results NUL-terminated string. It's strncpy() 
can produce not NUL-terminated string. Sorry for noise.


From christian at python.org  Sat Jul 20 16:04:08 2013
From: christian at python.org (Christian Heimes)
Date: Sat, 20 Jul 2013 16:04:08 +0200
Subject: [Python-Dev] cpython: Use strncat() instead of strcat() to
	silence some warnings.
In-Reply-To: <kse4u3$c3i$1@ger.gmane.org>
References: <3by7Hm4W8dz7LjM@mail.python.org> <ksdvgc$qt5$1@ger.gmane.org>
	<kse36u$r8v$1@ger.gmane.org> <kse4u3$c3i$1@ger.gmane.org>
Message-ID: <kse5ch$eak$1@ger.gmane.org>

Am 20.07.2013 15:56, schrieb Serhiy Storchaka:
> Oh, true. strncat() always results NUL-terminated string. It's strncpy()
> can produce not NUL-terminated string. Sorry for noise.

You're welcome! Better safe than sorry. :)



From ethan at stoneleaf.us  Sun Jul 21 03:37:57 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 20 Jul 2013 18:37:57 -0700
Subject: [Python-Dev] dict __contains__ raises TypeError on unhashable
 input
In-Reply-To: <94771240-354C-4240-BFCF-BDFCE6FE2E9C@mac.com>
References: <51E9D018.4030806@stoneleaf.us>
	<94771240-354C-4240-BFCF-BDFCE6FE2E9C@mac.com>
Message-ID: <51EB3B75.1070701@stoneleaf.us>

On 07/20/2013 03:21 AM, Ronald Oussoren wrote:
>
> On 20 Jul, 2013, at 1:47, Ethan Furman <ethan at stoneleaf.us> wrote:
>
>> While working on issue #18508 I stumbled across this:
>>
>> Traceback (most recent call last):
>> ...
>>   File "/usr/local/lib/python3.4/enum.py", line 417, in __new__
>>     if value in cls._value2member_map:
>> TypeError: unhashable type: 'list'
>>
>> I'll wrap it in a try-except block, but I must admit I was surprised the answer wasn't False.  After all, if the input is unhashable then obviously it's not in the dict; furthermore, if I were to compare the number 5 with a set() I would get False, not a TypeMismatch error, and dict keys are basically done by equality, the hash is just (?) a speed-up.
>
> Not quite, there are some objects that compare equal without both of them being hashable:
>
>>>> frozenset([1,2]) == set([1,2])
> True
>>>> dct = { frozenset([1,2]): 1 }
>>>> frozenset([1,2]) in dct
> True
>>>> set([1,2]) in dct
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
> TypeError: unhashable type: 'set'
>
> It would be strange if the last test would return False instead of raising an error.

Yeah, that would be unfortunate.

Okay, I can see dict.__contains__ and friends raising a TypeError, but 
what about dict.keys() ?  In 2.x it returned a list, in 3.x it returns a 
dict_keys, and dict_keys also raises an error instead of returning False.

--
~Ethan~

From guido at python.org  Sun Jul 21 04:37:02 2013
From: guido at python.org (Guido van Rossum)
Date: Sat, 20 Jul 2013 19:37:02 -0700
Subject: [Python-Dev] dict __contains__ raises TypeError on unhashable
	input
In-Reply-To: <51EB3B75.1070701@stoneleaf.us>
References: <51E9D018.4030806@stoneleaf.us>
	<94771240-354C-4240-BFCF-BDFCE6FE2E9C@mac.com>
	<51EB3B75.1070701@stoneleaf.us>
Message-ID: <CAP7+vJ+ZzQO0N+J0NvT3zN3gmqZ2LKRE17Fp2ighap-68y9RfA@mail.gmail.com>

On Sat, Jul 20, 2013 at 6:37 PM, Ethan Furman <ethan at stoneleaf.us> wrote:
> On 07/20/2013 03:21 AM, Ronald Oussoren wrote:
>>
>>
>> On 20 Jul, 2013, at 1:47, Ethan Furman <ethan at stoneleaf.us> wrote:
>>
>>> While working on issue #18508 I stumbled across this:
>>>
>>> Traceback (most recent call last):
>>> ...
>>>   File "/usr/local/lib/python3.4/enum.py", line 417, in __new__
>>>     if value in cls._value2member_map:
>>> TypeError: unhashable type: 'list'
>>>
>>> I'll wrap it in a try-except block, but I must admit I was surprised the
>>> answer wasn't False.  After all, if the input is unhashable then obviously
>>> it's not in the dict; furthermore, if I were to compare the number 5 with a
>>> set() I would get False, not a TypeMismatch error, and dict keys are
>>> basically done by equality, the hash is just (?) a speed-up.
>>
>>
>> Not quite, there are some objects that compare equal without both of them
>> being hashable:
>>
>>>>> frozenset([1,2]) == set([1,2])
>>
>> True
>>>>>
>>>>> dct = { frozenset([1,2]): 1 }
>>>>> frozenset([1,2]) in dct
>>
>> True
>>>>>
>>>>> set([1,2]) in dct
>>
>> Traceback (most recent call last):
>>    File "<stdin>", line 1, in <module>
>> TypeError: unhashable type: 'set'
>>
>> It would be strange if the last test would return False instead of raising
>> an error.
>
>
> Yeah, that would be unfortunate.
>
> Okay, I can see dict.__contains__ and friends raising a TypeError, but what
> about dict.keys() ?  In 2.x it returned a list, in 3.x it returns a
> dict_keys, and dict_keys also raises an error instead of returning False.

The same reasoning applies. And the Python 2 and 3 keys() methods are
so different that the backwards compatibility argument doesn't really
have any weight.

-- 
--Guido van Rossum (python.org/~guido)

From raymond.hettinger at gmail.com  Mon Jul 22 01:36:35 2013
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Sun, 21 Jul 2013 16:36:35 -0700
Subject: [Python-Dev] Building a Faster Python
Message-ID: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com>

Our current Mac OS X builds use GCC-4.2.

On Python2.7, I ran a comparison of gcc-4.2.1 builds
versus gcc-4.8.1 and found that the latter makes a much
faster Python.  PyBench2.0 shows the total running time
dropping from 5653ms to 4571ms.  The code is uniformly
better in just about every category.


Raymond








-------------------------------------------------------------------------------
PYBENCH 2.0
-------------------------------------------------------------------------------
* using CPython 2.7.5+ (2.7:f8df7c50132f, Jul 21 2013, 15:57:46) [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)]
* disabled garbage collection
* system check interval set to maximum: 2147483647
* using timer: time.time

Calibrating tests. Please wait... done.

Running 10 round(s) of the suite at warp factor 10:

* Round 1 done in 5.775 seconds.
* Round 2 done in 5.794 seconds.
* Round 3 done in 5.743 seconds.
* Round 4 done in 5.736 seconds.
* Round 5 done in 5.758 seconds.
* Round 6 done in 5.787 seconds.
* Round 7 done in 5.779 seconds.
* Round 8 done in 5.781 seconds.
* Round 9 done in 5.782 seconds.
* Round 10 done in 5.811 seconds.

-------------------------------------------------------------------------------
Benchmark: 2013-07-21 16:02:39
-------------------------------------------------------------------------------

    Rounds: 10
    Warp:   10
    Timer:  time.time

    Machine Details:
       Platform ID:    Darwin-12.4.0-x86_64-i386-64bit
       Processor:      i386

    Python:
       Implementation: CPython
       Executable:     /Users/raymondhettinger/py27/python.exe
       Version:        2.7.5+
       Compiler:       GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)
       Bits:           64bit
       Build:          Jul 21 2013 15:57:46 (#2.7:f8df7c50132f)
       Unicode:        UCS2


Test                             minimum  average  operation  overhead
-------------------------------------------------------------------------------
          BuiltinFunctionCalls:    112ms    114ms    0.22us    0.178ms
           BuiltinMethodLookup:     76ms     83ms    0.08us    0.207ms
                 CompareFloats:     78ms     80ms    0.07us    0.237ms
         CompareFloatsIntegers:     73ms     74ms    0.08us    0.178ms
               CompareIntegers:     86ms     87ms    0.05us    0.354ms
        CompareInternedStrings:     85ms     86ms    0.06us    0.895ms
                  CompareLongs:     72ms     72ms    0.07us    0.207ms
                CompareStrings:     66ms     66ms    0.07us    0.603ms
                CompareUnicode:     75ms     76ms    0.10us    0.455ms
    ComplexPythonFunctionCalls:     96ms     98ms    0.49us    0.298ms
                 ConcatStrings:    166ms    179ms    0.36us    0.346ms
                 ConcatUnicode:     80ms     82ms    0.27us    0.231ms
               CreateInstances:    111ms    115ms    1.03us    0.242ms
            CreateNewInstances:     84ms     87ms    1.04us    0.250ms
       CreateStringsWithConcat:     95ms     97ms    0.10us    0.602ms
       CreateUnicodeWithConcat:     70ms     72ms    0.18us    0.236ms
                  DictCreation:     77ms     77ms    0.19us    0.237ms
             DictWithFloatKeys:     81ms     82ms    0.09us    0.442ms
           DictWithIntegerKeys:     84ms     85ms    0.07us    0.591ms
            DictWithStringKeys:     78ms     79ms    0.07us    0.590ms
                      ForLoops:     69ms     70ms    2.81us    0.042ms
                    IfThenElse:     73ms     74ms    0.06us    0.445ms
                   ListSlicing:     69ms     70ms    4.98us    0.079ms
                NestedForLoops:     83ms     84ms    0.06us    0.016ms
      NestedListComprehensions:    103ms    105ms    8.78us    0.061ms
          NormalClassAttribute:     85ms     86ms    0.07us    0.298ms
       NormalInstanceAttribute:     81ms     81ms    0.07us    0.299ms
           PythonFunctionCalls:     94ms     95ms    0.29us    0.179ms
             PythonMethodCalls:    109ms    111ms    0.49us    0.093ms
                     Recursion:    134ms    137ms    2.74us    0.297ms
                  SecondImport:    110ms    111ms    1.11us    0.119ms
           SecondPackageImport:    113ms    115ms    1.15us    0.159ms
         SecondSubmoduleImport:    143ms    146ms    1.46us    0.119ms
       SimpleComplexArithmetic:     70ms     70ms    0.08us    0.237ms
        SimpleDictManipulation:     84ms     86ms    0.07us    0.297ms
         SimpleFloatArithmetic:     75ms     75ms    0.06us    0.355ms
      SimpleIntFloatArithmetic:     67ms     67ms    0.05us    0.355ms
       SimpleIntegerArithmetic:     68ms     68ms    0.05us    0.354ms
      SimpleListComprehensions:     93ms     94ms    7.82us    0.061ms
        SimpleListManipulation:     69ms     71ms    0.06us    0.386ms
          SimpleLongArithmetic:     81ms     82ms    0.12us    0.178ms
                    SmallLists:     94ms     98ms    0.14us    0.237ms
                   SmallTuples:     84ms     86ms    0.16us    0.267ms
         SpecialClassAttribute:     84ms     85ms    0.07us    0.299ms
      SpecialInstanceAttribute:    100ms    102ms    0.08us    0.299ms
                StringMappings:    440ms    446ms    1.77us    0.304ms
              StringPredicates:    150ms    156ms    0.22us    1.453ms
                 StringSlicing:    108ms    115ms    0.21us    0.500ms
                     TryExcept:     70ms     70ms    0.03us    0.444ms
                    TryFinally:    102ms    103ms    0.64us    0.238ms
                TryRaiseExcept:     79ms     81ms    1.27us    0.256ms
                  TupleSlicing:     95ms     98ms    0.37us    0.051ms
               UnicodeMappings:     75ms     76ms    2.10us    0.411ms
             UnicodePredicates:     77ms     80ms    0.15us    1.729ms
             UnicodeProperties:    122ms    123ms    0.31us    1.449ms
                UnicodeSlicing:    100ms    105ms    0.21us    0.442ms
                   WithFinally:    102ms    105ms    0.65us    0.238ms
               WithRaiseExcept:    155ms    157ms    1.96us    0.297ms
-------------------------------------------------------------------------------
Totals:                           5653ms   5774ms





-------------------------------------------------------------------------------
PYBENCH 2.0
-------------------------------------------------------------------------------
* using CPython 2.7.5+ (2.7:f8df7c50132f, Jul 21 2013, 16:11:35) [GCC 4.8.1]
* disabled garbage collection
* system check interval set to maximum: 2147483647
* using timer: time.time

Calibrating tests. Please wait... done.

Running 10 round(s) of the suite at warp factor 10:

* Round 1 done in 4.654 seconds.
* Round 2 done in 4.681 seconds.
* Round 3 done in 4.660 seconds.
* Round 4 done in 4.694 seconds.
* Round 5 done in 4.674 seconds.
* Round 6 done in 4.673 seconds.
* Round 7 done in 4.672 seconds.
* Round 8 done in 4.679 seconds.
* Round 9 done in 4.692 seconds.
* Round 10 done in 4.680 seconds.

-------------------------------------------------------------------------------
Benchmark: 2013-07-21 16:15:45
-------------------------------------------------------------------------------

    Rounds: 10
    Warp:   10
    Timer:  time.time

    Machine Details:
       Platform ID:    Darwin-12.4.0-x86_64-i386-64bit
       Processor:      i386

    Python:
       Implementation: CPython
       Executable:     /Users/raymondhettinger/py27/python.exe
       Version:        2.7.5+
       Compiler:       GCC 4.8.1
       Bits:           64bit
       Build:          Jul 21 2013 16:11:35 (#2.7:f8df7c50132f)
       Unicode:        UCS2


Test                             minimum  average  operation  overhead
-------------------------------------------------------------------------------
          BuiltinFunctionCalls:     90ms     92ms    0.18us    0.185ms
           BuiltinMethodLookup:     53ms     55ms    0.05us    0.146ms
                 CompareFloats:     60ms     60ms    0.05us    0.167ms
         CompareFloatsIntegers:     57ms     57ms    0.06us    0.125ms
               CompareIntegers:     67ms     67ms    0.04us    0.250ms
        CompareInternedStrings:     67ms     67ms    0.04us    0.631ms
                  CompareLongs:     60ms     62ms    0.06us    0.146ms
                CompareStrings:     52ms     53ms    0.05us    0.425ms
                CompareUnicode:     62ms     62ms    0.08us    0.350ms
    ComplexPythonFunctionCalls:     85ms     86ms    0.43us    0.210ms
                 ConcatStrings:    160ms    173ms    0.35us    0.252ms
                 ConcatUnicode:     74ms     75ms    0.25us    0.172ms
               CreateInstances:     81ms     84ms    0.75us    0.170ms
            CreateNewInstances:     62ms     64ms    0.77us    0.192ms
       CreateStringsWithConcat:     75ms     76ms    0.08us    0.418ms
       CreateUnicodeWithConcat:     64ms     65ms    0.16us    0.167ms
                  DictCreation:     56ms     57ms    0.14us    0.167ms
             DictWithFloatKeys:     60ms     61ms    0.07us    0.313ms
           DictWithIntegerKeys:     61ms     62ms    0.05us    0.418ms
            DictWithStringKeys:     55ms     56ms    0.05us    0.418ms
                      ForLoops:     43ms     43ms    1.72us    0.032ms
                    IfThenElse:     60ms     61ms    0.04us    0.313ms
                   ListSlicing:     72ms     90ms    6.40us    0.041ms
                NestedForLoops:     59ms     61ms    0.04us    0.013ms
      NestedListComprehensions:     79ms     81ms    6.77us    0.043ms
          NormalClassAttribute:     62ms     63ms    0.05us    0.210ms
       NormalInstanceAttribute:     55ms     56ms    0.05us    0.210ms
           PythonFunctionCalls:     80ms     81ms    0.25us    0.126ms
             PythonMethodCalls:     84ms     84ms    0.38us    0.065ms
                     Recursion:     97ms     97ms    1.95us    0.209ms
                  SecondImport:     82ms     83ms    0.83us    0.083ms
           SecondPackageImport:     88ms     90ms    0.90us    0.083ms
         SecondSubmoduleImport:    114ms    115ms    1.15us    0.083ms
       SimpleComplexArithmetic:     52ms     53ms    0.06us    0.167ms
        SimpleDictManipulation:     65ms     65ms    0.05us    0.209ms
         SimpleFloatArithmetic:     50ms     51ms    0.04us    0.251ms
      SimpleIntFloatArithmetic:     52ms     52ms    0.04us    0.251ms
       SimpleIntegerArithmetic:     52ms     52ms    0.04us    0.251ms
      SimpleListComprehensions:     67ms     70ms    5.82us    0.043ms
        SimpleListManipulation:     49ms     50ms    0.04us    0.272ms
          SimpleLongArithmetic:     48ms     48ms    0.07us    0.125ms
                    SmallLists:     80ms     83ms    0.12us    0.167ms
                   SmallTuples:     72ms     72ms    0.13us    0.221ms
         SpecialClassAttribute:     62ms     62ms    0.05us    0.210ms
      SpecialInstanceAttribute:    127ms    129ms    0.11us    0.210ms
                StringMappings:    396ms    399ms    1.58us    0.233ms
              StringPredicates:    125ms    127ms    0.18us    1.117ms
                 StringSlicing:     95ms    101ms    0.18us    0.355ms
                     TryExcept:     47ms     48ms    0.02us    0.313ms
                    TryFinally:     60ms     61ms    0.38us    0.167ms
                TryRaiseExcept:     62ms     64ms    0.99us    0.167ms
                  TupleSlicing:     85ms     88ms    0.33us    0.028ms
               UnicodeMappings:     93ms     93ms    2.59us    0.355ms
             UnicodePredicates:     66ms     66ms    0.12us    1.342ms
             UnicodeProperties:     86ms     88ms    0.22us    1.125ms
                UnicodeSlicing:     84ms     91ms    0.19us    0.315ms
                   WithFinally:     91ms     93ms    0.58us    0.167ms
               WithRaiseExcept:    127ms    129ms    1.61us    0.209ms
-------------------------------------------------------------------------------
Totals:                           4571ms   4676ms

From benhoyt at gmail.com  Mon Jul 22 01:46:40 2013
From: benhoyt at gmail.com (Ben Hoyt)
Date: Mon, 22 Jul 2013 11:46:40 +1200
Subject: [Python-Dev] Building a Faster Python
In-Reply-To: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com>
References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com>
Message-ID: <CAL9jXCGvFKUfUjs3x9E3XBf+_im1-gGakfhCG9h1YdwC9kNVAg@mail.gmail.com>

> PyBench2.0 shows the total running time dropping from 5653ms to 4571ms.

That's very cool -- a significant improvement. Is this the kind of change
that could go into 2.7.6 binaries?

As a Windows user, it makes me wonder if compiling with the latest version
of the Microsoft compiler would improve things similarly? (Though updating
project files to that is almost certainly a bigger project than the gcc
update.)

-Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130722/8b05dac1/attachment.html>

From brian at python.org  Mon Jul 22 02:21:23 2013
From: brian at python.org (Brian Curtin)
Date: Sun, 21 Jul 2013 19:21:23 -0500
Subject: [Python-Dev] Building a Faster Python
In-Reply-To: <CAL9jXCGvFKUfUjs3x9E3XBf+_im1-gGakfhCG9h1YdwC9kNVAg@mail.gmail.com>
References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com>
	<CAL9jXCGvFKUfUjs3x9E3XBf+_im1-gGakfhCG9h1YdwC9kNVAg@mail.gmail.com>
Message-ID: <CAD+XWwpTdgrk071ry-j32D+kjWrgP7z7jsYT8_3tPfgJtT5qZg@mail.gmail.com>

On Sun, Jul 21, 2013 at 6:46 PM, Ben Hoyt <benhoyt at gmail.com> wrote:
>> PyBench2.0 shows the total running time dropping from 5653ms to 4571ms.
>
> That's very cool -- a significant improvement. Is this the kind of change
> that could go into 2.7.6 binaries?
>
> As a Windows user, it makes me wonder if compiling with the latest version
> of the Microsoft compiler would improve things similarly? (Though updating
> project files to that is almost certainly a bigger project than the gcc
> update.)

I think I have a 3.3 build on VS2012 somewhere - maybe I'll refresh it
for default/3.4 and run the same benchmarks on it. The changes
couldn't go into 2.7 as far as I'm aware, at least when it comes to
changing Visual Studio versions.

From Steve.Dower at microsoft.com  Mon Jul 22 02:24:45 2013
From: Steve.Dower at microsoft.com (Steve Dower)
Date: Mon, 22 Jul 2013 00:24:45 +0000
Subject: [Python-Dev] Building a Faster Python
In-Reply-To: <CAL9jXCGvFKUfUjs3x9E3XBf+_im1-gGakfhCG9h1YdwC9kNVAg@mail.gmail.com>
References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com>
	<CAL9jXCGvFKUfUjs3x9E3XBf+_im1-gGakfhCG9h1YdwC9kNVAg@mail.gmail.com>
Message-ID: <b8ae1000f40a43d99b51c2a35c5ec900@BY2PR03MB206.namprd03.prod.outlook.com>



>From: Ben Hoyt
>> PyBench2.0 shows the total running time dropping from 5653ms to 4571ms.
>
> That's very cool -- a significant improvement. Is this the kind of change that could go into 2.7.6 binaries?
>
> As a Windows user, it makes me wonder if compiling with the latest version of the Microsoft compiler
> would improve things similarly?

I'd expect to see some improvement, based solely on the bugs fixed recently by the optimizer team. No idea how much, but I know that Martin builds with PGO, and that's also been improved recently.

> (Though updating project files to that is almost certainly a bigger project than the gcc update.)

No idea what the gcc update involved, but this is only accurate if it took less than two minutes :-) I've upgraded the CPython projects before and everything worked fine.

Steve



From nad at acm.org  Mon Jul 22 02:32:51 2013
From: nad at acm.org (Ned Deily)
Date: Sun, 21 Jul 2013 17:32:51 -0700
Subject: [Python-Dev] Building a Faster Python
References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com>
Message-ID: <nad-21E1F3.17325121072013@news.gmane.org>

In article <252C50D8-C23D-438D-BAE1-B22E0D65A21A at gmail.com>,
 Raymond Hettinger <raymond.hettinger at gmail.com> wrote:
> Our current Mac OS X builds use GCC-4.2.
> 
> On Python2.7, I ran a comparison of gcc-4.2.1 builds
> versus gcc-4.8.1 and found that the latter makes a much
> faster Python.  PyBench2.0 shows the total running time
> dropping from 5653ms to 4571ms.  The code is uniformly
> better in just about every category.

Interesting.  A note about compilers:  it looks like the benchmark used 
the Apple gcc4.2-frontend / LLVM-backend hybrid compiler that is shipped 
with Xcode 4.  We don't use that compiler for building any OS X binary 
installers, in fact, we blacklist it for Python 3.3+ because we couldn't 
get it to produce a working 3.3 interpreter:

   http://bugs.python.org/issue13241

Apple has deprecated use of that compiler (it hasn't been patched in 
ages) and has announced that it will no longer be shipped starting with 
the next major release of Xcode, which is expected with the upcoming 
release of OS X 10.9 (Mavericks).

What would be interesting is to run the benchmark with Apple's currently 
supported compiler for OS X, clang, which is the default "cc" in the 
most recent versions of Xcode 4 (4.6.3) for OS X 10.7 and 10.8.

  $ cc --version
  Apple LLVM version 4.2 (clang-425.0.28) (based on LLVM 3.2svn)

Clang is expected to be the only compiler available via Xcode for 10.9.  
Note that the most recent Python binary installers default to using 
clang to build user extension modules with Xcode 4 on 10.7 and 10.8.

Currently, for the Intel-only (64-bit/32-bit) installers we instead use 
the final Apple-supplied gcc-4.2 (gcc front- and backend, no LLVM) 
shipped with Xcode 3.2.6 on OS X 10.6:
   [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin]
This makes it easier to provide one Python that can be installed and 
works on multiple OS X versions, for the Intel-only variant: 10.6 
through 10.8.  As of Xcode 4, Apple no longer ships this compiler.  
Because of the post-4.2 switch to GPL-3 licensing for gcc, it seems that 
Apple has no plans to look at, much less ship, anything newer than 4.2 
from the gcc code base.  At some point, we will probably switch to using 
clang for the Python in the installer but, at the moment, there appear 
to still be a large number of systems out there running OS X 10.6.

(Also, we and most third-party open-source package managers, like 
MacPorts and Homebrew, stick to using Apple-supplied compilers for most 
things rather than vanilla Gnu gcc 4.x because, among other things, 
Apple includes additional support within their compiler build chain for 
features like universal binaries and SDK support.)

-- 
 Ned Deily,
 nad at acm.org


From raymond.hettinger at gmail.com  Mon Jul 22 03:04:16 2013
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Sun, 21 Jul 2013 18:04:16 -0700
Subject: [Python-Dev] Building a Faster Python
In-Reply-To: <nad-21E1F3.17325121072013@news.gmane.org>
References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com>
	<nad-21E1F3.17325121072013@news.gmane.org>
Message-ID: <E955CDAA-C34C-4C06-B027-F853884D527C@gmail.com>


On Jul 21, 2013, at 5:32 PM, Ned Deily <nad at acm.org> wrote:

> In article <252C50D8-C23D-438D-BAE1-B22E0D65A21A at gmail.com>,
> Raymond Hettinger <raymond.hettinger at gmail.com> wrote:
>> Our current Mac OS X builds use GCC-4.2.
>> 
>> On Python2.7, I ran a comparison of gcc-4.2.1 builds
>> versus gcc-4.8.1 and found that the latter makes a much
>> faster Python.  PyBench2.0 shows the total running time
>> dropping from 5653ms to 4571ms.  The code is uniformly
>> better in just about every category.
> 
> Interesting.  A note about compilers:  it looks like the benchmark used 
> the Apple gcc4.2-frontend / LLVM-backend hybrid compiler that is shipped 
> with Xcode 4.  We don't use that compiler for building any OS X binary 
> installers, in fact, we blacklist it for Python 3.3+ because we couldn't 
> get it to produce a working 3.3 interpreter:


The results aren't much different using the build published on python.org.

<insert some joke about an Apple-to-Apple comparsion ;-) >

Using a fresh OS X install from our website, PyBench clocks in at 5376ms 


Raymond



---------------------------

-------------------------------------------------------------------------------
PYBENCH 2.0
-------------------------------------------------------------------------------
* using CPython 2.7.5 (v2.7.5:ab05e7dd2788, May 13 2013, 13:18:45) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
* disabled garbage collection
* system check interval set to maximum: 2147483647
* using timer: time.time

Calibrating tests. Please wait... done.

Running 10 round(s) of the suite at warp factor 10:

* Round 1 done in 5.459 seconds.
* Round 2 done in 5.510 seconds.
* Round 3 done in 5.476 seconds.
* Round 4 done in 5.473 seconds.
* Round 5 done in 5.497 seconds.
* Round 6 done in 5.533 seconds.
* Round 7 done in 5.520 seconds.
* Round 8 done in 5.518 seconds.
* Round 9 done in 5.515 seconds.
* Round 10 done in 5.515 seconds.

-------------------------------------------------------------------------------
Benchmark: 2013-07-21 17:54:13
-------------------------------------------------------------------------------

    Rounds: 10
    Warp:   10
    Timer:  time.time

    Machine Details:
       Platform ID:    Darwin-12.4.0-x86_64-i386-64bit
       Processor:      i386

    Python:
       Implementation: CPython
       Executable:     /Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
       Version:        2.7.5
       Compiler:       GCC 4.2.1 (Apple Inc. build 5666) (dot 3)
       Bits:           64bit
       Build:          May 13 2013 13:18:45 (#v2.7.5:ab05e7dd2788)
       Unicode:        UCS2


Test                             minimum  average  operation  overhead
-------------------------------------------------------------------------------
          BuiltinFunctionCalls:    108ms    109ms    0.21us    0.166ms
           BuiltinMethodLookup:     74ms     77ms    0.07us    0.193ms
                 CompareFloats:     73ms     73ms    0.06us    0.229ms
         CompareFloatsIntegers:     71ms     71ms    0.08us    0.167ms
               CompareIntegers:     78ms     78ms    0.04us    0.334ms
        CompareInternedStrings:     76ms     77ms    0.05us    0.842ms
                  CompareLongs:     69ms     70ms    0.07us    0.196ms
                CompareStrings:     61ms     61ms    0.06us    0.566ms
                CompareUnicode:     66ms     68ms    0.09us    0.431ms
    ComplexPythonFunctionCalls:     98ms    101ms    0.51us    0.280ms
                 ConcatStrings:    166ms    179ms    0.36us    0.324ms
                 ConcatUnicode:     79ms     81ms    0.27us    0.220ms
               CreateInstances:    105ms    107ms    0.96us    0.232ms
            CreateNewInstances:     84ms     88ms    1.04us    0.233ms
       CreateStringsWithConcat:     95ms     96ms    0.10us    0.552ms
       CreateUnicodeWithConcat:     70ms     72ms    0.18us    0.222ms
                  DictCreation:     79ms     80ms    0.20us    0.221ms
             DictWithFloatKeys:     83ms     84ms    0.09us    0.414ms
           DictWithIntegerKeys:     87ms     88ms    0.07us    0.552ms
            DictWithStringKeys:     65ms     66ms    0.06us    0.552ms
                      ForLoops:     58ms     58ms    2.34us    0.039ms
                    IfThenElse:     65ms     66ms    0.05us    0.414ms
                   ListSlicing:     72ms     89ms    6.38us    0.055ms
                NestedForLoops:     79ms     82ms    0.05us    0.014ms
      NestedListComprehensions:     94ms     96ms    7.98us    0.056ms
          NormalClassAttribute:     81ms     81ms    0.07us    0.280ms
       NormalInstanceAttribute:     73ms     75ms    0.06us    0.274ms
           PythonFunctionCalls:     85ms     86ms    0.26us    0.166ms
             PythonMethodCalls:    100ms    103ms    0.46us    0.086ms
                     Recursion:    110ms    113ms    2.25us    0.276ms
                  SecondImport:    104ms    105ms    1.05us    0.112ms
           SecondPackageImport:    111ms    111ms    1.11us    0.110ms
         SecondSubmoduleImport:    138ms    139ms    1.39us    0.110ms
       SimpleComplexArithmetic:     71ms     71ms    0.08us    0.221ms
        SimpleDictManipulation:     91ms     92ms    0.08us    0.276ms
         SimpleFloatArithmetic:     71ms     72ms    0.05us    0.331ms
      SimpleIntFloatArithmetic:     64ms     64ms    0.05us    0.331ms
       SimpleIntegerArithmetic:     64ms     64ms    0.05us    0.337ms
      SimpleListComprehensions:     81ms     83ms    6.93us    0.056ms
        SimpleListManipulation:     61ms     61ms    0.05us    0.359ms
          SimpleLongArithmetic:     65ms     66ms    0.10us    0.168ms
                    SmallLists:     90ms     94ms    0.14us    0.221ms
                   SmallTuples:     77ms     79ms    0.15us    0.248ms
         SpecialClassAttribute:     79ms     79ms    0.07us    0.281ms
      SpecialInstanceAttribute:     96ms     98ms    0.08us    0.281ms
                StringMappings:    435ms    436ms    1.73us    0.291ms
              StringPredicates:    142ms    143ms    0.20us    1.499ms
                 StringSlicing:    106ms    114ms    0.20us    0.466ms
                     TryExcept:     68ms     68ms    0.03us    0.414ms
                    TryFinally:     72ms     73ms    0.46us    0.221ms
                TryRaiseExcept:     77ms     78ms    1.23us    0.221ms
                  TupleSlicing:     90ms     96ms    0.37us    0.036ms
               UnicodeMappings:    100ms    101ms    2.79us    0.402ms
             UnicodePredicates:     77ms     78ms    0.15us    1.820ms
             UnicodeProperties:     95ms     96ms    0.24us    1.503ms
                UnicodeSlicing:     91ms     98ms    0.20us    0.412ms
                   WithFinally:     98ms    100ms    0.62us    0.230ms
               WithRaiseExcept:    159ms    163ms    2.04us    0.282ms
-------------------------------------------------------------------------------
Totals:                           5376ms   5502ms


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130721/04ecd17d/attachment-0001.html>

From larry at hastings.org  Mon Jul 22 03:01:29 2013
From: larry at hastings.org (Larry Hastings)
Date: Sun, 21 Jul 2013 18:01:29 -0700
Subject: [Python-Dev] Building a Faster Python
In-Reply-To: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com>
References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com>
Message-ID: <51EC8469.1030608@hastings.org>

On 07/21/2013 04:36 PM, Raymond Hettinger wrote:
> Our current Mac OS X builds use GCC-4.2.
>
> On Python2.7, I ran a comparison of gcc-4.2.1 builds
> versus gcc-4.8.1 and found that the latter makes a much
> faster Python.  PyBench2.0 shows the total running time
> dropping from 5653ms to 4571ms.  The code is uniformly
> better in just about every category.

I know that newer Microsoft compilers tend to drop support for older 
operating systems, and often that means the binaries simply won't work 
on those operating systems.  (Yes, technically it's the libraries that 
break the older code.)

Are Apple's compilers the same?  Or do they tend to support all the old 
versions of OS X?


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130721/416756d8/attachment.html>

From rdmurray at bitdance.com  Mon Jul 22 04:11:42 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Sun, 21 Jul 2013 22:11:42 -0400
Subject: [Python-Dev]
	=?utf-8?q?=5BPython-checkins=5D_cpython_=282=2E7=29?=
	=?utf-8?q?=3A_Issue_=2318441=3A_Make_test=2Esupport=2Erequires=28?=
	=?utf-8?q?=27gui=27=29_skip_when_it_should=2E?=
In-Reply-To: <3bz3Fl3m9SzSdZ@mail.python.org>
References: <3bz3Fl3m9SzSdZ@mail.python.org>
Message-ID: <20130722021142.CC2862500A8@webabinitio.net>

On Mon, 22 Jul 2013 02:13:47 +0200, terry.reedy <python-checkins at python.org> wrote:
> +# If buildbot improperly sets gui resource (#18365, #18441), remove it
> +# so requires('gui') tests are skipped while non-gui tests still run.
> +if use_resources and 'gui' in use_resources:

Note that the buildbot cannot "improperly" set the GUI resource.
Setting a resource on the regrtest command line says "please try to run
these tests".  If they can't be run, they should then be skipped for
whatever reason they can't run.

--David

From nad at acm.org  Mon Jul 22 07:35:45 2013
From: nad at acm.org (Ned Deily)
Date: Sun, 21 Jul 2013 22:35:45 -0700
Subject: [Python-Dev] cpython (2.7): Issue #18441: Make
	test.support.requires('gui') skip when it should.
References: <3bz3Fl3m9SzSdZ@mail.python.org>
	<20130722021142.CC2862500A8@webabinitio.net>
	<51ECAE41.5060403@udel.edu>
Message-ID: <nad-CCECE5.22354421072013@ger.gmane.org>

In article <51ECAE41.5060403 at udel.edu>, Terry Reedy <tjreedy at udel.edu> 
wrote:
> On 7/21/2013 10:11 PM, R. David Murray wrote:
> > On Mon, 22 Jul 2013 02:13:47 +0200, terry.reedy 
> > <python-checkins at python.org> wrote:
> >> +# If buildbot improperly sets gui resource (#18365, #18441), remove it
> >> +# so requires('gui') tests are skipped while non-gui tests still run.
> >> +if use_resources and 'gui' in use_resources:
> >
> > Note that the buildbot cannot "improperly" set the GUI resource.
> 
> I disagree. I do understand though, that what I call improper setting on 
> the command line cannot be prevented. So I think regrtest (and unittest 
> if and when it supports resources) should intercept 'gui', test once and 
> for all whether it was set 'properly' (or 'appropriately', if you 
> prefer), which is to say, test whether there is a usable GUI, and ignore 
> 'gui' if not.
> 
> > Setting a resource on the regrtest command line says "please try to run
> > these tests".

Right.

> When doing so causes a failure at best and a crash at worst, and those 
> consequences and the means of avoiding them are not documented, that is 
> not a very considerate request. It has cost me over a day of my life.

Unfortunately, that's the way regrtest resources work.

> The printed out -h help says
>     "gui - Run tests that require a running GUI."
> Not 'try', but 'run'. That implies to me that there *is* 'a running GUI' 
> and that is it all right to run tests that *require* such. That was also 
> what I understood from the discussion on
> http://bugs.python.org/issue18103 Create a GUI test framework for Idle

This is exactly what Issue8716 was about.  The buildbot has no way of 
knowing ahead of time whether a test will cause a crash or not.  Yes, Tk 
should not crash but it does in some cases.  Speaking of #8716, that 
reminds me that there is an open issue with it (documented in 
Issue17496).  There is the start of a patch there to use a more general 
approach to testing for a working Tk that might be applicable on all 
platforms.

-- 
 Ned Deily,
 nad at acm.org


From ronaldoussoren at mac.com  Mon Jul 22 08:04:23 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Mon, 22 Jul 2013 08:04:23 +0200
Subject: [Python-Dev] Building a Faster Python
In-Reply-To: <51EC8469.1030608@hastings.org>
References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com>
	<51EC8469.1030608@hastings.org>
Message-ID: <9B1C8E04-4DA8-41CA-8FAA-A73F18EE4532@mac.com>


On 22 Jul, 2013, at 3:01, Larry Hastings <larry at hastings.org> wrote:

> On 07/21/2013 04:36 PM, Raymond Hettinger wrote:
>> Our current Mac OS X builds use GCC-4.2.
>> 
>> On Python2.7, I ran a comparison of gcc-4.2.1 builds
>> versus gcc-4.8.1 and found that the latter makes a much
>> faster Python.  PyBench2.0 shows the total running time
>> dropping from 5653ms to 4571ms.  The code is uniformly
>> better in just about every category.
>> 
> 
> I know that newer Microsoft compilers tend to drop support for older operating systems, and often that means the binaries simply won't work on those operating systems.  (Yes, technically it's the libraries that break the older code.)
> 
> Are Apple's compilers the same?  Or do they tend to support all the old versions of OS X?

The compiler's included with Xcode 4 cannot generate code for PowerPC machines, other than that it should be possible to generate code for older machines with the latest Xcode version.  I have a Python installation on my machine that I use to generate app distributions for a 10.5 machine, and AFAIK this would also work for deploying to 10.4.

The hard part is usually convincing autotools-using projects to not use APIs that are available on the build machine but not on the deployment machine.

Ronald

From ronaldoussoren at mac.com  Mon Jul 22 07:59:33 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Mon, 22 Jul 2013 07:59:33 +0200
Subject: [Python-Dev] Building a Faster Python
In-Reply-To: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com>
References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com>
Message-ID: <22243275-1AC3-4BC2-B112-D082134E10F7@mac.com>


On 22 Jul, 2013, at 1:36, Raymond Hettinger <raymond.hettinger at gmail.com> wrote:

> Our current Mac OS X builds use GCC-4.2.
> 
> On Python2.7, I ran a comparison of gcc-4.2.1 builds
> versus gcc-4.8.1 and found that the latter makes a much
> faster Python.  PyBench2.0 shows the total running time
> dropping from 5653ms to 4571ms.  The code is uniformly
> better in just about every category.

Have you tried using clang from the latest devtools as well?

The OSX binary installers are build using the developer tools from
Apple, that happen to use gcc 4.2 on the machine used to build at
least the 32-bit binary installer (as that's the latest Xcode
that includes PPC support). 

FWIW I'd like to test with "clang -O4" as well (this performs
link-time optimization), I've seen some speedup with other projects 
and this might help with CPython's speed as well.

Ronald


From ronaldoussoren at mac.com  Mon Jul 22 08:08:51 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Mon, 22 Jul 2013 08:08:51 +0200
Subject: [Python-Dev] Building a Faster Python
In-Reply-To: <CAL9jXCGvFKUfUjs3x9E3XBf+_im1-gGakfhCG9h1YdwC9kNVAg@mail.gmail.com>
References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com>
	<CAL9jXCGvFKUfUjs3x9E3XBf+_im1-gGakfhCG9h1YdwC9kNVAg@mail.gmail.com>
Message-ID: <38C7767E-E24C-42B8-A953-5FC989596373@mac.com>


On 22 Jul, 2013, at 1:46, Ben Hoyt <benhoyt at gmail.com> wrote:

> > PyBench2.0 shows the total running time dropping from 5653ms to 4571ms.
> 
> That's very cool -- a significant improvement. Is this the kind of change that could go into 2.7.6 binaries?

I'd prefer not to do that (but don't build the installers anymore).  The installers for OSX are build using the system compiler, using a different compiler makes it harder to build the installer.

I don't even know if upstream GCC could easily be used for the binary installers, does GCC 4.8 support building FAT binaries in its compiler driver?

Ronald


From ronaldoussoren at mac.com  Mon Jul 22 08:13:49 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Mon, 22 Jul 2013 08:13:49 +0200
Subject: [Python-Dev] cpython (2.7): Issue #18441: Make
	test.support.requires('gui') skip when it should.
In-Reply-To: <nad-CCECE5.22354421072013@ger.gmane.org>
References: <3bz3Fl3m9SzSdZ@mail.python.org>
	<20130722021142.CC2862500A8@webabinitio.net>
	<51ECAE41.5060403@udel.edu>
	<nad-CCECE5.22354421072013@ger.gmane.org>
Message-ID: <D4B48379-8502-49DF-8183-B96CD8BD7413@mac.com>


On 22 Jul, 2013, at 7:35, Ned Deily <nad at acm.org> wrote:

> In article <51ECAE41.5060403 at udel.edu>, Terry Reedy <tjreedy at udel.edu> 
> wrote:
>> 
> 
> This is exactly what Issue8716 was about.  The buildbot has no way of 
> knowing ahead of time whether a test will cause a crash or not.  Yes, Tk 
> should not crash but it does in some cases.  Speaking of #8716, that 
> reminds me that there is an open issue with it (documented in 
> Issue17496).  There is the start of a patch there to use a more general 
> approach to testing for a working Tk that might be applicable on all 
> platforms.

Issue17496 contains a patch that might fix the crashing problem
for OSX (and possibly other Unix-y platforms). I haven't been able to test 
this in an environment that's simular enough to the buildbots (that is, a 
system where the user running the test does not have GUI access).

That patch starts wish (the TCL shell) in a subprocess, and skips tests
when that doesn't work (on OSX that would be because Tk crashed).

Ronald



From solipsis at pitrou.net  Mon Jul 22 08:15:34 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 22 Jul 2013 08:15:34 +0200
Subject: [Python-Dev] Building a Faster Python
References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com>
Message-ID: <20130722081534.7cef6df0@fsol>

On Sun, 21 Jul 2013 16:36:35 -0700
Raymond Hettinger <raymond.hettinger at gmail.com> wrote:
> Our current Mac OS X builds use GCC-4.2.
> 
> On Python2.7, I ran a comparison of gcc-4.2.1 builds
> versus gcc-4.8.1 and found that the latter makes a much
> faster Python.  PyBench2.0 shows the total running time
> dropping from 5653ms to 4571ms.  The code is uniformly
> better in just about every category.

You could try running the benchmarks suite to see what that gives:
http://hg.python.org/benchmarks/

Regards

Antoine.



From fijall at gmail.com  Mon Jul 22 09:32:01 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 22 Jul 2013 09:32:01 +0200
Subject: [Python-Dev] Building a Faster Python
In-Reply-To: <20130722081534.7cef6df0@fsol>
References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com>
	<20130722081534.7cef6df0@fsol>
Message-ID: <CAK5idxQoenes6S8JkYOMh-UzKbMgk8yzHQFhQHjyHdLPQub+3g@mail.gmail.com>

On Mon, Jul 22, 2013 at 8:15 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sun, 21 Jul 2013 16:36:35 -0700
> Raymond Hettinger <raymond.hettinger at gmail.com> wrote:
>> Our current Mac OS X builds use GCC-4.2.
>>
>> On Python2.7, I ran a comparison of gcc-4.2.1 builds
>> versus gcc-4.8.1 and found that the latter makes a much
>> faster Python.  PyBench2.0 shows the total running time
>> dropping from 5653ms to 4571ms.  The code is uniformly
>> better in just about every category.
>
> You could try running the benchmarks suite to see what that gives:
> http://hg.python.org/benchmarks/
>
> Regards
>
> Antoine.

or pypy benchmark suite which is more comprehensive for python 2.7
(http://bitbucket.org/pypy/benchmarks)

From fijall at gmail.com  Mon Jul 22 09:32:25 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 22 Jul 2013 09:32:25 +0200
Subject: [Python-Dev] Building a Faster Python
In-Reply-To: <CAK5idxQoenes6S8JkYOMh-UzKbMgk8yzHQFhQHjyHdLPQub+3g@mail.gmail.com>
References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com>
	<20130722081534.7cef6df0@fsol>
	<CAK5idxQoenes6S8JkYOMh-UzKbMgk8yzHQFhQHjyHdLPQub+3g@mail.gmail.com>
Message-ID: <CAK5idxTrb43uHnfi8LPF13NOoL8G-ZnFnF5crtya1O2cJEi-HA@mail.gmail.com>

On Mon, Jul 22, 2013 at 9:32 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> On Mon, Jul 22, 2013 at 8:15 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> On Sun, 21 Jul 2013 16:36:35 -0700
>> Raymond Hettinger <raymond.hettinger at gmail.com> wrote:
>>> Our current Mac OS X builds use GCC-4.2.
>>>
>>> On Python2.7, I ran a comparison of gcc-4.2.1 builds
>>> versus gcc-4.8.1 and found that the latter makes a much
>>> faster Python.  PyBench2.0 shows the total running time
>>> dropping from 5653ms to 4571ms.  The code is uniformly
>>> better in just about every category.
>>
>> You could try running the benchmarks suite to see what that gives:
>> http://hg.python.org/benchmarks/
>>
>> Regards
>>
>> Antoine.
>
> or pypy benchmark suite which is more comprehensive for python 2.7
> (http://bitbucket.org/pypy/benchmarks)

Besides, is there any reason not to use clang by default on OS X?

From ethan at stoneleaf.us  Mon Jul 22 09:15:56 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 22 Jul 2013 00:15:56 -0700
Subject: [Python-Dev] dict __contains__ raises TypeError on unhashable
 input
In-Reply-To: <51E9D018.4030806@stoneleaf.us>
References: <51E9D018.4030806@stoneleaf.us>
Message-ID: <51ECDC2C.4090808@stoneleaf.us>

Thanks for the insights, everyone.

--
~Ethan~

From ronaldoussoren at mac.com  Mon Jul 22 09:50:53 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Mon, 22 Jul 2013 09:50:53 +0200
Subject: [Python-Dev] Building a Faster Python
In-Reply-To: <CAK5idxTrb43uHnfi8LPF13NOoL8G-ZnFnF5crtya1O2cJEi-HA@mail.gmail.com>
References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com>
	<20130722081534.7cef6df0@fsol>
	<CAK5idxQoenes6S8JkYOMh-UzKbMgk8yzHQFhQHjyHdLPQub+3g@mail.gmail.com>
	<CAK5idxTrb43uHnfi8LPF13NOoL8G-ZnFnF5crtya1O2cJEi-HA@mail.gmail.com>
Message-ID: <F5F4EEF5-19B5-4C6F-B465-EC47A7BAABB4@mac.com>


On 22 Jul, 2013, at 9:32, Maciej Fijalkowski <fijall at gmail.com> wrote:

> On Mon, Jul 22, 2013 at 9:32 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> On Mon, Jul 22, 2013 at 8:15 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>>> On Sun, 21 Jul 2013 16:36:35 -0700
>>> Raymond Hettinger <raymond.hettinger at gmail.com> wrote:
>>>> Our current Mac OS X builds use GCC-4.2.
>>>> 
>>>> On Python2.7, I ran a comparison of gcc-4.2.1 builds
>>>> versus gcc-4.8.1 and found that the latter makes a much
>>>> faster Python.  PyBench2.0 shows the total running time
>>>> dropping from 5653ms to 4571ms.  The code is uniformly
>>>> better in just about every category.
>>> 
>>> You could try running the benchmarks suite to see what that gives:
>>> http://hg.python.org/benchmarks/
>>> 
>>> Regards
>>> 
>>> Antoine.
>> 
>> or pypy benchmark suite which is more comprehensive for python 2.7
>> (http://bitbucket.org/pypy/benchmarks)
> 
> Besides, is there any reason not to use clang by default on OS X?

The 32-bit installer contains binaries that work on PPC, that's why those are built using
an older version of Xcode. I'd have to check if that version of Xcode supports clang,
and if that version of clang is good enough.

The "intel" installer can, and should, be build with clang (and preferably with the most
recent Xcode release to ensure that the latest supported compiler is used). 

Note that the CPython configure script, and distutils, already use clang by
default if you a recent Xcode but that's primarily because gcc is llvm-gcc when you use
Xcode and llvm-gcc is broken (it miscompiles at least the unicode implementation in Python 3.3),
this overrides the default behavior of configure (using gcc whenever it is available unless
the user explictly overrides).

Ronald


From martin at v.loewis.de  Mon Jul 22 14:24:01 2013
From: martin at v.loewis.de (martin at v.loewis.de)
Date: Mon, 22 Jul 2013 14:24:01 +0200
Subject: [Python-Dev] Building a Faster Python
In-Reply-To: <b8ae1000f40a43d99b51c2a35c5ec900@BY2PR03MB206.namprd03.prod.outlook.com>
References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com>
	<CAL9jXCGvFKUfUjs3x9E3XBf+_im1-gGakfhCG9h1YdwC9kNVAg@mail.gmail.com>
	<b8ae1000f40a43d99b51c2a35c5ec900@BY2PR03MB206.namprd03.prod.outlook.com>
Message-ID: <20130722142401.Horde.nqeJmFp2uS2W98k7RhbP4Q1@webmail.df.eu>


Quoting Steve Dower <Steve.Dower at microsoft.com>:

>> As a Windows user, it makes me wonder if compiling with the latest  
>> version of the Microsoft compiler
>> would improve things similarly?
>
> I'd expect to see some improvement, based solely on the bugs fixed  
> recently by the optimizer team. No idea how much, but I know that  
> Martin builds with PGO, and that's also been improved recently.

I would not expect to see any improvement, unfortunately. I used to
build with PGO, but the recent VS releases miscompiled code, which
was very hard to track down. So PGO has lost in my view for at least
3 VS releases :-(

>> (Though updating project files to that is almost certainly a bigger  
>> project than the gcc update.)
>
> No idea what the gcc update involved, but this is only accurate if  
> it took less than two minutes :-) I've upgraded the CPython projects  
> before and everything worked fine.

Still, we couldn't release 2.7 with any other compiler because of the CRT
dependency.

Regards,
Martin



From storchaka at gmail.com  Mon Jul 22 15:30:41 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 22 Jul 2013 16:30:41 +0300
Subject: [Python-Dev] Dash
In-Reply-To: <ks996h$p10$1@ger.gmane.org>
References: <ks996h$p10$1@ger.gmane.org>
Message-ID: <ksjc5r$gk5$1@ger.gmane.org>

I have opened an issue (http://bugs.python.org/issue18529) for patches.


From eric at trueblade.com  Mon Jul 22 16:09:15 2013
From: eric at trueblade.com (Eric V. Smith)
Date: Mon, 22 Jul 2013 10:09:15 -0400
Subject: [Python-Dev] [Python-checkins] cpython (3.3): let's not return
 NULL from functions that should return ints
In-Reply-To: <3byyDG02GdzQlq@mail.python.org>
References: <3byyDG02GdzQlq@mail.python.org>
Message-ID: <51ED3D0B.9080803@trueblade.com>

[re-sending to python-dev]

On 7/21/2013 4:27 PM, benjamin.peterson wrote:

> @@ -267,7 +267,7 @@
>          Py_DECREF(io);
>          Py_DECREF(binary);
>          PyMem_FREE(found_encoding);
> -        return PyErr_SetFromErrnoWithFilenameObject(PyExc_IOError, filename);
> +        return 0;
>      }
>      fob = _PyObject_CallMethodId(io, &PyId_TextIOWrapper, "Os", binary, encoding);
>      Py_DECREF(io);

Did you mean to remove the call to PyErr_SetFromErrnoWithFilenameObject?
Or just call it, then ignore its return value and return 0?


-- 
Eric.
_______________________________________________
Python-checkins mailing list
Python-checkins at python.org
http://mail.python.org/mailman/listinfo/python-checkins


From benjamin at python.org  Mon Jul 22 16:58:07 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Mon, 22 Jul 2013 07:58:07 -0700
Subject: [Python-Dev] [Python-checkins] cpython (3.3): let's not return
 NULL from functions that should return ints
In-Reply-To: <51ED3D0B.9080803@trueblade.com>
References: <3byyDG02GdzQlq@mail.python.org> <51ED3D0B.9080803@trueblade.com>
Message-ID: <CAPZV6o_2ynCJRwxxUaSTUW_ufDCT4orDebK9GDqi5vFrheN7tw@mail.gmail.com>

Yes, I meant to remove it. This function gnores errors in general.

2013/7/22 Eric V. Smith <eric at trueblade.com>:
> [re-sending to python-dev]
>
> On 7/21/2013 4:27 PM, benjamin.peterson wrote:
>
>> @@ -267,7 +267,7 @@
>>          Py_DECREF(io);
>>          Py_DECREF(binary);
>>          PyMem_FREE(found_encoding);
>> -        return PyErr_SetFromErrnoWithFilenameObject(PyExc_IOError, filename);
>> +        return 0;
>>      }
>>      fob = _PyObject_CallMethodId(io, &PyId_TextIOWrapper, "Os", binary, encoding);
>>      Py_DECREF(io);
>
> Did you mean to remove the call to PyErr_SetFromErrnoWithFilenameObject?
> Or just call it, then ignore its return value and return 0?
>
>
> --
> Eric.
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> http://mail.python.org/mailman/listinfo/python-checkins
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/benjamin%40python.org



-- 
Regards,
Benjamin

From dmalcolm at redhat.com  Mon Jul 22 17:08:32 2013
From: dmalcolm at redhat.com (David Malcolm)
Date: Mon, 22 Jul 2013 11:08:32 -0400
Subject: [Python-Dev] Building a Faster Python
In-Reply-To: <CAK5idxTrb43uHnfi8LPF13NOoL8G-ZnFnF5crtya1O2cJEi-HA@mail.gmail.com>
References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com>
	<20130722081534.7cef6df0@fsol>
	<CAK5idxQoenes6S8JkYOMh-UzKbMgk8yzHQFhQHjyHdLPQub+3g@mail.gmail.com>
	<CAK5idxTrb43uHnfi8LPF13NOoL8G-ZnFnF5crtya1O2cJEi-HA@mail.gmail.com>
Message-ID: <1374505712.3898.13.camel@surprise>

On Mon, 2013-07-22 at 09:32 +0200, Maciej Fijalkowski wrote:
> On Mon, Jul 22, 2013 at 9:32 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> > On Mon, Jul 22, 2013 at 8:15 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> >> On Sun, 21 Jul 2013 16:36:35 -0700
> >> Raymond Hettinger <raymond.hettinger at gmail.com> wrote:
> >>> Our current Mac OS X builds use GCC-4.2.
> >>>
> >>> On Python2.7, I ran a comparison of gcc-4.2.1 builds
> >>> versus gcc-4.8.1 and found that the latter makes a much
> >>> faster Python.  PyBench2.0 shows the total running time
> >>> dropping from 5653ms to 4571ms.  The code is uniformly
> >>> better in just about every category.
> >>
> >> You could try running the benchmarks suite to see what that gives:
> >> http://hg.python.org/benchmarks/
> >>
> >> Regards
> >>
> >> Antoine.
> >
> > or pypy benchmark suite which is more comprehensive for python 2.7
> > (http://bitbucket.org/pypy/benchmarks)
> 
> Besides, is there any reason not to use clang by default on OS X?

How did this thread go from:
  "for OS X, GCC 4.8.1 gives you significantly faster machine code
   than the system GCC 4.2.1"
to
  "let's just use clang"
?

Presumably if you want the faster possible machine code for the platform
the correct approach is to benchmark the available compilers trying
various options and to see which does best.  Note that the meaning of
e.g. "-O3" varies from compiler to compiler (I see someone else already
made the Apples-to-Apples pun).

(I should declare that I've been hacking on GCC for the last few months,
so I have an interest in this)

Hope this is constructive
Dave


From solipsis at pitrou.net  Mon Jul 22 17:15:08 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 22 Jul 2013 17:15:08 +0200
Subject: [Python-Dev] Building a Faster Python
References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com>
	<20130722081534.7cef6df0@fsol>
	<CAK5idxQoenes6S8JkYOMh-UzKbMgk8yzHQFhQHjyHdLPQub+3g@mail.gmail.com>
	<CAK5idxTrb43uHnfi8LPF13NOoL8G-ZnFnF5crtya1O2cJEi-HA@mail.gmail.com>
	<1374505712.3898.13.camel@surprise>
Message-ID: <20130722171508.43fa2ff7@pitrou.net>

Le Mon, 22 Jul 2013 11:08:32 -0400,
David Malcolm <dmalcolm at redhat.com> a ?crit :
> 
> How did this thread go from:
>   "for OS X, GCC 4.8.1 gives you significantly faster machine code
>    than the system GCC 4.2.1"
> to
>   "let's just use clang"
> ?
> 
> Presumably if you want the faster possible machine code for the
> platform the correct approach is to benchmark the available compilers
> trying various options and to see which does best.

Depends whether the goal is to choose the fastest compiler, or to
use the platform's official compiler. Linux distributions don't compile
packages with icc, AFAIK.

Regards

Antoine.



From dmalcolm at redhat.com  Mon Jul 22 17:24:34 2013
From: dmalcolm at redhat.com (David Malcolm)
Date: Mon, 22 Jul 2013 11:24:34 -0400
Subject: [Python-Dev] Building a Faster Python
In-Reply-To: <20130722171508.43fa2ff7@pitrou.net>
References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com>
	<20130722081534.7cef6df0@fsol>
	<CAK5idxQoenes6S8JkYOMh-UzKbMgk8yzHQFhQHjyHdLPQub+3g@mail.gmail.com>
	<CAK5idxTrb43uHnfi8LPF13NOoL8G-ZnFnF5crtya1O2cJEi-HA@mail.gmail.com>
	<1374505712.3898.13.camel@surprise>
	<20130722171508.43fa2ff7@pitrou.net>
Message-ID: <1374506674.3898.17.camel@surprise>

On Mon, 2013-07-22 at 17:15 +0200, Antoine Pitrou wrote:
> Le Mon, 22 Jul 2013 11:08:32 -0400,
> David Malcolm <dmalcolm at redhat.com> a ?crit :
> > 
> > How did this thread go from:
> >   "for OS X, GCC 4.8.1 gives you significantly faster machine code
> >    than the system GCC 4.2.1"
> > to
> >   "let's just use clang"
> > ?
> > 
> > Presumably if you want the faster possible machine code for the
> > platform the correct approach is to benchmark the available compilers
> > trying various options and to see which does best.
> 
> Depends whether the goal is to choose the fastest compiler, or to
> use the platform's official compiler. Linux distributions don't compile
> packages with icc, AFAIK.

If Intel put icc under a Free Software/Open Source license, maybe we
would ;-)

(only my own opinion, of course)
Dave


From ronaldoussoren at mac.com  Mon Jul 22 18:09:40 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Mon, 22 Jul 2013 18:09:40 +0200
Subject: [Python-Dev] Building a Faster Python
In-Reply-To: <1374505712.3898.13.camel@surprise>
References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com>
	<20130722081534.7cef6df0@fsol>
	<CAK5idxQoenes6S8JkYOMh-UzKbMgk8yzHQFhQHjyHdLPQub+3g@mail.gmail.com>
	<CAK5idxTrb43uHnfi8LPF13NOoL8G-ZnFnF5crtya1O2cJEi-HA@mail.gmail.com>
	<1374505712.3898.13.camel@surprise>
Message-ID: <F239B760-C40C-4F82-920D-5BB29BF002E6@mac.com>


On 22 Jul, 2013, at 17:08, David Malcolm <dmalcolm at redhat.com> wrote:

> On Mon, 2013-07-22 at 09:32 +0200, Maciej Fijalkowski wrote:
>> On Mon, Jul 22, 2013 at 9:32 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>>> On Mon, Jul 22, 2013 at 8:15 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>>>> On Sun, 21 Jul 2013 16:36:35 -0700
>>>> Raymond Hettinger <raymond.hettinger at gmail.com> wrote:
>>>>> Our current Mac OS X builds use GCC-4.2.
>>>>> 
>>>>> On Python2.7, I ran a comparison of gcc-4.2.1 builds
>>>>> versus gcc-4.8.1 and found that the latter makes a much
>>>>> faster Python.  PyBench2.0 shows the total running time
>>>>> dropping from 5653ms to 4571ms.  The code is uniformly
>>>>> better in just about every category.
>>>> 
>>>> You could try running the benchmarks suite to see what that gives:
>>>> http://hg.python.org/benchmarks/
>>>> 
>>>> Regards
>>>> 
>>>> Antoine.
>>> 
>>> or pypy benchmark suite which is more comprehensive for python 2.7
>>> (http://bitbucket.org/pypy/benchmarks)
>> 
>> Besides, is there any reason not to use clang by default on OS X?
> 
> How did this thread go from:
>  "for OS X, GCC 4.8.1 gives you significantly faster machine code
>   than the system GCC 4.2.1"
> to
>  "let's just use clang"
> ?

Because we use the system compiler for building the official binary packages.

I'm not looking forward to bootstrapping GCC multiple times[*] just to be able
to build a slightly faster python.  And more so because you have to be very
careful when using a alternative compiler when building the installer, it is
very easy to end up with a build that others cannot use to build extension
because they don't have /Users/ronald/Tools/Compiler/gcc-4.8/bin/gcc.

> 
> (I should declare that I've been hacking on GCC for the last few months,
> so I have an interest in this)

It would still be interesting to know which compiler would generate the
fastest code for CPython.  Apple tends to claim that clang generates better
code than GCC, buit AFAIK they compare the latest clang with the latest
version of GCC that they used to ship, which is ancient by now.

Ronald

[*] multiple times due to fat binaries.


From ben at redfrontdoor.org  Mon Jul 22 23:44:28 2013
From: ben at redfrontdoor.org (Ben North)
Date: Mon, 22 Jul 2013 22:44:28 +0100
Subject: [Python-Dev] Avoiding error from repr() of recursive dictview
Message-ID: <CAHG0eJUmYZEeVXfq4s+r-YeGaeYhEN7Zw6EJrhF_PnoOM0QscA@mail.gmail.com>

Hi,

A friend of mine, Ruadhan O'Flanagan, came across a bug which turned out
to be the one noted in [http://bugs.python.org/issue18019], i.e.:

>>> d={}
>>> d[42]=d.viewvalues()
>>> d
<segmentation fault>

This issue has been fixed in hg; the behaviour now is that a
RuntimeError is produced for a recursive dictionary view:

>>> d={}
>>> d[42]=d.viewvalues()
>>> d # (output line-broken:)
{42: Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: maximum recursion depth exceeded
  while getting the repr of a list

Before finding this, though, I'd investigated and made a patch which
produces a similar "..." output to a recursive dictionary.  Reworking
against current 2.7, the behaviour would be:

>>> x={}
>>> x[42]=x
>>> x # existing behaviour for dictionaries:
{42: {...}}

>>> d={}
>>> d[42]=d.viewvalues()
>>> d # new behaviour:
{42: dict_values([...])}
>>> d[43]=d.viewitems()
>>> d # (output line-broken:)
{42: dict_values([..., dict_items([(42, ...), (43, ...)])]),
 43: dict_items([(42, dict_values([..., ...])), (43, ...)])}

Attached is the patch, against current 2.7 branch.  If there is interest
in applying this, I will create a proper patch (changelog entry, fix to
Lib/test/test_dictviews.py, etc.).

Thanks,

Ben.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: non-error-recursive-dictview.patch
Type: application/octet-stream
Size: 974 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130722/1def81fd/attachment.obj>

From greg at krypto.org  Tue Jul 23 00:01:38 2013
From: greg at krypto.org (Gregory P. Smith)
Date: Mon, 22 Jul 2013 15:01:38 -0700
Subject: [Python-Dev] Avoiding error from repr() of recursive dictview
In-Reply-To: <CAHG0eJUmYZEeVXfq4s+r-YeGaeYhEN7Zw6EJrhF_PnoOM0QscA@mail.gmail.com>
References: <CAHG0eJUmYZEeVXfq4s+r-YeGaeYhEN7Zw6EJrhF_PnoOM0QscA@mail.gmail.com>
Message-ID: <CAGE7PNKP8p6rXiWC7_ogtFegZDm2v28J3ppcLhQWGS7wfkh53A@mail.gmail.com>

On Mon, Jul 22, 2013 at 2:44 PM, Ben North <ben at redfrontdoor.org> wrote:

> Hi,
>
> A friend of mine, Ruadhan O'Flanagan, came across a bug which turned out
> to be the one noted in [http://bugs.python.org/issue18019], i.e.:
>
> >>> d={}
> >>> d[42]=d.viewvalues()
> >>> d
> <segmentation fault>
>
> This issue has been fixed in hg; the behaviour now is that a
> RuntimeError is produced for a recursive dictionary view:
>
> >>> d={}
> >>> d[42]=d.viewvalues()
> >>> d # (output line-broken:)
> {42: Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> RuntimeError: maximum recursion depth exceeded
>   while getting the repr of a list
>
> Before finding this, though, I'd investigated and made a patch which
> produces a similar "..." output to a recursive dictionary.  Reworking
> against current 2.7, the behaviour would be:
>
> >>> x={}
> >>> x[42]=x
> >>> x # existing behaviour for dictionaries:
> {42: {...}}
>
> >>> d={}
> >>> d[42]=d.viewvalues()
> >>> d # new behaviour:
> {42: dict_values([...])}
> >>> d[43]=d.viewitems()
> >>> d # (output line-broken:)
> {42: dict_values([..., dict_items([(42, ...), (43, ...)])]),
>  43: dict_items([(42, dict_values([..., ...])), (43, ...)])}
>
> Attached is the patch, against current 2.7 branch.  If there is interest
> in applying this, I will create a proper patch (changelog entry, fix to
> Lib/test/test_dictviews.py, etc.).
>

Mailing lists are where patches go to get lost and die. :)  Post it on an
issue on bugs.python.org.  Given that the RuntimeError fix has been
released, your proposed ... behavior is arguably a new feature so I'd only
expect this to make sense for consideration in 3.4, not 2.7.  (if accepted
at all)

-gps


>
> Thanks,
>
> Ben.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130722/1ba4db5f/attachment.html>

From ben at redfrontdoor.org  Tue Jul 23 00:26:46 2013
From: ben at redfrontdoor.org (Ben North)
Date: Mon, 22 Jul 2013 23:26:46 +0100
Subject: [Python-Dev] Avoiding error from repr() of recursive dictview
In-Reply-To: <CAGE7PNKP8p6rXiWC7_ogtFegZDm2v28J3ppcLhQWGS7wfkh53A@mail.gmail.com>
References: <CAHG0eJUmYZEeVXfq4s+r-YeGaeYhEN7Zw6EJrhF_PnoOM0QscA@mail.gmail.com>
	<CAGE7PNKP8p6rXiWC7_ogtFegZDm2v28J3ppcLhQWGS7wfkh53A@mail.gmail.com>
Message-ID: <CAHG0eJVzp2sfuBH1pi6f_-PLbbNwjDBuA6qmGsJmOb_GgiGhmQ@mail.gmail.com>

Hi Greg,

Thanks for the feedback.

On 22 July 2013 23:01, Gregory P. Smith <greg at krypto.org> wrote:
> On Mon, Jul 22, 2013 at 2:44 PM, Ben North <ben at redfrontdoor.org> wrote:
>> [... proposed change of behaviour such that a recursive
>> dictview gives a repr() with "..." rather than a RuntimeError ...]
>
> [...] Post it on an issue on bugs.python.org.

http://bugs.python.org/issue18533

Ben.

From zuo at chopin.edu.pl  Tue Jul 23 00:30:12 2013
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Tue, 23 Jul 2013 00:30:12 +0200
Subject: [Python-Dev] Avoiding error from repr() of recursive dictview
In-Reply-To: <CAGE7PNKP8p6rXiWC7_ogtFegZDm2v28J3ppcLhQWGS7wfkh53A@mail.gmail.com>
References: <CAHG0eJUmYZEeVXfq4s+r-YeGaeYhEN7Zw6EJrhF_PnoOM0QscA@mail.gmail.com>
	<CAGE7PNKP8p6rXiWC7_ogtFegZDm2v28J3ppcLhQWGS7wfkh53A@mail.gmail.com>
Message-ID: <0f288d04d15c509860c14e6524468a69@chopin.edu.pl>

23.07.2013 00:01, Gregory P. Smith wrote:

> On Mon, Jul 22, 2013 at 2:44 PM, Ben North <ben at redfrontdoor.org> 
> wrote:
>
>> A friend of mine, Ruadhan O'Flanagan, came across a bug which turned 
>> out
>> to be the one noted in [http://bugs.python.org/issue18019 [1]], 
>> i.e.:
>>
>> >>> d={}
>> >>> d[42]=d.viewvalues()
>> >>> d
>> <segmentation fault>
>>
>> This issue has been fixed in hg; the behaviour now is that a
>> RuntimeError is produced for a recursive dictionary view:
>>
>> >>> d={}
>> >>> d[42]=d.viewvalues()
>> >>> d # (output line-broken:)
>> {42: Traceback (most recent call last):
>> ? File "<stdin>", line 1, in <module>
>> RuntimeError: maximum recursion depth exceeded
>> ? while getting the repr of a list
>>
>> Before finding this, though, I'd investigated and made a patch which
>> produces a similar "..." output to a recursive dictionary. 
>> ?Reworking
>> against current 2.7, the behaviour would be:
>>
>> >>> x={}
>> >>> x[42]=x
>> >>> x # existing behaviour for dictionaries:
>> {42: {...}}
>>
>> >>> d={}
>> >>> d[42]=d.viewvalues()
>> >>> d # new behaviour:
>> {42: dict_values([...])}
>> >>> d[43]=d.viewitems()
>> >>> d # (output line-broken:)
>> {42: dict_values([..., dict_items([(42, ...), (43, ...)])]),
>> ?43: dict_items([(42, dict_values([..., ...])), (43, ...)])}
>>
>> Attached is the patch, against current 2.7 branch. ?If there is 
>> interest
>> in applying this, I will create a proper patch (changelog entry, fix 
>> to
>> Lib/test/test_dictviews.py, etc.).
>
> Mailing lists are where patches go to get lost and die. :) ?Post it
> on an issue on bugs.python.org [4]. ?Given that the RuntimeError fix
> has been released, your proposed ... behavior is arguably a new
> feature so I'd only expect this to make sense for consideration in
> 3.4, not 2.7. ?(if accepted at all)

IMHO it's still a bug (even though not so painful as segfault) that 
should also be fixed in 2.7 and 3.2/3.3.

In other cases (such as `d={}; d[42]=d; repr(d)`) Python does its best 
to avoid an error -- why in this case (`d={}; 
d[42]=d.<Py2.x:view>values(); repr(d)`) should it raise an exception? 
IMHO it's an obvious oversight in implementation, not a feature that 
anybody would expect.

Regards.
*j


From victor.stinner at gmail.com  Tue Jul 23 01:36:47 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 23 Jul 2013 01:36:47 +0200
Subject: [Python-Dev] [Python-checkins] cpython: Fix #18530. Remove
 extra stat call from posixpath.ismount
In-Reply-To: <3bzW5x3YpTz7Lks@mail.python.org>
References: <3bzW5x3YpTz7Lks@mail.python.org>
Message-ID: <CAMpsgwZr9+om2L2ZGT=irYpQet1QRpQ9Ph6BSy7zujjg1Nk+hQ@mail.gmail.com>

Could you please keep the comment "# A symlink can never be a mount point"
? It is useful. (I didn't know that, I'm not a windows developer.)

Victor
Le 22 juil. 2013 20:08, "brian.curtin" <python-checkins at python.org> a
?crit :

> http://hg.python.org/cpython/rev/240adc564539
> changeset:   84791:240adc564539
> parent:      84788:84d6c1c0665e
> user:        Brian Curtin <brian at python.org>
> date:        Mon Jul 22 13:07:52 2013 -0500
> summary:
>   Fix #18530. Remove extra stat call from posixpath.ismount
>
> files:
>   Lib/posixpath.py |  22 ++++++++++++++--------
>   Misc/NEWS        |   3 +++
>   2 files changed, 17 insertions(+), 8 deletions(-)
>
>
> diff --git a/Lib/posixpath.py b/Lib/posixpath.py
> --- a/Lib/posixpath.py
> +++ b/Lib/posixpath.py
> @@ -182,18 +182,24 @@
>
>  def ismount(path):
>      """Test whether a path is a mount point"""
> -    if islink(path):
> -        # A symlink can never be a mount point
> -        return False
>      try:
>          s1 = os.lstat(path)
> -        if isinstance(path, bytes):
> -            parent = join(path, b'..')
> -        else:
> -            parent = join(path, '..')
> +    except OSError:
> +        # It doesn't exist -- so not a mount point. :-)
> +        return False
> +    else:
> +        if stat.S_ISLNK(s1.st_mode):
> +            return False
> +
> +    if isinstance(path, bytes):
> +        parent = join(path, b'..')
> +    else:
> +        parent = join(path, '..')
> +    try:
>          s2 = os.lstat(parent)
>      except OSError:
> -        return False # It doesn't exist -- so not a mount point :-)
> +        return False
> +
>      dev1 = s1.st_dev
>      dev2 = s2.st_dev
>      if dev1 != dev2:
> diff --git a/Misc/NEWS b/Misc/NEWS
> --- a/Misc/NEWS
> +++ b/Misc/NEWS
> @@ -162,6 +162,9 @@
>  Library
>  -------
>
> +- Issue #18530: Remove additional stat call from posixpath.ismount.
> +  Patch by Alex Gaynor.
> +
>  - Issue #18514: Fix unreachable Py_DECREF() call in PyCData_FromBaseObj()
>
>  - Issue #9177: Calling read() or write() now raises ValueError, not
>
> --
> Repository URL: http://hg.python.org/cpython
>
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> http://mail.python.org/mailman/listinfo/python-checkins
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130723/0df23566/attachment.html>

From victor.stinner at gmail.com  Tue Jul 23 01:43:30 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 23 Jul 2013 01:43:30 +0200
Subject: [Python-Dev] [Python-checkins] cpython: Issue #18520: Add a new
 PyStructSequence_InitType2() function, same than
In-Reply-To: <3bzcCt6wttz7Ljj@mail.python.org>
References: <3bzcCt6wttz7Ljj@mail.python.org>
Message-ID: <CAMpsgwb1HyFQyjVMig7yPgyDHHbTOfKKeiU5MEhRuUr7he_4UQ@mail.gmail.com>

"Add a new PyStructSequence_InitType2()"

I added a new function because I guess that it would break the API (and
ABI) to change the return type of a function in a minor release.

Tell me if you have a better name than PyStructSequence_InitType2() ;-)

"Ex" suffix is usually used when parameters are added. It is not the case
here.

Victor
Le 22 juil. 2013 23:59, "victor.stinner" <python-checkins at python.org> a
?crit :

> http://hg.python.org/cpython/rev/fc718c177ee6
> changeset:   84793:fc718c177ee6
> user:        Victor Stinner <victor.stinner at gmail.com>
> date:        Mon Jul 22 22:24:54 2013 +0200
> summary:
>   Issue #18520: Add a new PyStructSequence_InitType2() function, same than
> PyStructSequence_InitType() except that it has a return value (0 on
> success,
> -1 on error).
>
>  * PyStructSequence_InitType2() now raises MemoryError on memory
> allocation failure
>  * Fix also some calls to PyDict_SetItemString(): handle error
>
> files:
>   Include/pythonrun.h    |   2 +-
>   Include/structseq.h    |   2 +
>   Misc/NEWS              |   4 +++
>   Modules/_lsprof.c      |  10 ++++---
>   Modules/grpmodule.c    |  11 ++++++--
>   Modules/posixmodule.c  |  24 ++++++++++++------
>   Modules/pwdmodule.c    |   5 ++-
>   Modules/resource.c     |   9 ++++--
>   Modules/signalmodule.c |   7 +++--
>   Modules/spwdmodule.c   |   8 ++++--
>   Modules/timemodule.c   |   5 ++-
>   Objects/floatobject.c  |   9 ++++--
>   Objects/longobject.c   |   6 +++-
>   Objects/structseq.c    |  37 +++++++++++++++++++++--------
>   Python/pythonrun.c     |   3 +-
>   Python/sysmodule.c     |  23 ++++++++++++-----
>   Python/thread.c        |   6 +++-
>   17 files changed, 117 insertions(+), 54 deletions(-)
>
>
> diff --git a/Include/pythonrun.h b/Include/pythonrun.h
> --- a/Include/pythonrun.h
> +++ b/Include/pythonrun.h
> @@ -197,7 +197,7 @@
>  PyAPI_FUNC(void) _PyExc_Init(PyObject * bltinmod);
>  PyAPI_FUNC(void) _PyImportHooks_Init(void);
>  PyAPI_FUNC(int) _PyFrame_Init(void);
> -PyAPI_FUNC(void) _PyFloat_Init(void);
> +PyAPI_FUNC(int) _PyFloat_Init(void);
>  PyAPI_FUNC(int) PyByteArray_Init(void);
>  PyAPI_FUNC(void) _PyRandom_Init(void);
>  #endif
> diff --git a/Include/structseq.h b/Include/structseq.h
> --- a/Include/structseq.h
> +++ b/Include/structseq.h
> @@ -24,6 +24,8 @@
>  #ifndef Py_LIMITED_API
>  PyAPI_FUNC(void) PyStructSequence_InitType(PyTypeObject *type,
>                                             PyStructSequence_Desc *desc);
> +PyAPI_FUNC(int) PyStructSequence_InitType2(PyTypeObject *type,
> +                                           PyStructSequence_Desc *desc);
>  #endif
>  PyAPI_FUNC(PyTypeObject*) PyStructSequence_NewType(PyStructSequence_Desc
> *desc);
>
> diff --git a/Misc/NEWS b/Misc/NEWS
> --- a/Misc/NEWS
> +++ b/Misc/NEWS
> @@ -10,6 +10,10 @@
>  Core and Builtins
>  -----------------
>
> +- Issue #18520: Add a new PyStructSequence_InitType2() function, same than
> +  PyStructSequence_InitType() except that it has a return value (0 on
> success,
> +  -1 on error).
> +
>  - Issue #15905: Fix theoretical buffer overflow in handling of
> sys.argv[0],
>    prefix and exec_prefix if the operation system does not obey MAXPATHLEN.
>
> diff --git a/Modules/_lsprof.c b/Modules/_lsprof.c
> --- a/Modules/_lsprof.c
> +++ b/Modules/_lsprof.c
> @@ -884,10 +884,12 @@
>      PyDict_SetItemString(d, "Profiler", (PyObject *)&PyProfiler_Type);
>
>      if (!initialized) {
> -        PyStructSequence_InitType(&StatsEntryType,
> -                                  &profiler_entry_desc);
> -        PyStructSequence_InitType(&StatsSubEntryType,
> -                                  &profiler_subentry_desc);
> +        if (PyStructSequence_InitType2(&StatsEntryType,
> +                                       &profiler_entry_desc) < 0)
> +            return NULL;
> +        if (PyStructSequence_InitType2(&StatsSubEntryType,
> +                                       &profiler_subentry_desc) < 0)
> +            return NULL;
>      }
>      Py_INCREF((PyObject*) &StatsEntryType);
>      Py_INCREF((PyObject*) &StatsSubEntryType);
> diff --git a/Modules/grpmodule.c b/Modules/grpmodule.c
> --- a/Modules/grpmodule.c
> +++ b/Modules/grpmodule.c
> @@ -210,9 +210,14 @@
>      if (m == NULL)
>          return NULL;
>      d = PyModule_GetDict(m);
> -    if (!initialized)
> -            PyStructSequence_InitType(&StructGrpType,
> &struct_group_type_desc);
> -    PyDict_SetItemString(d, "struct_group", (PyObject *) &StructGrpType);
> +    if (!initialized) {
> +        if (PyStructSequence_InitType2(&StructGrpType,
> +                                       &struct_group_type_desc) < 0)
> +            return NULL;
> +    }
> +    if (PyDict_SetItemString(d, "struct_group",
> +                             (PyObject *)&StructGrpType) < 0)
> +        return NULL;
>      initialized = 1;
>      return m;
>  }
> diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
> --- a/Modules/posixmodule.c
> +++ b/Modules/posixmodule.c
> @@ -11518,19 +11518,23 @@
>      if (!initialized) {
>  #if defined(HAVE_WAITID) && !defined(__APPLE__)
>          waitid_result_desc.name = MODNAME ".waitid_result";
> -        PyStructSequence_InitType(&WaitidResultType, &waitid_result_desc);
> +        if (PyStructSequence_InitType2(&WaitidResultType,
> &waitid_result_desc) < 0)
> +            return NULL;
>  #endif
>
>          stat_result_desc.name = MODNAME ".stat_result";
>          stat_result_desc.fields[7].name = PyStructSequence_UnnamedField;
>          stat_result_desc.fields[8].name = PyStructSequence_UnnamedField;
>          stat_result_desc.fields[9].name = PyStructSequence_UnnamedField;
> -        PyStructSequence_InitType(&StatResultType, &stat_result_desc);
> +        if (PyStructSequence_InitType2(&StatResultType,
> &stat_result_desc) < 0)
> +            return NULL;
>          structseq_new = StatResultType.tp_new;
>          StatResultType.tp_new = statresult_new;
>
>          statvfs_result_desc.name = MODNAME ".statvfs_result";
> -        PyStructSequence_InitType(&StatVFSResultType,
> &statvfs_result_desc);
> +        if (PyStructSequence_InitType2(&StatVFSResultType,
> +                                       &statvfs_result_desc) < 0)
> +            return NULL;
>  #ifdef NEED_TICKS_PER_SECOND
>  #  if defined(HAVE_SYSCONF) && defined(_SC_CLK_TCK)
>          ticks_per_second = sysconf(_SC_CLK_TCK);
> @@ -11543,12 +11547,15 @@
>
>  #if defined(HAVE_SCHED_SETPARAM) || defined(HAVE_SCHED_SETSCHEDULER)
>          sched_param_desc.name = MODNAME ".sched_param";
> -        PyStructSequence_InitType(&SchedParamType, &sched_param_desc);
> +        if (PyStructSequence_InitType2(&SchedParamType,
> &sched_param_desc) < 0)
> +            return NULL;
>          SchedParamType.tp_new = sched_param_new;
>  #endif
>
>          /* initialize TerminalSize_info */
> -        PyStructSequence_InitType(&TerminalSizeType, &TerminalSize_desc);
> +        if (PyStructSequence_InitType2(&TerminalSizeType,
> +                                       &TerminalSize_desc) < 0)
> +            return NULL;
>      }
>  #if defined(HAVE_WAITID) && !defined(__APPLE__)
>      Py_INCREF((PyObject*) &WaitidResultType);
> @@ -11566,11 +11573,13 @@
>  #endif
>
>      times_result_desc.name = MODNAME ".times_result";
> -    PyStructSequence_InitType(&TimesResultType, &times_result_desc);
> +    if (PyStructSequence_InitType2(&TimesResultType, &times_result_desc)
> < 0)
> +        return NULL;
>      PyModule_AddObject(m, "times_result", (PyObject *)&TimesResultType);
>
>      uname_result_desc.name = MODNAME ".uname_result";
> -    PyStructSequence_InitType(&UnameResultType, &uname_result_desc);
> +    if (PyStructSequence_InitType2(&UnameResultType, &uname_result_desc)
> < 0)
> +        return NULL;
>      PyModule_AddObject(m, "uname_result", (PyObject *)&UnameResultType);
>
>  #ifdef __APPLE__
> @@ -11648,7 +11657,6 @@
>      initialized = 1;
>
>      return m;
> -
>  }
>
>  #ifdef __cplusplus
> diff --git a/Modules/pwdmodule.c b/Modules/pwdmodule.c
> --- a/Modules/pwdmodule.c
> +++ b/Modules/pwdmodule.c
> @@ -216,8 +216,9 @@
>          return NULL;
>
>      if (!initialized) {
> -        PyStructSequence_InitType(&StructPwdType,
> -                                  &struct_pwd_type_desc);
> +        if (PyStructSequence_InitType2(&StructPwdType,
> +                                       &struct_pwd_type_desc) < 0)
> +            return NULL;
>          initialized = 1;
>      }
>      Py_INCREF((PyObject *) &StructPwdType);
> diff --git a/Modules/resource.c b/Modules/resource.c
> --- a/Modules/resource.c
> +++ b/Modules/resource.c
> @@ -263,9 +263,12 @@
>      /* Add some symbolic constants to the module */
>      Py_INCREF(PyExc_OSError);
>      PyModule_AddObject(m, "error", PyExc_OSError);
> -    if (!initialized)
> -        PyStructSequence_InitType(&StructRUsageType,
> -                                  &struct_rusage_desc);
> +    if (!initialized) {
> +        if (PyStructSequence_InitType2(&StructRUsageType,
> +                                       &struct_rusage_desc) < 0)
> +            return NULL;
> +    }
> +
>      Py_INCREF(&StructRUsageType);
>      PyModule_AddObject(m, "struct_rusage",
>                         (PyObject*) &StructRUsageType);
> diff --git a/Modules/signalmodule.c b/Modules/signalmodule.c
> --- a/Modules/signalmodule.c
> +++ b/Modules/signalmodule.c
> @@ -978,9 +978,10 @@
>          return NULL;
>
>  #if defined(HAVE_SIGWAITINFO) || defined(HAVE_SIGTIMEDWAIT)
> -    if (!initialized)
> -        PyStructSequence_InitType(&SiginfoType, &struct_siginfo_desc);
> -
> +    if (!initialized) {
> +        if (PyStructSequence_InitType2(&SiginfoType,
> &struct_siginfo_desc) < 0)
> +            return NULL;
> +    }
>      Py_INCREF((PyObject*) &SiginfoType);
>      PyModule_AddObject(m, "struct_siginfo", (PyObject*) &SiginfoType);
>      initialized = 1;
> diff --git a/Modules/spwdmodule.c b/Modules/spwdmodule.c
> --- a/Modules/spwdmodule.c
> +++ b/Modules/spwdmodule.c
> @@ -196,9 +196,11 @@
>      m=PyModule_Create(&spwdmodule);
>      if (m == NULL)
>          return NULL;
> -    if (!initialized)
> -        PyStructSequence_InitType(&StructSpwdType,
> -                                  &struct_spwd_type_desc);
> +    if (!initialized) {
> +        if (PyStructSequence_InitType2(&StructSpwdType,
> +                                       &struct_spwd_type_desc) < 0)
> +            return NULL;
> +    }
>      Py_INCREF((PyObject *) &StructSpwdType);
>      PyModule_AddObject(m, "struct_spwd", (PyObject *) &StructSpwdType);
>      initialized = 1;
> diff --git a/Modules/timemodule.c b/Modules/timemodule.c
> --- a/Modules/timemodule.c
> +++ b/Modules/timemodule.c
> @@ -1476,8 +1476,9 @@
>      PyInit_timezone(m);
>
>      if (!initialized) {
> -        PyStructSequence_InitType(&StructTimeType,
> -                                  &struct_time_type_desc);
> +        if (PyStructSequence_InitType2(&StructTimeType,
> +                                       &struct_time_type_desc) < 0)
> +            return NULL;
>
>  #ifdef MS_WINDOWS
>          winver.dwOSVersionInfoSize = sizeof(winver);
> diff --git a/Objects/floatobject.c b/Objects/floatobject.c
> --- a/Objects/floatobject.c
> +++ b/Objects/floatobject.c
> @@ -1853,7 +1853,7 @@
>      float_new,                                  /* tp_new */
>  };
>
> -void
> +int
>  _PyFloat_Init(void)
>  {
>      /* We attempt to determine if this machine is using IEEE
> @@ -1903,8 +1903,11 @@
>      float_format = detected_float_format;
>
>      /* Init float info */
> -    if (FloatInfoType.tp_name == 0)
> -        PyStructSequence_InitType(&FloatInfoType, &floatinfo_desc);
> +    if (FloatInfoType.tp_name == NULL) {
> +        if (PyStructSequence_InitType2(&FloatInfoType, &floatinfo_desc) <
> 0)
> +            return 0;
> +    }
> +    return 1;
>  }
>
>  int
> diff --git a/Objects/longobject.c b/Objects/longobject.c
> --- a/Objects/longobject.c
> +++ b/Objects/longobject.c
> @@ -5059,8 +5059,10 @@
>      }
>  #endif
>      /* initialize int_info */
> -    if (Int_InfoType.tp_name == 0)
> -        PyStructSequence_InitType(&Int_InfoType, &int_info_desc);
> +    if (Int_InfoType.tp_name == NULL) {
> +        if (PyStructSequence_InitType2(&Int_InfoType, &int_info_desc) < 0)
> +            return 0;
> +    }
>
>      return 1;
>  }
> diff --git a/Objects/structseq.c b/Objects/structseq.c
> --- a/Objects/structseq.c
> +++ b/Objects/structseq.c
> @@ -320,12 +320,13 @@
>      structseq_new,                              /* tp_new */
>  };
>
> -void
> -PyStructSequence_InitType(PyTypeObject *type, PyStructSequence_Desc *desc)
> +int
> +PyStructSequence_InitType2(PyTypeObject *type, PyStructSequence_Desc
> *desc)
>  {
>      PyObject *dict;
>      PyMemberDef* members;
>      int n_members, n_unnamed_members, i, k;
> +    PyObject *v;
>
>  #ifdef Py_TRACE_REFS
>      /* if the type object was chained, unchain it first
> @@ -347,8 +348,10 @@
>      type->tp_doc = desc->doc;
>
>      members = PyMem_NEW(PyMemberDef, n_members-n_unnamed_members+1);
> -    if (members == NULL)
> -        return;
> +    if (members == NULL) {
> +        PyErr_NoMemory();
> +        return -1;
> +    }
>
>      for (i = k = 0; i < n_members; ++i) {
>          if (desc->fields[i].name == PyStructSequence_UnnamedField)
> @@ -366,22 +369,33 @@
>      type->tp_members = members;
>
>      if (PyType_Ready(type) < 0)
> -        return;
> +        return -1;
>      Py_INCREF(type);
>
>      dict = type->tp_dict;
>  #define SET_DICT_FROM_INT(key, value)                           \
>      do {                                                        \
> -        PyObject *v = PyLong_FromLong((long) value);            \
> -        if (v != NULL) {                                        \
> -            PyDict_SetItemString(dict, key, v);                 \
> +        v = PyLong_FromLong((long) value);                      \
> +        if (v == NULL)                                          \
> +            return -1;                                          \
> +        if (PyDict_SetItemString(dict, key, v) < 0) {           \
>              Py_DECREF(v);                                       \
> +            return -1;                                          \
>          }                                                       \
> +        Py_DECREF(v);                                           \
>      } while (0)
>
>      SET_DICT_FROM_INT(visible_length_key, desc->n_in_sequence);
>      SET_DICT_FROM_INT(real_length_key, n_members);
>      SET_DICT_FROM_INT(unnamed_fields_key, n_unnamed_members);
> +
> +    return 0;
> +}
> +
> +void
> +PyStructSequence_InitType(PyTypeObject *type, PyStructSequence_Desc *desc)
> +{
> +    (void)PyStructSequence_InitType2(type, desc);
>  }
>
>  PyTypeObject*
> @@ -390,8 +404,11 @@
>      PyTypeObject *result;
>
>      result = (PyTypeObject*)PyType_GenericAlloc(&PyType_Type, 0);
> -    if (result != NULL) {
> -        PyStructSequence_InitType(result, desc);
> +    if (result == NULL)
> +        return NULL;
> +    if (PyStructSequence_InitType2(result, desc) < 0) {
> +        Py_DECREF(result);
> +        return NULL;
>      }
>      return result;
>  }
> diff --git a/Python/pythonrun.c b/Python/pythonrun.c
> --- a/Python/pythonrun.c
> +++ b/Python/pythonrun.c
> @@ -328,7 +328,8 @@
>      if (!PyByteArray_Init())
>          Py_FatalError("Py_Initialize: can't init bytearray");
>
> -    _PyFloat_Init();
> +    if (!_PyFloat_Init())
> +        Py_FatalError("Py_Initialize: can't init float");
>
>      interp->modules = PyDict_New();
>      if (interp->modules == NULL)
> diff --git a/Python/sysmodule.c b/Python/sysmodule.c
> --- a/Python/sysmodule.c
> +++ b/Python/sysmodule.c
> @@ -1634,8 +1634,10 @@
>      SET_SYS_FROM_STRING("int_info",
>                          PyLong_GetInfo());
>      /* initialize hash_info */
> -    if (Hash_InfoType.tp_name == 0)
> -        PyStructSequence_InitType(&Hash_InfoType, &hash_info_desc);
> +    if (Hash_InfoType.tp_name == NULL) {
> +        if (PyStructSequence_InitType2(&Hash_InfoType, &hash_info_desc) <
> 0)
> +            return NULL;
> +    }
>      SET_SYS_FROM_STRING("hash_info",
>                          get_hash_info());
>      SET_SYS_FROM_STRING("maxunicode",
> @@ -1676,8 +1678,11 @@
>      }
>
>      /* version_info */
> -    if (VersionInfoType.tp_name == 0)
> -        PyStructSequence_InitType(&VersionInfoType, &version_info_desc);
> +    if (VersionInfoType.tp_name == NULL) {
> +        if (PyStructSequence_InitType2(&VersionInfoType,
> +                                       &version_info_desc) < 0)
> +            return NULL;
> +    }
>      version_info = make_version_info();
>      SET_SYS_FROM_STRING("version_info", version_info);
>      /* prevent user from creating new instances */
> @@ -1688,8 +1693,10 @@
>      SET_SYS_FROM_STRING("implementation", make_impl_info(version_info));
>
>      /* flags */
> -    if (FlagsType.tp_name == 0)
> -        PyStructSequence_InitType(&FlagsType, &flags_desc);
> +    if (FlagsType.tp_name == 0) {
> +        if (PyStructSequence_InitType2(&FlagsType, &flags_desc) < 0)
> +            return NULL;
> +    }
>      SET_SYS_FROM_STRING("flags", make_flags());
>      /* prevent user from creating new instances */
>      FlagsType.tp_init = NULL;
> @@ -1699,7 +1706,9 @@
>  #if defined(MS_WINDOWS)
>      /* getwindowsversion */
>      if (WindowsVersionType.tp_name == 0)
> -        PyStructSequence_InitType(&WindowsVersionType,
> &windows_version_desc);
> +        if (PyStructSequence_InitType2(&WindowsVersionType,
> +                                       &windows_version_desc) < 0)
> +            return NULL;
>      /* prevent user from creating new instances */
>      WindowsVersionType.tp_init = NULL;
>      WindowsVersionType.tp_new = NULL;
> diff --git a/Python/thread.c b/Python/thread.c
> --- a/Python/thread.c
> +++ b/Python/thread.c
> @@ -399,8 +399,10 @@
>      int len;
>  #endif
>
> -    if (ThreadInfoType.tp_name == 0)
> -        PyStructSequence_InitType(&ThreadInfoType, &threadinfo_desc);
> +    if (ThreadInfoType.tp_name == 0) {
> +        if (PyStructSequence_InitType2(&ThreadInfoType, &threadinfo_desc)
> < 0)
> +            return NULL;
> +    }
>
>      threadinfo = PyStructSequence_New(&ThreadInfoType);
>      if (threadinfo == NULL)
>
> --
> Repository URL: http://hg.python.org/cpython
>
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> http://mail.python.org/mailman/listinfo/python-checkins
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130723/b648345b/attachment-0001.html>

From benjamin at python.org  Tue Jul 23 02:01:46 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Mon, 22 Jul 2013 17:01:46 -0700
Subject: [Python-Dev] [Python-checkins] cpython: Issue #18520: Add a new
 PyStructSequence_InitType2() function, same than
In-Reply-To: <CAMpsgwb1HyFQyjVMig7yPgyDHHbTOfKKeiU5MEhRuUr7he_4UQ@mail.gmail.com>
References: <3bzcCt6wttz7Ljj@mail.python.org>
	<CAMpsgwb1HyFQyjVMig7yPgyDHHbTOfKKeiU5MEhRuUr7he_4UQ@mail.gmail.com>
Message-ID: <CAPZV6o8N4ajUFuTHadq3T756O-zyMf6uGCRRHVyTwDkqtK6A4g@mail.gmail.com>

We've cheerfully broken the ABI before on minor releases, though if
it's part of the stable ABI, we can't be cavaliar about that anymore.

2013/7/22 Victor Stinner <victor.stinner at gmail.com>:
> "Add a new PyStructSequence_InitType2()"
>
> I added a new function because I guess that it would break the API (and ABI)
> to change the return type of a function in a minor release.
>
> Tell me if you have a better name than PyStructSequence_InitType2() ;-)
>
> "Ex" suffix is usually used when parameters are added. It is not the case
> here.
>
> Victor
>
> Le 22 juil. 2013 23:59, "victor.stinner" <python-checkins at python.org> a
> ?crit :
>>
>> http://hg.python.org/cpython/rev/fc718c177ee6
>> changeset:   84793:fc718c177ee6
>> user:        Victor Stinner <victor.stinner at gmail.com>
>> date:        Mon Jul 22 22:24:54 2013 +0200
>> summary:
>>   Issue #18520: Add a new PyStructSequence_InitType2() function, same than
>> PyStructSequence_InitType() except that it has a return value (0 on
>> success,
>> -1 on error).
>>
>>  * PyStructSequence_InitType2() now raises MemoryError on memory
>> allocation failure
>>  * Fix also some calls to PyDict_SetItemString(): handle error
>>
>> files:
>>   Include/pythonrun.h    |   2 +-
>>   Include/structseq.h    |   2 +
>>   Misc/NEWS              |   4 +++
>>   Modules/_lsprof.c      |  10 ++++---
>>   Modules/grpmodule.c    |  11 ++++++--
>>   Modules/posixmodule.c  |  24 ++++++++++++------
>>   Modules/pwdmodule.c    |   5 ++-
>>   Modules/resource.c     |   9 ++++--
>>   Modules/signalmodule.c |   7 +++--
>>   Modules/spwdmodule.c   |   8 ++++--
>>   Modules/timemodule.c   |   5 ++-
>>   Objects/floatobject.c  |   9 ++++--
>>   Objects/longobject.c   |   6 +++-
>>   Objects/structseq.c    |  37 +++++++++++++++++++++--------
>>   Python/pythonrun.c     |   3 +-
>>   Python/sysmodule.c     |  23 ++++++++++++-----
>>   Python/thread.c        |   6 +++-
>>   17 files changed, 117 insertions(+), 54 deletions(-)
>>
>>
>> diff --git a/Include/pythonrun.h b/Include/pythonrun.h
>> --- a/Include/pythonrun.h
>> +++ b/Include/pythonrun.h
>> @@ -197,7 +197,7 @@
>>  PyAPI_FUNC(void) _PyExc_Init(PyObject * bltinmod);
>>  PyAPI_FUNC(void) _PyImportHooks_Init(void);
>>  PyAPI_FUNC(int) _PyFrame_Init(void);
>> -PyAPI_FUNC(void) _PyFloat_Init(void);
>> +PyAPI_FUNC(int) _PyFloat_Init(void);
>>  PyAPI_FUNC(int) PyByteArray_Init(void);
>>  PyAPI_FUNC(void) _PyRandom_Init(void);
>>  #endif
>> diff --git a/Include/structseq.h b/Include/structseq.h
>> --- a/Include/structseq.h
>> +++ b/Include/structseq.h
>> @@ -24,6 +24,8 @@
>>  #ifndef Py_LIMITED_API
>>  PyAPI_FUNC(void) PyStructSequence_InitType(PyTypeObject *type,
>>                                             PyStructSequence_Desc *desc);
>> +PyAPI_FUNC(int) PyStructSequence_InitType2(PyTypeObject *type,
>> +                                           PyStructSequence_Desc *desc);
>>  #endif
>>  PyAPI_FUNC(PyTypeObject*) PyStructSequence_NewType(PyStructSequence_Desc
>> *desc);
>>
>> diff --git a/Misc/NEWS b/Misc/NEWS
>> --- a/Misc/NEWS
>> +++ b/Misc/NEWS
>> @@ -10,6 +10,10 @@
>>  Core and Builtins
>>  -----------------
>>
>> +- Issue #18520: Add a new PyStructSequence_InitType2() function, same
>> than
>> +  PyStructSequence_InitType() except that it has a return value (0 on
>> success,
>> +  -1 on error).
>> +
>>  - Issue #15905: Fix theoretical buffer overflow in handling of
>> sys.argv[0],
>>    prefix and exec_prefix if the operation system does not obey
>> MAXPATHLEN.
>>
>> diff --git a/Modules/_lsprof.c b/Modules/_lsprof.c
>> --- a/Modules/_lsprof.c
>> +++ b/Modules/_lsprof.c
>> @@ -884,10 +884,12 @@
>>      PyDict_SetItemString(d, "Profiler", (PyObject *)&PyProfiler_Type);
>>
>>      if (!initialized) {
>> -        PyStructSequence_InitType(&StatsEntryType,
>> -                                  &profiler_entry_desc);
>> -        PyStructSequence_InitType(&StatsSubEntryType,
>> -                                  &profiler_subentry_desc);
>> +        if (PyStructSequence_InitType2(&StatsEntryType,
>> +                                       &profiler_entry_desc) < 0)
>> +            return NULL;
>> +        if (PyStructSequence_InitType2(&StatsSubEntryType,
>> +                                       &profiler_subentry_desc) < 0)
>> +            return NULL;
>>      }
>>      Py_INCREF((PyObject*) &StatsEntryType);
>>      Py_INCREF((PyObject*) &StatsSubEntryType);
>> diff --git a/Modules/grpmodule.c b/Modules/grpmodule.c
>> --- a/Modules/grpmodule.c
>> +++ b/Modules/grpmodule.c
>> @@ -210,9 +210,14 @@
>>      if (m == NULL)
>>          return NULL;
>>      d = PyModule_GetDict(m);
>> -    if (!initialized)
>> -            PyStructSequence_InitType(&StructGrpType,
>> &struct_group_type_desc);
>> -    PyDict_SetItemString(d, "struct_group", (PyObject *) &StructGrpType);
>> +    if (!initialized) {
>> +        if (PyStructSequence_InitType2(&StructGrpType,
>> +                                       &struct_group_type_desc) < 0)
>> +            return NULL;
>> +    }
>> +    if (PyDict_SetItemString(d, "struct_group",
>> +                             (PyObject *)&StructGrpType) < 0)
>> +        return NULL;
>>      initialized = 1;
>>      return m;
>>  }
>> diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
>> --- a/Modules/posixmodule.c
>> +++ b/Modules/posixmodule.c
>> @@ -11518,19 +11518,23 @@
>>      if (!initialized) {
>>  #if defined(HAVE_WAITID) && !defined(__APPLE__)
>>          waitid_result_desc.name = MODNAME ".waitid_result";
>> -        PyStructSequence_InitType(&WaitidResultType,
>> &waitid_result_desc);
>> +        if (PyStructSequence_InitType2(&WaitidResultType,
>> &waitid_result_desc) < 0)
>> +            return NULL;
>>  #endif
>>
>>          stat_result_desc.name = MODNAME ".stat_result";
>>          stat_result_desc.fields[7].name = PyStructSequence_UnnamedField;
>>          stat_result_desc.fields[8].name = PyStructSequence_UnnamedField;
>>          stat_result_desc.fields[9].name = PyStructSequence_UnnamedField;
>> -        PyStructSequence_InitType(&StatResultType, &stat_result_desc);
>> +        if (PyStructSequence_InitType2(&StatResultType,
>> &stat_result_desc) < 0)
>> +            return NULL;
>>          structseq_new = StatResultType.tp_new;
>>          StatResultType.tp_new = statresult_new;
>>
>>          statvfs_result_desc.name = MODNAME ".statvfs_result";
>> -        PyStructSequence_InitType(&StatVFSResultType,
>> &statvfs_result_desc);
>> +        if (PyStructSequence_InitType2(&StatVFSResultType,
>> +                                       &statvfs_result_desc) < 0)
>> +            return NULL;
>>  #ifdef NEED_TICKS_PER_SECOND
>>  #  if defined(HAVE_SYSCONF) && defined(_SC_CLK_TCK)
>>          ticks_per_second = sysconf(_SC_CLK_TCK);
>> @@ -11543,12 +11547,15 @@
>>
>>  #if defined(HAVE_SCHED_SETPARAM) || defined(HAVE_SCHED_SETSCHEDULER)
>>          sched_param_desc.name = MODNAME ".sched_param";
>> -        PyStructSequence_InitType(&SchedParamType, &sched_param_desc);
>> +        if (PyStructSequence_InitType2(&SchedParamType,
>> &sched_param_desc) < 0)
>> +            return NULL;
>>          SchedParamType.tp_new = sched_param_new;
>>  #endif
>>
>>          /* initialize TerminalSize_info */
>> -        PyStructSequence_InitType(&TerminalSizeType, &TerminalSize_desc);
>> +        if (PyStructSequence_InitType2(&TerminalSizeType,
>> +                                       &TerminalSize_desc) < 0)
>> +            return NULL;
>>      }
>>  #if defined(HAVE_WAITID) && !defined(__APPLE__)
>>      Py_INCREF((PyObject*) &WaitidResultType);
>> @@ -11566,11 +11573,13 @@
>>  #endif
>>
>>      times_result_desc.name = MODNAME ".times_result";
>> -    PyStructSequence_InitType(&TimesResultType, &times_result_desc);
>> +    if (PyStructSequence_InitType2(&TimesResultType, &times_result_desc)
>> < 0)
>> +        return NULL;
>>      PyModule_AddObject(m, "times_result", (PyObject *)&TimesResultType);
>>
>>      uname_result_desc.name = MODNAME ".uname_result";
>> -    PyStructSequence_InitType(&UnameResultType, &uname_result_desc);
>> +    if (PyStructSequence_InitType2(&UnameResultType, &uname_result_desc)
>> < 0)
>> +        return NULL;
>>      PyModule_AddObject(m, "uname_result", (PyObject *)&UnameResultType);
>>
>>  #ifdef __APPLE__
>> @@ -11648,7 +11657,6 @@
>>      initialized = 1;
>>
>>      return m;
>> -
>>  }
>>
>>  #ifdef __cplusplus
>> diff --git a/Modules/pwdmodule.c b/Modules/pwdmodule.c
>> --- a/Modules/pwdmodule.c
>> +++ b/Modules/pwdmodule.c
>> @@ -216,8 +216,9 @@
>>          return NULL;
>>
>>      if (!initialized) {
>> -        PyStructSequence_InitType(&StructPwdType,
>> -                                  &struct_pwd_type_desc);
>> +        if (PyStructSequence_InitType2(&StructPwdType,
>> +                                       &struct_pwd_type_desc) < 0)
>> +            return NULL;
>>          initialized = 1;
>>      }
>>      Py_INCREF((PyObject *) &StructPwdType);
>> diff --git a/Modules/resource.c b/Modules/resource.c
>> --- a/Modules/resource.c
>> +++ b/Modules/resource.c
>> @@ -263,9 +263,12 @@
>>      /* Add some symbolic constants to the module */
>>      Py_INCREF(PyExc_OSError);
>>      PyModule_AddObject(m, "error", PyExc_OSError);
>> -    if (!initialized)
>> -        PyStructSequence_InitType(&StructRUsageType,
>> -                                  &struct_rusage_desc);
>> +    if (!initialized) {
>> +        if (PyStructSequence_InitType2(&StructRUsageType,
>> +                                       &struct_rusage_desc) < 0)
>> +            return NULL;
>> +    }
>> +
>>      Py_INCREF(&StructRUsageType);
>>      PyModule_AddObject(m, "struct_rusage",
>>                         (PyObject*) &StructRUsageType);
>> diff --git a/Modules/signalmodule.c b/Modules/signalmodule.c
>> --- a/Modules/signalmodule.c
>> +++ b/Modules/signalmodule.c
>> @@ -978,9 +978,10 @@
>>          return NULL;
>>
>>  #if defined(HAVE_SIGWAITINFO) || defined(HAVE_SIGTIMEDWAIT)
>> -    if (!initialized)
>> -        PyStructSequence_InitType(&SiginfoType, &struct_siginfo_desc);
>> -
>> +    if (!initialized) {
>> +        if (PyStructSequence_InitType2(&SiginfoType,
>> &struct_siginfo_desc) < 0)
>> +            return NULL;
>> +    }
>>      Py_INCREF((PyObject*) &SiginfoType);
>>      PyModule_AddObject(m, "struct_siginfo", (PyObject*) &SiginfoType);
>>      initialized = 1;
>> diff --git a/Modules/spwdmodule.c b/Modules/spwdmodule.c
>> --- a/Modules/spwdmodule.c
>> +++ b/Modules/spwdmodule.c
>> @@ -196,9 +196,11 @@
>>      m=PyModule_Create(&spwdmodule);
>>      if (m == NULL)
>>          return NULL;
>> -    if (!initialized)
>> -        PyStructSequence_InitType(&StructSpwdType,
>> -                                  &struct_spwd_type_desc);
>> +    if (!initialized) {
>> +        if (PyStructSequence_InitType2(&StructSpwdType,
>> +                                       &struct_spwd_type_desc) < 0)
>> +            return NULL;
>> +    }
>>      Py_INCREF((PyObject *) &StructSpwdType);
>>      PyModule_AddObject(m, "struct_spwd", (PyObject *) &StructSpwdType);
>>      initialized = 1;
>> diff --git a/Modules/timemodule.c b/Modules/timemodule.c
>> --- a/Modules/timemodule.c
>> +++ b/Modules/timemodule.c
>> @@ -1476,8 +1476,9 @@
>>      PyInit_timezone(m);
>>
>>      if (!initialized) {
>> -        PyStructSequence_InitType(&StructTimeType,
>> -                                  &struct_time_type_desc);
>> +        if (PyStructSequence_InitType2(&StructTimeType,
>> +                                       &struct_time_type_desc) < 0)
>> +            return NULL;
>>
>>  #ifdef MS_WINDOWS
>>          winver.dwOSVersionInfoSize = sizeof(winver);
>> diff --git a/Objects/floatobject.c b/Objects/floatobject.c
>> --- a/Objects/floatobject.c
>> +++ b/Objects/floatobject.c
>> @@ -1853,7 +1853,7 @@
>>      float_new,                                  /* tp_new */
>>  };
>>
>> -void
>> +int
>>  _PyFloat_Init(void)
>>  {
>>      /* We attempt to determine if this machine is using IEEE
>> @@ -1903,8 +1903,11 @@
>>      float_format = detected_float_format;
>>
>>      /* Init float info */
>> -    if (FloatInfoType.tp_name == 0)
>> -        PyStructSequence_InitType(&FloatInfoType, &floatinfo_desc);
>> +    if (FloatInfoType.tp_name == NULL) {
>> +        if (PyStructSequence_InitType2(&FloatInfoType, &floatinfo_desc) <
>> 0)
>> +            return 0;
>> +    }
>> +    return 1;
>>  }
>>
>>  int
>> diff --git a/Objects/longobject.c b/Objects/longobject.c
>> --- a/Objects/longobject.c
>> +++ b/Objects/longobject.c
>> @@ -5059,8 +5059,10 @@
>>      }
>>  #endif
>>      /* initialize int_info */
>> -    if (Int_InfoType.tp_name == 0)
>> -        PyStructSequence_InitType(&Int_InfoType, &int_info_desc);
>> +    if (Int_InfoType.tp_name == NULL) {
>> +        if (PyStructSequence_InitType2(&Int_InfoType, &int_info_desc) <
>> 0)
>> +            return 0;
>> +    }
>>
>>      return 1;
>>  }
>> diff --git a/Objects/structseq.c b/Objects/structseq.c
>> --- a/Objects/structseq.c
>> +++ b/Objects/structseq.c
>> @@ -320,12 +320,13 @@
>>      structseq_new,                              /* tp_new */
>>  };
>>
>> -void
>> -PyStructSequence_InitType(PyTypeObject *type, PyStructSequence_Desc
>> *desc)
>> +int
>> +PyStructSequence_InitType2(PyTypeObject *type, PyStructSequence_Desc
>> *desc)
>>  {
>>      PyObject *dict;
>>      PyMemberDef* members;
>>      int n_members, n_unnamed_members, i, k;
>> +    PyObject *v;
>>
>>  #ifdef Py_TRACE_REFS
>>      /* if the type object was chained, unchain it first
>> @@ -347,8 +348,10 @@
>>      type->tp_doc = desc->doc;
>>
>>      members = PyMem_NEW(PyMemberDef, n_members-n_unnamed_members+1);
>> -    if (members == NULL)
>> -        return;
>> +    if (members == NULL) {
>> +        PyErr_NoMemory();
>> +        return -1;
>> +    }
>>
>>      for (i = k = 0; i < n_members; ++i) {
>>          if (desc->fields[i].name == PyStructSequence_UnnamedField)
>> @@ -366,22 +369,33 @@
>>      type->tp_members = members;
>>
>>      if (PyType_Ready(type) < 0)
>> -        return;
>> +        return -1;
>>      Py_INCREF(type);
>>
>>      dict = type->tp_dict;
>>  #define SET_DICT_FROM_INT(key, value)                           \
>>      do {                                                        \
>> -        PyObject *v = PyLong_FromLong((long) value);            \
>> -        if (v != NULL) {                                        \
>> -            PyDict_SetItemString(dict, key, v);                 \
>> +        v = PyLong_FromLong((long) value);                      \
>> +        if (v == NULL)                                          \
>> +            return -1;                                          \
>> +        if (PyDict_SetItemString(dict, key, v) < 0) {           \
>>              Py_DECREF(v);                                       \
>> +            return -1;                                          \
>>          }                                                       \
>> +        Py_DECREF(v);                                           \
>>      } while (0)
>>
>>      SET_DICT_FROM_INT(visible_length_key, desc->n_in_sequence);
>>      SET_DICT_FROM_INT(real_length_key, n_members);
>>      SET_DICT_FROM_INT(unnamed_fields_key, n_unnamed_members);
>> +
>> +    return 0;
>> +}
>> +
>> +void
>> +PyStructSequence_InitType(PyTypeObject *type, PyStructSequence_Desc
>> *desc)
>> +{
>> +    (void)PyStructSequence_InitType2(type, desc);
>>  }
>>
>>  PyTypeObject*
>> @@ -390,8 +404,11 @@
>>      PyTypeObject *result;
>>
>>      result = (PyTypeObject*)PyType_GenericAlloc(&PyType_Type, 0);
>> -    if (result != NULL) {
>> -        PyStructSequence_InitType(result, desc);
>> +    if (result == NULL)
>> +        return NULL;
>> +    if (PyStructSequence_InitType2(result, desc) < 0) {
>> +        Py_DECREF(result);
>> +        return NULL;
>>      }
>>      return result;
>>  }
>> diff --git a/Python/pythonrun.c b/Python/pythonrun.c
>> --- a/Python/pythonrun.c
>> +++ b/Python/pythonrun.c
>> @@ -328,7 +328,8 @@
>>      if (!PyByteArray_Init())
>>          Py_FatalError("Py_Initialize: can't init bytearray");
>>
>> -    _PyFloat_Init();
>> +    if (!_PyFloat_Init())
>> +        Py_FatalError("Py_Initialize: can't init float");
>>
>>      interp->modules = PyDict_New();
>>      if (interp->modules == NULL)
>> diff --git a/Python/sysmodule.c b/Python/sysmodule.c
>> --- a/Python/sysmodule.c
>> +++ b/Python/sysmodule.c
>> @@ -1634,8 +1634,10 @@
>>      SET_SYS_FROM_STRING("int_info",
>>                          PyLong_GetInfo());
>>      /* initialize hash_info */
>> -    if (Hash_InfoType.tp_name == 0)
>> -        PyStructSequence_InitType(&Hash_InfoType, &hash_info_desc);
>> +    if (Hash_InfoType.tp_name == NULL) {
>> +        if (PyStructSequence_InitType2(&Hash_InfoType, &hash_info_desc) <
>> 0)
>> +            return NULL;
>> +    }
>>      SET_SYS_FROM_STRING("hash_info",
>>                          get_hash_info());
>>      SET_SYS_FROM_STRING("maxunicode",
>> @@ -1676,8 +1678,11 @@
>>      }
>>
>>      /* version_info */
>> -    if (VersionInfoType.tp_name == 0)
>> -        PyStructSequence_InitType(&VersionInfoType, &version_info_desc);
>> +    if (VersionInfoType.tp_name == NULL) {
>> +        if (PyStructSequence_InitType2(&VersionInfoType,
>> +                                       &version_info_desc) < 0)
>> +            return NULL;
>> +    }
>>      version_info = make_version_info();
>>      SET_SYS_FROM_STRING("version_info", version_info);
>>      /* prevent user from creating new instances */
>> @@ -1688,8 +1693,10 @@
>>      SET_SYS_FROM_STRING("implementation", make_impl_info(version_info));
>>
>>      /* flags */
>> -    if (FlagsType.tp_name == 0)
>> -        PyStructSequence_InitType(&FlagsType, &flags_desc);
>> +    if (FlagsType.tp_name == 0) {
>> +        if (PyStructSequence_InitType2(&FlagsType, &flags_desc) < 0)
>> +            return NULL;
>> +    }
>>      SET_SYS_FROM_STRING("flags", make_flags());
>>      /* prevent user from creating new instances */
>>      FlagsType.tp_init = NULL;
>> @@ -1699,7 +1706,9 @@
>>  #if defined(MS_WINDOWS)
>>      /* getwindowsversion */
>>      if (WindowsVersionType.tp_name == 0)
>> -        PyStructSequence_InitType(&WindowsVersionType,
>> &windows_version_desc);
>> +        if (PyStructSequence_InitType2(&WindowsVersionType,
>> +                                       &windows_version_desc) < 0)
>> +            return NULL;
>>      /* prevent user from creating new instances */
>>      WindowsVersionType.tp_init = NULL;
>>      WindowsVersionType.tp_new = NULL;
>> diff --git a/Python/thread.c b/Python/thread.c
>> --- a/Python/thread.c
>> +++ b/Python/thread.c
>> @@ -399,8 +399,10 @@
>>      int len;
>>  #endif
>>
>> -    if (ThreadInfoType.tp_name == 0)
>> -        PyStructSequence_InitType(&ThreadInfoType, &threadinfo_desc);
>> +    if (ThreadInfoType.tp_name == 0) {
>> +        if (PyStructSequence_InitType2(&ThreadInfoType, &threadinfo_desc)
>> < 0)
>> +            return NULL;
>> +    }
>>
>>      threadinfo = PyStructSequence_New(&ThreadInfoType);
>>      if (threadinfo == NULL)
>>
>> --
>> Repository URL: http://hg.python.org/cpython
>>
>> _______________________________________________
>> Python-checkins mailing list
>> Python-checkins at python.org
>> http://mail.python.org/mailman/listinfo/python-checkins
>>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/benjamin%40python.org
>



-- 
Regards,
Benjamin

From brian at python.org  Tue Jul 23 02:06:59 2013
From: brian at python.org (Brian Curtin)
Date: Mon, 22 Jul 2013 19:06:59 -0500
Subject: [Python-Dev] [Python-checkins] cpython: Fix #18530. Remove
 extra stat call from posixpath.ismount
In-Reply-To: <CAMpsgwZr9+om2L2ZGT=irYpQet1QRpQ9Ph6BSy7zujjg1Nk+hQ@mail.gmail.com>
References: <3bzW5x3YpTz7Lks@mail.python.org>
	<CAMpsgwZr9+om2L2ZGT=irYpQet1QRpQ9Ph6BSy7zujjg1Nk+hQ@mail.gmail.com>
Message-ID: <CAD+XWwpvEBOPp-_6tb3T47ROWQtout2ghRf4uR8jGiwzdi3FPA@mail.gmail.com>

On Mon, Jul 22, 2013 at 6:36 PM, Victor Stinner
<victor.stinner at gmail.com> wrote:
> Could you please keep the comment "# A symlink can never be a mount point" ?
> It is useful. (I didn't know that, I'm not a windows developer.)

I don't think that's specific to Windows, but I added it back in d6213012d87b.

From dwightdhutto at gmail.com  Tue Jul 23 07:39:03 2013
From: dwightdhutto at gmail.com (David Hutto)
Date: Tue, 23 Jul 2013 01:39:03 -0400
Subject: [Python-Dev] Building a Faster Python
In-Reply-To: <F239B760-C40C-4F82-920D-5BB29BF002E6@mac.com>
References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com>
	<20130722081534.7cef6df0@fsol>
	<CAK5idxQoenes6S8JkYOMh-UzKbMgk8yzHQFhQHjyHdLPQub+3g@mail.gmail.com>
	<CAK5idxTrb43uHnfi8LPF13NOoL8G-ZnFnF5crtya1O2cJEi-HA@mail.gmail.com>
	<1374505712.3898.13.camel@surprise>
	<F239B760-C40C-4F82-920D-5BB29BF002E6@mac.com>
Message-ID: <CA+vVgJWvbW0gaRh_A_3bC4i458X_RyG8bU37QjXtsphEzWrTfg@mail.gmail.com>

Just a suggestion from working with an assembly language stepper from a
while back with Intel x86...lost to an HD crash, but couldn't you
disassemble the binary, run through the assembly, and look for specific
instructions that you could refine into a simpler, smaller cycling time to
improve upon with a stepper that counts cycles?

I've seen other assembly steppers after I started mine for electrical
engineering of PCB's, but I don't know if they're counting cycles, and
suggesting other actions in an editor typ/optimization mode that the
compiler might miss.


On Mon, Jul 22, 2013 at 12:09 PM, Ronald Oussoren <ronaldoussoren at mac.com>wrote:

>
> On 22 Jul, 2013, at 17:08, David Malcolm <dmalcolm at redhat.com> wrote:
>
> > On Mon, 2013-07-22 at 09:32 +0200, Maciej Fijalkowski wrote:
> >> On Mon, Jul 22, 2013 at 9:32 AM, Maciej Fijalkowski <fijall at gmail.com>
> wrote:
> >>> On Mon, Jul 22, 2013 at 8:15 AM, Antoine Pitrou <solipsis at pitrou.net>
> wrote:
> >>>> On Sun, 21 Jul 2013 16:36:35 -0700
> >>>> Raymond Hettinger <raymond.hettinger at gmail.com> wrote:
> >>>>> Our current Mac OS X builds use GCC-4.2.
> >>>>>
> >>>>> On Python2.7, I ran a comparison of gcc-4.2.1 builds
> >>>>> versus gcc-4.8.1 and found that the latter makes a much
> >>>>> faster Python.  PyBench2.0 shows the total running time
> >>>>> dropping from 5653ms to 4571ms.  The code is uniformly
> >>>>> better in just about every category.
> >>>>
> >>>> You could try running the benchmarks suite to see what that gives:
> >>>> http://hg.python.org/benchmarks/
> >>>>
> >>>> Regards
> >>>>
> >>>> Antoine.
> >>>
> >>> or pypy benchmark suite which is more comprehensive for python 2.7
> >>> (http://bitbucket.org/pypy/benchmarks)
> >>
> >> Besides, is there any reason not to use clang by default on OS X?
> >
> > How did this thread go from:
> >  "for OS X, GCC 4.8.1 gives you significantly faster machine code
> >   than the system GCC 4.2.1"
> > to
> >  "let's just use clang"
> > ?
>
> Because we use the system compiler for building the official binary
> packages.
>
> I'm not looking forward to bootstrapping GCC multiple times[*] just to be
> able
> to build a slightly faster python.  And more so because you have to be very
> careful when using a alternative compiler when building the installer, it
> is
> very easy to end up with a build that others cannot use to build extension
> because they don't have /Users/ronald/Tools/Compiler/gcc-4.8/bin/gcc.
>
> >
> > (I should declare that I've been hacking on GCC for the last few months,
> > so I have an interest in this)
>
> It would still be interesting to know which compiler would generate the
> fastest code for CPython.  Apple tends to claim that clang generates better
> code than GCC, buit AFAIK they compare the latest clang with the latest
> version of GCC that they used to ship, which is ancient by now.
>
> Ronald
>
> [*] multiple times due to fat binaries.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/dwightdhutto%40gmail.com
>



-- 
Best Regards,
David Hutto
*CEO:* *http://www.hitwebdevelopment.com*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130723/765ab114/attachment.html>

From ronaldoussoren at mac.com  Tue Jul 23 08:15:29 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Tue, 23 Jul 2013 08:15:29 +0200
Subject: [Python-Dev] [Python-checkins] cpython: Issue #18520: Add a new
	PyStructSequence_InitType2() function, same than
In-Reply-To: <CAPZV6o8N4ajUFuTHadq3T756O-zyMf6uGCRRHVyTwDkqtK6A4g@mail.gmail.com>
References: <3bzcCt6wttz7Ljj@mail.python.org>
	<CAMpsgwb1HyFQyjVMig7yPgyDHHbTOfKKeiU5MEhRuUr7he_4UQ@mail.gmail.com>
	<CAPZV6o8N4ajUFuTHadq3T756O-zyMf6uGCRRHVyTwDkqtK6A4g@mail.gmail.com>
Message-ID: <C3C886D8-A4DE-4E24-A65D-D728FF7ED206@mac.com>


On 23 Jul, 2013, at 2:01, Benjamin Peterson <benjamin at python.org> wrote:

> We've cheerfully broken the ABI before on minor releases, though if
> it's part of the stable ABI, we can't be cavaliar about that anymore.

It is not part of the stable ABI. Given that the implementation of 
PyStructSequence_InitType() in the patch just calls PyStructSequence_InitType2()
and ignores the return value you could change the return value of ..InitType().

This may or may not break existing extensions using the function (depending on
platform ABI details, AFAIK this is not a problem on x86/x86_64), but reusing
extensions across python feature releases is not supported anyway.  There are
no problems when compiling code, most C compilers won't even warn about ignored
return values unless you explicitly ask for it.

Ronald

> 
> 2013/7/22 Victor Stinner <victor.stinner at gmail.com>:
>> "Add a new PyStructSequence_InitType2()"
>> 
>> I added a new function because I guess that it would break the API (and ABI)
>> to change the return type of a function in a minor release.
>> 
>> Tell me if you have a better name than PyStructSequence_InitType2() ;-)
>> 
>> "Ex" suffix is usually used when parameters are added. It is not the case
>> here.
>> 
>> Victor
>> 
>> Le 22 juil. 2013 23:59, "victor.stinner" <python-checkins at python.org> a
>> ?crit :
>>> 
>>> http://hg.python.org/cpython/rev/fc718c177ee6
>>> changeset:   84793:fc718c177ee6
>>> user:        Victor Stinner <victor.stinner at gmail.com>
>>> date:        Mon Jul 22 22:24:54 2013 +0200
>>> summary:
>>>  Issue #18520: Add a new PyStructSequence_InitType2() function, same than
>>> PyStructSequence_InitType() except that it has a return value (0 on
>>> success,
>>> -1 on error).
>>> 
>>> * PyStructSequence_InitType2() now raises MemoryError on memory
>>> allocation failure
>>> * Fix also some calls to PyDict_SetItemString(): handle error
>>> 
>>> files:
>>>  Include/pythonrun.h    |   2 +-
>>>  Include/structseq.h    |   2 +
>>>  Misc/NEWS              |   4 +++
>>>  Modules/_lsprof.c      |  10 ++++---
>>>  Modules/grpmodule.c    |  11 ++++++--
>>>  Modules/posixmodule.c  |  24 ++++++++++++------
>>>  Modules/pwdmodule.c    |   5 ++-
>>>  Modules/resource.c     |   9 ++++--
>>>  Modules/signalmodule.c |   7 +++--
>>>  Modules/spwdmodule.c   |   8 ++++--
>>>  Modules/timemodule.c   |   5 ++-
>>>  Objects/floatobject.c  |   9 ++++--
>>>  Objects/longobject.c   |   6 +++-
>>>  Objects/structseq.c    |  37 +++++++++++++++++++++--------
>>>  Python/pythonrun.c     |   3 +-
>>>  Python/sysmodule.c     |  23 ++++++++++++-----
>>>  Python/thread.c        |   6 +++-
>>>  17 files changed, 117 insertions(+), 54 deletions(-)
>>> 
>>> 
>>> diff --git a/Include/pythonrun.h b/Include/pythonrun.h
>>> --- a/Include/pythonrun.h
>>> +++ b/Include/pythonrun.h
>>> @@ -197,7 +197,7 @@
>>> PyAPI_FUNC(void) _PyExc_Init(PyObject * bltinmod);
>>> PyAPI_FUNC(void) _PyImportHooks_Init(void);
>>> PyAPI_FUNC(int) _PyFrame_Init(void);
>>> -PyAPI_FUNC(void) _PyFloat_Init(void);
>>> +PyAPI_FUNC(int) _PyFloat_Init(void);
>>> PyAPI_FUNC(int) PyByteArray_Init(void);
>>> PyAPI_FUNC(void) _PyRandom_Init(void);
>>> #endif
>>> diff --git a/Include/structseq.h b/Include/structseq.h
>>> --- a/Include/structseq.h
>>> +++ b/Include/structseq.h
>>> @@ -24,6 +24,8 @@
>>> #ifndef Py_LIMITED_API
>>> PyAPI_FUNC(void) PyStructSequence_InitType(PyTypeObject *type,
>>>                                            PyStructSequence_Desc *desc);
>>> +PyAPI_FUNC(int) PyStructSequence_InitType2(PyTypeObject *type,
>>> +                                           PyStructSequence_Desc *desc);
>>> #endif
>>> PyAPI_FUNC(PyTypeObject*) PyStructSequence_NewType(PyStructSequence_Desc
>>> *desc);
>>> 
>>> diff --git a/Misc/NEWS b/Misc/NEWS
>>> --- a/Misc/NEWS
>>> +++ b/Misc/NEWS
>>> @@ -10,6 +10,10 @@
>>> Core and Builtins
>>> -----------------
>>> 
>>> +- Issue #18520: Add a new PyStructSequence_InitType2() function, same
>>> than
>>> +  PyStructSequence_InitType() except that it has a return value (0 on
>>> success,
>>> +  -1 on error).
>>> +
>>> - Issue #15905: Fix theoretical buffer overflow in handling of
>>> sys.argv[0],
>>>   prefix and exec_prefix if the operation system does not obey
>>> MAXPATHLEN.
>>> 
>>> diff --git a/Modules/_lsprof.c b/Modules/_lsprof.c
>>> --- a/Modules/_lsprof.c
>>> +++ b/Modules/_lsprof.c
>>> @@ -884,10 +884,12 @@
>>>     PyDict_SetItemString(d, "Profiler", (PyObject *)&PyProfiler_Type);
>>> 
>>>     if (!initialized) {
>>> -        PyStructSequence_InitType(&StatsEntryType,
>>> -                                  &profiler_entry_desc);
>>> -        PyStructSequence_InitType(&StatsSubEntryType,
>>> -                                  &profiler_subentry_desc);
>>> +        if (PyStructSequence_InitType2(&StatsEntryType,
>>> +                                       &profiler_entry_desc) < 0)
>>> +            return NULL;
>>> +        if (PyStructSequence_InitType2(&StatsSubEntryType,
>>> +                                       &profiler_subentry_desc) < 0)
>>> +            return NULL;
>>>     }
>>>     Py_INCREF((PyObject*) &StatsEntryType);
>>>     Py_INCREF((PyObject*) &StatsSubEntryType);
>>> diff --git a/Modules/grpmodule.c b/Modules/grpmodule.c
>>> --- a/Modules/grpmodule.c
>>> +++ b/Modules/grpmodule.c
>>> @@ -210,9 +210,14 @@
>>>     if (m == NULL)
>>>         return NULL;
>>>     d = PyModule_GetDict(m);
>>> -    if (!initialized)
>>> -            PyStructSequence_InitType(&StructGrpType,
>>> &struct_group_type_desc);
>>> -    PyDict_SetItemString(d, "struct_group", (PyObject *) &StructGrpType);
>>> +    if (!initialized) {
>>> +        if (PyStructSequence_InitType2(&StructGrpType,
>>> +                                       &struct_group_type_desc) < 0)
>>> +            return NULL;
>>> +    }
>>> +    if (PyDict_SetItemString(d, "struct_group",
>>> +                             (PyObject *)&StructGrpType) < 0)
>>> +        return NULL;
>>>     initialized = 1;
>>>     return m;
>>> }
>>> diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
>>> --- a/Modules/posixmodule.c
>>> +++ b/Modules/posixmodule.c
>>> @@ -11518,19 +11518,23 @@
>>>     if (!initialized) {
>>> #if defined(HAVE_WAITID) && !defined(__APPLE__)
>>>         waitid_result_desc.name = MODNAME ".waitid_result";
>>> -        PyStructSequence_InitType(&WaitidResultType,
>>> &waitid_result_desc);
>>> +        if (PyStructSequence_InitType2(&WaitidResultType,
>>> &waitid_result_desc) < 0)
>>> +            return NULL;
>>> #endif
>>> 
>>>         stat_result_desc.name = MODNAME ".stat_result";
>>>         stat_result_desc.fields[7].name = PyStructSequence_UnnamedField;
>>>         stat_result_desc.fields[8].name = PyStructSequence_UnnamedField;
>>>         stat_result_desc.fields[9].name = PyStructSequence_UnnamedField;
>>> -        PyStructSequence_InitType(&StatResultType, &stat_result_desc);
>>> +        if (PyStructSequence_InitType2(&StatResultType,
>>> &stat_result_desc) < 0)
>>> +            return NULL;
>>>         structseq_new = StatResultType.tp_new;
>>>         StatResultType.tp_new = statresult_new;
>>> 
>>>         statvfs_result_desc.name = MODNAME ".statvfs_result";
>>> -        PyStructSequence_InitType(&StatVFSResultType,
>>> &statvfs_result_desc);
>>> +        if (PyStructSequence_InitType2(&StatVFSResultType,
>>> +                                       &statvfs_result_desc) < 0)
>>> +            return NULL;
>>> #ifdef NEED_TICKS_PER_SECOND
>>> #  if defined(HAVE_SYSCONF) && defined(_SC_CLK_TCK)
>>>         ticks_per_second = sysconf(_SC_CLK_TCK);
>>> @@ -11543,12 +11547,15 @@
>>> 
>>> #if defined(HAVE_SCHED_SETPARAM) || defined(HAVE_SCHED_SETSCHEDULER)
>>>         sched_param_desc.name = MODNAME ".sched_param";
>>> -        PyStructSequence_InitType(&SchedParamType, &sched_param_desc);
>>> +        if (PyStructSequence_InitType2(&SchedParamType,
>>> &sched_param_desc) < 0)
>>> +            return NULL;
>>>         SchedParamType.tp_new = sched_param_new;
>>> #endif
>>> 
>>>         /* initialize TerminalSize_info */
>>> -        PyStructSequence_InitType(&TerminalSizeType, &TerminalSize_desc);
>>> +        if (PyStructSequence_InitType2(&TerminalSizeType,
>>> +                                       &TerminalSize_desc) < 0)
>>> +            return NULL;
>>>     }
>>> #if defined(HAVE_WAITID) && !defined(__APPLE__)
>>>     Py_INCREF((PyObject*) &WaitidResultType);
>>> @@ -11566,11 +11573,13 @@
>>> #endif
>>> 
>>>     times_result_desc.name = MODNAME ".times_result";
>>> -    PyStructSequence_InitType(&TimesResultType, &times_result_desc);
>>> +    if (PyStructSequence_InitType2(&TimesResultType, &times_result_desc)
>>> < 0)
>>> +        return NULL;
>>>     PyModule_AddObject(m, "times_result", (PyObject *)&TimesResultType);
>>> 
>>>     uname_result_desc.name = MODNAME ".uname_result";
>>> -    PyStructSequence_InitType(&UnameResultType, &uname_result_desc);
>>> +    if (PyStructSequence_InitType2(&UnameResultType, &uname_result_desc)
>>> < 0)
>>> +        return NULL;
>>>     PyModule_AddObject(m, "uname_result", (PyObject *)&UnameResultType);
>>> 
>>> #ifdef __APPLE__
>>> @@ -11648,7 +11657,6 @@
>>>     initialized = 1;
>>> 
>>>     return m;
>>> -
>>> }
>>> 
>>> #ifdef __cplusplus
>>> diff --git a/Modules/pwdmodule.c b/Modules/pwdmodule.c
>>> --- a/Modules/pwdmodule.c
>>> +++ b/Modules/pwdmodule.c
>>> @@ -216,8 +216,9 @@
>>>         return NULL;
>>> 
>>>     if (!initialized) {
>>> -        PyStructSequence_InitType(&StructPwdType,
>>> -                                  &struct_pwd_type_desc);
>>> +        if (PyStructSequence_InitType2(&StructPwdType,
>>> +                                       &struct_pwd_type_desc) < 0)
>>> +            return NULL;
>>>         initialized = 1;
>>>     }
>>>     Py_INCREF((PyObject *) &StructPwdType);
>>> diff --git a/Modules/resource.c b/Modules/resource.c
>>> --- a/Modules/resource.c
>>> +++ b/Modules/resource.c
>>> @@ -263,9 +263,12 @@
>>>     /* Add some symbolic constants to the module */
>>>     Py_INCREF(PyExc_OSError);
>>>     PyModule_AddObject(m, "error", PyExc_OSError);
>>> -    if (!initialized)
>>> -        PyStructSequence_InitType(&StructRUsageType,
>>> -                                  &struct_rusage_desc);
>>> +    if (!initialized) {
>>> +        if (PyStructSequence_InitType2(&StructRUsageType,
>>> +                                       &struct_rusage_desc) < 0)
>>> +            return NULL;
>>> +    }
>>> +
>>>     Py_INCREF(&StructRUsageType);
>>>     PyModule_AddObject(m, "struct_rusage",
>>>                        (PyObject*) &StructRUsageType);
>>> diff --git a/Modules/signalmodule.c b/Modules/signalmodule.c
>>> --- a/Modules/signalmodule.c
>>> +++ b/Modules/signalmodule.c
>>> @@ -978,9 +978,10 @@
>>>         return NULL;
>>> 
>>> #if defined(HAVE_SIGWAITINFO) || defined(HAVE_SIGTIMEDWAIT)
>>> -    if (!initialized)
>>> -        PyStructSequence_InitType(&SiginfoType, &struct_siginfo_desc);
>>> -
>>> +    if (!initialized) {
>>> +        if (PyStructSequence_InitType2(&SiginfoType,
>>> &struct_siginfo_desc) < 0)
>>> +            return NULL;
>>> +    }
>>>     Py_INCREF((PyObject*) &SiginfoType);
>>>     PyModule_AddObject(m, "struct_siginfo", (PyObject*) &SiginfoType);
>>>     initialized = 1;
>>> diff --git a/Modules/spwdmodule.c b/Modules/spwdmodule.c
>>> --- a/Modules/spwdmodule.c
>>> +++ b/Modules/spwdmodule.c
>>> @@ -196,9 +196,11 @@
>>>     m=PyModule_Create(&spwdmodule);
>>>     if (m == NULL)
>>>         return NULL;
>>> -    if (!initialized)
>>> -        PyStructSequence_InitType(&StructSpwdType,
>>> -                                  &struct_spwd_type_desc);
>>> +    if (!initialized) {
>>> +        if (PyStructSequence_InitType2(&StructSpwdType,
>>> +                                       &struct_spwd_type_desc) < 0)
>>> +            return NULL;
>>> +    }
>>>     Py_INCREF((PyObject *) &StructSpwdType);
>>>     PyModule_AddObject(m, "struct_spwd", (PyObject *) &StructSpwdType);
>>>     initialized = 1;
>>> diff --git a/Modules/timemodule.c b/Modules/timemodule.c
>>> --- a/Modules/timemodule.c
>>> +++ b/Modules/timemodule.c
>>> @@ -1476,8 +1476,9 @@
>>>     PyInit_timezone(m);
>>> 
>>>     if (!initialized) {
>>> -        PyStructSequence_InitType(&StructTimeType,
>>> -                                  &struct_time_type_desc);
>>> +        if (PyStructSequence_InitType2(&StructTimeType,
>>> +                                       &struct_time_type_desc) < 0)
>>> +            return NULL;
>>> 
>>> #ifdef MS_WINDOWS
>>>         winver.dwOSVersionInfoSize = sizeof(winver);
>>> diff --git a/Objects/floatobject.c b/Objects/floatobject.c
>>> --- a/Objects/floatobject.c
>>> +++ b/Objects/floatobject.c
>>> @@ -1853,7 +1853,7 @@
>>>     float_new,                                  /* tp_new */
>>> };
>>> 
>>> -void
>>> +int
>>> _PyFloat_Init(void)
>>> {
>>>     /* We attempt to determine if this machine is using IEEE
>>> @@ -1903,8 +1903,11 @@
>>>     float_format = detected_float_format;
>>> 
>>>     /* Init float info */
>>> -    if (FloatInfoType.tp_name == 0)
>>> -        PyStructSequence_InitType(&FloatInfoType, &floatinfo_desc);
>>> +    if (FloatInfoType.tp_name == NULL) {
>>> +        if (PyStructSequence_InitType2(&FloatInfoType, &floatinfo_desc) <
>>> 0)
>>> +            return 0;
>>> +    }
>>> +    return 1;
>>> }
>>> 
>>> int
>>> diff --git a/Objects/longobject.c b/Objects/longobject.c
>>> --- a/Objects/longobject.c
>>> +++ b/Objects/longobject.c
>>> @@ -5059,8 +5059,10 @@
>>>     }
>>> #endif
>>>     /* initialize int_info */
>>> -    if (Int_InfoType.tp_name == 0)
>>> -        PyStructSequence_InitType(&Int_InfoType, &int_info_desc);
>>> +    if (Int_InfoType.tp_name == NULL) {
>>> +        if (PyStructSequence_InitType2(&Int_InfoType, &int_info_desc) <
>>> 0)
>>> +            return 0;
>>> +    }
>>> 
>>>     return 1;
>>> }
>>> diff --git a/Objects/structseq.c b/Objects/structseq.c
>>> --- a/Objects/structseq.c
>>> +++ b/Objects/structseq.c
>>> @@ -320,12 +320,13 @@
>>>     structseq_new,                              /* tp_new */
>>> };
>>> 
>>> -void
>>> -PyStructSequence_InitType(PyTypeObject *type, PyStructSequence_Desc
>>> *desc)
>>> +int
>>> +PyStructSequence_InitType2(PyTypeObject *type, PyStructSequence_Desc
>>> *desc)
>>> {
>>>     PyObject *dict;
>>>     PyMemberDef* members;
>>>     int n_members, n_unnamed_members, i, k;
>>> +    PyObject *v;
>>> 
>>> #ifdef Py_TRACE_REFS
>>>     /* if the type object was chained, unchain it first
>>> @@ -347,8 +348,10 @@
>>>     type->tp_doc = desc->doc;
>>> 
>>>     members = PyMem_NEW(PyMemberDef, n_members-n_unnamed_members+1);
>>> -    if (members == NULL)
>>> -        return;
>>> +    if (members == NULL) {
>>> +        PyErr_NoMemory();
>>> +        return -1;
>>> +    }
>>> 
>>>     for (i = k = 0; i < n_members; ++i) {
>>>         if (desc->fields[i].name == PyStructSequence_UnnamedField)
>>> @@ -366,22 +369,33 @@
>>>     type->tp_members = members;
>>> 
>>>     if (PyType_Ready(type) < 0)
>>> -        return;
>>> +        return -1;
>>>     Py_INCREF(type);
>>> 
>>>     dict = type->tp_dict;
>>> #define SET_DICT_FROM_INT(key, value)                           \
>>>     do {                                                        \
>>> -        PyObject *v = PyLong_FromLong((long) value);            \
>>> -        if (v != NULL) {                                        \
>>> -            PyDict_SetItemString(dict, key, v);                 \
>>> +        v = PyLong_FromLong((long) value);                      \
>>> +        if (v == NULL)                                          \
>>> +            return -1;                                          \
>>> +        if (PyDict_SetItemString(dict, key, v) < 0) {           \
>>>             Py_DECREF(v);                                       \
>>> +            return -1;                                          \
>>>         }                                                       \
>>> +        Py_DECREF(v);                                           \
>>>     } while (0)
>>> 
>>>     SET_DICT_FROM_INT(visible_length_key, desc->n_in_sequence);
>>>     SET_DICT_FROM_INT(real_length_key, n_members);
>>>     SET_DICT_FROM_INT(unnamed_fields_key, n_unnamed_members);
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +void
>>> +PyStructSequence_InitType(PyTypeObject *type, PyStructSequence_Desc
>>> *desc)
>>> +{
>>> +    (void)PyStructSequence_InitType2(type, desc);
>>> }
>>> 
>>> PyTypeObject*
>>> @@ -390,8 +404,11 @@
>>>     PyTypeObject *result;
>>> 
>>>     result = (PyTypeObject*)PyType_GenericAlloc(&PyType_Type, 0);
>>> -    if (result != NULL) {
>>> -        PyStructSequence_InitType(result, desc);
>>> +    if (result == NULL)
>>> +        return NULL;
>>> +    if (PyStructSequence_InitType2(result, desc) < 0) {
>>> +        Py_DECREF(result);
>>> +        return NULL;
>>>     }
>>>     return result;
>>> }
>>> diff --git a/Python/pythonrun.c b/Python/pythonrun.c
>>> --- a/Python/pythonrun.c
>>> +++ b/Python/pythonrun.c
>>> @@ -328,7 +328,8 @@
>>>     if (!PyByteArray_Init())
>>>         Py_FatalError("Py_Initialize: can't init bytearray");
>>> 
>>> -    _PyFloat_Init();
>>> +    if (!_PyFloat_Init())
>>> +        Py_FatalError("Py_Initialize: can't init float");
>>> 
>>>     interp->modules = PyDict_New();
>>>     if (interp->modules == NULL)
>>> diff --git a/Python/sysmodule.c b/Python/sysmodule.c
>>> --- a/Python/sysmodule.c
>>> +++ b/Python/sysmodule.c
>>> @@ -1634,8 +1634,10 @@
>>>     SET_SYS_FROM_STRING("int_info",
>>>                         PyLong_GetInfo());
>>>     /* initialize hash_info */
>>> -    if (Hash_InfoType.tp_name == 0)
>>> -        PyStructSequence_InitType(&Hash_InfoType, &hash_info_desc);
>>> +    if (Hash_InfoType.tp_name == NULL) {
>>> +        if (PyStructSequence_InitType2(&Hash_InfoType, &hash_info_desc) <
>>> 0)
>>> +            return NULL;
>>> +    }
>>>     SET_SYS_FROM_STRING("hash_info",
>>>                         get_hash_info());
>>>     SET_SYS_FROM_STRING("maxunicode",
>>> @@ -1676,8 +1678,11 @@
>>>     }
>>> 
>>>     /* version_info */
>>> -    if (VersionInfoType.tp_name == 0)
>>> -        PyStructSequence_InitType(&VersionInfoType, &version_info_desc);
>>> +    if (VersionInfoType.tp_name == NULL) {
>>> +        if (PyStructSequence_InitType2(&VersionInfoType,
>>> +                                       &version_info_desc) < 0)
>>> +            return NULL;
>>> +    }
>>>     version_info = make_version_info();
>>>     SET_SYS_FROM_STRING("version_info", version_info);
>>>     /* prevent user from creating new instances */
>>> @@ -1688,8 +1693,10 @@
>>>     SET_SYS_FROM_STRING("implementation", make_impl_info(version_info));
>>> 
>>>     /* flags */
>>> -    if (FlagsType.tp_name == 0)
>>> -        PyStructSequence_InitType(&FlagsType, &flags_desc);
>>> +    if (FlagsType.tp_name == 0) {
>>> +        if (PyStructSequence_InitType2(&FlagsType, &flags_desc) < 0)
>>> +            return NULL;
>>> +    }
>>>     SET_SYS_FROM_STRING("flags", make_flags());
>>>     /* prevent user from creating new instances */
>>>     FlagsType.tp_init = NULL;
>>> @@ -1699,7 +1706,9 @@
>>> #if defined(MS_WINDOWS)
>>>     /* getwindowsversion */
>>>     if (WindowsVersionType.tp_name == 0)
>>> -        PyStructSequence_InitType(&WindowsVersionType,
>>> &windows_version_desc);
>>> +        if (PyStructSequence_InitType2(&WindowsVersionType,
>>> +                                       &windows_version_desc) < 0)
>>> +            return NULL;
>>>     /* prevent user from creating new instances */
>>>     WindowsVersionType.tp_init = NULL;
>>>     WindowsVersionType.tp_new = NULL;
>>> diff --git a/Python/thread.c b/Python/thread.c
>>> --- a/Python/thread.c
>>> +++ b/Python/thread.c
>>> @@ -399,8 +399,10 @@
>>>     int len;
>>> #endif
>>> 
>>> -    if (ThreadInfoType.tp_name == 0)
>>> -        PyStructSequence_InitType(&ThreadInfoType, &threadinfo_desc);
>>> +    if (ThreadInfoType.tp_name == 0) {
>>> +        if (PyStructSequence_InitType2(&ThreadInfoType, &threadinfo_desc)
>>> < 0)
>>> +            return NULL;
>>> +    }
>>> 
>>>     threadinfo = PyStructSequence_New(&ThreadInfoType);
>>>     if (threadinfo == NULL)
>>> 
>>> --
>>> Repository URL: http://hg.python.org/cpython
>>> 
>>> _______________________________________________
>>> Python-checkins mailing list
>>> Python-checkins at python.org
>>> http://mail.python.org/mailman/listinfo/python-checkins
>>> 
>> 
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> http://mail.python.org/mailman/options/python-dev/benjamin%40python.org
>> 
> 
> 
> 
> -- 
> Regards,
> Benjamin
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> http://mail.python.org/mailman/listinfo/python-checkins


From solipsis at pitrou.net  Tue Jul 23 08:27:27 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 23 Jul 2013 08:27:27 +0200
Subject: [Python-Dev] cpython: Add modeling file for Coverity Scan.
References: <3bzfGN573Hz7Lkf@mail.python.org>
Message-ID: <20130723082727.634ee6cb@fsol>

On Tue, 23 Jul 2013 01:31:24 +0200 (CEST)
christian.heimes <python-checkins at python.org> wrote:
> +
> +typedef int sdigit;
> +typedef long Py_ssize_t;

Can't you write "typedef ssize_t Py_ssize_t" instead?

Regards

Antoine.



From solipsis at pitrou.net  Tue Jul 23 08:23:45 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 23 Jul 2013 08:23:45 +0200
Subject: [Python-Dev] cpython: Issue #18520: Add a new
 PyStructSequence_InitType2() function, same than
References: <3bzcCt6wttz7Ljj@mail.python.org>
	<CAMpsgwb1HyFQyjVMig7yPgyDHHbTOfKKeiU5MEhRuUr7he_4UQ@mail.gmail.com>
	<CAPZV6o8N4ajUFuTHadq3T756O-zyMf6uGCRRHVyTwDkqtK6A4g@mail.gmail.com>
	<C3C886D8-A4DE-4E24-A65D-D728FF7ED206@mac.com>
	<20130723082230.57a5f5b2@fsol>
Message-ID: <20130723082345.037d0921@fsol>

On Tue, 23 Jul 2013 08:22:30 +0200
Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Tue, 23 Jul 2013 08:15:29 +0200
> Ronald Oussoren <ronaldoussoren at mac.com> wrote:
> > 
> > On 23 Jul, 2013, at 2:01, Benjamin Peterson <benjamin at python.org> wrote:
> > 
> > > We've cheerfully broken the ABI before on minor releases, though if
> > > it's part of the stable ABI, we can't be cavaliar about that anymore.
> > 
> > It is not part of the stable ABI. Given that the implementation of 
> > PyStructSequence_InitType() in the patch just calls PyStructSequence_InitType2()
> > and ignores the return value you could change the return value of ..InitType().
> > 
> > This may or may not break existing extensions using the function (depending on
> > platform ABI details, AFAIK this is not a problem on x86/x86_64), but reusing
> > extensions across python feature releases is not supported anyway.
> 
> Not supported? It should certainly be supported accross bugfix
> versions. Otherwise, installing a new bugfix version would force you to
> recompile all independently-installed extension modules.

Ah, but I see that "minor release" was used in the sense of "feature
release". My bad. (Grrr!)

Regards

Antoine.



From christian at python.org  Tue Jul 23 12:00:51 2013
From: christian at python.org (Christian Heimes)
Date: Tue, 23 Jul 2013 12:00:51 +0200
Subject: [Python-Dev] cpython (3.3): return NULL here
In-Reply-To: <3bznlW6d69z7LkD@mail.python.org>
References: <3bznlW6d69z7LkD@mail.python.org>
Message-ID: <51EE5453.4010209@python.org>

Am 23.07.2013 07:08, schrieb benjamin.peterson:
> http://hg.python.org/cpython/rev/042ff9325c5e
> changeset:   84804:042ff9325c5e
> branch:      3.3
> parent:      84789:bb63f813a00f
> user:        Benjamin Peterson <benjamin at python.org>
> date:        Mon Jul 22 22:08:09 2013 -0700
> summary:
>   return NULL here
> 
> files:
>   Python/dynload_shlib.c |  3 ++-
>   1 files changed, 2 insertions(+), 1 deletions(-)
> 
> 
> diff --git a/Python/dynload_shlib.c b/Python/dynload_shlib.c
> --- a/Python/dynload_shlib.c
> +++ b/Python/dynload_shlib.c
> @@ -91,7 +91,8 @@
>          int i;
>          struct stat statb;
>          if (fstat(fileno(fp), &statb) == -1) {
> -            return PyErr_SetFromErrno(PyExc_IOError);
> +            PyErr_SetFromErrno(PyExc_IOError);
> +            return NULL;
>          }

PyErr_SetFromErrno() already and always returns NULL. Or do you prefer
to return NULL explicitly?

Christian

From benjamin at python.org  Tue Jul 23 17:10:40 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Tue, 23 Jul 2013 08:10:40 -0700
Subject: [Python-Dev] [Python-checkins] cpython (3.3): return NULL here
In-Reply-To: <51EE5453.4010209@python.org>
References: <3bznlW6d69z7LkD@mail.python.org>
	<51EE5453.4010209@python.org>
Message-ID: <CAPZV6o_SFj39y42q=KpPmqY3oLwA7dFh1au50ZWLYEx-xw_hPg@mail.gmail.com>

2013/7/23 Christian Heimes <christian at python.org>:
> Am 23.07.2013 07:08, schrieb benjamin.peterson:
>> http://hg.python.org/cpython/rev/042ff9325c5e
>> changeset:   84804:042ff9325c5e
>> branch:      3.3
>> parent:      84789:bb63f813a00f
>> user:        Benjamin Peterson <benjamin at python.org>
>> date:        Mon Jul 22 22:08:09 2013 -0700
>> summary:
>>   return NULL here
>>
>> files:
>>   Python/dynload_shlib.c |  3 ++-
>>   1 files changed, 2 insertions(+), 1 deletions(-)
>>
>>
>> diff --git a/Python/dynload_shlib.c b/Python/dynload_shlib.c
>> --- a/Python/dynload_shlib.c
>> +++ b/Python/dynload_shlib.c
>> @@ -91,7 +91,8 @@
>>          int i;
>>          struct stat statb;
>>          if (fstat(fileno(fp), &statb) == -1) {
>> -            return PyErr_SetFromErrno(PyExc_IOError);
>> +            PyErr_SetFromErrno(PyExc_IOError);
>> +            return NULL;
>>          }
>
> PyErr_SetFromErrno() already and always returns NULL. Or do you prefer
> to return NULL explicitly?

It might always return NULL, but the compiler sees (PyObject *)NULL
when this function returns dl_funcptr.


--
Regards,
Benjamin

From pelhamny at gmail.com  Tue Jul 23 12:17:20 2013
From: pelhamny at gmail.com (Nicholas Hart)
Date: Tue, 23 Jul 2013 06:17:20 -0400
Subject: [Python-Dev] tuple index out of range
Message-ID: <CAGVtho5OWd_1qgAatknYhYJSZwMrQKeAgmExy0UaKbt8bfrZKw@mail.gmail.com>

Hi,

I am new to this list and to troubleshooting python.  I hope someone can
help me.  I am getting this tuple index out of range error while running a
test call to my python code.  Not sure what this error really means and was
hoping someone might shed some light on how to fix this.  Also was
wondering why much of my .py files are not getting compiled to .pyc upon
first run.. is this unusual or need I not worry?

Running python 2.5.2 on fedora.

Thanks,
Nick
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130723/c82d1625/attachment.html>

From ronaldoussoren at mac.com  Tue Jul 23 17:30:09 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Tue, 23 Jul 2013 17:30:09 +0200
Subject: [Python-Dev] tuple index out of range
In-Reply-To: <CAGVtho5OWd_1qgAatknYhYJSZwMrQKeAgmExy0UaKbt8bfrZKw@mail.gmail.com>
References: <CAGVtho5OWd_1qgAatknYhYJSZwMrQKeAgmExy0UaKbt8bfrZKw@mail.gmail.com>
Message-ID: <7E7D3ED0-8306-4F53-9D67-D1D7CACAAA87@mac.com>


On 23 Jul, 2013, at 12:17, Nicholas Hart <pelhamny at gmail.com> wrote:

> Hi,
> 
> I am new to this list and to troubleshooting python.  I hope someone can help me.  I am getting this tuple index out of range error while running a test call to my python code.  Not sure what this error really means and was hoping someone might shed some light on how to fix this.  Also was wondering why much of my .py files are not getting compiled to .pyc upon first run.. is this unusual or need I not worry?  
> 
> Running python 2.5.2 on fedora.

Nick,

This list is focussed on the development of Python, not on development with Python.  The python-list list is more appropriate for asking questions like yours.

Regards,

  Ronald

> 
> Thanks,
> Nick
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com


From christian at python.org  Tue Jul 23 17:36:34 2013
From: christian at python.org (Christian Heimes)
Date: Tue, 23 Jul 2013 17:36:34 +0200
Subject: [Python-Dev] cpython (3.3): return NULL here
In-Reply-To: <CAPZV6o_SFj39y42q=KpPmqY3oLwA7dFh1au50ZWLYEx-xw_hPg@mail.gmail.com>
References: <3bznlW6d69z7LkD@mail.python.org> <51EE5453.4010209@python.org>
	<CAPZV6o_SFj39y42q=KpPmqY3oLwA7dFh1au50ZWLYEx-xw_hPg@mail.gmail.com>
Message-ID: <51EEA302.7070008@python.org>

Am 23.07.2013 17:10, schrieb Benjamin Peterson:
>> PyErr_SetFromErrno() already and always returns NULL. Or do you prefer
>> to return NULL explicitly?
> 
> It might always return NULL, but the compiler sees (PyObject *)NULL
> when this function returns dl_funcptr.

Oh, you are right. I must have missed the compiler warning. How about we
turn type return and type assignment warnings into fatal errors?

Christian


From ronaldoussoren at mac.com  Tue Jul 23 17:46:26 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Tue, 23 Jul 2013 17:46:26 +0200
Subject: [Python-Dev] cpython (3.3): return NULL here
In-Reply-To: <51EEA302.7070008@python.org>
References: <3bznlW6d69z7LkD@mail.python.org> <51EE5453.4010209@python.org>
	<CAPZV6o_SFj39y42q=KpPmqY3oLwA7dFh1au50ZWLYEx-xw_hPg@mail.gmail.com>
	<51EEA302.7070008@python.org>
Message-ID: <9C38BD1B-24FF-4360-A729-6DC215238A6C@mac.com>


On 23 Jul, 2013, at 17:36, Christian Heimes <christian at python.org> wrote:

> Am 23.07.2013 17:10, schrieb Benjamin Peterson:
>>> PyErr_SetFromErrno() already and always returns NULL. Or do you prefer
>>> to return NULL explicitly?
>> 
>> It might always return NULL, but the compiler sees (PyObject *)NULL
>> when this function returns dl_funcptr.
> 
> Oh, you are right. I must have missed the compiler warning. How about we
> turn type return and type assignment warnings into fatal errors?

That's probably possible with a '-Werror=....' argument. But please consider
issue 18211 before unconditionally adding such a flag, as that issue mentions
new compiler flags also get used when compiling 3th-party extensions.

I guess there needs to be (yet) another CFLAGS_xxx variable in the Makefile that
gets added to $(CFLAGS) during the build of Python itself, but is ignored by
distutils and sysconfig.

Ronald

> 
> Christian
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com


From martin at v.loewis.de  Tue Jul 23 18:11:26 2013
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 23 Jul 2013 18:11:26 +0200
Subject: [Python-Dev] Adding Python scripts to PATHEXT on Windows
In-Reply-To: <CACac1F9WtTj69gBv2SKR3chxFdduhNtxNOmPqrt=mK5kHb_ZMA@mail.gmail.com>
References: <CACac1F9WtTj69gBv2SKR3chxFdduhNtxNOmPqrt=mK5kHb_ZMA@mail.gmail.com>
Message-ID: <51EEAB2E.9030804@v.loewis.de>

Am 15.07.13 10:26, schrieb Paul Moore:
> Does anyone have any objections to this? I could try to write a patch,
> but I know next to nothing about building MSIs, so if any of the
> installer experts could help that would be fantastic.

It's fine with me. I could write the patch, but will likely forget - so
please remind me with every alpha that misses this. Also, contributions
are welcome.

Regards,
Martin



From p.f.moore at gmail.com  Tue Jul 23 18:33:15 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 23 Jul 2013 17:33:15 +0100
Subject: [Python-Dev] Adding Python scripts to PATHEXT on Windows
In-Reply-To: <51EEAB2E.9030804@v.loewis.de>
References: <CACac1F9WtTj69gBv2SKR3chxFdduhNtxNOmPqrt=mK5kHb_ZMA@mail.gmail.com>
	<51EEAB2E.9030804@v.loewis.de>
Message-ID: <CACac1F9xLhfEPZuJ3-s8RaV-fi+UGaRMxp3SrzyupSkWWPG9OA@mail.gmail.com>

On 23 July 2013 17:11, "Martin v. L?wis" <martin at v.loewis.de> wrote:

> Am 15.07.13 10:26, schrieb Paul Moore:
> > Does anyone have any objections to this? I could try to write a patch,
> > but I know next to nothing about building MSIs, so if any of the
> > installer experts could help that would be fantastic.
>
> It's fine with me. I could write the patch, but will likely forget - so
> please remind me with every alpha that misses this. Also, contributions
> are welcome.
>

Thanks.

I think I have the basic idea of what's needed, so I'll write an initial
patch. If you can check it that'd be great.

Paul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130723/b496f758/attachment.html>

From christian at python.org  Tue Jul 23 21:52:47 2013
From: christian at python.org (Christian Heimes)
Date: Tue, 23 Jul 2013 21:52:47 +0200
Subject: [Python-Dev] cpython: Add modeling file for Coverity Scan.
In-Reply-To: <20130723082727.634ee6cb@fsol>
References: <3bzfGN573Hz7Lkf@mail.python.org> <20130723082727.634ee6cb@fsol>
Message-ID: <51EEDF0F.8020105@python.org>

Am 23.07.2013 08:27, schrieb Antoine Pitrou:
> On Tue, 23 Jul 2013 01:31:24 +0200 (CEST)
> christian.heimes <python-checkins at python.org> wrote:
>> +
>> +typedef int sdigit;
>> +typedef long Py_ssize_t;
> 
> Can't you write "typedef ssize_t Py_ssize_t" instead?

No, but it really doesn't matter. Coverity just needs a similar type for
modeling.

"modeling_file.c", line 23: error #20:
          identifier "ssize_t" is undefined
  typedef ssize_t Py_ssize_t;
          ^

1 error detected in the compilation of "modeling_file.c".
ERROR: cov-translate returned with code 2

From victor.stinner at gmail.com  Wed Jul 24 00:45:22 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 24 Jul 2013 00:45:22 +0200
Subject: [Python-Dev] Inherance of file descriptor and handles on Windows
	(PEP 446)
Message-ID: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>

Hi,

Guido van Rossum and others asked me details on how file descriptors
and handles are inherited on Windows, for the PEP 446.
http://www.python.org/dev/peps/pep-0446/

I hacked Python 3.4 to add a os.get_cloexec() function (extracted from
my implementation of the PEP 433), here are some results.

Python functions open(), os.open() and os.dup() create file
descriptors with the HANDLE_FLAG_INHERIT flag set (cloexec=False),
whereas os.pipe() creates 2 file descriptors with the
HANDLE_FLAG_INHERIT flag unset (cloexec=False, see also issue #4708).

Even if the HANDLE_FLAG_INHERIT flag is set, all handles are closed if
subprocess is used with close_fds=True (which is the default value of
the parameter), and all file descriptors are closed except 0, 1 and 2.

If close_fds=False, handles with the HANDLE_FLAG_INHERIT flag set are
inherited, but all file descriptors are still closed except 0, 1 and
2.

(I didn't check if file descriptors 0, 1 and 2 are inherited,
duplicated or new file descriptors.)

The PEP 466 allows you to control which handles are inherited to child
process when you use subprocess with close_fds=False. (The subprocess
parameter should be called "close_handles" on Windows to avoid
confusion.)

Said differently: the HANDLE_FLAG_INHERIT flag only has an effect on
*handles*, as indicated in its name. On Windows, file *descriptors*
are never inherited (are always closed) in child processes. I don't
think that it is possible to inherit file descriptors on Windows.

By the way, using pass_fds on Windows raises an assertion error
("pass_fds not supported on Windows").

Another example in Python:
---
import subprocess, sys

code = """
import os, sys
fd = int(sys.argv[1])
f = os.fdopen(fd, "rb")
print(f.read())
"""

f = open(__file__, "rb")
fd = f.fileno()
subprocess.call([sys.executable, "-c", code, str(fd)], close_fds=False)
---

On Unix, the child process will write the script into stdout. On
Windows, you just get an OSError(9, "Bad file descriptor") exception.

To fix this example on Windows, you have to:

* Retrieve the handle of the file using msvcrt.get_osfhandle() ;
* Pass the handle, instead of the file descriptor, to the child ;
* Create a file descriptor from the handle using
msvcrt.open_osfhandle() in the child.

The fix would be simpler if Python would provide the handle of a file
object (ex: in a method) and if open() supported opening a handle as
it does with file descriptors on UNIX.

Victor

From shibturn at gmail.com  Wed Jul 24 01:21:18 2013
From: shibturn at gmail.com (Richard Oudkerk)
Date: Wed, 24 Jul 2013 00:21:18 +0100
Subject: [Python-Dev] Inherance of file descriptor and handles on
	Windows (PEP 446)
In-Reply-To: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
References: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
Message-ID: <ksn359$q3a$1@ger.gmane.org>

On 23/07/2013 11:45pm, Victor Stinner wrote:
> Said differently: the HANDLE_FLAG_INHERIT flag only has an effect on
> *handles*, as indicated in its name. On Windows, file*descriptors*
> are never inherited (are always closed) in child processes. I don't
> think that it is possible to inherit file descriptors on Windows.

Actually, you can inherit fds if you use os.spawnv() instead of 
subprocess.Popen().

-- 
Richard


From victor.stinner at gmail.com  Wed Jul 24 01:40:18 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 24 Jul 2013 01:40:18 +0200
Subject: [Python-Dev] Inherance of file descriptor and handles on
	Windows (PEP 446)
In-Reply-To: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
References: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
Message-ID: <CAMpsgwZVdVhOWWPJf2mXxRi9PoOxg81Rg5kPvggAn3R++9FgZw@mail.gmail.com>

The multiprocessing module is an example of use case relying on
inherance of handles. It calls CreateProcess() with
bInheritHandles=TRUE to share a pipe between the manager (parent) and
the worker (child process).

Note: subprocess and multiprocess have their own function to set the
HANDLE_FLAG_INHERIT flag: they use DuplicateHandle(), whereas
SetHandleInformation() could be used (to reuse the existing handle
instead of creating a new handle).

2013/7/24 Victor Stinner <victor.stinner at gmail.com>:
> Python functions open(), os.open() and os.dup() create file
> descriptors with the HANDLE_FLAG_INHERIT flag set (cloexec=False),
> whereas os.pipe() creates 2 file descriptors with the
> HANDLE_FLAG_INHERIT flag unset (cloexec=False, see also issue #4708).
> (...)
> If close_fds=False, handles with the HANDLE_FLAG_INHERIT flag set are
> inherited, but all file descriptors are still closed except 0, 1 and
> 2.

Leaking handles in child processes is also an issue on Windows. Random examples.

http://bugs.python.org/issue17634
"Win32: shutil.copy leaks file handles to child processes"
"Win32's native CopyFile API call doesn't leak file handles to child processes."

http://ghc.haskell.org/trac/ghc/ticket/2650
"The case in which I originally ran into this was
System.Directory.copyFile intermittently reporting a "permission
denied" error for a temp file it was using. I think it was trying to
delete it, but failing because a child process somewhere was hanging
on to the Handle."
According to the issue, GHC calls CreateProcess with
bInheritHandles=TRUE (as Python did until Python 3.2).

http://support.microsoft.com/kb/315939
"This behavior can occur if two threads simultaneously create child
processes and redirect the STD handles through pipes. In this
scenario, there is a race condition during the creation of the pipes
and processes, in which it is possible for one child to inherit file
handles intended for the other child."
=> Python looks to be correct, it uses the (StartupInfo)
"STARTF_USESTDHANDLES" flag

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6428742
Java is still calling CreateProcess() with bInheritHandles=TRUE which
causes issues like "6347873: (so) Ports opened with
ServerSocketChannel blocks when using Runtime.exec". Interesting
comment: "Submitter has a point.  Very risky to fix." :-/


For the record, the default value of close_fds parameter was also set
to True on Windows by the following changeset:

changeset:   66889:59b43dc34158
user:        Gregory P. Smith <greg at mad-scientist.com>
date:        Tue Dec 14 13:43:30 2010 +0000
files:       Doc/library/subprocess.rst Lib/subprocess.py
Lib/test/test_subprocess.py Misc/NEWS Modules/_posixsubprocess.c
description:
Issue #6559: fix the subprocess.Popen pass_fds implementation. Add a unittest.
Issue #7213: Change the close_fds default on Windows to better match the new
default on POSIX.  True when possible (False if stdin/stdout/stderr are
supplied).

Update the documentation to reflect all of the above.

The changeset was written to fix http://bugs.python.org/issue7213 ;
another example of leak of handles.


> The PEP 466 allows you to control which handles are inherited to child
> process when you use subprocess with close_fds=False. (The subprocess
> parameter should be called "close_handles" on Windows to avoid
> confusion.)

Another advantage of the PEP 446 is that most of the time, the
HANDLE_FLAG_INHERIT flag value can be set atomatically at the creation
of the file descriptor (creation of the handle). It is a nice
enhancement to fight against race conditions with threads ;-) "Most of
the time": for example, socket are inherited by default,
WSA_FLAG_NO_HANDLE_INHERIT flag was only added to Windows Vista.

Victor

From p.f.moore at gmail.com  Wed Jul 24 08:29:44 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 24 Jul 2013 07:29:44 +0100
Subject: [Python-Dev] Inherance of file descriptor and handles on
 Windows (PEP 446)
In-Reply-To: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
References: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
Message-ID: <CACac1F8UXrjyBZ7r6aCUbUJT48OKwJJk-ij9tMmH1CwW73drAQ@mail.gmail.com>

On 23 July 2013 23:45, Victor Stinner <victor.stinner at gmail.com> wrote:

> Said differently: the HANDLE_FLAG_INHERIT flag only has an effect on
> *handles*, as indicated in its name. On Windows, file *descriptors*
> are never inherited (are always closed) in child processes. I don't
> think that it is possible to inherit file descriptors on Windows.
>

That is correct - handles are the OS-level concept, fds are implemented in
the CRT. So code that uses raw Windows APIs to create a new process won't
have any means to inherit fds.


> The fix would be simpler if Python would provide the handle of a file
> object (ex: in a method) and if open() supported opening a handle as
> it does with file descriptors on UNIX.
>

That would give a similar level of functionality to Unix. Whether it is
used sufficiently often to be worth it, is a separate question, of course...

Paul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130724/dae97907/attachment.html>

From greg at krypto.org  Wed Jul 24 08:43:59 2013
From: greg at krypto.org (Gregory P. Smith)
Date: Tue, 23 Jul 2013 23:43:59 -0700
Subject: [Python-Dev] [Python-checkins] cpython (3.3): return NULL here
In-Reply-To: <9C38BD1B-24FF-4360-A729-6DC215238A6C@mac.com>
References: <3bznlW6d69z7LkD@mail.python.org> <51EE5453.4010209@python.org>
	<CAPZV6o_SFj39y42q=KpPmqY3oLwA7dFh1au50ZWLYEx-xw_hPg@mail.gmail.com>
	<51EEA302.7070008@python.org>
	<9C38BD1B-24FF-4360-A729-6DC215238A6C@mac.com>
Message-ID: <CAGE7PN+ZcNLz66caT4tUEQuJt53nffJ2C+W9SBwRsc-nX78YmQ@mail.gmail.com>

On Tue, Jul 23, 2013 at 8:46 AM, Ronald Oussoren <ronaldoussoren at mac.com>wrote:

>
> On 23 Jul, 2013, at 17:36, Christian Heimes <christian at python.org> wrote:
>
> > Am 23.07.2013 17:10, schrieb Benjamin Peterson:
> >>> PyErr_SetFromErrno() already and always returns NULL. Or do you prefer
> >>> to return NULL explicitly?
> >>
> >> It might always return NULL, but the compiler sees (PyObject *)NULL
> >> when this function returns dl_funcptr.
> >
> > Oh, you are right. I must have missed the compiler warning. How about we
> > turn type return and type assignment warnings into fatal errors?
>
> That's probably possible with a '-Werror=....' argument. But please
> consider
> issue 18211 before unconditionally adding such a flag, as that issue
> mentions
> new compiler flags also get used when compiling 3th-party extensions.
>
> I guess there needs to be (yet) another CFLAGS_xxx variable in the
> Makefile that
> gets added to $(CFLAGS) during the build of Python itself, but is ignored
> by
> distutils and sysconfig.
>

It seems fair to turn those on in 3.4 and require that third party
extensions clean up their code when porting from 3.3 to 3.4.


> Ronald
>
> >
> > Christian
> >
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > http://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com
>
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> http://mail.python.org/mailman/listinfo/python-checkins
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130723/5168cb3d/attachment-0001.html>

From ronaldoussoren at mac.com  Wed Jul 24 09:01:30 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Wed, 24 Jul 2013 09:01:30 +0200
Subject: [Python-Dev] [Python-checkins] cpython (3.3): return NULL here
In-Reply-To: <CAGE7PN+ZcNLz66caT4tUEQuJt53nffJ2C+W9SBwRsc-nX78YmQ@mail.gmail.com>
References: <3bznlW6d69z7LkD@mail.python.org> <51EE5453.4010209@python.org>
	<CAPZV6o_SFj39y42q=KpPmqY3oLwA7dFh1au50ZWLYEx-xw_hPg@mail.gmail.com>
	<51EEA302.7070008@python.org>
	<9C38BD1B-24FF-4360-A729-6DC215238A6C@mac.com>
	<CAGE7PN+ZcNLz66caT4tUEQuJt53nffJ2C+W9SBwRsc-nX78YmQ@mail.gmail.com>
Message-ID: <568DFFEE-F257-4EA2-A749-DA8BCECCDDF0@mac.com>


On 24 Jul, 2013, at 8:43, Gregory P. Smith <greg at krypto.org> wrote:

> 
> On Tue, Jul 23, 2013 at 8:46 AM, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
> 
> On 23 Jul, 2013, at 17:36, Christian Heimes <christian at python.org> wrote:
> 
> > Am 23.07.2013 17:10, schrieb Benjamin Peterson:
> >>> PyErr_SetFromErrno() already and always returns NULL. Or do you prefer
> >>> to return NULL explicitly?
> >>
> >> It might always return NULL, but the compiler sees (PyObject *)NULL
> >> when this function returns dl_funcptr.
> >
> > Oh, you are right. I must have missed the compiler warning. How about we
> > turn type return and type assignment warnings into fatal errors?
> 
> That's probably possible with a '-Werror=....' argument. But please consider
> issue 18211 before unconditionally adding such a flag, as that issue mentions
> new compiler flags also get used when compiling 3th-party extensions.
> 
> I guess there needs to be (yet) another CFLAGS_xxx variable in the Makefile that
> gets added to $(CFLAGS) during the build of Python itself, but is ignored by
> distutils and sysconfig.
> 
> It seems fair to turn those on in 3.4 and require that third party extensions clean up their code when porting from 3.3 to 3.4.

In this case its "just" code cleanup, the issue I filed (see above) is for another -Werror flag
that causes compile errors with some valid C99 code that isn't valid C89. That's good for
CPython itself because its source code is explicitly C89, but is not good when building 3th-party
extensions.

A proper fix requires tweaking the configure script, Makefile and distutils and that's not really
a fun prospect ;-)

Ronald

> 
> 
> Ronald
> 
> >
> > Christian
> >
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > http://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com
> 
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> http://mail.python.org/mailman/listinfo/python-checkins
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com


From solipsis at pitrou.net  Wed Jul 24 10:17:59 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 24 Jul 2013 10:17:59 +0200
Subject: [Python-Dev] [Python-checkins] cpython (3.3): return NULL here
References: <3bznlW6d69z7LkD@mail.python.org> <51EE5453.4010209@python.org>
	<CAPZV6o_SFj39y42q=KpPmqY3oLwA7dFh1au50ZWLYEx-xw_hPg@mail.gmail.com>
	<51EEA302.7070008@python.org>
	<9C38BD1B-24FF-4360-A729-6DC215238A6C@mac.com>
	<CAGE7PN+ZcNLz66caT4tUEQuJt53nffJ2C+W9SBwRsc-nX78YmQ@mail.gmail.com>
	<568DFFEE-F257-4EA2-A749-DA8BCECCDDF0@mac.com>
Message-ID: <20130724101759.7f608f12@pitrou.net>

Le Wed, 24 Jul 2013 09:01:30 +0200,
Ronald Oussoren <ronaldoussoren at mac.com> a ?crit :
> 
> On 24 Jul, 2013, at 8:43, Gregory P. Smith <greg at krypto.org> wrote:
> 
> > 
> > On Tue, Jul 23, 2013 at 8:46 AM, Ronald Oussoren
> > <ronaldoussoren at mac.com> wrote:
> > 
> > On 23 Jul, 2013, at 17:36, Christian Heimes <christian at python.org>
> > wrote:
> > 
> > > Am 23.07.2013 17:10, schrieb Benjamin Peterson:
> > >>> PyErr_SetFromErrno() already and always returns NULL. Or do you
> > >>> prefer to return NULL explicitly?
> > >>
> > >> It might always return NULL, but the compiler sees (PyObject
> > >> *)NULL when this function returns dl_funcptr.
> > >
> > > Oh, you are right. I must have missed the compiler warning. How
> > > about we turn type return and type assignment warnings into fatal
> > > errors?
> > 
> > That's probably possible with a '-Werror=....' argument. But please
> > consider issue 18211 before unconditionally adding such a flag, as
> > that issue mentions new compiler flags also get used when compiling
> > 3th-party extensions.
> > 
> > I guess there needs to be (yet) another CFLAGS_xxx variable in the
> > Makefile that gets added to $(CFLAGS) during the build of Python
> > itself, but is ignored by distutils and sysconfig.
> > 
> > It seems fair to turn those on in 3.4 and require that third party
> > extensions clean up their code when porting from 3.3 to 3.4.
> 
> In this case its "just" code cleanup, the issue I filed (see above)
> is for another -Werror flag that causes compile errors with some
> valid C99 code that isn't valid C89. That's good for CPython itself
> because its source code is explicitly C89, but is not good when
> building 3th-party extensions.

Agreed. We shouldn't impose specific error flags on 3rd-party extension
writers.

Regards

Antoine.



From bkabrda at redhat.com  Wed Jul 24 11:12:59 2013
From: bkabrda at redhat.com (Bohuslav Kabrda)
Date: Wed, 24 Jul 2013 05:12:59 -0400 (EDT)
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
Message-ID: <1903981217.6011829.1374657179869.JavaMail.root@redhat.com>

Hi all,
in recent days, there has been a discussion on fedora-devel (see thread [1]) about moving to Python 3 as a default.
I'd really love to hear opinions on the matter from the upstream, mainly regarding these two points (that are not that clearly defined in my original proposal and have been formed during the discussion):

- Should we point /usr/bin/python to Python 3 when we make the move?
I know that pep 394 [2] deals with this and it says that /usr/bin/python may refer to Python 3 on some bleeding edge distributions - supposedly, this was added to the pep because of what Arch Linux did, not the other way round.
As the pep says, the recommendation of pointing /usr/bin/python to Python 2 may be changed after the Python 3 ecosystem is sufficiently mature. I'm wondering if there are any more specific criteria - list of big projects migrated/ported or something like that - or will this be judged by what I'd call "overall spirit" in Python community (I hope you know what I mean by this)?
In Fedora, we have two concerns that clash in this decision - being "First" (e.g. actively promote and use new technologies and also suggest them to our users) vs. not breaking user expectations. So we figured it'd be a good idea to ask upstream to get more opinions on this.

- What should user get after using "yum install python"?
There are basically few ways of coping with this:
1) Just keep doing what we do, eventually far in the future drop "python" package and never provide it again (= go on only with python3/python4/... while having "yum install python" do nothing).
2) Do what is in 1), but when "python" is dropped, use virtual provide (*) "python" for python3 package, so that "yum install python" installs python3.
3), 4) Rename python to python2 and {don't add, add} virtual provide "python" in the same way that is in 1), 2)
5) Rename python to python2 and python3 to python at one point. This makes sense to me from the traditional "one version in distro + possibly compat package shipping the old" approach in Linux, but some say that Python 2 and Python 3 are just different languages [3] and this should never be done.
All of the approaches have their pros and cons, but generally it is all about what user should get when he tries to install python - either nothing or python2 for now and python3 in future - and how we as a distro cope with that on the technical side (and when we should actually do the switch).
Just as a sidenote, IMO the package that gets installed as "python" (if any) should point to /usr/bin/python, which makes consider these two points very closely coupled.

Thank you all for your suggestions and opinions.
Slavek.

-- 
Regards,
Bohuslav "Slavek" Kabrda.

(*) You can think of a virtual provide as of a "packaging symlink" - adding "Provides: foo" to package makes "yum install foo" install this package, even if this package name is "bar".

[1] https://lists.fedoraproject.org/pipermail/devel/2013-July/186098.html
[2] http://www.python.org/dev/peps/pep-0394/
[3] https://lists.fedoraproject.org/pipermail/devel/2013-July/186186.html

From rosuav at gmail.com  Wed Jul 24 12:16:33 2013
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 24 Jul 2013 20:16:33 +1000
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
Message-ID: <CAPTjJmoDG29uZyKGK1W7pc=jfubmmdwAVtADyP7EYxMEvCVTUQ@mail.gmail.com>

On Wed, Jul 24, 2013 at 7:12 PM, Bohuslav Kabrda <bkabrda at redhat.com> wrote:
> - What should user get after using "yum install python"?
> There are basically few ways of coping with this:
> 1) Just keep doing what we do, eventually far in the future drop "python" package and never provide it again (= go on only with python3/python4/... while having "yum install python" do nothing).
> 2) Do what is in 1), but when "python" is dropped, use virtual provide (*) "python" for python3 package, so that "yum install python" installs python3.
> 3), 4) Rename python to python2 and {don't add, add} virtual provide "python" in the same way that is in 1), 2)
> 5) Rename python to python2 and python3 to python at one point. This makes sense to me from the traditional "one version in distro + possibly compat package shipping the old" approach in Linux, but some say that Python 2 and Python 3 are just different languages [3] and this should never be done.


If a user's (rather than dev's) perspective is of value, I'm a Debian
and Ubuntu user who wields Python as a multi-context weapon. Typing
"packagemanager install python" (for me that's apt-get, for you yum,
same diff) should get the one against which most other packages are
going to depend. Since your policy is to move people forward, that's
most likely going to be Python 3, unless you have a lot of packages
that simply haven't yet moved.

Your fifth option seems messy. I'd rather see python2 and python3 as
viable packages, and then use the virtual provide to symlink 'python'
to one of them, probably py3. So I'd vote for option 4. Python 2 and 3
aren't so different as to be called separate languages, there just
happen to be some backward-incompatible changes.

ChrisA

From dirkjan at ochtman.nl  Wed Jul 24 12:33:07 2013
From: dirkjan at ochtman.nl (Dirkjan Ochtman)
Date: Wed, 24 Jul 2013 12:33:07 +0200
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
Message-ID: <CAKmKYaAnpc0V2pa=5fc7AuY1w48PHNMU+Jz4MBJRCxrMT0rMBA@mail.gmail.com>

On Wed, Jul 24, 2013 at 11:12 AM, Bohuslav Kabrda <bkabrda at redhat.com> wrote:
> - What should user get after using "yum install python"?
> There are basically few ways of coping with this:
> 1) Just keep doing what we do, eventually far in the future drop "python" package and never provide it again (= go on only with python3/python4/... while having "yum install python" do nothing).
> 2) Do what is in 1), but when "python" is dropped, use virtual provide (*) "python" for python3 package, so that "yum install python" installs python3.
> 3), 4) Rename python to python2 and {don't add, add} virtual provide "python" in the same way that is in 1), 2)
> 5) Rename python to python2 and python3 to python at one point. This makes sense to me from the traditional "one version in distro + possibly compat package shipping the old" approach in Linux, but some say that Python 2 and Python 3 are just different languages [3] and this should never be done.
> All of the approaches have their pros and cons, but generally it is all about what user should get when he tries to install python - either nothing or python2 for now and python3 in future - and how we as a distro cope with that on the technical side (and when we should actually do the switch).
> Just as a sidenote, IMO the package that gets installed as "python" (if any) should point to /usr/bin/python, which makes consider these two points very closely coupled.

On Gentoo we get python2 and python3 executables and have user-level
tools to change what the 'python' symlink points to. I think we
default to only having Python 3 installed (and Python 3 is even made
the default Python), though it's currently always the case that Python
2 gets pulled in by some core dependencies.

Cheers,

Dirkjan

From p.f.moore at gmail.com  Wed Jul 24 12:36:25 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 24 Jul 2013 11:36:25 +0100
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
Message-ID: <CACac1F8XjTLVWm6QzcyN3exQGxvRDgDZeATjGzNdGREmBTR66Q@mail.gmail.com>

On 24 July 2013 10:12, Bohuslav Kabrda <bkabrda at redhat.com> wrote:

> - What should user get after using "yum install python"?
> There are basically few ways of coping with this:
> 1) Just keep doing what we do, eventually far in the future drop "python"
> package and never provide it again (= go on only with python3/python4/...
> while having "yum install python" do nothing).
> 2) Do what is in 1), but when "python" is dropped, use virtual provide (*)
> "python" for python3 package, so that "yum install python" installs python3.
> 3), 4) Rename python to python2 and {don't add, add} virtual provide
> "python" in the same way that is in 1), 2)
> 5) Rename python to python2 and python3 to python at one point. This makes
> sense to me from the traditional "one version in distro + possibly compat
> package shipping the old" approach in Linux, but some say that Python 2 and
> Python 3 are just different languages [3] and this should never be done.
>

I'm not a Unix user, but IMO any approach that has the simple package name
and command "python" disappear or become deprecated, makes me sad. I think
that "yum install python" should install Python, and "python" should run
Python. Frankly, the differences between Python 2 and Python 3 are *not*
big enough that users writing simple scripts and applications for personal
or local use can't cope with this. Not everyone writes major libraries, and
it's the casual users you should be looking at.

There may be a debate over *when* this usage should give the user Python 3
rather than Python 2 (I'm very much an early adopter by instinct, so I'd
say sooner rather than later, but others may differ) but that's secondary.

Paul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130724/0a8dbb53/attachment.html>

From fijall at gmail.com  Wed Jul 24 13:53:22 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 24 Jul 2013 13:53:22 +0200
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <CACac1F8XjTLVWm6QzcyN3exQGxvRDgDZeATjGzNdGREmBTR66Q@mail.gmail.com>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
	<CACac1F8XjTLVWm6QzcyN3exQGxvRDgDZeATjGzNdGREmBTR66Q@mail.gmail.com>
Message-ID: <CAK5idxSzxGuX88crOqMLwabbboecHrZu=hzDaZqMo0sonC8sQw@mail.gmail.com>

On Wed, Jul 24, 2013 at 12:36 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 24 July 2013 10:12, Bohuslav Kabrda <bkabrda at redhat.com> wrote:
>>
>> - What should user get after using "yum install python"?
>> There are basically few ways of coping with this:
>> 1) Just keep doing what we do, eventually far in the future drop "python"
>> package and never provide it again (= go on only with python3/python4/...
>> while having "yum install python" do nothing).
>> 2) Do what is in 1), but when "python" is dropped, use virtual provide (*)
>> "python" for python3 package, so that "yum install python" installs python3.
>> 3), 4) Rename python to python2 and {don't add, add} virtual provide
>> "python" in the same way that is in 1), 2)
>> 5) Rename python to python2 and python3 to python at one point. This makes
>> sense to me from the traditional "one version in distro + possibly compat
>> package shipping the old" approach in Linux, but some say that Python 2 and
>> Python 3 are just different languages [3] and this should never be done.
>
>
> I'm not a Unix user, but IMO any approach that has the simple package name
> and command "python" disappear or become deprecated, makes me sad. I think
> that "yum install python" should install Python, and "python" should run
> Python. Frankly, the differences between Python 2 and Python 3 are *not* big
> enough that users writing simple scripts and applications for personal or
> local use can't cope with this. Not everyone writes major libraries, and
> it's the casual users you should be looking at.

the problem is not that. The problem is that people *use* major libraries a lot.

From brett at python.org  Wed Jul 24 15:34:11 2013
From: brett at python.org (Brett Cannon)
Date: Wed, 24 Jul 2013 09:34:11 -0400
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
Message-ID: <CAP1=2W7y6w_f9D69czQCDv--580D4yvOD+6qenOL3T4E6b2ryg@mail.gmail.com>

On Wed, Jul 24, 2013 at 5:12 AM, Bohuslav Kabrda <bkabrda at redhat.com> wrote:

> Hi all,
> in recent days, there has been a discussion on fedora-devel (see thread
> [1]) about moving to Python 3 as a default.
> I'd really love to hear opinions on the matter from the upstream, mainly
> regarding these two points (that are not that clearly defined in my
> original proposal and have been formed during the discussion):
>
> - Should we point /usr/bin/python to Python 3 when we make the move?
> I know that pep 394 [2] deals with this and it says that /usr/bin/python
> may refer to Python 3 on some bleeding edge distributions - supposedly,
> this was added to the pep because of what Arch Linux did, not the other way
> round.
> As the pep says, the recommendation of pointing /usr/bin/python to Python
> 2 may be changed after the Python 3 ecosystem is sufficiently mature. I'm
> wondering if there are any more specific criteria - list of big projects
> migrated/ported or something like that - or will this be judged by what I'd
> call "overall spirit" in Python community (I hope you know what I mean by
> this)?
> In Fedora, we have two concerns that clash in this decision - being
> "First" (e.g. actively promote and use new technologies and also suggest
> them to our users) vs. not breaking user expectations. So we figured it'd
> be a good idea to ask upstream to get more opinions on this.
>
> - What should user get after using "yum install python"?
> There are basically few ways of coping with this:
> 1) Just keep doing what we do, eventually far in the future drop "python"
> package and never provide it again (= go on only with python3/python4/...
> while having "yum install python" do nothing).
> 2) Do what is in 1), but when "python" is dropped, use virtual provide (*)
> "python" for python3 package, so that "yum install python" installs python3.
> 3), 4) Rename python to python2 and {don't add, add} virtual provide
> "python" in the same way that is in 1), 2)
> 5) Rename python to python2 and python3 to python at one point. This makes
> sense to me from the traditional "one version in distro + possibly compat
> package shipping the old" approach in Linux, but some say that Python 2 and
> Python 3 are just different languages [3] and this should never be done.
> All of the approaches have their pros and cons, but generally it is all
> about what user should get when he tries to install python - either nothing
> or python2 for now and python3 in future - and how we as a distro cope with
> that on the technical side (and when we should actually do the switch).
> Just as a sidenote, IMO the package that gets installed as "python" (if
> any) should point to /usr/bin/python, which makes consider these two points
> very closely coupled.
>

A similar discussion broke out when Arch Linux switched python to point to
python3. This led to http://www.python.org/dev/peps/pep-0394/ which says
have python2/python3, and have python point at whatever makes the most
sense to you based on your users and version uptake (option 3/4). The key,
though, is adding python2 and getting your code to use that
binary  specifically so that shifting the default name is more of a
convenience than something which might break existing code not ready for
the switch.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130724/140e87e3/attachment.html>

From regebro at gmail.com  Wed Jul 24 15:54:29 2013
From: regebro at gmail.com (Lennart Regebro)
Date: Wed, 24 Jul 2013 15:54:29 +0200
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
Message-ID: <CAL0kPAW97HnYU1ubD2bD64=ZazgrSTGBMajRA0eGxH4DiWkrfg@mail.gmail.com>

On Wed, Jul 24, 2013 at 11:12 AM, Bohuslav Kabrda <bkabrda at redhat.com> wrote:
> - Should we point /usr/bin/python to Python 3 when we make the move?

No.

> - What should user get after using "yum install python"?

Will a base install include Python 3? If it does, I think yum install
python should mean python3, and hence already be installed.

//Lennart

From dwightdhutto at gmail.com  Wed Jul 24 16:09:11 2013
From: dwightdhutto at gmail.com (David Hutto)
Date: Wed, 24 Jul 2013 10:09:11 -0400
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <CAL0kPAW97HnYU1ubD2bD64=ZazgrSTGBMajRA0eGxH4DiWkrfg@mail.gmail.com>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
	<CAL0kPAW97HnYU1ubD2bD64=ZazgrSTGBMajRA0eGxH4DiWkrfg@mail.gmail.com>
Message-ID: <CA+vVgJW_HKU2AATJ5Wz_xKH+qgMm7A1O2PBDoHuK=8W7Dktx0g@mail.gmail.com>

Haven't been over to yum land in a while, but usually they stabilize the
distro in synergy to the python version. Ubuntu came with 2.7, and I just
use sudo-apt get install, for 3.2(or a gui package manager), and then can
use 3.2 with using the command python3.2 in command line, or you can
compile from source with a newer python not available with your distro's
package manager/repository.

It's really up to your distro's base linux, and python version decided on
top of a specific distro they define as compatible with the distros system.



On Wed, Jul 24, 2013 at 9:54 AM, Lennart Regebro <regebro at gmail.com> wrote:

> On Wed, Jul 24, 2013 at 11:12 AM, Bohuslav Kabrda <bkabrda at redhat.com>
> wrote:
> > - Should we point /usr/bin/python to Python 3 when we make the move?
>
> No.
>
> > - What should user get after using "yum install python"?
>
> Will a base install include Python 3? If it does, I think yum install
> python should mean python3, and hence already be installed.
>
> //Lennart
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/dwightdhutto%40gmail.com
>



-- 
Best Regards,
David Hutto
*CEO:* *http://www.hitwebdevelopment.com*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130724/7ee5218b/attachment.html>

From martin at v.loewis.de  Wed Jul 24 17:10:27 2013
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 24 Jul 2013 17:10:27 +0200
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
References: <1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
Message-ID: <51EFEE63.5080702@v.loewis.de>

Am 24.07.13 11:12, schrieb Bohuslav Kabrda:
> - Should we point /usr/bin/python to Python 3 when we make the move?

This should depend on the answer to this question:
- for how long have you been providing /usr/bin/python2 binaries?

Users "should" have been explicit in declaring scripts as
/usr/bin/python2 for quite some time now, except that this would break
on distros which don't provide a python2 symlink.

Ideally, you "should" have warned users to be explicit with python2 if
the script wouldn't work on python3. But I'd wave this requirement, as
there is already the upstream PEP.

So (IMO) if it the last three Fedora releases had been providing python2
binaries, it should be allowed to switch python to be python3. The exact
number can be debated; it should depend on what releases are still in
active use so that a script would run on multiple releases.

Then you can tell users that possible breakage is easy to fix: just
install python2, and change the shebang line.

Regards,
Martin


From a.badger at gmail.com  Wed Jul 24 17:26:44 2013
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Wed, 24 Jul 2013 08:26:44 -0700
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <CAP1=2W7y6w_f9D69czQCDv--580D4yvOD+6qenOL3T4E6b2ryg@mail.gmail.com>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
	<CAP1=2W7y6w_f9D69czQCDv--580D4yvOD+6qenOL3T4E6b2ryg@mail.gmail.com>
Message-ID: <20130724152644.GI11402@unaka.lan>

Note: I'm the opposite number to bkabrda in the discussion on the Fedora
Lists about how quickly we should be breaking end-user expectations of what
"python" means.

On Wed, Jul 24, 2013 at 09:34:11AM -0400, Brett Cannon wrote:
> 
> 
> 
> On Wed, Jul 24, 2013 at 5:12 AM, Bohuslav Kabrda <bkabrda at redhat.com> wrote:
> 
>     Hi all,
>     in recent days, there has been a discussion on fedora-devel (see thread
>     [1]) about moving to Python 3 as a default.
>     I'd really love to hear opinions on the matter from the upstream, mainly
>     regarding these two points (that are not that clearly defined in my
>     original proposal and have been formed during the discussion):
> 
Note that the proposal is for Fedora 22.  So the timeframe for making the
switch in development is approximately 8 months from now.  Timeframe for
that release to be public is October 2014.

>     - Should we point /usr/bin/python to Python 3 when we make the move?
>     I know that pep 394 [2] deals with this and it says that /usr/bin/python
>     may refer to Python 3 on some bleeding edge distributions - supposedly,
>     this was added to the pep because of what Arch Linux did, not the other way
>     round.
>     As the pep says, the recommendation of pointing /usr/bin/python to Python 2
>     may be changed after the Python 3 ecosystem is sufficiently mature. I'm
>     wondering if there are any more specific criteria - list of big projects
>     migrated/ported or something like that - or will this be judged by what I'd
>     call "overall spirit" in Python community (I hope you know what I mean by
>     this)?
>     In Fedora, we have two concerns that clash in this decision - being "First"
>     (e.g. actively promote and use new technologies and also suggest them to
>     our users) vs. not breaking user expectations. So we figured it'd be a good
>     idea to ask upstream to get more opinions on this.
> 
>     - What should user get after using "yum install python"?
>     There are basically few ways of coping with this:
>     1) Just keep doing what we do, eventually far in the future drop "python"
>     package and never provide it again (= go on only with python3/python4/...
>     while having "yum install python" do nothing).
>     2) Do what is in 1), but when "python" is dropped, use virtual provide (*)
>     "python" for python3 package, so that "yum install python" installs
>     python3.
>     3), 4) Rename python to python2 and {don't add, add} virtual provide
>     "python" in the same way that is in 1), 2)
>
4) Is my preference: python package becomes python2; Virtual Provide: python
means you'd get the python package is what I'd promote for now.  Users still
expect python2 when they talk about "python".  At some point in the future,
people will come to pycon and talks will apply to python3 unless otherwise
specified.  People writing new blog posts will say "python" and the code
they use as samples won't run on the python2 interpreter.  Expecting for
that to be the case in 12 months seems premature.

>     5) Rename python to python2 and python3 to python at one point. This makes
>     sense to me from the traditional "one version in distro + possibly compat
>     package shipping the old" approach in Linux, but some say that Python 2 and
>     Python 3 are just different languages [3] and this should never be done.
>     All of the approaches have their pros and cons, but generally it is all
>     about what user should get when he tries to install python - either nothing
>     or python2 for now and python3 in future - and how we as a distro cope with
>     that on the technical side (and when we should actually do the switch).
>     Just as a sidenote, IMO the package that gets installed as "python" (if
>     any) should point to /usr/bin/python, which makes consider these two points
>     very closely coupled.
> 
> 
> A similar discussion broke out when Arch Linux switched python to point to
> python3. This led to?http://www.python.org/dev/peps/pep-0394/?which says have
> python2/python3, and have python point at whatever makes the most sense to you
> based on your users and version uptake (option 3/4).

<nod>  I think bkabrda is looking for some clarification on PEP-394.  My
reading and participation in the previous discussions lead me to believe
that while PEP-394 wants to be diplomatic, the message it wants to get
across is:

1) warn distributions that what Arch was doing was premature.
2) provide a means to get them to switch at roughly the same time (when the
   recommendation in the PEP is flipped to suggest linking /usr/bin/python
   to /usr/bin/python3)

This is especially my reading from the Recommendations section of the PEP.
Unfortunately, we're getting stuck in the Abstract section which has this
bullet point:

* python should refer to the same target as python2 but may refer to python3
on some bleeding edge distributions

Knowing the history, I read this in two parts:
* Recommendation to distributions: "python should refer to the same target
  as python2".
* Statement of fact: "but may refer to python3 on some bleeding edge
  *ahem*Arch*ahem* distributions"

However, other people are reading this as one recommendation for
distributions:

If you are a conservative, slow moving distro (like RHEL or Debian Stable)
then you should point to python2.  If you are a fast moving distro like
Arch or Fedora or Ubuntu then you should point to python3.

So -- is there some opinion on which of these is correct?

> The key, though, is adding
> python2 and getting your code to use that binary??specifically so that shifting
> the default name is more of a convenience than something which might break
> existing code not ready for the switch.

<nod>  yeah, this is something that I think should be in the distribution's
one year plan regardless of whether the binary is switched now or not.  Note
that I (and from Nick's last email on the fedora thread, I think he would
agree) consider this a prerequisite but not the primary gating factor for
shifting /usr/bin/python to point to /usr/bin/python3.  In addition to the
applications managed by the distribution there's also the mass of local
scripts written for each site that will break if the interpreter is
switched.

I proposed above some of the things that would indicate user expectation
has switched and make it appropriate to break scripts that haven't ported
(because those users and admins would look at the world around them and
see that they were overdue on changing their expectation that python meant
python2 instead of python3.) Barry Warsaw, on IRC, also proposed that
letting the upstream python core team make the decision by switching at the
distribution level when the core team switched the recommendation in PEP-394
made sense and I can agree with that as well.

-Toshio
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130724/d4e732e0/attachment.pgp>

From ncoghlan at gmail.com  Wed Jul 24 17:41:33 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 25 Jul 2013 01:41:33 +1000
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <20130724152644.GI11402@unaka.lan>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
	<CAP1=2W7y6w_f9D69czQCDv--580D4yvOD+6qenOL3T4E6b2ryg@mail.gmail.com>
	<20130724152644.GI11402@unaka.lan>
Message-ID: <CADiSq7feUW3-WbNTRLz5o+c2s1xLwrMn3i8bJy2WA3own7YxdQ@mail.gmail.com>

On 25 July 2013 01:26, Toshio Kuratomi <a.badger at gmail.com> wrote:
> On Wed, Jul 24, 2013 at 09:34:11AM -0400, Brett Cannon wrote:
>> A similar discussion broke out when Arch Linux switched python to point to
>> python3. This led to http://www.python.org/dev/peps/pep-0394/ which says have
>> python2/python3, and have python point at whatever makes the most sense to you
>> based on your users and version uptake (option 3/4).
>
> <nod>  I think bkabrda is looking for some clarification on PEP-394.  My
> reading and participation in the previous discussions lead me to believe
> that while PEP-394 wants to be diplomatic, the message it wants to get
> across is:
>
> 1) warn distributions that what Arch was doing was premature.
> 2) provide a means to get them to switch at roughly the same time (when the
>    recommendation in the PEP is flipped to suggest linking /usr/bin/python
>    to /usr/bin/python3)
>
> This is especially my reading from the Recommendations section of the PEP.
> Unfortunately, we're getting stuck in the Abstract section which has this
> bullet point:
>
> * python should refer to the same target as python2 but may refer to python3
> on some bleeding edge distributions
>
> Knowing the history, I read this in two parts:
> * Recommendation to distributions: "python should refer to the same target
>   as python2".
> * Statement of fact: "but may refer to python3 on some bleeding edge
>   *ahem*Arch*ahem* distributions"
>
> However, other people are reading this as one recommendation for
> distributions:
>
> If you are a conservative, slow moving distro (like RHEL or Debian Stable)
> then you should point to python2.  If you are a fast moving distro like
> Arch or Fedora or Ubuntu then you should point to python3.
>
> So -- is there some opinion on which of these is correct?

It's the first one, but I can definitely see how it could be read as
the second one.

How's this for an updated wording in the abstract:

  * for the time being, all distributions should ensure that python
refers to the same target as python2
  * however, users should be aware that python refers to python3 on at
least Arch Linux (that change is
    what prompted the creation of this PEP), so "python" should be
used in the shebang line only for
    scripts that are source compatible with both Python 2 and 3

That's what the previous bullet point was *trying* to say, but it
wasn't very clear, since the first half of the sentence was aimed at
distributions, while the second was aimed at end users.

Cheers,
Nick.



-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Wed Jul 24 17:45:28 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 25 Jul 2013 01:45:28 +1000
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <CADiSq7feUW3-WbNTRLz5o+c2s1xLwrMn3i8bJy2WA3own7YxdQ@mail.gmail.com>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
	<CAP1=2W7y6w_f9D69czQCDv--580D4yvOD+6qenOL3T4E6b2ryg@mail.gmail.com>
	<20130724152644.GI11402@unaka.lan>
	<CADiSq7feUW3-WbNTRLz5o+c2s1xLwrMn3i8bJy2WA3own7YxdQ@mail.gmail.com>
Message-ID: <CADiSq7cCSgRxQxtSTfEAt=oyft6B1ZPd749bozhUXQz=Sz-xYw@mail.gmail.com>

On 25 July 2013 01:41, Nick Coghlan <ncoghlan at gmail.com> wrote:
> How's this for an updated wording in the abstract:
>
>   * for the time being, all distributions should ensure that python
> refers to the same target as python2
>   * however, users should be aware that python refers to python3 on at
> least Arch Linux (that change is
>     what prompted the creation of this PEP), so "python" should be
> used in the shebang line only for
>     scripts that are source compatible with both Python 2 and 3
>
> That's what the previous bullet point was *trying* to say, but it
> wasn't very clear, since the first half of the sentence was aimed at
> distributions, while the second was aimed at end users.

I'll also note that this is deliberately consistent with the behaviour
of the "py" launcher on Windows (as defined in PEP 397): if you invoke
that from the command with both Python 2.x and Python 3.x installed,
it will default to using the most recently released version of Python
2, *unless* you pass a "-3" on the command line, or the shebang line
indicates the script is a Python 3 script.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From regebro at gmail.com  Wed Jul 24 17:56:08 2013
From: regebro at gmail.com (Lennart Regebro)
Date: Wed, 24 Jul 2013 17:56:08 +0200
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <CAL0kPAW97HnYU1ubD2bD64=ZazgrSTGBMajRA0eGxH4DiWkrfg@mail.gmail.com>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
	<CAL0kPAW97HnYU1ubD2bD64=ZazgrSTGBMajRA0eGxH4DiWkrfg@mail.gmail.com>
Message-ID: <CAL0kPAU9-RR6OZiYerd5nJ7=70zDA5sQ1xt7tPDpG_2mr+1BqA@mail.gmail.com>

On Wed, Jul 24, 2013 at 3:54 PM, Lennart Regebro <regebro at gmail.com> wrote:
> On Wed, Jul 24, 2013 at 11:12 AM, Bohuslav Kabrda <bkabrda at redhat.com> wrote:
>> - Should we point /usr/bin/python to Python 3 when we make the move?
>
> No.

To be more explicit. I think it's perfectly fine to not provide a
/usr/bin/python at all, but I think pointing it to Python 3 will
provide unhelpful error messages. Also having a /usr/bin/python that
does nothing but say "You should use either /usr/bin/python2 or
/usr/bin/python3" would work.

But pointing it to Python 3 will make for unhelpful and confusing
error messages for anyone having Python scripts using /usr/bin/python,
which is the majority of people having python scripts at all.

//Lennart

From barry at python.org  Wed Jul 24 18:08:20 2013
From: barry at python.org (Barry Warsaw)
Date: Wed, 24 Jul 2013 12:08:20 -0400
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
Message-ID: <20130724120820.342b6075@anarchist>

On Jul 24, 2013, at 05:12 AM, Bohuslav Kabrda wrote:

>in recent days, there has been a discussion on fedora-devel (see thread [1])
>about moving to Python 3 as a default.

I've been lurking via Gmane. :)

>- Should we point /usr/bin/python to Python 3 when we make the move?

Over in Debian (inherited by Ubuntu), we have no plans to do this.  Currently,
/usr/bin/python points to pythonX.Y which for us is 2.7.  Our /usr/bin/python2
symlink also points to python2.7, and our /usr/bin/python3 points to python3.3.

I'm very much in favor of promoting Python 3 adoption through the entire
ecosystem.  We've been working in Ubuntu to banish Python 2 off the images,
eventually only in universe by the next LTS (14.04 - in April 2014).  For
practical reasons, I don't think we'll quite get there, but it's good to have
goals, and I've been really impressed with the progress made by distro
developers and upstreams.

"Porting is one honking great idea -- let's do more of those!"

We have a mandate that all new work (certainly where Canonical is the upstream
for Ubuntu packages) should be Python 3 unless there are specific dependencies
for which porting is impractical and there are no alternatives.  We want no
Python 2 on the Ubuntu Touch images.  Some of the new work will retain Python
2 compatibility for now, again for practical reasons.

Code we control is the easy part.  There is just so much out there that isn't
even in the distro, stuff that no distro maintainer will ever even know about
(unless we break stuff ;), and all of that is expecting /usr/bin/python to be
Python 2.  If they want Python 3, they are explicitly shebanging
/usr/bin/python3.  I think Nick in one of the Fedora threads correctly
identified the vast number of one-off scripts out there.

At this time, I think keeping /usr/bin/python pointing to Python 2 (2.7
preferably) strikes the right balance between what we in Python land would
like to see happen, with the least amount of pain, and friendly encouragement
to our distro users.

I hope someday we can remember Python 2 as fondly as we remember 1.5.2
(<cough>1.6.1</cough>).  I think there's a ton of things we can do to get us
there, but forcing our users through unnecessary pain too early won't help the
cause.  The last year or so has really shown a very positive shift toward
Python 3, in code, mindshare, and enthusiasm.

>I know that pep 394 [2] deals with this and it says that /usr/bin/python may
>refer to Python 3 on some bleeding edge distributions - supposedly, this was
>added to the pep because of what Arch Linux did, not the other way round.

Correct.  I think we should consider rewording the text to make this more
clear.

>As the pep says, the recommendation of pointing /usr/bin/python to Python 2
>may be changed after the Python 3 ecosystem is sufficiently mature. I'm
>wondering if there are any more specific criteria - list of big projects
>migrated/ported or something like that - or will this be judged by what I'd
>call "overall spirit" in Python community (I hope you know what I mean by
>this)?

In the discussions I've had at Pycon and various other forum, it seems like
Python 2.7's scheduled security-only transition in May 2015 would be an
appropriate time to make the switch.  If there's broad agreement to do this,
we have plenty of time to prepare our users, and all of our software stacks.

>- What should user get after using "yum install python"?

I can only say that for Debian and Ubuntu, `apt-get install python` will get
you the default Python 2 version (e.g. 2.7).  `apt-get install python3` will
get you the default Python 3 version (e.g. 3.3).  I don't think changing this
really provides much user benefit, and I've not seen any complaints/bugs
indicating user confusion or frustration.  I'd advocate for being user driven
here, i.e. IIABDFI[1].

>All of the approaches have their pros and cons, but generally it is all about
>what user should get when he tries to install python - either nothing or
>python2 for now and python3 in future - and how we as a distro cope with that
>on the technical side (and when we should actually do the switch).

I'm glad Arch is out there hacking at the vines.  For whatever part I can play
and for whatever my opinion matters, I would like to keep the Fedora/RH and
Debian/Ubuntu ecosystem in sync.  PEP 394 is the instrument we should rally
around.

Cheers,
-Barry

[1] The secret 20th Zen of Python?  If It Ain't Broke Don't Fix It.

From lgautier at gmail.com  Wed Jul 24 18:21:54 2013
From: lgautier at gmail.com (Laurent Gautier)
Date: Wed, 24 Jul 2013 18:21:54 +0200
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <CAL0kPAU9-RR6OZiYerd5nJ7=70zDA5sQ1xt7tPDpG_2mr+1BqA@mail.gmail.com>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
	<CAL0kPAW97HnYU1ubD2bD64=ZazgrSTGBMajRA0eGxH4DiWkrfg@mail.gmail.com>
	<CAL0kPAU9-RR6OZiYerd5nJ7=70zDA5sQ1xt7tPDpG_2mr+1BqA@mail.gmail.com>
Message-ID: <51EFFF22.1070603@gmail.com>

On 07/24/2013 05:56 PM, Lennart Regebro wrote:
> On Wed, Jul 24, 2013 at 3:54 PM, Lennart Regebro <regebro at gmail.com> wrote:
>> On Wed, Jul 24, 2013 at 11:12 AM, Bohuslav Kabrda <bkabrda at redhat.com> wrote:
>>> - Should we point /usr/bin/python to Python 3 when we make the move?
>> No.
> To be more explicit. I think it's perfectly fine to not provide a
> /usr/bin/python at all, but I think pointing it to Python 3 will
> provide unhelpful error messages. Also having a /usr/bin/python that
> does nothing but say "You should use either /usr/bin/python2 or
> /usr/bin/python3" would work.
>
> But pointing it to Python 3 will make for unhelpful and confusing
> error messages for anyone having Python scripts using /usr/bin/python,
> which is the majority of people having python scripts at all.

I guess that the underlying idea is to make the majority of python 
scripts working with Python 3.

What about the following idea ?

- errors that are typical of "Python 2 script running with Python 
3"-specific are probably limited (e.g., use of unicode, use of xrange, 
etc...)

- /usr/bin/python could be a wrapper that tries to run it with Python 3 
and display an informative message if the errors above are occurring
(e.g., "Python 3 is the current release, and this script might be 
designed for the older Python 2 - please explicitly use python2 if you 
need it"

- /usr/bin/python could issue a warning in any case about the need to be 
explicit about python2/python3

>
> //Lennart
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/lgautier%40gmail.com


From martin at v.loewis.de  Wed Jul 24 18:05:26 2013
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 24 Jul 2013 18:05:26 +0200
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <CAL0kPAU9-RR6OZiYerd5nJ7=70zDA5sQ1xt7tPDpG_2mr+1BqA@mail.gmail.com>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
	<CAL0kPAW97HnYU1ubD2bD64=ZazgrSTGBMajRA0eGxH4DiWkrfg@mail.gmail.com>
	<CAL0kPAU9-RR6OZiYerd5nJ7=70zDA5sQ1xt7tPDpG_2mr+1BqA@mail.gmail.com>
Message-ID: <51EFFB46.2040300@v.loewis.de>

Am 24.07.13 17:56, schrieb Lennart Regebro:> On Wed, Jul 24, 2013 at
3:54 PM, Lennart Regebro <regebro at gmail.com> wrote:
>> On Wed, Jul 24, 2013 at 11:12 AM, Bohuslav Kabrda
<bkabrda at redhat.com> wrote:
>>> - Should we point /usr/bin/python to Python 3 when we make the move?
>>
>> No.
>
> To be more explicit. I think it's perfectly fine to not provide a
> /usr/bin/python at all

I disagree. For interactive use, it's good if there is something called
"python" in the path.

Regards,
Martin


From rosuav at gmail.com  Wed Jul 24 18:30:51 2013
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 25 Jul 2013 02:30:51 +1000
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <51EFFF22.1070603@gmail.com>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
	<CAL0kPAW97HnYU1ubD2bD64=ZazgrSTGBMajRA0eGxH4DiWkrfg@mail.gmail.com>
	<CAL0kPAU9-RR6OZiYerd5nJ7=70zDA5sQ1xt7tPDpG_2mr+1BqA@mail.gmail.com>
	<51EFFF22.1070603@gmail.com>
Message-ID: <CAPTjJmosvJf4mwiqny=22No1e3FhV2UT52TdjCUogTc9LvrE+A@mail.gmail.com>

On Thu, Jul 25, 2013 at 2:21 AM, Laurent Gautier <lgautier at gmail.com> wrote:
> - errors that are typical of "Python 2 script running with Python
> 3"-specific are probably limited (e.g., use of unicode, use of xrange,
> etc...)
>

The most common, in interactive scripts at least, is likely to be:

>>> print "Hello, world!"
SyntaxError: invalid syntax

How helpful it's possible to make that one, I don't know. Is it safe
to presume that it's more likely a syntax error will come from an
interpreter version mismatch than a code bug?

ChrisA

From brian at python.org  Wed Jul 24 18:34:58 2013
From: brian at python.org (Brian Curtin)
Date: Wed, 24 Jul 2013 11:34:58 -0500
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <CAL0kPAU9-RR6OZiYerd5nJ7=70zDA5sQ1xt7tPDpG_2mr+1BqA@mail.gmail.com>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
	<CAL0kPAW97HnYU1ubD2bD64=ZazgrSTGBMajRA0eGxH4DiWkrfg@mail.gmail.com>
	<CAL0kPAU9-RR6OZiYerd5nJ7=70zDA5sQ1xt7tPDpG_2mr+1BqA@mail.gmail.com>
Message-ID: <CAD+XWwpJSM3X3KfmaC5rfrtV+pV=yAsn1fsrZAa_FhKAvyDQDg@mail.gmail.com>

On Wed, Jul 24, 2013 at 10:56 AM, Lennart Regebro <regebro at gmail.com> wrote:
> On Wed, Jul 24, 2013 at 3:54 PM, Lennart Regebro <regebro at gmail.com> wrote:
>> On Wed, Jul 24, 2013 at 11:12 AM, Bohuslav Kabrda <bkabrda at redhat.com> wrote:
>>> - Should we point /usr/bin/python to Python 3 when we make the move?
>>
>> No.
>
> To be more explicit. I think it's perfectly fine to not provide a
> /usr/bin/python at all, but I think pointing it to Python 3 will
> provide unhelpful error messages.

-1. For one example, this is going to add non-productive and
potentially confusing steps before any educator can use Python to
teach anything.

From barry at python.org  Wed Jul 24 18:40:15 2013
From: barry at python.org (Barry Warsaw)
Date: Wed, 24 Jul 2013 12:40:15 -0400
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <20130724152644.GI11402@unaka.lan>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
	<CAP1=2W7y6w_f9D69czQCDv--580D4yvOD+6qenOL3T4E6b2ryg@mail.gmail.com>
	<20130724152644.GI11402@unaka.lan>
Message-ID: <20130724124015.05d79275@anarchist>

On Jul 24, 2013, at 08:26 AM, Toshio Kuratomi wrote:

><nod>  I think bkabrda is looking for some clarification on PEP-394.  My
>reading and participation in the previous discussions lead me to believe
>that while PEP-394 wants to be diplomatic, the message it wants to get
>across is:
>
>1) warn distributions that what Arch was doing was premature.
>2) provide a means to get them to switch at roughly the same time (when the
>   recommendation in the PEP is flipped to suggest linking /usr/bin/python
>   to /usr/bin/python3)

+1

>This is especially my reading from the Recommendations section of the PEP.
>Unfortunately, we're getting stuck in the Abstract section which has this
>bullet point:
>
>* python should refer to the same target as python2 but may refer to python3
>on some bleeding edge distributions

Yes, that wording is confusing and should be fixed.

>Knowing the history, I read this in two parts:
>* Recommendation to distributions: "python should refer to the same target
>  as python2".
>* Statement of fact: "but may refer to python3 on some bleeding edge
>  *ahem*Arch*ahem* distributions"
>
>However, other people are reading this as one recommendation for
>distributions:
>
>If you are a conservative, slow moving distro (like RHEL or Debian Stable)
>then you should point to python2.  If you are a fast moving distro like
>Arch or Fedora or Ubuntu then you should point to python3.
>
>So -- is there some opinion on which of these is correct?

You know how I'm going to answer. :)

I like the way you've split that out into recommendations and
statements-of-fact.

>> The key, though, is adding python2 and getting your code to use that
>> binary??specifically so that shifting the default name is more of a
>> convenience than something which might break existing code not ready for
>> the switch.
>
><nod>  yeah, this is something that I think should be in the distribution's
>one year plan regardless of whether the binary is switched now or not.

We need to do that in Debian.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130724/2f70d238/attachment-0001.pgp>

From barry at python.org  Wed Jul 24 18:42:09 2013
From: barry at python.org (Barry Warsaw)
Date: Wed, 24 Jul 2013 12:42:09 -0400
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <CADiSq7feUW3-WbNTRLz5o+c2s1xLwrMn3i8bJy2WA3own7YxdQ@mail.gmail.com>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
	<CAP1=2W7y6w_f9D69czQCDv--580D4yvOD+6qenOL3T4E6b2ryg@mail.gmail.com>
	<20130724152644.GI11402@unaka.lan>
	<CADiSq7feUW3-WbNTRLz5o+c2s1xLwrMn3i8bJy2WA3own7YxdQ@mail.gmail.com>
Message-ID: <20130724124209.61f8a8ed@anarchist>

On Jul 25, 2013, at 01:41 AM, Nick Coghlan wrote:

>How's this for an updated wording in the abstract:
>
>  * for the time being, all distributions should ensure that python
>refers to the same target as python2
>  * however, users should be aware that python refers to python3 on at
>least Arch Linux (that change is
>    what prompted the creation of this PEP), so "python" should be
>used in the shebang line only for
>    scripts that are source compatible with both Python 2 and 3

+1

-Barry

From tjreedy at udel.edu  Wed Jul 24 19:04:43 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 24 Jul 2013 13:04:43 -0400
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
Message-ID: <ksp1f7$jtl$1@ger.gmane.org>

On 7/24/2013 5:12 AM, Bohuslav Kabrda wrote:
> Hi all, in recent days, there has been a discussion on fedora-devel
> (see thread [1]) about moving to Python 3 as a default.

Default-shift is a known natural language phenomenon.
https://en.wikipedia.org/wiki/Retronym
It is inevitably messy in the middle. Even worse for us, "Python' is 
both a natural and artificial language term and the default meaning will 
not necessarily be synchronized across the two domains.

Not being a current *nix user, I have no opinion on the specific 
alternatives. However, I am concerned about newcomers being mis-served 
by being mislead into thinking that 'Python' is python 2 rather python 
3. We see this in some newbie posts on Python list. (This is especially 
a concern when default Python is still, say, 2.5.)  I have no idea, 
though, how to avoid breaking scripts while directing newcomers in the 
proper direction.

-- 
Terry Jan Reedy


From gjcarneiro at gmail.com  Wed Jul 24 19:16:00 2013
From: gjcarneiro at gmail.com (Gustavo Carneiro)
Date: Wed, 24 Jul 2013 18:16:00 +0100
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <ksp1f7$jtl$1@ger.gmane.org>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
	<ksp1f7$jtl$1@ger.gmane.org>
Message-ID: <CAO-CpE+audZ7iBhJV2pZ4J-QeVYrYFM_TJiREyWt4HOxNbkRGg@mail.gmail.com>

On Wed, Jul 24, 2013 at 6:04 PM, Terry Reedy <tjreedy at udel.edu> wrote:

> On 7/24/2013 5:12 AM, Bohuslav Kabrda wrote:
>
>> Hi all, in recent days, there has been a discussion on fedora-devel
>> (see thread [1]) about moving to Python 3 as a default.
>>
>
> Default-shift is a known natural language phenomenon.
> https://en.wikipedia.org/wiki/**Retronym<https://en.wikipedia.org/wiki/Retronym>
> It is inevitably messy in the middle. Even worse for us, "Python' is both
> a natural and artificial language term and the default meaning will not
> necessarily be synchronized across the two domains.
>
> Not being a current *nix user, I have no opinion on the specific
> alternatives. However, I am concerned about newcomers being mis-served by
> being mislead into thinking that 'Python' is python 2 rather python 3. We
> see this in some newbie posts on Python list. (This is especially a concern
> when default Python is still, say, 2.5.)  I have no idea, though, how to
> avoid breaking scripts while directing newcomers in the proper direction.


+1

Maybe we could make interactive python2 scripts print a big warning
advising users to use python 3 instead of 2, if possible?  I mean, if
stdout is a TTY.  This way, even if 'python' is python2, newbies will see
the warning.  I think every developer (newbie or not) sooner or later will
try a python interactive shell, and so will see the warning.  It is
important also for experienced python developers to be told explicitly that
they should be thinking about switching to python 3, at some point.



>
>
> --
> Terry Jan Reedy
>
>
> ______________________________**_________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/**mailman/listinfo/python-dev<http://mail.python.org/mailman/listinfo/python-dev>
> Unsubscribe: http://mail.python.org/**mailman/options/python-dev/**
> gjcarneiro%40gmail.com<http://mail.python.org/mailman/options/python-dev/gjcarneiro%40gmail.com>
>



-- 
Gustavo J. A. M. Carneiro
"The universe is always one step beyond logic." -- Frank Herbert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130724/449a7b4c/attachment.html>

From lgautier at gmail.com  Wed Jul 24 19:22:35 2013
From: lgautier at gmail.com (Laurent Gautier)
Date: Wed, 24 Jul 2013 19:22:35 +0200
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <CAPTjJmosvJf4mwiqny=22No1e3FhV2UT52TdjCUogTc9LvrE+A@mail.gmail.com>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
	<CAL0kPAW97HnYU1ubD2bD64=ZazgrSTGBMajRA0eGxH4DiWkrfg@mail.gmail.com>
	<CAL0kPAU9-RR6OZiYerd5nJ7=70zDA5sQ1xt7tPDpG_2mr+1BqA@mail.gmail.com>
	<51EFFF22.1070603@gmail.com>
	<CAPTjJmosvJf4mwiqny=22No1e3FhV2UT52TdjCUogTc9LvrE+A@mail.gmail.com>
Message-ID: <51F00D5B.9060005@gmail.com>

On 07/24/2013 06:30 PM, Chris Angelico wrote:
> On Thu, Jul 25, 2013 at 2:21 AM, Laurent Gautier <lgautier at gmail.com> wrote:
>> - errors that are typical of "Python 2 script running with Python
>> 3"-specific are probably limited (e.g., use of unicode, use of xrange,
>> etc...)
>>
> The most common, in interactive scripts at least, is likely to be:
>
>>>> print "Hello, world!"
> SyntaxError: invalid syntax
>
> How helpful it's possible to make that one, I don't know. Is it safe
> to presume that it's more likely a syntax error will come from an
> interpreter version mismatch than a code bug?

The wrapper in /usr/bin/python:
- could use what is in 2to3. I think that most of the cases are solved 
there.
- whenever interactive, could have an intermediate layer between the 
input in the console and execution.


>
> ChrisA
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/lgautier%40gmail.com


From stefan_ml at behnel.de  Wed Jul 24 19:33:42 2013
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 24 Jul 2013 19:33:42 +0200
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <CAO-CpE+audZ7iBhJV2pZ4J-QeVYrYFM_TJiREyWt4HOxNbkRGg@mail.gmail.com>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
	<ksp1f7$jtl$1@ger.gmane.org>
	<CAO-CpE+audZ7iBhJV2pZ4J-QeVYrYFM_TJiREyWt4HOxNbkRGg@mail.gmail.com>
Message-ID: <ksp35f$8h2$1@ger.gmane.org>

Gustavo Carneiro, 24.07.2013 19:16:
> On Wed, Jul 24, 2013 at 6:04 PM, Terry Reedy wrote:
>> On 7/24/2013 5:12 AM, Bohuslav Kabrda wrote:
>>> Hi all, in recent days, there has been a discussion on fedora-devel
>>> (see thread [1]) about moving to Python 3 as a default.
>>
>> Not being a current *nix user, I have no opinion on the specific
>> alternatives. However, I am concerned about newcomers being mis-served by
>> being mislead into thinking that 'Python' is python 2 rather python 3. We
>> see this in some newbie posts on Python list. (This is especially a concern
>> when default Python is still, say, 2.5.)  I have no idea, though, how to
>> avoid breaking scripts while directing newcomers in the proper direction.
> 
> Maybe we could make interactive python2 scripts print a big warning
> advising users to use python 3 instead of 2, if possible?

Well, what I think could be done is to have the Py2 interpreter output a
warning if an unpostfixed shebang is found in an executed script. That
would eventually push users into either using the explicit "/usr/bin/env
python2", or port to Py3 straight away (or both).

Stefan



From guido at python.org  Wed Jul 24 19:56:05 2013
From: guido at python.org (Guido van Rossum)
Date: Wed, 24 Jul 2013 10:56:05 -0700
Subject: [Python-Dev] Inherance of file descriptor and handles on
 Windows (PEP 446)
In-Reply-To: <ksn359$q3a$1@ger.gmane.org>
References: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
	<ksn359$q3a$1@ger.gmane.org>
Message-ID: <CAP7+vJKGUS9cseDz+LgV0Z5DKJyYzEru6fAr1=Ct9GMBYB7i_w@mail.gmail.com>

On Tue, Jul 23, 2013 at 4:21 PM, Richard Oudkerk <shibturn at gmail.com> wrote:
> On 23/07/2013 11:45pm, Victor Stinner wrote:
>>
>> Said differently: the HANDLE_FLAG_INHERIT flag only has an effect on
>> *handles*, as indicated in its name. On Windows, file*descriptors*
>> are never inherited (are always closed) in child processes. I don't
>> think that it is possible to inherit file descriptors on Windows.
>
>
> Actually, you can inherit fds if you use os.spawnv() instead of
> subprocess.Popen().

Wow. Indeed you can -- I just tested this myself. How is this
accomplished? I guess the CRT has a backchannel to talk to itself when
it creates a process using spawn*?

This is the only reason I can think of for the odd default in the CRT
of opening file descriptors inheritable by default, which Victor
discovered. (But it doesn't explain why os.pipe() creates
uninheritable fds.)

If it weren't for this I would definitely vote to change the default
on Windows throughout the stdlib to create file descriptors whose
handles aren't inheritable. (Perhaps with a different policy for
stdin/stdout/stderr, which seem to be treated specially at the handle
level.)

I'm about ready to give up hope that we'll ever have a decent way to
deal with this.

But I'm also ready to propose that all this is such a mess that we
*should* change the default fd/handle inheritance to False, *across
platforms*, and damn the torpedoes -- i.e. accept breaking all
existing 3rd party UNIX code for subprocess creation that bypasses the
subprocess module, as well as breaking uses of os.spawn*() on both
platforms that depend on FD inheritance beyond stdin/stdout/stderr).

With the new, sane default, all we need instead of PEP 446 is a way to
make an FD inheritable after it's been created, which can be a single
os.make_inheritable(fd) call that you must apply to the fileno() of
the stream or socket object you want inherited (or directly to a FD
you created otherwise, e.g. with os.pipe()). On Windows, this should
probably only work with os.spawn*(), since otherwise you need *handle*
inheritance, not *FD* inheritance, and that's a non-portable concept
anyway.

We can fix multiprocessing any anything else in the stdlib that this
breaks, I presume.

To reduce the need for 3rd party subprocess creation code, we should
have better daemon creation code in the stdlib -- I wrote some damn
robust code for this purpose in my previous job, but it never saw the
light of day.

-- 
--Guido van Rossum (python.org/~guido)

From shibturn at gmail.com  Wed Jul 24 20:13:58 2013
From: shibturn at gmail.com (Richard Oudkerk)
Date: Wed, 24 Jul 2013 19:13:58 +0100
Subject: [Python-Dev] Inherance of file descriptor and handles on
 Windows (PEP 446)
In-Reply-To: <CAP7+vJKGUS9cseDz+LgV0Z5DKJyYzEru6fAr1=Ct9GMBYB7i_w@mail.gmail.com>
References: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
	<ksn359$q3a$1@ger.gmane.org>
	<CAP7+vJKGUS9cseDz+LgV0Z5DKJyYzEru6fAr1=Ct9GMBYB7i_w@mail.gmail.com>
Message-ID: <51F01966.7080908@gmail.com>

 > Wow. Indeed you can -- I just tested this myself. How is this
 > accomplished? I guess the CRT has a backchannel to talk to itself when
 > it creates a process using spawn*?

CreateProcess() takes a STARTUPINFO argument with undocumented fields 
cbReserved2, lpReserved2.  They are used to pass an array of fds.

 > But I'm also ready to propose that all this is such a mess that we
> *should* change the default fd/handle inheritance to False, *across
> platforms*, and damn the torpedoes -- i.e. accept breaking all
> existing 3rd party UNIX code for subprocess creation that bypasses the
> subprocess module, as well as breaking uses of os.spawn*() on both
> platforms that depend on FD inheritance beyond stdin/stdout/stderr).

+1

> We can fix multiprocessing any anything else in the stdlib that this
> breaks, I presume.

In the experimental branch of multiprocessing, child processes no longer 
inherit unnecessary handles.

--
Richard

From guido at python.org  Wed Jul 24 20:17:13 2013
From: guido at python.org (Guido van Rossum)
Date: Wed, 24 Jul 2013 11:17:13 -0700
Subject: [Python-Dev] Inherance of file descriptor and handles on
 Windows (PEP 446)
In-Reply-To: <51F01966.7080908@gmail.com>
References: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
	<ksn359$q3a$1@ger.gmane.org>
	<CAP7+vJKGUS9cseDz+LgV0Z5DKJyYzEru6fAr1=Ct9GMBYB7i_w@mail.gmail.com>
	<51F01966.7080908@gmail.com>
Message-ID: <CAP7+vJ+cRbA1jDaxHQo2b9DCgNv5LEyLd3UJ-Z+1gaiHu=WsFw@mail.gmail.com>

On Wed, Jul 24, 2013 at 11:13 AM, Richard Oudkerk <shibturn at gmail.com> wrote:
>> Wow. Indeed you can -- I just tested this myself. How is this
>> accomplished? I guess the CRT has a backchannel to talk to itself when
>> it creates a process using spawn*?
>
> CreateProcess() takes a STARTUPINFO argument with undocumented fields
> cbReserved2, lpReserved2.  They are used to pass an array of fds.

Does it also inherit sockets (which take up a different namespace than
regular FDs in CRT, unlike UNIX)?

>> But I'm also ready to propose that all this is such a mess that we
>>
>> *should* change the default fd/handle inheritance to False, *across
>> platforms*, and damn the torpedoes -- i.e. accept breaking all
>> existing 3rd party UNIX code for subprocess creation that bypasses the
>> subprocess module, as well as breaking uses of os.spawn*() on both
>> platforms that depend on FD inheritance beyond stdin/stdout/stderr).
>
> +1

Thanks! This was a difficult conclusion to come to. "Damn the
torpedoes" is occasionally a nice meme. :-(

>> We can fix multiprocessing any anything else in the stdlib that this
>> breaks, I presume.
>
> In the experimental branch of multiprocessing, child processes no longer
> inherit unnecessary handles.

And if we default individual handles to uninheritable, we can
presumably fix the ones that multiprocessing creates with the express
purpose of being inherited easily. (If it even uses that -- I haven't
read the source code, maybe it uses named pipes? :-)

-- 
--Guido van Rossum (python.org/~guido)

From shibturn at gmail.com  Wed Jul 24 20:36:48 2013
From: shibturn at gmail.com (Richard Oudkerk)
Date: Wed, 24 Jul 2013 19:36:48 +0100
Subject: [Python-Dev] Inherance of file descriptor and handles on
	Windows (PEP 446)
In-Reply-To: <CAP7+vJ+cRbA1jDaxHQo2b9DCgNv5LEyLd3UJ-Z+1gaiHu=WsFw@mail.gmail.com>
References: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
	<ksn359$q3a$1@ger.gmane.org>
	<CAP7+vJKGUS9cseDz+LgV0Z5DKJyYzEru6fAr1=Ct9GMBYB7i_w@mail.gmail.com>
	<51F01966.7080908@gmail.com>
	<CAP7+vJ+cRbA1jDaxHQo2b9DCgNv5LEyLd3UJ-Z+1gaiHu=WsFw@mail.gmail.com>
Message-ID: <ksp6rr$jg3$1@ger.gmane.org>

On 24/07/2013 7:17pm, Guido van Rossum wrote:
 > Does it also inherit sockets (which take up a different namespace than
 > regular FDs in CRT, unlike UNIX)?

Not reliably.  Processes created with CreateProcess() seems to inherit 
socket handles just like normal handles on my computer, but on some 
other computers -- with the same Windows version! -- it appears not to 
work.  See http://bugs.python.org/issue17399.

I think WSADuplicateSocket() should be used instead.

 > And if we default individual handles to uninheritable, we can
 > presumably fix the ones that multiprocessing creates with the express
 > purpose of being inherited easily. (If it even uses that -- I haven't
 > read the source code, maybe it uses named pipes?

multiprocessing never really needs to create any inheritable handles: it 
can use DuplicateHandle() to transfer each handle directly to the child 
process.

-- 
Richard


From a.badger at gmail.com  Wed Jul 24 21:27:16 2013
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Wed, 24 Jul 2013 12:27:16 -0700
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <20130724124209.61f8a8ed@anarchist>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
	<CAP1=2W7y6w_f9D69czQCDv--580D4yvOD+6qenOL3T4E6b2ryg@mail.gmail.com>
	<20130724152644.GI11402@unaka.lan>
	<CADiSq7feUW3-WbNTRLz5o+c2s1xLwrMn3i8bJy2WA3own7YxdQ@mail.gmail.com>
	<20130724124209.61f8a8ed@anarchist>
Message-ID: <20130724192716.GJ11402@unaka.lan>

On Wed, Jul 24, 2013 at 12:42:09PM -0400, Barry Warsaw wrote:
> On Jul 25, 2013, at 01:41 AM, Nick Coghlan wrote:
> 
> >How's this for an updated wording in the abstract:
> >
> >  * for the time being, all distributions should ensure that python
> >refers to the same target as python2
> >  * however, users should be aware that python refers to python3 on at
> >least Arch Linux (that change is
> >    what prompted the creation of this PEP), so "python" should be
> >used in the shebang line only for
> >    scripts that are source compatible with both Python 2 and 3
> 
> +1
> 
+1 as well.  Much clearer.

-Toshio
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130724/8846ff8f/attachment.pgp>

From regebro at gmail.com  Wed Jul 24 21:34:01 2013
From: regebro at gmail.com (Lennart Regebro)
Date: Wed, 24 Jul 2013 21:34:01 +0200
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <51F00D5B.9060005@gmail.com>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
	<CAL0kPAW97HnYU1ubD2bD64=ZazgrSTGBMajRA0eGxH4DiWkrfg@mail.gmail.com>
	<CAL0kPAU9-RR6OZiYerd5nJ7=70zDA5sQ1xt7tPDpG_2mr+1BqA@mail.gmail.com>
	<51EFFF22.1070603@gmail.com>
	<CAPTjJmosvJf4mwiqny=22No1e3FhV2UT52TdjCUogTc9LvrE+A@mail.gmail.com>
	<51F00D5B.9060005@gmail.com>
Message-ID: <CAL0kPAVSuWp_0965TKTjMQR+TxPLkat5AgXq-EdyUHkuyXi22g@mail.gmail.com>

On Wed, Jul 24, 2013 at 7:22 PM, Laurent Gautier <lgautier at gmail.com> wrote:
> The wrapper in /usr/bin/python:
> - could use what is in 2to3. I think that most of the cases are solved
> there.

Only the most trivial cases are solved completely by 2to3, and the
error messages you would get would be hard to understand, and running
the scripts would be very slow.

//Lennart

From victor.stinner at gmail.com  Wed Jul 24 23:50:08 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 24 Jul 2013 23:50:08 +0200
Subject: [Python-Dev] Inherance of file descriptor and handles on
 Windows (PEP 446)
In-Reply-To: <51F01966.7080908@gmail.com>
References: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
	<ksn359$q3a$1@ger.gmane.org>
	<CAP7+vJKGUS9cseDz+LgV0Z5DKJyYzEru6fAr1=Ct9GMBYB7i_w@mail.gmail.com>
	<51F01966.7080908@gmail.com>
Message-ID: <CAMpsgwZJw6zgR2gRGPXV5R_Y=R2bzBXB=UgWB4bQtxL9UEe_uw@mail.gmail.com>

2013/7/24 Richard Oudkerk <shibturn at gmail.com>:
>> Wow. Indeed you can -- I just tested this myself. How is this
>> accomplished? I guess the CRT has a backchannel to talk to itself when
>> it creates a process using spawn*?
>
> CreateProcess() takes a STARTUPINFO argument with undocumented fields
> cbReserved2, lpReserved2.  They are used to pass an array of fds.

So would it be possible to implement the pass_fds parameter of
subprocess using spawnl() or the undocumented fields?

And is it possible to close all handles except one (implement "pass_handles")?

The idea would be to use something subprocess.Popen(cmd,
pass_fds=[pipe_rfd], close_fds=True) or subprocess.Popen(cmd,
pass_handles=[pipe_rhandle], close_handles=True) instead of
subprocess.Popen(cmd, close_fds=False).

> In the experimental branch of multiprocessing, child processes
> no longer inherit unnecessary handles.

Where is this branch? How did you create the channel between the
manager and the worker?

Victor

From solipsis at pitrou.net  Wed Jul 24 23:57:33 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 24 Jul 2013 23:57:33 +0200
Subject: [Python-Dev] Inherance of file descriptor and handles on
	Windows (PEP 446)
References: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
	<ksn359$q3a$1@ger.gmane.org>
	<CAP7+vJKGUS9cseDz+LgV0Z5DKJyYzEru6fAr1=Ct9GMBYB7i_w@mail.gmail.com>
Message-ID: <20130724235733.3505fab0@fsol>

On Wed, 24 Jul 2013 10:56:05 -0700
Guido van Rossum <guido at python.org> wrote:
> 
> But I'm also ready to propose that all this is such a mess that we
> *should* change the default fd/handle inheritance to False, *across
> platforms*, and damn the torpedoes -- i.e. accept breaking all
> existing 3rd party UNIX code for subprocess creation that bypasses the
> subprocess module, as well as breaking uses of os.spawn*() on both
> platforms that depend on FD inheritance beyond stdin/stdout/stderr).

So I suppose you mean "change it to False except for
stdin/stdout/stderr"?

> With the new, sane default, all we need instead of PEP 446 is a way to
> make an FD inheritable after it's been created, which can be a single
> os.make_inheritable(fd) call that you must apply to the fileno() of
> the stream or socket object you want inherited (or directly to a FD
> you created otherwise, e.g. with os.pipe()). On Windows, this should
> probably only work with os.spawn*(), since otherwise you need *handle*
> inheritance, not *FD* inheritance, and that's a non-portable concept
> anyway.

I'm not sure how *fd* inheritance could work without *handle*
inheritance.
(since a fd seems to just be a proxy for a handle)

> To reduce the need for 3rd party subprocess creation code, we should
> have better daemon creation code in the stdlib -- I wrote some damn
> robust code for this purpose in my previous job, but it never saw the
> light of day.

What do you call "daemon"? An actual Unix-like daemon?

Regards

Antoine.



From guido at python.org  Thu Jul 25 00:25:50 2013
From: guido at python.org (Guido van Rossum)
Date: Wed, 24 Jul 2013 15:25:50 -0700
Subject: [Python-Dev] Inherance of file descriptor and handles on
 Windows (PEP 446)
In-Reply-To: <20130724235733.3505fab0@fsol>
References: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
	<ksn359$q3a$1@ger.gmane.org>
	<CAP7+vJKGUS9cseDz+LgV0Z5DKJyYzEru6fAr1=Ct9GMBYB7i_w@mail.gmail.com>
	<20130724235733.3505fab0@fsol>
Message-ID: <CAP7+vJJcweugSYKL3WqEPvB=vado29qdbKVKPZSmLdQsr6kdJQ@mail.gmail.com>

On Wed, Jul 24, 2013 at 2:57 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Wed, 24 Jul 2013 10:56:05 -0700
> Guido van Rossum <guido at python.org> wrote:
>>
>> But I'm also ready to propose that all this is such a mess that we
>> *should* change the default fd/handle inheritance to False, *across
>> platforms*, and damn the torpedoes -- i.e. accept breaking all
>> existing 3rd party UNIX code for subprocess creation that bypasses the
>> subprocess module, as well as breaking uses of os.spawn*() on both
>> platforms that depend on FD inheritance beyond stdin/stdout/stderr).
>
> So I suppose you mean "change it to False except for
> stdin/stdout/stderr"?

Hm, that's a tricky detail. I expect that on UNIX those are pre-opened
and inherited from the shell, and we should not change their cloexec
status; IIRC on Windows inheritance of the stdin/out/err handles is
guided by a separate flag to CreateProcess().

>> With the new, sane default, all we need instead of PEP 446 is a way to
>> make an FD inheritable after it's been created, which can be a single
>> os.make_inheritable(fd) call that you must apply to the fileno() of
>> the stream or socket object you want inherited (or directly to a FD
>> you created otherwise, e.g. with os.pipe()). On Windows, this should
>> probably only work with os.spawn*(), since otherwise you need *handle*
>> inheritance, not *FD* inheritance, and that's a non-portable concept
>> anyway.
>
> I'm not sure how *fd* inheritance could work without *handle*
> inheritance.
> (since a fd seems to just be a proxy for a handle)

The MS spawn() implementation takes care of this -- I presume they
make the handles inheritable (I think some of the bugs probably refer
to this fact) and then use the "internal" interface to pass the
mapping from FDs to handles to the subprocess.

>> To reduce the need for 3rd party subprocess creation code, we should
>> have better daemon creation code in the stdlib -- I wrote some damn
>> robust code for this purpose in my previous job, but it never saw the
>> light of day.
>
> What do you call "daemon"? An actual Unix-like daemon?

Yeah, a background process with parent PID 1 and not associated with
any terminal group.

-- 
--Guido van Rossum (python.org/~guido)

From solipsis at pitrou.net  Thu Jul 25 00:35:15 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 25 Jul 2013 00:35:15 +0200
Subject: [Python-Dev] Inherance of file descriptor and handles on
	Windows (PEP 446)
In-Reply-To: <CAP7+vJJcweugSYKL3WqEPvB=vado29qdbKVKPZSmLdQsr6kdJQ@mail.gmail.com>
References: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
	<ksn359$q3a$1@ger.gmane.org>
	<CAP7+vJKGUS9cseDz+LgV0Z5DKJyYzEru6fAr1=Ct9GMBYB7i_w@mail.gmail.com>
	<20130724235733.3505fab0@fsol>
	<CAP7+vJJcweugSYKL3WqEPvB=vado29qdbKVKPZSmLdQsr6kdJQ@mail.gmail.com>
Message-ID: <20130725003515.4b79ff17@fsol>

On Wed, 24 Jul 2013 15:25:50 -0700
Guido van Rossum <guido at python.org> wrote:
> >> To reduce the need for 3rd party subprocess creation code, we should
> >> have better daemon creation code in the stdlib -- I wrote some damn
> >> robust code for this purpose in my previous job, but it never saw the
> >> light of day.
> >
> > What do you call "daemon"? An actual Unix-like daemon?
> 
> Yeah, a background process with parent PID 1 and not associated with
> any terminal group.

But is that relevant to the PEP? A daemon only uses fork(), not exec().
Or have I misunderstood your concern?

Regards

Antoine.
 


From shibturn at gmail.com  Thu Jul 25 00:52:29 2013
From: shibturn at gmail.com (Richard Oudkerk)
Date: Wed, 24 Jul 2013 23:52:29 +0100
Subject: [Python-Dev] Inherance of file descriptor and handles on
	Windows (PEP 446)
In-Reply-To: <CAMpsgwZJw6zgR2gRGPXV5R_Y=R2bzBXB=UgWB4bQtxL9UEe_uw@mail.gmail.com>
References: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
	<ksn359$q3a$1@ger.gmane.org>
	<CAP7+vJKGUS9cseDz+LgV0Z5DKJyYzEru6fAr1=Ct9GMBYB7i_w@mail.gmail.com>
	<51F01966.7080908@gmail.com>
	<CAMpsgwZJw6zgR2gRGPXV5R_Y=R2bzBXB=UgWB4bQtxL9UEe_uw@mail.gmail.com>
Message-ID: <ksplr8$vrq$1@ger.gmane.org>

On 24/07/2013 10:50pm, Victor Stinner wrote:
> So would it be possible to implement the pass_fds parameter of
> subprocess using spawnl() or the undocumented fields?

Not in a non-racy way.

spawnv() calls CreateProcess() with bInheritHandles=TRUE, so *all* 
inheritable handles are inherited by the child.

The passing of the arrays of fds just makes the fds in the child process 
match the fds in the parent.

If you have a Visual Studio installed then the relevant code is in

    .../Microsoft Visual Studio 10.0/VC/crt/src/dospawn.c

> And is it possible to close all handles except one (implement "pass_handles")?

I don't know how to do that.

>> In the experimental branch of multiprocessing, child processes
>> no longer inherit unnecessary handles.
>
> Where is this branch? How did you create the channel between the
> manager and the worker?

     http://hg.python.org/sandbox/sbt/

The parent creates a pipe and starts the child process.  The pid of the 
parent, and the handle for the read end of the pipe are passed on the 
command line.  Then the child "steals" the handle from the parent using 
OpenProcess() and DuplicateHandle() using the DUPLICATE_CLOSE_SOURCE flag.

-- 
Richard


From greg.ewing at canterbury.ac.nz  Thu Jul 25 01:04:33 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 25 Jul 2013 11:04:33 +1200
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <CAPTjJmosvJf4mwiqny=22No1e3FhV2UT52TdjCUogTc9LvrE+A@mail.gmail.com>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
	<CAL0kPAW97HnYU1ubD2bD64=ZazgrSTGBMajRA0eGxH4DiWkrfg@mail.gmail.com>
	<CAL0kPAU9-RR6OZiYerd5nJ7=70zDA5sQ1xt7tPDpG_2mr+1BqA@mail.gmail.com>
	<51EFFF22.1070603@gmail.com>
	<CAPTjJmosvJf4mwiqny=22No1e3FhV2UT52TdjCUogTc9LvrE+A@mail.gmail.com>
Message-ID: <51F05D81.3020901@canterbury.ac.nz>

Chris Angelico wrote:
>>>>print "Hello, world!"
> SyntaxError: invalid syntax
> 
> Is it safe
> to presume that it's more likely a syntax error will come from an
> interpreter version mismatch than a code bug?

Maybe look at sys.args[0], and if it ends in "python" with
no version number, add something to the error message about
possibly using the wrong version?

-- 
Greg

From eric at trueblade.com  Thu Jul 25 01:12:02 2013
From: eric at trueblade.com (Eric V. Smith)
Date: Wed, 24 Jul 2013 19:12:02 -0400
Subject: [Python-Dev] Inherance of file descriptor and handles on
 Windows (PEP 446)
In-Reply-To: <CAP7+vJJcweugSYKL3WqEPvB=vado29qdbKVKPZSmLdQsr6kdJQ@mail.gmail.com>
References: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
	<ksn359$q3a$1@ger.gmane.org>
	<CAP7+vJKGUS9cseDz+LgV0Z5DKJyYzEru6fAr1=Ct9GMBYB7i_w@mail.gmail.com>
	<20130724235733.3505fab0@fsol>
	<CAP7+vJJcweugSYKL3WqEPvB=vado29qdbKVKPZSmLdQsr6kdJQ@mail.gmail.com>
Message-ID: <51F05F42.6070200@trueblade.com>

On 7/24/2013 6:25 PM, Guido van Rossum wrote:
>>> To reduce the need for 3rd party subprocess creation code, we should
>>> have better daemon creation code in the stdlib -- I wrote some damn
>>> robust code for this purpose in my previous job, but it never saw the
>>> light of day.
>>
>> What do you call "daemon"? An actual Unix-like daemon?
> 
> Yeah, a background process with parent PID 1 and not associated with
> any terminal group.

There's PEP 3143 and https://pypi.python.org/pypi/python-daemon. I've
used it often, with great success.

-- 
Eric.

From ncoghlan at gmail.com  Thu Jul 25 02:06:11 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 25 Jul 2013 10:06:11 +1000
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <20130724192716.GJ11402@unaka.lan>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
	<CAP1=2W7y6w_f9D69czQCDv--580D4yvOD+6qenOL3T4E6b2ryg@mail.gmail.com>
	<20130724152644.GI11402@unaka.lan>
	<CADiSq7feUW3-WbNTRLz5o+c2s1xLwrMn3i8bJy2WA3own7YxdQ@mail.gmail.com>
	<20130724124209.61f8a8ed@anarchist>
	<20130724192716.GJ11402@unaka.lan>
Message-ID: <CADiSq7cDzLUsjfYpB5hO4V8yOrK_D71cDnSgOAiCS-kUu_6wrw@mail.gmail.com>

On 25 Jul 2013 05:30, "Toshio Kuratomi" <a.badger at gmail.com> wrote:
>
> On Wed, Jul 24, 2013 at 12:42:09PM -0400, Barry Warsaw wrote:
> > On Jul 25, 2013, at 01:41 AM, Nick Coghlan wrote:
> >
> > >How's this for an updated wording in the abstract:
> > >
> > >  * for the time being, all distributions should ensure that python
> > >refers to the same target as python2
> > >  * however, users should be aware that python refers to python3 on at
> > >least Arch Linux (that change is
> > >    what prompted the creation of this PEP), so "python" should be
> > >used in the shebang line only for
> > >    scripts that are source compatible with both Python 2 and 3
> >
> > +1
> >
> +1 as well.  Much clearer.

OK, unless someone gets to it before me, I'll update it tonight.

Cheers,
Nick.

>
> -Toshio
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130725/f5304aff/attachment.html>

From cs at zip.com.au  Thu Jul 25 01:54:24 2013
From: cs at zip.com.au (Cameron Simpson)
Date: Thu, 25 Jul 2013 09:54:24 +1000
Subject: [Python-Dev] Inherance of file descriptor and handles on
 Windows (PEP 446)
In-Reply-To: <20130725003515.4b79ff17@fsol>
References: <20130725003515.4b79ff17@fsol>
Message-ID: <20130724235424.GA50053@cskk.homeip.net>

On 25Jul2013 00:35, Antoine Pitrou <solipsis at pitrou.net> wrote:
| On Wed, 24 Jul 2013 15:25:50 -0700
| Guido van Rossum <guido at python.org> wrote:
| > >> To reduce the need for 3rd party subprocess creation code, we should
| > >> have better daemon creation code in the stdlib -- I wrote some damn
| > >> robust code for this purpose in my previous job, but it never saw the
| > >> light of day.
| > >
| > > What do you call "daemon"? An actual Unix-like daemon?
| > 
| > Yeah, a background process with parent PID 1 and not associated with
| > any terminal group.
| 
| But is that relevant to the PEP? A daemon only uses fork(), not exec().

Not necessarily.

I routinely run other commands in a daemonlike mode, without the
command itself having "make myself a daemon" functionality. If I
were writing the launcher in Python (and my "setsid" script is
Python already), I would be doing all the daemon bits in Python
including the fork, then execing the other tool.

Cheers,
-- 

If it can't be turned off, it's not a feature. - Karl Heuer

From guido at python.org  Thu Jul 25 02:41:22 2013
From: guido at python.org (Guido van Rossum)
Date: Wed, 24 Jul 2013 17:41:22 -0700
Subject: [Python-Dev] Inherance of file descriptor and handles on
	Windows (PEP 446)
In-Reply-To: <20130725003515.4b79ff17@fsol>
References: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
	<ksn359$q3a$1@ger.gmane.org>
	<CAP7+vJKGUS9cseDz+LgV0Z5DKJyYzEru6fAr1=Ct9GMBYB7i_w@mail.gmail.com>
	<20130724235733.3505fab0@fsol>
	<CAP7+vJJcweugSYKL3WqEPvB=vado29qdbKVKPZSmLdQsr6kdJQ@mail.gmail.com>
	<20130725003515.4b79ff17@fsol>
Message-ID: <CAP7+vJKB9Tx8G_FvG5ZZDjxGifRde2EKgrFhH4DDaO1JpOGqgQ@mail.gmail.com>

On Wed, Jul 24, 2013 at 3:35 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Wed, 24 Jul 2013 15:25:50 -0700
> Guido van Rossum <guido at python.org> wrote:
>> >> To reduce the need for 3rd party subprocess creation code, we should
>> >> have better daemon creation code in the stdlib -- I wrote some damn
>> >> robust code for this purpose in my previous job, but it never saw the
>> >> light of day.
>> >
>> > What do you call "daemon"? An actual Unix-like daemon?
>>
>> Yeah, a background process with parent PID 1 and not associated with
>> any terminal group.
>
> But is that relevant to the PEP? A daemon only uses fork(), not exec().
> Or have I misunderstood your concern?

Actually it's common for "daemonization" libraries to run an arbitrary
executable as a daemon, and then it would be relevant. E.g. zdaemon
(https://pypi.python.org/pypi/zdaemon) does this. (Disclosure: I wrote
the first version of zdaemon when I worked for Zope Corp over a decade
ago. :-)

-- 
--Guido van Rossum (python.org/~guido)

From ben+python at benfinney.id.au  Thu Jul 25 04:08:18 2013
From: ben+python at benfinney.id.au (Ben Finney)
Date: Thu, 25 Jul 2013 12:08:18 +1000
Subject: [Python-Dev] Daemon creation code in the standard library (was:
	Inherance of file descriptor and handles on Windows (PEP 446))
References: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
	<ksn359$q3a$1@ger.gmane.org>
	<CAP7+vJKGUS9cseDz+LgV0Z5DKJyYzEru6fAr1=Ct9GMBYB7i_w@mail.gmail.com>
Message-ID: <7wy58vs86l.fsf_-_@benfinney.id.au>

Guido van Rossum <guido at python.org> writes:

> To reduce the need for 3rd party subprocess creation code, we should
> have better daemon creation code in the stdlib -- I wrote some damn
> robust code for this purpose in my previous job, but it never saw the
> light of day.

Work continues on the PEP 3143-compatible ?python-daemon?, porting it to
Python 3 and aiming for inclusion in the standard library.

Interested parties are invited to join us on the discussion forums
<URL:http://lists.alioth.debian.org/cgi-bin/mailman/admin/python-daemon-devel>.

-- 
 \            ?Politics is not the art of the possible. It consists in |
  `\       choosing between the disastrous and the unpalatable.? ?John |
_o__)                                    Kenneth Galbraith, 1962-03-02 |
Ben Finney


From ben+python at benfinney.id.au  Thu Jul 25 04:18:38 2013
From: ben+python at benfinney.id.au (Ben Finney)
Date: Thu, 25 Jul 2013 12:18:38 +1000
Subject: [Python-Dev] Daemon creation code in the standard library
References: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
	<ksn359$q3a$1@ger.gmane.org>
	<CAP7+vJKGUS9cseDz+LgV0Z5DKJyYzEru6fAr1=Ct9GMBYB7i_w@mail.gmail.com>
	<7wy58vs86l.fsf_-_@benfinney.id.au>
Message-ID: <7w1u6njsap.fsf@benfinney.id.au>

Ben Finney <ben+python at benfinney.id.au> writes:

> Work continues on the PEP 3143-compatible ?python-daemon?, porting it to
> Python 3 and aiming for inclusion in the standard library.

At PyPI <URL:http://pypi.python.org/pypi/python-daemon/>, and
development co-ordinated at Alioth
<URL:https://alioth.debian.org/projects/python-daemon/>.

> Interested parties are invited to join us on the discussion forums

The correct link for the ?python-daemon-devel? forum is
<URL:http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/python-daemon-devel>.
For announcements only, we have 
<URL:http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/python-daemon-announce>.

-- 
 \        ?This sentence contradicts itself ? no actually it doesn't.? |
  `\                                               ?Douglas Hofstadter |
_o__)                                                                  |
Ben Finney


From bkabrda at redhat.com  Thu Jul 25 07:50:07 2013
From: bkabrda at redhat.com (Bohuslav Kabrda)
Date: Thu, 25 Jul 2013 01:50:07 -0400 (EDT)
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <CAL0kPAW97HnYU1ubD2bD64=ZazgrSTGBMajRA0eGxH4DiWkrfg@mail.gmail.com>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
	<CAL0kPAW97HnYU1ubD2bD64=ZazgrSTGBMajRA0eGxH4DiWkrfg@mail.gmail.com>
Message-ID: <872938422.6471844.1374731407688.JavaMail.root@redhat.com>

----- Original Message -----
> On Wed, Jul 24, 2013 at 11:12 AM, Bohuslav Kabrda <bkabrda at redhat.com> wrote:
> > - Should we point /usr/bin/python to Python 3 when we make the move?
> 
> No.
> 
> > - What should user get after using "yum install python"?
> 
> Will a base install include Python 3? If it does, I think yum install
> python should mean python3, and hence already be installed.
> 

Yep, base will include Python 3.
This is not just about python package, but also all other python extension modules (e.g. python-foo vs. python2-foo vs. python3-foo...).

> //Lennart
> 

-- 
Regards,
Bohuslav "Slavek" Kabrda.

From bkabrda at redhat.com  Thu Jul 25 08:47:08 2013
From: bkabrda at redhat.com (Bohuslav Kabrda)
Date: Thu, 25 Jul 2013 02:47:08 -0400 (EDT)
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <51EFEE63.5080702@v.loewis.de>
References: <1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
	<51EFEE63.5080702@v.loewis.de>
Message-ID: <2124741396.6500012.1374734828603.JavaMail.root@redhat.com>

----- Original Message -----
> Am 24.07.13 11:12, schrieb Bohuslav Kabrda:
> > - Should we point /usr/bin/python to Python 3 when we make the move?
> 
> This should depend on the answer to this question:
> - for how long have you been providing /usr/bin/python2 binaries?
> 

Huh, I don't know exactly, but quick git blame of our python.spec shows that /usr/bin/python2 has been there since July 2009, but probably even longer.

> Users "should" have been explicit in declaring scripts as
> /usr/bin/python2 for quite some time now, except that this would break
> on distros which don't provide a python2 symlink.
> 
> Ideally, you "should" have warned users to be explicit with python2 if
> the script wouldn't work on python3. But I'd wave this requirement, as
> there is already the upstream PEP.
> 
> So (IMO) if it the last three Fedora releases had been providing python2
> binaries, it should be allowed to switch python to be python3. The exact
> number can be debated; it should depend on what releases are still in
> active use so that a script would run on multiple releases.
> 

Yep, it seems like 6+ fedora releases and certainly the currently active ones have that. We will however need to make it clear that everyone, especially package maintainers, points to /usr/bin/python{2,3}.

> Then you can tell users that possible breakage is easy to fix: just
> install python2, and change the shebang line.
> 
> Regards,
> Martin
> 
> 

-- 
Regards,
Bohuslav "Slavek" Kabrda.

From fijall at gmail.com  Thu Jul 25 11:45:10 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 25 Jul 2013 11:45:10 +0200
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <51F00D5B.9060005@gmail.com>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
	<CAL0kPAW97HnYU1ubD2bD64=ZazgrSTGBMajRA0eGxH4DiWkrfg@mail.gmail.com>
	<CAL0kPAU9-RR6OZiYerd5nJ7=70zDA5sQ1xt7tPDpG_2mr+1BqA@mail.gmail.com>
	<51EFFF22.1070603@gmail.com>
	<CAPTjJmosvJf4mwiqny=22No1e3FhV2UT52TdjCUogTc9LvrE+A@mail.gmail.com>
	<51F00D5B.9060005@gmail.com>
Message-ID: <CAK5idxTgMGNbzFaGzDis78M6pq4NM9K2mLTUvUF-ieDkKSC8jQ@mail.gmail.com>

On Wed, Jul 24, 2013 at 7:22 PM, Laurent Gautier <lgautier at gmail.com> wrote:
> On 07/24/2013 06:30 PM, Chris Angelico wrote:
>>
>> On Thu, Jul 25, 2013 at 2:21 AM, Laurent Gautier <lgautier at gmail.com>
>> wrote:
>>>
>>> - errors that are typical of "Python 2 script running with Python
>>> 3"-specific are probably limited (e.g., use of unicode, use of xrange,
>>> etc...)
>>>
>> The most common, in interactive scripts at least, is likely to be:
>>
>>>>> print "Hello, world!"
>>
>> SyntaxError: invalid syntax
>>
>> How helpful it's possible to make that one, I don't know. Is it safe
>> to presume that it's more likely a syntax error will come from an
>> interpreter version mismatch than a code bug?
>
>
> The wrapper in /usr/bin/python:
> - could use what is in 2to3. I think that most of the cases are solved
> there.
> - whenever interactive, could have an intermediate layer between the input
> in the console and execution.

So you suggest that instead of a clear SyntaxError you should end up
with a confusing error (something has no attribute xyz or so) after a
while (if say somone tries to load twisted via 2to3).

From a.badger at gmail.com  Thu Jul 25 12:38:46 2013
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Thu, 25 Jul 2013 03:38:46 -0700
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <CAP1=2W7y6w_f9D69czQCDv--580D4yvOD+6qenOL3T4E6b2ryg@mail.gmail.com>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
	<CAP1=2W7y6w_f9D69czQCDv--580D4yvOD+6qenOL3T4E6b2ryg@mail.gmail.com>
Message-ID: <CABVPEKp92VHGbLJjV_svedkLz5T22sxSHu6JMxKGm7Rx-VoB8Q@mail.gmail.com>

On Jul 24, 2013 6:37 AM, "Brett Cannon" <brett at python.org> wrote:
> The key, though, is adding python2 and getting your code to use that
binary  specifically so that shifting the default name is more of a
convenience than something which might break existing code not ready for
the switch.
>
Applicable to this, does anyone know whether distutils, setuptools,
distlib, or any of the other standard build+install tools are doing shebang
requiring?  Are they doing the right thing wrt python vs python2?

-Toshio
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130725/22ad3690/attachment.html>

From lgautier at gmail.com  Thu Jul 25 12:41:35 2013
From: lgautier at gmail.com (Laurent Gautier)
Date: Thu, 25 Jul 2013 12:41:35 +0200
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <CAK5idxTgMGNbzFaGzDis78M6pq4NM9K2mLTUvUF-ieDkKSC8jQ@mail.gmail.com>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
	<CAL0kPAW97HnYU1ubD2bD64=ZazgrSTGBMajRA0eGxH4DiWkrfg@mail.gmail.com>
	<CAL0kPAU9-RR6OZiYerd5nJ7=70zDA5sQ1xt7tPDpG_2mr+1BqA@mail.gmail.com>
	<51EFFF22.1070603@gmail.com>
	<CAPTjJmosvJf4mwiqny=22No1e3FhV2UT52TdjCUogTc9LvrE+A@mail.gmail.com>
	<51F00D5B.9060005@gmail.com>
	<CAK5idxTgMGNbzFaGzDis78M6pq4NM9K2mLTUvUF-ieDkKSC8jQ@mail.gmail.com>
Message-ID: <51F100DF.70204@gmail.com>

On 07/25/2013 11:45 AM, Maciej Fijalkowski wrote:
> On Wed, Jul 24, 2013 at 7:22 PM, Laurent Gautier <lgautier at gmail.com> wrote:
>> On 07/24/2013 06:30 PM, Chris Angelico wrote:
>>> On Thu, Jul 25, 2013 at 2:21 AM, Laurent Gautier <lgautier at gmail.com>
>>> wrote:
>>>> - errors that are typical of "Python 2 script running with Python
>>>> 3"-specific are probably limited (e.g., use of unicode, use of xrange,
>>>> etc...)
>>>>
>>> The most common, in interactive scripts at least, is likely to be:
>>>
>>>>>> print "Hello, world!"
>>> SyntaxError: invalid syntax
>>>
>>> How helpful it's possible to make that one, I don't know. Is it safe
>>> to presume that it's more likely a syntax error will come from an
>>> interpreter version mismatch than a code bug?
>>
>> The wrapper in /usr/bin/python:
>> - could use what is in 2to3. I think that most of the cases are solved
>> there.
>> - whenever interactive, could have an intermediate layer between the input
>> in the console and execution.
> So you suggest that instead of a clear SyntaxError you should end up
> with a confusing error (something has no attribute xyz or so) after a
> while (if say somone tries to load twisted via 2to3).

In a sense, yes.

I think that the a priori (expectations) a user has plays a role in 
whether something is confusing or not.
Consider the two following situations:
- a programmer is working on code, and is testing it. If the program 
fails, this is because he/she made a mistake and the error message 
should indicate where the error originates from
- a user is running a python script (he expects to be working), and is 
using the default /usr/bin/python (formerly Python 2, now Python 3). If 
the program fails because of obvious Python 2-only idioms, reporting 
this rather that the SyntaxError is much less confusing.

Having that said, the comments are pointing out that this is may be not 
a completely good idea (most notably because of how slow 2to3 is).
A wrapper producing an unconditional warning about the default python 
being changed to Python 3, and differences between Python 2 and Python 3 
the possible source of errors, (as I think it was suggested) is going to 
be a better idea.

From regebro at gmail.com  Thu Jul 25 13:19:29 2013
From: regebro at gmail.com (Lennart Regebro)
Date: Thu, 25 Jul 2013 13:19:29 +0200
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <51F100DF.70204@gmail.com>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
	<CAL0kPAW97HnYU1ubD2bD64=ZazgrSTGBMajRA0eGxH4DiWkrfg@mail.gmail.com>
	<CAL0kPAU9-RR6OZiYerd5nJ7=70zDA5sQ1xt7tPDpG_2mr+1BqA@mail.gmail.com>
	<51EFFF22.1070603@gmail.com>
	<CAPTjJmosvJf4mwiqny=22No1e3FhV2UT52TdjCUogTc9LvrE+A@mail.gmail.com>
	<51F00D5B.9060005@gmail.com>
	<CAK5idxTgMGNbzFaGzDis78M6pq4NM9K2mLTUvUF-ieDkKSC8jQ@mail.gmail.com>
	<51F100DF.70204@gmail.com>
Message-ID: <CAL0kPAWBabP5UNb+43LSnpjiX9_mSsp=bRumCXdswz0UPA=rKg@mail.gmail.com>

On Thu, Jul 25, 2013 at 12:41 PM, Laurent Gautier <lgautier at gmail.com> wrote:
> - a user is running a python script (he expects to be working), and is using
> the default /usr/bin/python (formerly Python 2, now Python 3). If the
> program fails because of obvious Python 2-only idioms, reporting this rather
> that the SyntaxError is much less confusing.

2to3 will not report this. It will get rid of the SyntaxError (which
at least is fairly clear) and give you a completely different and even
more obscure error. You have replaced a somewhat unclear error with a
very unclear error.

If we want to report a problem, then /usr/bin/python should just
report that you should use /usr/bin/python2 or /usr/bin/python3.

That's clear.

//Lennart

From lgautier at gmail.com  Thu Jul 25 13:53:33 2013
From: lgautier at gmail.com (Laurent Gautier)
Date: Thu, 25 Jul 2013 13:53:33 +0200
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <CAL0kPAWBabP5UNb+43LSnpjiX9_mSsp=bRumCXdswz0UPA=rKg@mail.gmail.com>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
	<CAL0kPAW97HnYU1ubD2bD64=ZazgrSTGBMajRA0eGxH4DiWkrfg@mail.gmail.com>
	<CAL0kPAU9-RR6OZiYerd5nJ7=70zDA5sQ1xt7tPDpG_2mr+1BqA@mail.gmail.com>
	<51EFFF22.1070603@gmail.com>
	<CAPTjJmosvJf4mwiqny=22No1e3FhV2UT52TdjCUogTc9LvrE+A@mail.gmail.com>
	<51F00D5B.9060005@gmail.com>
	<CAK5idxTgMGNbzFaGzDis78M6pq4NM9K2mLTUvUF-ieDkKSC8jQ@mail.gmail.com>
	<51F100DF.70204@gmail.com>
	<CAL0kPAWBabP5UNb+43LSnpjiX9_mSsp=bRumCXdswz0UPA=rKg@mail.gmail.com>
Message-ID: <51F111BD.5030008@gmail.com>

On 07/25/2013 01:19 PM, Lennart Regebro wrote:
> On Thu, Jul 25, 2013 at 12:41 PM, Laurent Gautier <lgautier at gmail.com> wrote:
>> - a user is running a python script (he expects to be working), and is using
>> the default /usr/bin/python (formerly Python 2, now Python 3). If the
>> program fails because of obvious Python 2-only idioms, reporting this rather
>> that the SyntaxError is much less confusing.
> 2to3 will not report this.

My meaning was the use of 2to3's machinery (and fire a message if a 
translation occurs) not 2to3 itself, obviously.

>   It will get rid of the SyntaxError (which
> at least is fairly clear) and give you a completely different and even
> more obscure error. You have replaced a somewhat unclear error with a
> very unclear error.
>
> If we want to report a problem, then /usr/bin/python should just
> report that you should use /usr/bin/python2 or /usr/bin/python3.
>
> That's clear.
>
> //Lennart


From ncoghlan at gmail.com  Thu Jul 25 14:25:26 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 25 Jul 2013 22:25:26 +1000
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <CABVPEKp92VHGbLJjV_svedkLz5T22sxSHu6JMxKGm7Rx-VoB8Q@mail.gmail.com>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
	<CAP1=2W7y6w_f9D69czQCDv--580D4yvOD+6qenOL3T4E6b2ryg@mail.gmail.com>
	<CABVPEKp92VHGbLJjV_svedkLz5T22sxSHu6JMxKGm7Rx-VoB8Q@mail.gmail.com>
Message-ID: <CADiSq7eB71Or=nkzo7JLkYBz=AHHh9jkuWwZ1o5GTh48_6ZRvg@mail.gmail.com>

On 25 July 2013 20:38, Toshio Kuratomi <a.badger at gmail.com> wrote:
>
> On Jul 24, 2013 6:37 AM, "Brett Cannon" <brett at python.org> wrote:
>> The key, though, is adding python2 and getting your code to use that
>> binary  specifically so that shifting the default name is more of a
>> convenience than something which might break existing code not ready for the
>> switch.
>>
> Applicable to this, does anyone know whether distutils, setuptools, distlib,
> or any of the other standard build+install tools are doing shebang
> requiring?  Are they doing the right thing wrt python vs python2?

It occurs to me they're almost certainly using "sys.executable" to set
the shebang line, so probably not :(

distutils-sig could probably offer a better answer though, most of the
packaging folks don't hang out here.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Thu Jul 25 14:35:28 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 25 Jul 2013 22:35:28 +1000
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <CADiSq7cDzLUsjfYpB5hO4V8yOrK_D71cDnSgOAiCS-kUu_6wrw@mail.gmail.com>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
	<CAP1=2W7y6w_f9D69czQCDv--580D4yvOD+6qenOL3T4E6b2ryg@mail.gmail.com>
	<20130724152644.GI11402@unaka.lan>
	<CADiSq7feUW3-WbNTRLz5o+c2s1xLwrMn3i8bJy2WA3own7YxdQ@mail.gmail.com>
	<20130724124209.61f8a8ed@anarchist>
	<20130724192716.GJ11402@unaka.lan>
	<CADiSq7cDzLUsjfYpB5hO4V8yOrK_D71cDnSgOAiCS-kUu_6wrw@mail.gmail.com>
Message-ID: <CADiSq7dx+o7Jm3STDMJYzxSV2fxGSUQ1H3PseMWBs77UXa_s8A@mail.gmail.com>

On 25 July 2013 10:06, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
> On 25 Jul 2013 05:30, "Toshio Kuratomi" <a.badger at gmail.com> wrote:
>>
>> On Wed, Jul 24, 2013 at 12:42:09PM -0400, Barry Warsaw wrote:
>> > On Jul 25, 2013, at 01:41 AM, Nick Coghlan wrote:
>> >
>> > >How's this for an updated wording in the abstract:
>> > >
>> > >  * for the time being, all distributions should ensure that python
>> > >refers to the same target as python2
>> > >  * however, users should be aware that python refers to python3 on at
>> > >least Arch Linux (that change is
>> > >    what prompted the creation of this PEP), so "python" should be
>> > >used in the shebang line only for
>> > >    scripts that are source compatible with both Python 2 and 3
>> >
>> > +1
>> >
>> +1 as well.  Much clearer.
>
> OK, unless someone gets to it before me, I'll update it tonight.

Update is here: http://hg.python.org/peps/rev/4b379a690ae2

I added one final bullet point to the abstract with advice for users:

* in preparation for an eventual change in the default version of
Python, Python 2 only scripts should either be updated to be source
compatible with Python 3 or else to use python2 in the shebang line.

I also rewrote the first two bullet points in the Migration Notes
section, as I realised they contributed heavily to the notion that
distros like Fedora should switch aggressively, while only the
conservative distros like RHEL and Debian stable should preserve the
old behaviour (which wasn't the intent at all).

Finally, I added a couple of other notes:

* one pointing out the kind of time frame we're thinking about (i.e.
years) by mentioning the 2.7 switch to security fix only mode
* one noting that the Python launcher for Windows also default to
Python 2 if both 2.x and 3.x are installed, so we're also aiming for
cross platform consistency here

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From eliben at gmail.com  Thu Jul 25 16:29:29 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Thu, 25 Jul 2013 07:29:29 -0700
Subject: [Python-Dev] Official github mirror for CPython?
Message-ID: <CAF-Rda-=wFp_pWSa2gqFxet1M7pf68si=LKZ7FyGU7UrfRS3FA@mail.gmail.com>

Hi all,

I've been looking for a Github mirror for Python, and found two:

* https://github.com/python-git/python has a lot of forks/watches/starts
but seems to be very out of date (last updated 4 years ago)
* https://github.com/python-mirror/python doesn't appear to be very popular
but is updated daily

Are some of you the owners of these repositories? Should we consolidate to
a single "semi-official" mirror?

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130725/5eca6846/attachment.html>

From rosuav at gmail.com  Thu Jul 25 16:33:07 2013
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 26 Jul 2013 00:33:07 +1000
Subject: [Python-Dev] Official github mirror for CPython?
In-Reply-To: <CAF-Rda-=wFp_pWSa2gqFxet1M7pf68si=LKZ7FyGU7UrfRS3FA@mail.gmail.com>
References: <CAF-Rda-=wFp_pWSa2gqFxet1M7pf68si=LKZ7FyGU7UrfRS3FA@mail.gmail.com>
Message-ID: <CAPTjJmrabD03i9AhkxYXu3iawRt0kG6asOwNLd-D++o4JM=pkg@mail.gmail.com>

On Fri, Jul 26, 2013 at 12:29 AM, Eli Bendersky <eliben at gmail.com> wrote:
> Hi all,
>
> I've been looking for a Github mirror for Python, and found two:
>
> * https://github.com/python-git/python has a lot of forks/watches/starts but
> seems to be very out of date (last updated 4 years ago)
> * https://github.com/python-mirror/python doesn't appear to be very popular
> but is updated daily

Also https://github.com/akheron/cpython seems to be the same again.

ChrisA

From christian at python.org  Thu Jul 25 16:37:02 2013
From: christian at python.org (Christian Heimes)
Date: Thu, 25 Jul 2013 16:37:02 +0200
Subject: [Python-Dev] Official github mirror for CPython?
In-Reply-To: <CAF-Rda-=wFp_pWSa2gqFxet1M7pf68si=LKZ7FyGU7UrfRS3FA@mail.gmail.com>
References: <CAF-Rda-=wFp_pWSa2gqFxet1M7pf68si=LKZ7FyGU7UrfRS3FA@mail.gmail.com>
Message-ID: <51F1380E.9070608@python.org>

Am 25.07.2013 16:29, schrieb Eli Bendersky:
> Hi all,
> 
> I've been looking for a Github mirror for Python, and found two:
> 
> * https://github.com/python-git/python has a lot of forks/watches/starts
> but seems to be very out of date (last updated 4 years ago)
> * https://github.com/python-mirror/python doesn't appear to be very
> popular but is updated daily
> 
> Are some of you the owners of these repositories? Should we consolidate
> to a single "semi-official" mirror?

+1

Does the PSF have an official account on github? We have one on bitbucket...

Christian


From brian at python.org  Thu Jul 25 16:48:40 2013
From: brian at python.org (Brian Curtin)
Date: Thu, 25 Jul 2013 09:48:40 -0500
Subject: [Python-Dev] Official github mirror for CPython?
In-Reply-To: <51F1380E.9070608@python.org>
References: <CAF-Rda-=wFp_pWSa2gqFxet1M7pf68si=LKZ7FyGU7UrfRS3FA@mail.gmail.com>
	<51F1380E.9070608@python.org>
Message-ID: <CAD+XWwrCD_tLM0b1OFea5ZZq7oH+1TYR_yNyp=mNDKJe7ZBALQ@mail.gmail.com>

On Thu, Jul 25, 2013 at 9:37 AM, Christian Heimes <christian at python.org> wrote:
> Am 25.07.2013 16:29, schrieb Eli Bendersky:
>> Hi all,
>>
>> I've been looking for a Github mirror for Python, and found two:
>>
>> * https://github.com/python-git/python has a lot of forks/watches/starts
>> but seems to be very out of date (last updated 4 years ago)
>> * https://github.com/python-mirror/python doesn't appear to be very
>> popular but is updated daily
>>
>> Are some of you the owners of these repositories? Should we consolidate
>> to a single "semi-official" mirror?
>
> +1
>
> Does the PSF have an official account on github? We have one on bitbucket...

I don't remember who runs this, and I thought I was in it (maybe just
on BB), but: https://github.com/python

From christian at python.org  Thu Jul 25 16:58:46 2013
From: christian at python.org (Christian Heimes)
Date: Thu, 25 Jul 2013 16:58:46 +0200
Subject: [Python-Dev] Official github mirror for CPython?
In-Reply-To: <CAD+XWwrCD_tLM0b1OFea5ZZq7oH+1TYR_yNyp=mNDKJe7ZBALQ@mail.gmail.com>
References: <CAF-Rda-=wFp_pWSa2gqFxet1M7pf68si=LKZ7FyGU7UrfRS3FA@mail.gmail.com>
	<51F1380E.9070608@python.org>
	<CAD+XWwrCD_tLM0b1OFea5ZZq7oH+1TYR_yNyp=mNDKJe7ZBALQ@mail.gmail.com>
Message-ID: <51F13D26.9090603@python.org>

Am 25.07.2013 16:48, schrieb Brian Curtin:
> On Thu, Jul 25, 2013 at 9:37 AM, Christian Heimes <christian at python.org> wrote:
>> Am 25.07.2013 16:29, schrieb Eli Bendersky:
>>> Hi all,
>>>
>>> I've been looking for a Github mirror for Python, and found two:
>>>
>>> * https://github.com/python-git/python has a lot of forks/watches/starts
>>> but seems to be very out of date (last updated 4 years ago)
>>> * https://github.com/python-mirror/python doesn't appear to be very
>>> popular but is updated daily
>>>
>>> Are some of you the owners of these repositories? Should we consolidate
>>> to a single "semi-official" mirror?
>>
>> +1
>>
>> Does the PSF have an official account on github? We have one on bitbucket...
> 
> I don't remember who runs this, and I thought I was in it (maybe just
> on BB), but: https://github.com/python

Jesse might know more about it. He is an admin of
https://bitbucket.org/PSF/, too.

Christian


From eliben at gmail.com  Thu Jul 25 17:18:34 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Thu, 25 Jul 2013 08:18:34 -0700
Subject: [Python-Dev] Official github mirror for CPython?
In-Reply-To: <CAD+XWwrCD_tLM0b1OFea5ZZq7oH+1TYR_yNyp=mNDKJe7ZBALQ@mail.gmail.com>
References: <CAF-Rda-=wFp_pWSa2gqFxet1M7pf68si=LKZ7FyGU7UrfRS3FA@mail.gmail.com>
	<51F1380E.9070608@python.org>
	<CAD+XWwrCD_tLM0b1OFea5ZZq7oH+1TYR_yNyp=mNDKJe7ZBALQ@mail.gmail.com>
Message-ID: <CAF-Rda82reci5WoY7J_1eZ_pBv+58sxXBaZ_MKSHef3onTuuRA@mail.gmail.com>

On Thu, Jul 25, 2013 at 7:48 AM, Brian Curtin <brian at python.org> wrote:

> On Thu, Jul 25, 2013 at 9:37 AM, Christian Heimes <christian at python.org>
> wrote:
> > Am 25.07.2013 16:29, schrieb Eli Bendersky:
> >> Hi all,
> >>
> >> I've been looking for a Github mirror for Python, and found two:
> >>
> >> * https://github.com/python-git/python has a lot of
> forks/watches/starts
> >> but seems to be very out of date (last updated 4 years ago)
> >> * https://github.com/python-mirror/python doesn't appear to be very
> >> popular but is updated daily
> >>
> >> Are some of you the owners of these repositories? Should we consolidate
> >> to a single "semi-official" mirror?
> >
> > +1
> >
> > Does the PSF have an official account on github? We have one on
> bitbucket...
>
> I don't remember who runs this, and I thought I was in it (maybe just
> on BB), but: https://github.com/python
>

This appears to be a good place, but it doesn't have an actual mirror
(yet?) ;-)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130725/8deaa0af/attachment.html>

From solipsis at pitrou.net  Thu Jul 25 17:26:58 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 25 Jul 2013 17:26:58 +0200
Subject: [Python-Dev] Daemon creation code in the standard library (was:
 Inherance of file descriptor and handles on Windows (PEP 446))
References: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
	<ksn359$q3a$1@ger.gmane.org>
	<CAP7+vJKGUS9cseDz+LgV0Z5DKJyYzEru6fAr1=Ct9GMBYB7i_w@mail.gmail.com>
	<7wy58vs86l.fsf_-_@benfinney.id.au>
Message-ID: <20130725172658.60fe7982@pitrou.net>

Le Thu, 25 Jul 2013 12:08:18 +1000,
Ben Finney <ben+python at benfinney.id.au> a ?crit :
> Guido van Rossum <guido at python.org> writes:
> 
> > To reduce the need for 3rd party subprocess creation code, we should
> > have better daemon creation code in the stdlib -- I wrote some damn
> > robust code for this purpose in my previous job, but it never saw
> > the light of day.
> 
> Work continues on the PEP 3143-compatible ?python-daemon?, porting it
> to Python 3 and aiming for inclusion in the standard library.

The PEP hasn't been formally accepted yet, however.
Skimming back through the archives, one sticking point was the default
value of the "umask" parameter. Setting the umask to 0 if the user
didn't ask for something else is a disaster, security-wise.

Another problem I've had when using it is that the `pidfile` combines
the notion of pidfile and process lock in one unique attribute. This
is quite inflexible when you're using something else than Skip
Montanaro's "lockfile" library. I'm using a separate lock based on
locket.py:
https://github.com/mwilliamson/locket.py
because it is based on POSIX advisory locks, and therefore doesn't
suffer from the "stale pid file" issues you get when a process
(or the whole system) crashes.

Therefore I'd be -1 on the PEP until those issues are alleviated.

Regards

Antoine.



From brett at python.org  Thu Jul 25 17:30:45 2013
From: brett at python.org (Brett Cannon)
Date: Thu, 25 Jul 2013 11:30:45 -0400
Subject: [Python-Dev] Official github mirror for CPython?
In-Reply-To: <CAF-Rda82reci5WoY7J_1eZ_pBv+58sxXBaZ_MKSHef3onTuuRA@mail.gmail.com>
References: <CAF-Rda-=wFp_pWSa2gqFxet1M7pf68si=LKZ7FyGU7UrfRS3FA@mail.gmail.com>
	<51F1380E.9070608@python.org>
	<CAD+XWwrCD_tLM0b1OFea5ZZq7oH+1TYR_yNyp=mNDKJe7ZBALQ@mail.gmail.com>
	<CAF-Rda82reci5WoY7J_1eZ_pBv+58sxXBaZ_MKSHef3onTuuRA@mail.gmail.com>
Message-ID: <CAP1=2W4rM8CVAn45Mz5L=eic+ivuJOGSwqQrEFucTUp=nop=UA@mail.gmail.com>

On Thu, Jul 25, 2013 at 11:18 AM, Eli Bendersky <eliben at gmail.com> wrote:

>
>
>
> On Thu, Jul 25, 2013 at 7:48 AM, Brian Curtin <brian at python.org> wrote:
>
>> On Thu, Jul 25, 2013 at 9:37 AM, Christian Heimes <christian at python.org>
>> wrote:
>> > Am 25.07.2013 16:29, schrieb Eli Bendersky:
>> >> Hi all,
>> >>
>> >> I've been looking for a Github mirror for Python, and found two:
>> >>
>> >> * https://github.com/python-git/python has a lot of
>> forks/watches/starts
>> >> but seems to be very out of date (last updated 4 years ago)
>> >> * https://github.com/python-mirror/python doesn't appear to be very
>> >> popular but is updated daily
>> >>
>> >> Are some of you the owners of these repositories? Should we consolidate
>> >> to a single "semi-official" mirror?
>> >
>> > +1
>> >
>> > Does the PSF have an official account on github? We have one on
>> bitbucket...
>>
>> I don't remember who runs this, and I thought I was in it (maybe just
>> on BB), but: https://github.com/python
>>
>
> This appears to be a good place, but it doesn't have an actual mirror
> (yet?) ;-)
>

Based on the list of people who are members of github.com/python it's as
official as it's going to get (depends on who of that group owns it).

But assuming whomever owns it is okay with hosting a mirror, what exactly
is going to be required to have it updated regularly? Someone is going to
have to write the scripts to pull from the hg repo and then push up to
github else it's just going to end up out-of-date on top of maintaining
whatever setup is devised.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130725/bfe3ba72/attachment.html>

From regebro at gmail.com  Thu Jul 25 18:03:42 2013
From: regebro at gmail.com (Lennart Regebro)
Date: Thu, 25 Jul 2013 18:03:42 +0200
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <51F111BD.5030008@gmail.com>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
	<CAL0kPAW97HnYU1ubD2bD64=ZazgrSTGBMajRA0eGxH4DiWkrfg@mail.gmail.com>
	<CAL0kPAU9-RR6OZiYerd5nJ7=70zDA5sQ1xt7tPDpG_2mr+1BqA@mail.gmail.com>
	<51EFFF22.1070603@gmail.com>
	<CAPTjJmosvJf4mwiqny=22No1e3FhV2UT52TdjCUogTc9LvrE+A@mail.gmail.com>
	<51F00D5B.9060005@gmail.com>
	<CAK5idxTgMGNbzFaGzDis78M6pq4NM9K2mLTUvUF-ieDkKSC8jQ@mail.gmail.com>
	<51F100DF.70204@gmail.com>
	<CAL0kPAWBabP5UNb+43LSnpjiX9_mSsp=bRumCXdswz0UPA=rKg@mail.gmail.com>
	<51F111BD.5030008@gmail.com>
Message-ID: <CAL0kPAV4fDCN26PYFEBQafGD7Zo7KwC4nTjm-iAk8D2t9ueSMw@mail.gmail.com>

On Thu, Jul 25, 2013 at 1:53 PM, Laurent Gautier <lgautier at gmail.com> wrote:
> My meaning was the use of 2to3's machinery (and fire a message if a
> translation occurs) not 2to3 itself, obviously.

I don't understand what you mean is the difference.

//Lennart

From a.badger at gmail.com  Thu Jul 25 18:08:14 2013
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Thu, 25 Jul 2013 09:08:14 -0700
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <CADiSq7eB71Or=nkzo7JLkYBz=AHHh9jkuWwZ1o5GTh48_6ZRvg@mail.gmail.com>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
	<CAP1=2W7y6w_f9D69czQCDv--580D4yvOD+6qenOL3T4E6b2ryg@mail.gmail.com>
	<CABVPEKp92VHGbLJjV_svedkLz5T22sxSHu6JMxKGm7Rx-VoB8Q@mail.gmail.com>
	<CADiSq7eB71Or=nkzo7JLkYBz=AHHh9jkuWwZ1o5GTh48_6ZRvg@mail.gmail.com>
Message-ID: <20130725160814.GM11402@unaka.lan>

On Thu, Jul 25, 2013 at 10:25:26PM +1000, Nick Coghlan wrote:
> On 25 July 2013 20:38, Toshio Kuratomi <a.badger at gmail.com> wrote:
> >
> > On Jul 24, 2013 6:37 AM, "Brett Cannon" <brett at python.org> wrote:
> >> The key, though, is adding python2 and getting your code to use that
> >> binary  specifically so that shifting the default name is more of a
> >> convenience than something which might break existing code not ready for the
> >> switch.
> >>
> > Applicable to this, does anyone know whether distutils, setuptools, distlib,
> > or any of the other standard build+install tools are doing shebang
> > requiring?  Are they doing the right thing wrt python vs python2?
> 
> It occurs to me they're almost certainly using "sys.executable" to set
> the shebang line, so probably not :(
> 
> distutils-sig could probably offer a better answer though, most of the
> packaging folks don't hang out here.
> 
Thanks!

For other Linux distributors following along, here's my message to
distutils-sig:

http://mail.python.org/pipermail/distutils-sig/2013-July/022001.html

-Toshio
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130725/aebd5ae7/attachment.pgp>

From fijall at gmail.com  Thu Jul 25 18:59:17 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 25 Jul 2013 18:59:17 +0200
Subject: [Python-Dev] Official github mirror for CPython?
In-Reply-To: <CAP1=2W4rM8CVAn45Mz5L=eic+ivuJOGSwqQrEFucTUp=nop=UA@mail.gmail.com>
References: <CAF-Rda-=wFp_pWSa2gqFxet1M7pf68si=LKZ7FyGU7UrfRS3FA@mail.gmail.com>
	<51F1380E.9070608@python.org>
	<CAD+XWwrCD_tLM0b1OFea5ZZq7oH+1TYR_yNyp=mNDKJe7ZBALQ@mail.gmail.com>
	<CAF-Rda82reci5WoY7J_1eZ_pBv+58sxXBaZ_MKSHef3onTuuRA@mail.gmail.com>
	<CAP1=2W4rM8CVAn45Mz5L=eic+ivuJOGSwqQrEFucTUp=nop=UA@mail.gmail.com>
Message-ID: <CAK5idxRCqnYidX-PE3-VBOLMPr35UT1uCrO9jqB9EBqReNgg9A@mail.gmail.com>

On Thu, Jul 25, 2013 at 5:30 PM, Brett Cannon <brett at python.org> wrote:
>
>
>
> On Thu, Jul 25, 2013 at 11:18 AM, Eli Bendersky <eliben at gmail.com> wrote:
>>
>>
>>
>>
>> On Thu, Jul 25, 2013 at 7:48 AM, Brian Curtin <brian at python.org> wrote:
>>>
>>> On Thu, Jul 25, 2013 at 9:37 AM, Christian Heimes <christian at python.org>
>>> wrote:
>>> > Am 25.07.2013 16:29, schrieb Eli Bendersky:
>>> >> Hi all,
>>> >>
>>> >> I've been looking for a Github mirror for Python, and found two:
>>> >>
>>> >> * https://github.com/python-git/python has a lot of
>>> >> forks/watches/starts
>>> >> but seems to be very out of date (last updated 4 years ago)
>>> >> * https://github.com/python-mirror/python doesn't appear to be very
>>> >> popular but is updated daily
>>> >>
>>> >> Are some of you the owners of these repositories? Should we
>>> >> consolidate
>>> >> to a single "semi-official" mirror?
>>> >
>>> > +1
>>> >
>>> > Does the PSF have an official account on github? We have one on
>>> > bitbucket...
>>>
>>> I don't remember who runs this, and I thought I was in it (maybe just
>>> on BB), but: https://github.com/python
>>
>>
>> This appears to be a good place, but it doesn't have an actual mirror
>> (yet?) ;-)
>
>
> Based on the list of people who are members of github.com/python it's as
> official as it's going to get (depends on who of that group owns it).
>
> But assuming whomever owns it is okay with hosting a mirror, what exactly is
> going to be required to have it updated regularly? Someone is going to have
> to write the scripts to pull from the hg repo and then push up to github
> else it's just going to end up out-of-date on top of maintaining whatever
> setup is devised.
>

It's slightly more work than this. You need to keep all the features
relying on hg work, like the revision version it was compiled from
etc.

From christian at python.org  Thu Jul 25 20:48:02 2013
From: christian at python.org (Christian Heimes)
Date: Thu, 25 Jul 2013 20:48:02 +0200
Subject: [Python-Dev] Coverity Scan
Message-ID: <ksrrsr$apk$1@ger.gmane.org>

Hello,

this is an update on my work and the current status of Coverity Scan.

Maybe you have noticed a checkins made be me that end with the line "CID
#". These are checkins that fix an issue that was discovered by the
static code analyzer Coverity. Coverity is a commercial product but it's
a free service for some Open Source projects. Python has been analyzed
by Coverity since about 2007. Guido, Neal, Brett, Stefan and some other
developers have used Coverity before I took over. I fixed a couple of
issues before 3.3 reached the RC phase and more bugs in the last couple
of months.

Coverity is really great and its web GUI is fun to use, too. I was able
to identify and fix resource leaks, NULL pointer issues, buffer
overflows and missing checks all over the place. Because it's a static
analyzer that follows data-flows and control-flows the tool can detect
issues in error paths that are hardly visited at all. I have started to
document Coverity here:

  http://docs.python.org/devguide/coverity.html


Interview
---------

A week ago I was contacted by Coverity. They have started a series of
articles and press releases about Open Source projects that use their
free service Coverity Scan, see


http://www.coverity.com/company/press-releases/read/coverity-introduces-monthly-spotlight-series-for-coverity-scan-open-source-projects

Two days ago I had a lovely phone interview about my involvement in the
Python project and our development style. They are going to release a
nice article in a couple of weeks. In the mean time we have time to fix
the remaining couple issues. We *might* be able to reach the highest
coverity integrity level! I have dealt with all major issues so we just
have to fix a couple of issues.


Current stats
-------------

Lines of Code:	396,179
Defect Density:	0.05
Total defects:	1,054
Outstanding:	   21 (Coverity Connect shows less)
Dismissed:	  222
Fixed:		  811

http://i.imgur.com/NoELjcj.jpg
http://i.imgur.com/eJSzTUX.jpg


open issues
-----------

http://bugs.python.org/issue17899
http://bugs.python.org/issue18556
http://bugs.python.org/issue18555
http://bugs.python.org/issue18552
http://bugs.python.org/issue18551
http://bugs.python.org/issue18550
http://bugs.python.org/issue18528


Christian


From lgautier at gmail.com  Thu Jul 25 21:16:56 2013
From: lgautier at gmail.com (Laurent Gautier)
Date: Thu, 25 Jul 2013 21:16:56 +0200
Subject: [Python-Dev] Python 3 as a Default in Linux Distros
In-Reply-To: <CAL0kPAV4fDCN26PYFEBQafGD7Zo7KwC4nTjm-iAk8D2t9ueSMw@mail.gmail.com>
References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com>
	<1903981217.6011829.1374657179869.JavaMail.root@redhat.com>
	<CAL0kPAW97HnYU1ubD2bD64=ZazgrSTGBMajRA0eGxH4DiWkrfg@mail.gmail.com>
	<CAL0kPAU9-RR6OZiYerd5nJ7=70zDA5sQ1xt7tPDpG_2mr+1BqA@mail.gmail.com>
	<51EFFF22.1070603@gmail.com>
	<CAPTjJmosvJf4mwiqny=22No1e3FhV2UT52TdjCUogTc9LvrE+A@mail.gmail.com>
	<51F00D5B.9060005@gmail.com>
	<CAK5idxTgMGNbzFaGzDis78M6pq4NM9K2mLTUvUF-ieDkKSC8jQ@mail.gmail.com>
	<51F100DF.70204@gmail.com>
	<CAL0kPAWBabP5UNb+43LSnpjiX9_mSsp=bRumCXdswz0UPA=rKg@mail.gmail.com>
	<51F111BD.5030008@gmail.com>
	<CAL0kPAV4fDCN26PYFEBQafGD7Zo7KwC4nTjm-iAk8D2t9ueSMw@mail.gmail.com>
Message-ID: <51F179A8.3030505@gmail.com>

On 07/25/2013 06:03 PM, Lennart Regebro wrote:
> On Thu, Jul 25, 2013 at 1:53 PM, Laurent Gautier <lgautier at gmail.com> wrote:
>> My meaning was the use of 2to3's machinery (and fire a message if a
>> translation occurs) not 2to3 itself, obviously.
> I don't understand what you mean is the difference.
>
> //Lennart

bullet-point version:

- 2to3 is a code translator

- to function, it needs to identify Python 2-only idioms so they can be 
translated

- the idea _was_ (*) to re-use that engine (identification-only, not 
translation) so the presence of Python 2-only idioms would cause a 
"Python 3 is now the default for "python", buddy. This script was 
seemingly written for Python 2, so you should run it with the command 
python2" sort of error message. The speed issue could be smaller than an 
full run of 2to3 since the error would be triggered at the first snippet 
of Python 2-only code encountered, but *.

(*: I withdrew the suggestion few emails back)

L.

From tjreedy at udel.edu  Fri Jul 26 00:00:55 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 25 Jul 2013 18:00:55 -0400
Subject: [Python-Dev] Coverity Scan
In-Reply-To: <ksrrsr$apk$1@ger.gmane.org>
References: <ksrrsr$apk$1@ger.gmane.org>
Message-ID: <kss76k$5l9$1@ger.gmane.org>

On 7/25/2013 2:48 PM, Christian Heimes wrote:
> Hello,
>
> this is an update on my work and the current status of Coverity Scan.

Great work.

>
> Maybe you have noticed a checkins made be me that end with the line "CID
> #". These are checkins that fix an issue that was discovered by the
> static code analyzer Coverity. Coverity is a commercial product but it's
> a free service for some Open Source projects. Python has been analyzed
> by Coverity since about 2007. Guido, Neal, Brett, Stefan and some other
> developers have used Coverity before I took over. I fixed a couple of
> issues before 3.3 reached the RC phase and more bugs in the last couple
> of months.

The benefit for us is not just improving Python having external 
verification of its excellence in relation both to other open-source 
projects and commercial software.

> Coverity is really great and its web GUI is fun to use, too. I was able
> to identify and fix resource leaks, NULL pointer issues, buffer
> overflows and missing checks all over the place. Because it's a static
> analyzer that follows data-flows and control-flows the tool can detect
> issues in error paths that are hardly visited at all. I have started to
> document Coverity here:
>
>    http://docs.python.org/devguide/coverity.html
>
>
> Interview
> ---------
>
> A week ago I was contacted by Coverity. They have started a series of
> articles and press releases about Open Source projects that use their
> free service Coverity Scan, see
>
> http://www.coverity.com/company/press-releases/read/coverity-introduces-monthly-spotlight-series-for-coverity-scan-open-source-projects

The intention is to promote the best of open source to industry.

> Two days ago I had a lovely phone interview about my involvement in the
> Python project and our development style. They are going to release a
> nice article in a couple of weeks. In the mean time we have time to fix
> the remaining couple issues. We *might* be able to reach the highest
> coverity integrity level! I have dealt with all major issues so we just
> have to fix a couple of issues.


> Current stats
> -------------
>
> Lines of Code:	396,179

C only? or does Python code now count as 'source code'?

> Defect Density:	0.05

= defects per thousand lines = 20/400

Anything under 1 is good. The release above reports Samba now at .6.
http://www.pcworld.com/article/2038244/linux-code-is-the-benchmark-of-quality-study-concludes.html
reports Linux 3.8 as having the same for 7.6 million lines.

> Total defects:	1,054
> Outstanding:	   21 (Coverity Connect shows less)
> Dismissed:	  222

This implies that they accept our designation of some things as False 
Positives or Intentional. Does Coverity do any review of such 
designations, so a project cannot cheat?

> Fixed:		  811
>
> http://i.imgur.com/NoELjcj.jpg
> http://i.imgur.com/eJSzTUX.jpg
>
>
> open issues
> -----------
>
> http://bugs.python.org/issue17899
> http://bugs.python.org/issue18556
> http://bugs.python.org/issue18555
> http://bugs.python.org/issue18552
> http://bugs.python.org/issue18551
> http://bugs.python.org/issue18550
> http://bugs.python.org/issue18528

-- 
Terry Jan Reedy


From tjreedy at udel.edu  Fri Jul 26 00:32:08 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 25 Jul 2013 18:32:08 -0400
Subject: [Python-Dev] Coverity Scan
In-Reply-To: <kss76k$5l9$1@ger.gmane.org>
References: <ksrrsr$apk$1@ger.gmane.org> <kss76k$5l9$1@ger.gmane.org>
Message-ID: <kss914$tar$1@ger.gmane.org>

On 7/25/2013 6:00 PM, Terry Reedy wrote:

>> Defect Density:    0.05
>
> = defects per thousand lines = 20/400
>
> Anything under 1 is good. The release above reports Samba now at .6.
> http://www.pcworld.com/article/2038244/linux-code-is-the-benchmark-of-quality-study-concludes.html
>
> reports Linux 3.8 as having the same for 7.6 million lines.
>
>> Total defects:    1,054
>> Outstanding:       21 (Coverity Connect shows less)
>> Dismissed:      222
>
> This implies that they accept our designation of some things as False
> Positives or Intentional. Does Coverity do any review of such
> designations, so a project cannot cheat?

I found the answer here
https://docs.google.com/file/d/0B5wQCOK_TiRiMWVqQ0xPaDEzbkU/edit
Coverity Integrity Level 1 is 1 (defect/1000 lines)
Level 2 is .1 (we have passed that)
Level 3 is .01 + no major defects + <20% (all all defects?) false 
positives as that is their normal rate.#

A higher false positive rates requires auditing by Coverity. They claim 
"A higher false positive rate indicates misconfiguration, usage of 
unusual idioms, or incorrect diagnosis of a large number of defects." 
They else add "or a flaw in our analysis."

# Since false positives should stay constant as true positives are 
reduced toward 0, false / all should tend toward 1 (100%) if I 
understand the ratio correctly.

>
>> Fixed:          811
>>
>> http://i.imgur.com/NoELjcj.jpg
>> http://i.imgur.com/eJSzTUX.jpg

-- 
Terry Jan Reedy


From solipsis at pitrou.net  Fri Jul 26 00:50:13 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 26 Jul 2013 00:50:13 +0200
Subject: [Python-Dev] Coverity Scan
References: <ksrrsr$apk$1@ger.gmane.org>
	<kss76k$5l9$1@ger.gmane.org>
Message-ID: <20130726005013.022300ff@fsol>

On Thu, 25 Jul 2013 18:00:55 -0400
Terry Reedy <tjreedy at udel.edu> wrote:
> On 7/25/2013 2:48 PM, Christian Heimes wrote:
> > Hello,
> >
> > this is an update on my work and the current status of Coverity Scan.
> 
> Great work.
> 
> >
> > Maybe you have noticed a checkins made be me that end with the line "CID
> > #". These are checkins that fix an issue that was discovered by the
> > static code analyzer Coverity. Coverity is a commercial product but it's
> > a free service for some Open Source projects. Python has been analyzed
> > by Coverity since about 2007. Guido, Neal, Brett, Stefan and some other
> > developers have used Coverity before I took over. I fixed a couple of
> > issues before 3.3 reached the RC phase and more bugs in the last couple
> > of months.
> 
> The benefit for us is not just improving Python having external 
> verification of its excellence in relation both to other open-source 
> projects and commercial software.

"Excellence"? The term is too weak, I would say "perfection" at least,
but perhaps we should go as far as "divinity".

Regards

Antoine.



From christian at python.org  Fri Jul 26 00:56:30 2013
From: christian at python.org (Christian Heimes)
Date: Fri, 26 Jul 2013 00:56:30 +0200
Subject: [Python-Dev] Coverity Scan
In-Reply-To: <kss914$tar$1@ger.gmane.org>
References: <ksrrsr$apk$1@ger.gmane.org> <kss76k$5l9$1@ger.gmane.org>
	<kss914$tar$1@ger.gmane.org>
Message-ID: <51F1AD1E.70309@python.org>

Am 26.07.2013 00:32, schrieb Terry Reedy:
> I found the answer here
> https://docs.google.com/file/d/0B5wQCOK_TiRiMWVqQ0xPaDEzbkU/edit
> Coverity Integrity Level 1 is 1 (defect/1000 lines)
> Level 2 is .1 (we have passed that)
> Level 3 is .01 + no major defects + <20% (all all defects?) false
> positives as that is their normal rate.#
> 
> A higher false positive rates requires auditing by Coverity. They claim
> "A higher false positive rate indicates misconfiguration, usage of
> unusual idioms, or incorrect diagnosis of a large number of defects."
> They else add "or a flaw in our analysis."
> 
> # Since false positives should stay constant as true positives are
> reduced toward 0, false / all should tend toward 1 (100%) if I
> understand the ratio correctly.

About 40% of the dismissed cases are cause by a handful of issues. I
have documented these issues as "known limitations"
http://docs.python.org/devguide/coverity.html#known-limitations .

For example about 35 false positives are related to PyLong_FromLong()
and our small integer optimization. A correct modeling file would
eliminate the false positive defects. My attempts don't work as hoped
and I don't have access to all professional coverity tools to debug my
trials.

Nearly 20 false positives are caused by Py_BuildValue("N"). I'm still
astonished that Coverity understands Python's reference counting most of
the time. :)

Did I mention that we have almost reached Level 3? All major defects
have been dealt with (one of them locally on the test machine until
Larry pushes his patch soonish), 4 of 7 minor issues must be closed and
our dismissed rate is just little over 20% (222 out of 1054 = 21%).

Christian




From christian at python.org  Fri Jul 26 01:02:12 2013
From: christian at python.org (Christian Heimes)
Date: Fri, 26 Jul 2013 01:02:12 +0200
Subject: [Python-Dev] Coverity Scan
In-Reply-To: <20130726005013.022300ff@fsol>
References: <ksrrsr$apk$1@ger.gmane.org> <kss76k$5l9$1@ger.gmane.org>
	<20130726005013.022300ff@fsol>
Message-ID: <51F1AE74.8050401@python.org>

Am 26.07.2013 00:50, schrieb Antoine Pitrou:
> "Excellence"? The term is too weak, I would say "perfection" at least,
> but perhaps we should go as far as "divinity".

Don't forget that Python can offer lots of places to keep your bike
clean and dry ... *scnr*

From christian at python.org  Fri Jul 26 01:20:01 2013
From: christian at python.org (Christian Heimes)
Date: Fri, 26 Jul 2013 01:20:01 +0200
Subject: [Python-Dev] Coverity Scan
In-Reply-To: <kss76k$5l9$1@ger.gmane.org>
References: <ksrrsr$apk$1@ger.gmane.org> <kss76k$5l9$1@ger.gmane.org>
Message-ID: <kssbqq$os$1@ger.gmane.org>

Am 26.07.2013 00:00, schrieb Terry Reedy:
>> http://www.coverity.com/company/press-releases/read/coverity-introduces-monthly-spotlight-series-for-coverity-scan-open-source-projects
>>
> 
> The intention is to promote the best of open source to industry.

I think it's also a marketing tool. They like to sell their product. I
don't have a problem with that. After all Coverity provides a useful
service for free that supplements our own debugging tools.

>> Lines of Code:    396,179
> 
> C only? or does Python code now count as 'source code'?

It's just C code and headers. Coverity doesn't analyze Python code.
According to cloc Python has 296707 + 78126 == 374833 lines of code in C
and header files. I'm not sure why Coverity detects more.

> 
>> Defect Density:    0.05
> 
> = defects per thousand lines = 20/400
> 
> Anything under 1 is good. The release above reports Samba now at .6.
> http://www.pcworld.com/article/2038244/linux-code-is-the-benchmark-of-quality-study-concludes.html
> 
> reports Linux 3.8 as having the same for 7.6 million lines.

These are amazing numbers. Python is much smaller.

> 
>> Total defects:    1,054
>> Outstanding:       21 (Coverity Connect shows less)
>> Dismissed:      222
> 
> This implies that they accept our designation of some things as False
> Positives or Intentional. Does Coverity do any review of such
> designations, so a project cannot cheat?

What's the point of cheating? :)

I could dismiss any remaining defect as intentionally or false positive
but that would only harm ourselves. As you already pointed out Coverity
reserves the right to inspect dismissed bugs for their highest ranking.

I'm in the process of looking through all dismissed defects. Some of
them are relics of deleted files and removed code. Some other may go
away with proper modeling.

Christian


From tjreedy at udel.edu  Fri Jul 26 01:24:34 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 25 Jul 2013 19:24:34 -0400
Subject: [Python-Dev] Coverity Scan
In-Reply-To: <51F1AD1E.70309@python.org>
References: <ksrrsr$apk$1@ger.gmane.org> <kss76k$5l9$1@ger.gmane.org>
	<kss914$tar$1@ger.gmane.org> <51F1AD1E.70309@python.org>
Message-ID: <kssc3e$5eq$1@ger.gmane.org>

On 7/25/2013 6:56 PM, Christian Heimes wrote:
> Am 26.07.2013 00:32, schrieb Terry Reedy:

>> # Since false positives should stay constant as true positives are
>> reduced toward 0, false / all should tend toward 1 (100%) if I
>> understand the ratio correctly.

Which I did not ;-).

> About 40% of the dismissed cases are cause by a handful of issues. I
> have documented these issues as "known limitations"
> http://docs.python.org/devguide/coverity.html#known-limitations .
>
> For example about 35 false positives are related to PyLong_FromLong()
> and our small integer optimization. A correct modeling file would
> eliminate the false positive defects. My attempts don't work as hoped
> and I don't have access to all professional coverity tools to debug my
> trials.

Perhaps Coverity will help when doing an audit.

> Nearly 20 false positives are caused by Py_BuildValue("N"). I'm still
> astonished that Coverity understands Python's reference counting most of
> the time. :)
>
> Did I mention that we have almost reached Level 3? All major defects

It is hard to measure the benefit of preventitive medicine, but I 
imagine that we should see fewer mysterious crashes and heisenbugs than 
we would have. In any case, Level 3 certification should help people 
promoting the use of Python in organizational settings, whether as 
employees or consultants.

> have been dealt with (one of them locally on the test machine until
> Larry pushes his patch soonish), 4 of 7 minor issues must be closed and

.1 * 390 allows 3 defects (or 4 if they round up) -- astonishingly good!

> our dismissed rate is just little over 20% (222 out of 1054 = 21%).

So merely verifying the 35 PyLong_FromLong dismissals will put us under.
Thanks for clarifying the proper denominator -- all defects ever found. 
It seems obvious in retrospect, but I was focused on current stats, not 
the history.

-- 
Terry Jan Reedy


From ncoghlan at gmail.com  Fri Jul 26 01:31:50 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 26 Jul 2013 09:31:50 +1000
Subject: [Python-Dev] Official github mirror for CPython?
In-Reply-To: <CAK5idxRCqnYidX-PE3-VBOLMPr35UT1uCrO9jqB9EBqReNgg9A@mail.gmail.com>
References: <CAF-Rda-=wFp_pWSa2gqFxet1M7pf68si=LKZ7FyGU7UrfRS3FA@mail.gmail.com>
	<51F1380E.9070608@python.org>
	<CAD+XWwrCD_tLM0b1OFea5ZZq7oH+1TYR_yNyp=mNDKJe7ZBALQ@mail.gmail.com>
	<CAF-Rda82reci5WoY7J_1eZ_pBv+58sxXBaZ_MKSHef3onTuuRA@mail.gmail.com>
	<CAP1=2W4rM8CVAn45Mz5L=eic+ivuJOGSwqQrEFucTUp=nop=UA@mail.gmail.com>
	<CAK5idxRCqnYidX-PE3-VBOLMPr35UT1uCrO9jqB9EBqReNgg9A@mail.gmail.com>
Message-ID: <CADiSq7fj3RtyfB688JBYF9myc_U4B8BhED2EUjuM3y+xkKvddQ@mail.gmail.com>

On 26 Jul 2013 03:02, "Maciej Fijalkowski" <fijall at gmail.com> wrote:
>
> On Thu, Jul 25, 2013 at 5:30 PM, Brett Cannon <brett at python.org> wrote:
> >
> >
> >
> > On Thu, Jul 25, 2013 at 11:18 AM, Eli Bendersky <eliben at gmail.com>
wrote:
> >>
> >>
> >>
> >>
> >> On Thu, Jul 25, 2013 at 7:48 AM, Brian Curtin <brian at python.org> wrote:
> >>>
> >>> On Thu, Jul 25, 2013 at 9:37 AM, Christian Heimes <
christian at python.org>
> >>> wrote:
> >>> > Am 25.07.2013 16:29, schrieb Eli Bendersky:
> >>> >> Hi all,
> >>> >>
> >>> >> I've been looking for a Github mirror for Python, and found two:
> >>> >>
> >>> >> * https://github.com/python-git/python has a lot of
> >>> >> forks/watches/starts
> >>> >> but seems to be very out of date (last updated 4 years ago)
> >>> >> * https://github.com/python-mirror/python doesn't appear to be very
> >>> >> popular but is updated daily
> >>> >>
> >>> >> Are some of you the owners of these repositories? Should we
> >>> >> consolidate
> >>> >> to a single "semi-official" mirror?
> >>> >
> >>> > +1
> >>> >
> >>> > Does the PSF have an official account on github? We have one on
> >>> > bitbucket...
> >>>
> >>> I don't remember who runs this, and I thought I was in it (maybe just
> >>> on BB), but: https://github.com/python
> >>
> >>
> >> This appears to be a good place, but it doesn't have an actual mirror
> >> (yet?) ;-)
> >
> >
> > Based on the list of people who are members of github.com/python it's as
> > official as it's going to get (depends on who of that group owns it).
> >
> > But assuming whomever owns it is okay with hosting a mirror, what
exactly is
> > going to be required to have it updated regularly? Someone is going to
have
> > to write the scripts to pull from the hg repo and then push up to github
> > else it's just going to end up out-of-date on top of maintaining
whatever
> > setup is devised.
> >
>
> It's slightly more work than this. You need to keep all the features
> relying on hg work, like the revision version it was compiled from
> etc.

To be honest, if people are going to spend time tinkering with our VCS
infrastructure, one of the most interesting things we could do is explore
what would be involved in setting up RhodeCode on hg.python.org :)

(For those that haven't seen it, RhodeCode seems broadly comparable to
BitBucket feature wise, but because of the way it is licensed, the source
code is freely available to all, and running your own instance is
free-as-in-beer for non-profits and open source projects).

Cheers,
Nick.

> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130726/17cb12e0/attachment.html>

From cs at zip.com.au  Fri Jul 26 05:48:14 2013
From: cs at zip.com.au (Cameron Simpson)
Date: Fri, 26 Jul 2013 13:48:14 +1000
Subject: [Python-Dev] Daemon creation code in the standard library (was:
 Inherance of file descriptor and handles on Windows (PEP 446))
In-Reply-To: <20130725172658.60fe7982@pitrou.net>
References: <20130725172658.60fe7982@pitrou.net>
Message-ID: <20130726034814.GA58699@cskk.homeip.net>

On 25Jul2013 17:26, Antoine Pitrou <solipsis at pitrou.net> wrote:
| Le Thu, 25 Jul 2013 12:08:18 +1000,
| Ben Finney <ben+python at benfinney.id.au> a ?crit :
| > Work continues on the PEP 3143-compatible ?python-daemon?, porting it
| > to Python 3 and aiming for inclusion in the standard library.
| 
| The PEP hasn't been formally accepted yet, however.
| Skimming back through the archives, one sticking point was the default
| value of the "umask" parameter. Setting the umask to 0 if the user
| didn't ask for something else is a disaster, security-wise.

I have to say, +10 here.

I have always found the convention that daemons have a umask of 0
to be utterly bogus, because almost all library code relies on the
umask to set default security policy for initial file permissions.

Prone to rant on this at length if required...

Cheers,
-- 
Cameron Simpson <cs at zip.com.au>

Cordless hoses have been around for quite some time. They're called buckets.
        - Dan Prener <prener at watson.ibm.com>

From ben+python at benfinney.id.au  Fri Jul 26 09:11:35 2013
From: ben+python at benfinney.id.au (Ben Finney)
Date: Fri, 26 Jul 2013 17:11:35 +1000
Subject: [Python-Dev] Daemon creation code in the standard library
References: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
	<ksn359$q3a$1@ger.gmane.org>
	<CAP7+vJKGUS9cseDz+LgV0Z5DKJyYzEru6fAr1=Ct9GMBYB7i_w@mail.gmail.com>
	<7wy58vs86l.fsf_-_@benfinney.id.au>
	<20130725172658.60fe7982@pitrou.net>
Message-ID: <7wsiz1bxso.fsf@benfinney.id.au>

Antoine Pitrou <solipsis at pitrou.net> writes:

> Therefore I'd be -1 on [PEP 3143] until those issues are alleviated.

Cameron Simpson <cs at zip.com.au> writes:

> I have always found the convention that daemons have a umask of 0
> to be utterly bogus, because almost all library code relies on the
> umask to set default security policy for initial file permissions.
>
> Prone to rant on this at length if required...


Thanks folks. We'd love to have this discussion over at the
?python-daemon-devel? discussion forum if you want to have it in more
detail.

-- 
 \     ?We are all agreed that your theory is crazy. The question that |
  `\      divides us is whether it is crazy enough to have a chance of |
_o__)            being correct.? ?Niels Bohr (to Wolfgang Pauli), 1958 |
Ben Finney


From ronaldoussoren at mac.com  Fri Jul 26 09:38:10 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Fri, 26 Jul 2013 09:38:10 +0200
Subject: [Python-Dev] Daemon creation code in the standard library
In-Reply-To: <7w1u6njsap.fsf@benfinney.id.au>
References: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
	<ksn359$q3a$1@ger.gmane.org>
	<CAP7+vJKGUS9cseDz+LgV0Z5DKJyYzEru6fAr1=Ct9GMBYB7i_w@mail.gmail.com>
	<7wy58vs86l.fsf_-_@benfinney.id.au>
	<7w1u6njsap.fsf@benfinney.id.au>
Message-ID: <54C49643-17F7-4CAE-AAB4-1FD143DC22E5@mac.com>


On 25 Jul, 2013, at 4:18, Ben Finney <ben+python at benfinney.id.au> wrote:

> Ben Finney <ben+python at benfinney.id.au> writes:
> 
>> Work continues on the PEP 3143-compatible ?python-daemon?, porting it to
>> Python 3 and aiming for inclusion in the standard library.

At first glance the library appears to close all open files, with an option
to exclude some specific file descriptors (that is, you need to pass a list
of files that shouldn't be closed). 

That makes it a lot harder to do some initialization before daemonizing.
I prefer to perform at least some initialization early in program startup to
be able to give sensible error messages. I've had too many initscripts that
claimed to have started a daemon sucessfully, only to have that daemon stop
right away because it noticed a problem right after it detached itself.

Ronald

> 
> At PyPI <URL:http://pypi.python.org/pypi/python-daemon/>, and
> development co-ordinated at Alioth
> <URL:https://alioth.debian.org/projects/python-daemon/>.
> 
>> Interested parties are invited to join us on the discussion forums
> 
> The correct link for the ?python-daemon-devel? forum is
> <URL:http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/python-daemon-devel>.
> For announcements only, we have 
> <URL:http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/python-daemon-announce>.
> 
> -- 
> \        ?This sentence contradicts itself ? no actually it doesn't.? |
>  `\                                               ?Douglas Hofstadter |
> _o__)                                                                  |
> Ben Finney
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com


From solipsis at pitrou.net  Fri Jul 26 09:50:47 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 26 Jul 2013 09:50:47 +0200
Subject: [Python-Dev] Official github mirror for CPython?
References: <CAF-Rda-=wFp_pWSa2gqFxet1M7pf68si=LKZ7FyGU7UrfRS3FA@mail.gmail.com>
	<51F1380E.9070608@python.org>
	<CAD+XWwrCD_tLM0b1OFea5ZZq7oH+1TYR_yNyp=mNDKJe7ZBALQ@mail.gmail.com>
	<CAF-Rda82reci5WoY7J_1eZ_pBv+58sxXBaZ_MKSHef3onTuuRA@mail.gmail.com>
	<CAP1=2W4rM8CVAn45Mz5L=eic+ivuJOGSwqQrEFucTUp=nop=UA@mail.gmail.com>
	<CAK5idxRCqnYidX-PE3-VBOLMPr35UT1uCrO9jqB9EBqReNgg9A@mail.gmail.com>
	<CADiSq7fj3RtyfB688JBYF9myc_U4B8BhED2EUjuM3y+xkKvddQ@mail.gmail.com>
Message-ID: <20130726095047.7c55e7d2@pitrou.net>

Le Fri, 26 Jul 2013 09:31:50 +1000,
Nick Coghlan <ncoghlan at gmail.com> a ?crit :
> 
> To be honest, if people are going to spend time tinkering with our VCS
> infrastructure, one of the most interesting things we could do is
> explore what would be involved in setting up RhodeCode on
> hg.python.org :)
> 
> (For those that haven't seen it, RhodeCode seems broadly comparable to
> BitBucket feature wise, but because of the way it is licensed, the
> source code is freely available to all, and running your own instance
> is free-as-in-beer for non-profits and open source projects).

By "freely available", do you mean actual open source / free software?

Regards

Antoine.



From p.f.moore at gmail.com  Fri Jul 26 10:04:39 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 26 Jul 2013 09:04:39 +0100
Subject: [Python-Dev] Official github mirror for CPython?
In-Reply-To: <20130726095047.7c55e7d2@pitrou.net>
References: <CAF-Rda-=wFp_pWSa2gqFxet1M7pf68si=LKZ7FyGU7UrfRS3FA@mail.gmail.com>
	<51F1380E.9070608@python.org>
	<CAD+XWwrCD_tLM0b1OFea5ZZq7oH+1TYR_yNyp=mNDKJe7ZBALQ@mail.gmail.com>
	<CAF-Rda82reci5WoY7J_1eZ_pBv+58sxXBaZ_MKSHef3onTuuRA@mail.gmail.com>
	<CAP1=2W4rM8CVAn45Mz5L=eic+ivuJOGSwqQrEFucTUp=nop=UA@mail.gmail.com>
	<CAK5idxRCqnYidX-PE3-VBOLMPr35UT1uCrO9jqB9EBqReNgg9A@mail.gmail.com>
	<CADiSq7fj3RtyfB688JBYF9myc_U4B8BhED2EUjuM3y+xkKvddQ@mail.gmail.com>
	<20130726095047.7c55e7d2@pitrou.net>
Message-ID: <CACac1F-MX7Yuv2op4tR0xMO1nMGpupTdvx2vW8mzTSHa80=cQw@mail.gmail.com>

On 26 July 2013 08:50, Antoine Pitrou <solipsis at pitrou.net> wrote:

> > (For those that haven't seen it, RhodeCode seems broadly comparable to
> > BitBucket feature wise, but because of the way it is licensed, the
> > source code is freely available to all, and running your own instance
> > is free-as-in-beer for non-profits and open source projects).
>
> By "freely available", do you mean actual open source / free software?
>

Looks like it - from the website: "Open source source control management
system for Mercurial and GIT with code-reviews, built in push/pull server,
LDAP/AD, permissions system and full text search."

Paul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130726/ac6ca70c/attachment.html>

From ronaldoussoren at mac.com  Fri Jul 26 10:05:28 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Fri, 26 Jul 2013 10:05:28 +0200
Subject: [Python-Dev] Official github mirror for CPython?
In-Reply-To: <20130726095047.7c55e7d2@pitrou.net>
References: <CAF-Rda-=wFp_pWSa2gqFxet1M7pf68si=LKZ7FyGU7UrfRS3FA@mail.gmail.com>
	<51F1380E.9070608@python.org>
	<CAD+XWwrCD_tLM0b1OFea5ZZq7oH+1TYR_yNyp=mNDKJe7ZBALQ@mail.gmail.com>
	<CAF-Rda82reci5WoY7J_1eZ_pBv+58sxXBaZ_MKSHef3onTuuRA@mail.gmail.com>
	<CAP1=2W4rM8CVAn45Mz5L=eic+ivuJOGSwqQrEFucTUp=nop=UA@mail.gmail.com>
	<CAK5idxRCqnYidX-PE3-VBOLMPr35UT1uCrO9jqB9EBqReNgg9A@mail.gmail.com>
	<CADiSq7fj3RtyfB688JBYF9myc_U4B8BhED2EUjuM3y+xkKvddQ@mail.gmail.com>
	<20130726095047.7c55e7d2@pitrou.net>
Message-ID: <3A462190-3B65-44ED-B699-9B0B8B1BC775@mac.com>


On 26 Jul, 2013, at 9:50, Antoine Pitrou <solipsis at pitrou.net> wrote:

> Le Fri, 26 Jul 2013 09:31:50 +1000,
> Nick Coghlan <ncoghlan at gmail.com> a ?crit :
>> 
>> To be honest, if people are going to spend time tinkering with our VCS
>> infrastructure, one of the most interesting things we could do is
>> explore what would be involved in setting up RhodeCode on
>> hg.python.org :)
>> 
>> (For those that haven't seen it, RhodeCode seems broadly comparable to
>> BitBucket feature wise, but because of the way it is licensed, the
>> source code is freely available to all, and running your own instance
>> is free-as-in-beer for non-profits and open source projects).
> 
> By "freely available", do you mean actual open source / free software?

It appears to be GPLv3, with a for-pay enterprise edition. The latter is
free-as-in-beer for non-profits and open source projects.

Ronald

(See <https://secure.rhodecode.org/rhodecode/files/433d6385b216da52f68fa871ed1ff99f8d618613/README.rst>)


From dirkjan at ochtman.nl  Fri Jul 26 10:03:26 2013
From: dirkjan at ochtman.nl (Dirkjan Ochtman)
Date: Fri, 26 Jul 2013 10:03:26 +0200
Subject: [Python-Dev] Official github mirror for CPython?
In-Reply-To: <20130726095047.7c55e7d2@pitrou.net>
References: <CAF-Rda-=wFp_pWSa2gqFxet1M7pf68si=LKZ7FyGU7UrfRS3FA@mail.gmail.com>
	<51F1380E.9070608@python.org>
	<CAD+XWwrCD_tLM0b1OFea5ZZq7oH+1TYR_yNyp=mNDKJe7ZBALQ@mail.gmail.com>
	<CAF-Rda82reci5WoY7J_1eZ_pBv+58sxXBaZ_MKSHef3onTuuRA@mail.gmail.com>
	<CAP1=2W4rM8CVAn45Mz5L=eic+ivuJOGSwqQrEFucTUp=nop=UA@mail.gmail.com>
	<CAK5idxRCqnYidX-PE3-VBOLMPr35UT1uCrO9jqB9EBqReNgg9A@mail.gmail.com>
	<CADiSq7fj3RtyfB688JBYF9myc_U4B8BhED2EUjuM3y+xkKvddQ@mail.gmail.com>
	<20130726095047.7c55e7d2@pitrou.net>
Message-ID: <CAKmKYaC3Y-+zG3FJxiMzJ4bzrd4mt3YYq1hoWK-R_vrFAdG=ig@mail.gmail.com>

On Fri, Jul 26, 2013 at 9:50 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> (For those that haven't seen it, RhodeCode seems broadly comparable to
>> BitBucket feature wise, but because of the way it is licensed, the
>> source code is freely available to all, and running your own instance
>> is free-as-in-beer for non-profits and open source projects).
>
> By "freely available", do you mean actual open source / free software?

It seems to be licensed under the GPLv3.

https://secure.rhodecode.org/rhodecode/files/433d6385b216da52f68fa871ed1ff99f8d618613/COPYING
https://rhodecode.com/blog/25/new-rhodecode-licensing

Cheers,

Dirkjan

From solipsis at pitrou.net  Fri Jul 26 10:14:27 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 26 Jul 2013 10:14:27 +0200
Subject: [Python-Dev] Official github mirror for CPython?
References: <CAF-Rda-=wFp_pWSa2gqFxet1M7pf68si=LKZ7FyGU7UrfRS3FA@mail.gmail.com>
	<51F1380E.9070608@python.org>
	<CAD+XWwrCD_tLM0b1OFea5ZZq7oH+1TYR_yNyp=mNDKJe7ZBALQ@mail.gmail.com>
	<CAF-Rda82reci5WoY7J_1eZ_pBv+58sxXBaZ_MKSHef3onTuuRA@mail.gmail.com>
	<CAP1=2W4rM8CVAn45Mz5L=eic+ivuJOGSwqQrEFucTUp=nop=UA@mail.gmail.com>
	<CAK5idxRCqnYidX-PE3-VBOLMPr35UT1uCrO9jqB9EBqReNgg9A@mail.gmail.com>
	<CADiSq7fj3RtyfB688JBYF9myc_U4B8BhED2EUjuM3y+xkKvddQ@mail.gmail.com>
	<20130726095047.7c55e7d2@pitrou.net>
	<CAKmKYaC3Y-+zG3FJxiMzJ4bzrd4mt3YYq1hoWK-R_vrFAdG=ig@mail.gmail.com>
Message-ID: <20130726101427.6ecb58be@pitrou.net>

Le Fri, 26 Jul 2013 10:03:26 +0200,
Dirkjan Ochtman <dirkjan at ochtman.nl> a ?crit :
> On Fri, Jul 26, 2013 at 9:50 AM, Antoine Pitrou <solipsis at pitrou.net>
> wrote:
> >> (For those that haven't seen it, RhodeCode seems broadly
> >> comparable to BitBucket feature wise, but because of the way it is
> >> licensed, the source code is freely available to all, and running
> >> your own instance is free-as-in-beer for non-profits and open
> >> source projects).
> >
> > By "freely available", do you mean actual open source / free
> > software?
> 
> It seems to be licensed under the GPLv3.
> 
> https://secure.rhodecode.org/rhodecode/files/433d6385b216da52f68fa871ed1ff99f8d618613/COPYING
> https://rhodecode.com/blog/25/new-rhodecode-licensing

Ah, right. Apparently a certain @ncoghlan_dev contributed to that...

Regards

Antoine.



From ncoghlan at gmail.com  Fri Jul 26 12:51:16 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 26 Jul 2013 20:51:16 +1000
Subject: [Python-Dev] Official github mirror for CPython?
In-Reply-To: <20130726101427.6ecb58be@pitrou.net>
References: <CAF-Rda-=wFp_pWSa2gqFxet1M7pf68si=LKZ7FyGU7UrfRS3FA@mail.gmail.com>
	<51F1380E.9070608@python.org>
	<CAD+XWwrCD_tLM0b1OFea5ZZq7oH+1TYR_yNyp=mNDKJe7ZBALQ@mail.gmail.com>
	<CAF-Rda82reci5WoY7J_1eZ_pBv+58sxXBaZ_MKSHef3onTuuRA@mail.gmail.com>
	<CAP1=2W4rM8CVAn45Mz5L=eic+ivuJOGSwqQrEFucTUp=nop=UA@mail.gmail.com>
	<CAK5idxRCqnYidX-PE3-VBOLMPr35UT1uCrO9jqB9EBqReNgg9A@mail.gmail.com>
	<CADiSq7fj3RtyfB688JBYF9myc_U4B8BhED2EUjuM3y+xkKvddQ@mail.gmail.com>
	<20130726095047.7c55e7d2@pitrou.net>
	<CAKmKYaC3Y-+zG3FJxiMzJ4bzrd4mt3YYq1hoWK-R_vrFAdG=ig@mail.gmail.com>
	<20130726101427.6ecb58be@pitrou.net>
Message-ID: <CADiSq7d_a+b0jjZm1x2-caRKN+9FkjisKacO+xFpLTRakgNQ1g@mail.gmail.com>

On 26 July 2013 18:14, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Le Fri, 26 Jul 2013 10:03:26 +0200,
> Dirkjan Ochtman <dirkjan at ochtman.nl> a ?crit :
>> On Fri, Jul 26, 2013 at 9:50 AM, Antoine Pitrou <solipsis at pitrou.net>
>> wrote:
>> >> (For those that haven't seen it, RhodeCode seems broadly
>> >> comparable to BitBucket feature wise, but because of the way it is
>> >> licensed, the source code is freely available to all, and running
>> >> your own instance is free-as-in-beer for non-profits and open
>> >> source projects).
>> >
>> > By "freely available", do you mean actual open source / free
>> > software?
>>
>> It seems to be licensed under the GPLv3.
>>
>> https://secure.rhodecode.org/rhodecode/files/433d6385b216da52f68fa871ed1ff99f8d618613/COPYING
>> https://rhodecode.com/blog/25/new-rhodecode-licensing
>
> Ah, right. Apparently a certain @ncoghlan_dev contributed to that...

Yeah, I've been publicly lamenting the closed source nature of GitHub
and BitBucket for a while now, dislike the Allura UI enough that "but
it's open source!" wasn't enough to win me over, and most of the other
free hosting options don't support Mercurial (or have anything
resembling a coherent business model). When RhodeCode brought their
hosted version to my attention, it aligned so well with what I had
been saying I wanted (a code hosting service making a genuine
commitment to openness, with a good user experience and an apparently
viable business model) that I felt *obliged* to check it out. While I
haven't migrated all my personal repos yet (due to some domain name
issues specific to the Hosted version), I was quite happy with the
user experience of the tool itself and plan to do that migration
eventually :)

As far as the licensing goes, rereading the comments below the
licensing announcement as well as the licenses themselves:

1. Versions prior to 2.0 (latest: 1.7.1) are already published under GPLv3
2. RhodeCode 2.0 (aka "RhodeCode Enterprise") will be the first
published using a Business Source license
3. The Business Source licensing means that 2.0 *will* become open
source under "GPLv2 or later" terms in August 2015 - RhodeCode won't
have the option of changing their mind and deciding not to open source
it after all. For the next two years, even though the source will be
made available, it's not technically an open source project yet, since
free usage under the business source license is limited to
installations with no more than 20 Active users. More users than that
requires a commercial license, which you either pay for, or is offered
for free to schools and universities, public open source projects and
also for development, environment and human rights NGOs.

I believe Hosted is already running 2.0, so that's the one that
impressed me. I'm not sure what 1.7.1 is like, since I have never used
it :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From solipsis at pitrou.net  Fri Jul 26 13:46:14 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 26 Jul 2013 13:46:14 +0200
Subject: [Python-Dev] Official github mirror for CPython?
References: <CAF-Rda-=wFp_pWSa2gqFxet1M7pf68si=LKZ7FyGU7UrfRS3FA@mail.gmail.com>
	<51F1380E.9070608@python.org>
	<CAD+XWwrCD_tLM0b1OFea5ZZq7oH+1TYR_yNyp=mNDKJe7ZBALQ@mail.gmail.com>
	<CAF-Rda82reci5WoY7J_1eZ_pBv+58sxXBaZ_MKSHef3onTuuRA@mail.gmail.com>
	<CAP1=2W4rM8CVAn45Mz5L=eic+ivuJOGSwqQrEFucTUp=nop=UA@mail.gmail.com>
	<CAK5idxRCqnYidX-PE3-VBOLMPr35UT1uCrO9jqB9EBqReNgg9A@mail.gmail.com>
	<CADiSq7fj3RtyfB688JBYF9myc_U4B8BhED2EUjuM3y+xkKvddQ@mail.gmail.com>
	<20130726095047.7c55e7d2@pitrou.net>
	<CAKmKYaC3Y-+zG3FJxiMzJ4bzrd4mt3YYq1hoWK-R_vrFAdG=ig@mail.gmail.com>
	<20130726101427.6ecb58be@pitrou.net>
	<CADiSq7d_a+b0jjZm1x2-caRKN+9FkjisKacO+xFpLTRakgNQ1g@mail.gmail.com>
Message-ID: <20130726134614.5de465b6@pitrou.net>

Le Fri, 26 Jul 2013 20:51:16 +1000,
Nick Coghlan <ncoghlan at gmail.com> a ?crit :
> 
> I believe Hosted is already running 2.0, so that's the one that
> impressed me.

Strange. Apparently, free trial accounts get a 1.7.1 version:
"RhodeCode 1.7.1.hosted.4356".
From my POV, it would need a ton of UI polish before being pleasant
to use...

Also, it's not obvious whether you can integrate your repo with a
bug tracker and a code review tool.

Regards

Antoine.



From ncoghlan at gmail.com  Fri Jul 26 14:06:55 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 26 Jul 2013 22:06:55 +1000
Subject: [Python-Dev] Official github mirror for CPython?
In-Reply-To: <20130726134614.5de465b6@pitrou.net>
References: <CAF-Rda-=wFp_pWSa2gqFxet1M7pf68si=LKZ7FyGU7UrfRS3FA@mail.gmail.com>
	<51F1380E.9070608@python.org>
	<CAD+XWwrCD_tLM0b1OFea5ZZq7oH+1TYR_yNyp=mNDKJe7ZBALQ@mail.gmail.com>
	<CAF-Rda82reci5WoY7J_1eZ_pBv+58sxXBaZ_MKSHef3onTuuRA@mail.gmail.com>
	<CAP1=2W4rM8CVAn45Mz5L=eic+ivuJOGSwqQrEFucTUp=nop=UA@mail.gmail.com>
	<CAK5idxRCqnYidX-PE3-VBOLMPr35UT1uCrO9jqB9EBqReNgg9A@mail.gmail.com>
	<CADiSq7fj3RtyfB688JBYF9myc_U4B8BhED2EUjuM3y+xkKvddQ@mail.gmail.com>
	<20130726095047.7c55e7d2@pitrou.net>
	<CAKmKYaC3Y-+zG3FJxiMzJ4bzrd4mt3YYq1hoWK-R_vrFAdG=ig@mail.gmail.com>
	<20130726101427.6ecb58be@pitrou.net>
	<CADiSq7d_a+b0jjZm1x2-caRKN+9FkjisKacO+xFpLTRakgNQ1g@mail.gmail.com>
	<20130726134614.5de465b6@pitrou.net>
Message-ID: <CADiSq7ccCCyQXv5EnYak7JL4pa5ZP2f3ivG_xaArrg6yiNcy9A@mail.gmail.com>

On 26 July 2013 21:46, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Le Fri, 26 Jul 2013 20:51:16 +1000,
> Nick Coghlan <ncoghlan at gmail.com> a ?crit :
>>
>> I believe Hosted is already running 2.0, so that's the one that
>> impressed me.
>
> Strange. Apparently, free trial accounts get a 1.7.1 version:
> "RhodeCode 1.7.1.hosted.4356".
> From my POV, it would need a ton of UI polish before being pleasant
> to use...

I tolerate Bugzilla and Gerrit every work day, which may have affected
my standards for what I judge to be an acceptable user interface ;)

> Also, it's not obvious whether you can integrate your repo with a
> bug tracker and a code review tool.

A self-hosted instance still has Mercurial on the backend, so any
Mercurial based integration should still work, even if it isn't
configurable through the front end.

However, I hadn't actually brought it up previously because I know
there would be a *lot* of work in taking it from "vague notion" to
"concrete infrastructure update proposal with a solid rationale behind
it" and I'm not in a position to do that work myself. I just couldn't
resist mentioning it once the idea of improved GitHub syncing was
brought up :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From victor.stinner at gmail.com  Fri Jul 26 14:08:35 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Fri, 26 Jul 2013 14:08:35 +0200
Subject: [Python-Dev] Inherance of file descriptor and handles on
 Windows (PEP 446)
In-Reply-To: <CAP7+vJKGUS9cseDz+LgV0Z5DKJyYzEru6fAr1=Ct9GMBYB7i_w@mail.gmail.com>
References: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
	<ksn359$q3a$1@ger.gmane.org>
	<CAP7+vJKGUS9cseDz+LgV0Z5DKJyYzEru6fAr1=Ct9GMBYB7i_w@mail.gmail.com>
Message-ID: <CAMpsgwZq9qR-BBSoXNDwZq=VYmYcv5AMgjv7bPnHBhPLfse6HA@mail.gmail.com>

2013/7/24 Guido van Rossum <guido at python.org>:
> But I'm also ready to propose that all this is such a mess that we
> *should* change the default fd/handle inheritance to False, *across
> platforms*, and damn the torpedoes -- i.e. accept breaking all
> existing 3rd party UNIX code for subprocess creation that bypasses the
> subprocess module, as well as breaking uses of os.spawn*() on both
> platforms that depend on FD inheritance beyond stdin/stdout/stderr).
>
> With the new, sane default, all we need instead of PEP 446 is a way to
> make an FD inheritable after it's been created, which can be a single
> os.make_inheritable(fd) call that you must apply to the fileno() of
> the stream or socket object you want inherited (or directly to a FD
> you created otherwise, e.g. with os.pipe()). On Windows, this should
> probably only work with os.spawn*(), since otherwise you need *handle*
> inheritance, not *FD* inheritance, and that's a non-portable concept
> anyway.

After having written 2 PEP on the topic, I slowly agree that make all
file descriptors non-inheritable is the best *compromise*. It solves
most, or all, issues.

The main drawback is the additionnal syscalls: on some platforms, 2
additional syscalls are need to make a file descriptor non-inheritable
for each creation of file descriptor. According to my benchmark on the
implementation of the PEP 433: the overhead of making a file
descriptor non-inheritable is between 1% and 3% (7.8 ?s => 7.9 or 8.0
?s) on Linux 3.6.
http://www.python.org/dev/peps/pep-0433/#performances

Having to make a file descriptor inheritable after creating it
non-inheritable is also not optimal. Making it first non-inheritable
requires 0, 1 or 2 extra syscalls, and making it inheritable again
require also 1 or 2 syscalls. So f=open(...);
os.make_inheritable(f.fileno()) can take up to 5 syscalls (1 open + 4
fnctl), whereas it can be done in only 1 syscall (1 open). One of the
motivation of the PEP 433 an 446 is to reduce the number of syscalls,
even if the use case was to make sockets *non-inheritable*.

If we consider that the most common case is to use non-inheritable
file descriptors, having to call os.make_inheritable() may be
acceptable. Windows and recent operating syscalls support creating
file descriptor directly non-inheritable in a single syscalls. ioctl()
can be also be used instead of fcntl() to use 1 syscall instead of 2.

> We can fix multiprocessing any anything else in the stdlib that this
> breaks, I presume.

The CGI code rely on inheritance of file descriptors 0, 1 and 2 which
are pipes. The file descriptors 0, 1 and 2 are replaced with the pipes
using os.dup2().

Victor

From solipsis at pitrou.net  Fri Jul 26 14:25:25 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 26 Jul 2013 14:25:25 +0200
Subject: [Python-Dev] Inherance of file descriptor and handles on
	Windows (PEP 446)
References: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
	<ksn359$q3a$1@ger.gmane.org>
	<CAP7+vJKGUS9cseDz+LgV0Z5DKJyYzEru6fAr1=Ct9GMBYB7i_w@mail.gmail.com>
	<CAMpsgwZq9qR-BBSoXNDwZq=VYmYcv5AMgjv7bPnHBhPLfse6HA@mail.gmail.com>
Message-ID: <20130726142525.79448c82@pitrou.net>

Le Fri, 26 Jul 2013 14:08:35 +0200,
Victor Stinner <victor.stinner at gmail.com> a ?crit :
> 
> After having written 2 PEP on the topic, I slowly agree that make all
> file descriptors non-inheritable is the best *compromise*. It solves
> most, or all, issues.

Even stdin/stdout/stderr? I think inheriting them is the sane default.

> The main drawback is the additionnal syscalls: on some platforms, 2
> additional syscalls are need to make a file descriptor non-inheritable
> for each creation of file descriptor. According to my benchmark on the
> implementation of the PEP 433: the overhead of making a file
> descriptor non-inheritable is between 1% and 3% (7.8 ?s => 7.9 or 8.0
> ?s) on Linux 3.6.

1% and 3% of what?
You're telling us there's a 0.1?s overhead. It's positively tiny.

Regards

Antoine.



From eliben at gmail.com  Fri Jul 26 14:56:25 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Fri, 26 Jul 2013 05:56:25 -0700
Subject: [Python-Dev] Coverity Scan
In-Reply-To: <ksrrsr$apk$1@ger.gmane.org>
References: <ksrrsr$apk$1@ger.gmane.org>
Message-ID: <CAF-Rda-_uEEqTGNiTnoxOrXw=ibsV+TKOXdyfzsz21pVEcDwsw@mail.gmail.com>

Just a quick question - is there a chance to convince Coverity to detect
Python refcounting leaks in C API code :-) ? This could be useful not only
for Python but for extensions too. As it stands now, Coverity's leak
detection is Python must be pretty weak because almost everything is done
via PyObject refcounts.

Eli


On Thu, Jul 25, 2013 at 11:48 AM, Christian Heimes <christian at python.org>wrote:

> Hello,
>
> this is an update on my work and the current status of Coverity Scan.
>
> Maybe you have noticed a checkins made be me that end with the line "CID
> #". These are checkins that fix an issue that was discovered by the
> static code analyzer Coverity. Coverity is a commercial product but it's
> a free service for some Open Source projects. Python has been analyzed
> by Coverity since about 2007. Guido, Neal, Brett, Stefan and some other
> developers have used Coverity before I took over. I fixed a couple of
> issues before 3.3 reached the RC phase and more bugs in the last couple
> of months.
>
> Coverity is really great and its web GUI is fun to use, too. I was able
> to identify and fix resource leaks, NULL pointer issues, buffer
> overflows and missing checks all over the place. Because it's a static
> analyzer that follows data-flows and control-flows the tool can detect
> issues in error paths that are hardly visited at all. I have started to
> document Coverity here:
>
>   http://docs.python.org/devguide/coverity.html
>
>
> Interview
> ---------
>
> A week ago I was contacted by Coverity. They have started a series of
> articles and press releases about Open Source projects that use their
> free service Coverity Scan, see
>
>
>
> http://www.coverity.com/company/press-releases/read/coverity-introduces-monthly-spotlight-series-for-coverity-scan-open-source-projects
>
> Two days ago I had a lovely phone interview about my involvement in the
> Python project and our development style. They are going to release a
> nice article in a couple of weeks. In the mean time we have time to fix
> the remaining couple issues. We *might* be able to reach the highest
> coverity integrity level! I have dealt with all major issues so we just
> have to fix a couple of issues.
>
>
> Current stats
> -------------
>
> Lines of Code:  396,179
> Defect Density: 0.05
> Total defects:  1,054
> Outstanding:       21 (Coverity Connect shows less)
> Dismissed:        222
> Fixed:            811
>
> http://i.imgur.com/NoELjcj.jpg
> http://i.imgur.com/eJSzTUX.jpg
>
>
> open issues
> -----------
>
> http://bugs.python.org/issue17899
> http://bugs.python.org/issue18556
> http://bugs.python.org/issue18555
> http://bugs.python.org/issue18552
> http://bugs.python.org/issue18551
> http://bugs.python.org/issue18550
> http://bugs.python.org/issue18528
>
>
> Christian
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/eliben%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130726/f4ad2c13/attachment.html>

From christian at python.org  Fri Jul 26 16:29:59 2013
From: christian at python.org (Christian Heimes)
Date: Fri, 26 Jul 2013 16:29:59 +0200
Subject: [Python-Dev] Coverity Scan
In-Reply-To: <CAF-Rda-_uEEqTGNiTnoxOrXw=ibsV+TKOXdyfzsz21pVEcDwsw@mail.gmail.com>
References: <ksrrsr$apk$1@ger.gmane.org>
	<CAF-Rda-_uEEqTGNiTnoxOrXw=ibsV+TKOXdyfzsz21pVEcDwsw@mail.gmail.com>
Message-ID: <51F287E7.6020702@python.org>

Am 26.07.2013 14:56, schrieb Eli Bendersky:
> Just a quick question - is there a chance to convince Coverity to detect
> Python refcounting leaks in C API code :-) ? This could be useful not
> only for Python but for extensions too. As it stands now, Coverity's
> leak detection is Python must be pretty weak because almost everything
> is done via PyObject refcounts.

Coverity is able to detect some cases of refcount leaks. I don't know if
the software is able to keep track of all reference counts. But it
understands missing Py_DECREF() in error branches.

For example:

PyObject *n = PyLong_FromLong(0);
PyObject *u = PyUnicode_FromString("example");

if (u == NULL) {
    return NULL;
    /* Coverity detects that 'n' leaks memory */
}

Christian

From brett at python.org  Fri Jul 26 16:29:40 2013
From: brett at python.org (Brett Cannon)
Date: Fri, 26 Jul 2013 10:29:40 -0400
Subject: [Python-Dev] Coverity Scan
In-Reply-To: <51F1AD1E.70309@python.org>
References: <ksrrsr$apk$1@ger.gmane.org> <kss76k$5l9$1@ger.gmane.org>
	<kss914$tar$1@ger.gmane.org> <51F1AD1E.70309@python.org>
Message-ID: <CAP1=2W7=_3==+oR5QKQML6KVX3+0GF3fPSEfh8+rV7mUC0bi3g@mail.gmail.com>

On Thu, Jul 25, 2013 at 6:56 PM, Christian Heimes <christian at python.org>wrote:

> Am 26.07.2013 00:32, schrieb Terry Reedy:
> > I found the answer here
> > https://docs.google.com/file/d/0B5wQCOK_TiRiMWVqQ0xPaDEzbkU/edit
> > Coverity Integrity Level 1 is 1 (defect/1000 lines)
> > Level 2 is .1 (we have passed that)
> > Level 3 is .01 + no major defects + <20% (all all defects?) false
> > positives as that is their normal rate.#
> >
> > A higher false positive rates requires auditing by Coverity. They claim
> > "A higher false positive rate indicates misconfiguration, usage of
> > unusual idioms, or incorrect diagnosis of a large number of defects."
> > They else add "or a flaw in our analysis."
> >
> > # Since false positives should stay constant as true positives are
> > reduced toward 0, false / all should tend toward 1 (100%) if I
> > understand the ratio correctly.
>
> About 40% of the dismissed cases are cause by a handful of issues. I
> have documented these issues as "known limitations"
> http://docs.python.org/devguide/coverity.html#known-limitations .
>
> For example about 35 false positives are related to PyLong_FromLong()
> and our small integer optimization. A correct modeling file would
> eliminate the false positive defects. My attempts don't work as hoped
> and I don't have access to all professional coverity tools to debug my
> trials.
>

Have you tried asking for help from Coverity? They have been rather nice so
far and they may be willing to just give us free help in getting the
modeling file set up properly.

-Brett


>
> Nearly 20 false positives are caused by Py_BuildValue("N"). I'm still
> astonished that Coverity understands Python's reference counting most of
> the time. :)
>
> Did I mention that we have almost reached Level 3? All major defects
> have been dealt with (one of them locally on the test machine until
> Larry pushes his patch soonish), 4 of 7 minor issues must be closed and
> our dismissed rate is just little over 20% (222 out of 1054 = 21%).
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130726/3b6f07f0/attachment.html>

From brett at python.org  Fri Jul 26 16:30:58 2013
From: brett at python.org (Brett Cannon)
Date: Fri, 26 Jul 2013 10:30:58 -0400
Subject: [Python-Dev] Coverity Scan
In-Reply-To: <CAF-Rda-_uEEqTGNiTnoxOrXw=ibsV+TKOXdyfzsz21pVEcDwsw@mail.gmail.com>
References: <ksrrsr$apk$1@ger.gmane.org>
	<CAF-Rda-_uEEqTGNiTnoxOrXw=ibsV+TKOXdyfzsz21pVEcDwsw@mail.gmail.com>
Message-ID: <CAP1=2W50F5yj7+1yTLXNnpc8F8Q-TZKwzLTY61F83gg7bwGZyw@mail.gmail.com>

On Fri, Jul 26, 2013 at 8:56 AM, Eli Bendersky <eliben at gmail.com> wrote:

> Just a quick question - is there a chance to convince Coverity to detect
> Python refcounting leaks in C API code :-) ?
>

You can always ask. =)


> This could be useful not only for Python but for extensions too. As it
> stands now, Coverity's leak detection is Python must be pretty weak because
> almost everything is done via PyObject refcounts.
>

Just an FYI (mostly for others  since I think Eli was at PyCon in the
relevant talk), David Malcolm has his work with gcc and refleak detection.
But yes, it would be nice if it was in Coverity as it would then be part of
the daily check.

-Brett


>
> Eli
>
>
> On Thu, Jul 25, 2013 at 11:48 AM, Christian Heimes <christian at python.org>wrote:
>
>> Hello,
>>
>> this is an update on my work and the current status of Coverity Scan.
>>
>> Maybe you have noticed a checkins made be me that end with the line "CID
>> #". These are checkins that fix an issue that was discovered by the
>> static code analyzer Coverity. Coverity is a commercial product but it's
>> a free service for some Open Source projects. Python has been analyzed
>> by Coverity since about 2007. Guido, Neal, Brett, Stefan and some other
>> developers have used Coverity before I took over. I fixed a couple of
>> issues before 3.3 reached the RC phase and more bugs in the last couple
>> of months.
>>
>> Coverity is really great and its web GUI is fun to use, too. I was able
>> to identify and fix resource leaks, NULL pointer issues, buffer
>> overflows and missing checks all over the place. Because it's a static
>> analyzer that follows data-flows and control-flows the tool can detect
>> issues in error paths that are hardly visited at all. I have started to
>> document Coverity here:
>>
>>   http://docs.python.org/devguide/coverity.html
>>
>>
>> Interview
>> ---------
>>
>> A week ago I was contacted by Coverity. They have started a series of
>> articles and press releases about Open Source projects that use their
>> free service Coverity Scan, see
>>
>>
>>
>> http://www.coverity.com/company/press-releases/read/coverity-introduces-monthly-spotlight-series-for-coverity-scan-open-source-projects
>>
>> Two days ago I had a lovely phone interview about my involvement in the
>> Python project and our development style. They are going to release a
>> nice article in a couple of weeks. In the mean time we have time to fix
>> the remaining couple issues. We *might* be able to reach the highest
>> coverity integrity level! I have dealt with all major issues so we just
>> have to fix a couple of issues.
>>
>>
>> Current stats
>> -------------
>>
>> Lines of Code:  396,179
>> Defect Density: 0.05
>> Total defects:  1,054
>> Outstanding:       21 (Coverity Connect shows less)
>> Dismissed:        222
>> Fixed:            811
>>
>> http://i.imgur.com/NoELjcj.jpg
>> http://i.imgur.com/eJSzTUX.jpg
>>
>>
>> open issues
>> -----------
>>
>> http://bugs.python.org/issue17899
>> http://bugs.python.org/issue18556
>> http://bugs.python.org/issue18555
>> http://bugs.python.org/issue18552
>> http://bugs.python.org/issue18551
>> http://bugs.python.org/issue18550
>> http://bugs.python.org/issue18528
>>
>>
>> Christian
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> http://mail.python.org/mailman/options/python-dev/eliben%40gmail.com
>>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130726/423888ea/attachment-0001.html>

From solipsis at pitrou.net  Fri Jul 26 16:48:03 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 26 Jul 2013 16:48:03 +0200
Subject: [Python-Dev] Coverity Scan
References: <ksrrsr$apk$1@ger.gmane.org>
	<CAF-Rda-_uEEqTGNiTnoxOrXw=ibsV+TKOXdyfzsz21pVEcDwsw@mail.gmail.com>
	<51F287E7.6020702@python.org>
Message-ID: <20130726164803.66f9887e@pitrou.net>

Le Fri, 26 Jul 2013 16:29:59 +0200,
Christian Heimes <christian at python.org> a ?crit :
> Coverity is able to detect some cases of refcount leaks. I don't know
> if the software is able to keep track of all reference counts. But it
> understands missing Py_DECREF() in error branches.
> 
> For example:
> 
> PyObject *n = PyLong_FromLong(0);
> PyObject *u = PyUnicode_FromString("example");
> 
> if (u == NULL) {
>     return NULL;
>     /* Coverity detects that 'n' leaks memory */
> }

But 'n' doesn't leak memory since PyLong_FromLong(0) is statically
allocated ;-)

More generally, in similar cases (e.g. replace "0" with a non-small
integer), you don't need any knowledge of reference counts to infer
that there is a memory leak. When the code discards the only existing
pointer to a heap-allocated memory area, there's a leak.

What we call "refcount leaks" is generally when an area is still
pointer-accessible, but failure to decrement the reference count
appropriately means it will never be released.

Regards

Antoine.



From eliben at gmail.com  Fri Jul 26 17:13:03 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Fri, 26 Jul 2013 08:13:03 -0700
Subject: [Python-Dev] Coverity Scan
In-Reply-To: <51F287E7.6020702@python.org>
References: <ksrrsr$apk$1@ger.gmane.org>
	<CAF-Rda-_uEEqTGNiTnoxOrXw=ibsV+TKOXdyfzsz21pVEcDwsw@mail.gmail.com>
	<51F287E7.6020702@python.org>
Message-ID: <CAF-Rda8rpaqO84zRYWs=gkOeUKUsioAYOopqJZatrAaX2XpVoQ@mail.gmail.com>

On Fri, Jul 26, 2013 at 7:29 AM, Christian Heimes <christian at python.org>wrote:

> Am 26.07.2013 14:56, schrieb Eli Bendersky:
> > Just a quick question - is there a chance to convince Coverity to detect
> > Python refcounting leaks in C API code :-) ? This could be useful not
> > only for Python but for extensions too. As it stands now, Coverity's
> > leak detection is Python must be pretty weak because almost everything
> > is done via PyObject refcounts.
>
> Coverity is able to detect some cases of refcount leaks. I don't know if
> the software is able to keep track of all reference counts. But it
> understands missing Py_DECREF() in error branches.
>
> For example:
>
> PyObject *n = PyLong_FromLong(0);
> PyObject *u = PyUnicode_FromString("example");
>
> if (u == NULL) {
>     return NULL;
>     /* Coverity detects that 'n' leaks memory */
> }
>

Interesting.

I was thinking of something more general though. Especially if we can mark
function arguments and return values as stealing references / creating new
ones / etc, many many common refcount bugs can be detected with static
analysis. This is definitely research-y, probably too much for our current
stage of relationship with Coverity :)

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130726/0ac43d6f/attachment.html>

From status at bugs.python.org  Fri Jul 26 18:07:38 2013
From: status at bugs.python.org (Python tracker)
Date: Fri, 26 Jul 2013 18:07:38 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20130726160738.291AD56A1A@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2013-07-19 - 2013-07-26)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    4114 (+22)
  closed 26220 (+35)
  total  30334 (+57)

Open issues with patches: 1856 


Issues opened (37)
==================

#18505: Duplicate function names in test_email.py
http://bugs.python.org/issue18505  opened by vajrasky

#18507: import_init() should not use Py_FatalError() but return an err
http://bugs.python.org/issue18507  opened by haypo

#18509: CJK decoders should return MBERR_EXCEPTION on PyUnicodeWriter 
http://bugs.python.org/issue18509  opened by haypo

#18512: sys.stdout.write does not allow bytes in Python 3.x
http://bugs.python.org/issue18512  opened by ntrrgc

#18515: zipfile._ZipDecryptor generates wasteful crc32 table on import
http://bugs.python.org/issue18515  opened by dholth

#18516: Typos in Lib/email/generator.py and Lib/email/architecture.rst
http://bugs.python.org/issue18516  opened by vajrasky

#18517: "xxlimited" extension declared incorrectly in setup.py
http://bugs.python.org/issue18517  opened by ned.deily

#18518: return-ing within code timed with timeit.timeit causes wrong r
http://bugs.python.org/issue18518  opened by icedream91

#18519: test_sqlite crashes on OS X tiger 3.x buildbot
http://bugs.python.org/issue18519  opened by ned.deily

#18521: [cppcheck] Full report
http://bugs.python.org/issue18521  opened by serval2412

#18526: Add resource management/guarding to unittest
http://bugs.python.org/issue18526  opened by zach.ware

#18527: Upgrade Modules/zlib to 1.2.8
http://bugs.python.org/issue18527  opened by christian.heimes

#18528: Possible fd leak in socketmodule
http://bugs.python.org/issue18528  opened by christian.heimes

#18529: Use long dash
http://bugs.python.org/issue18529  opened by serhiy.storchaka

#18530: posixpath.ismount performs extra lstat calls
http://bugs.python.org/issue18530  opened by alex

#18531: Undocumented different between METH_KEYWORDS and **kws
http://bugs.python.org/issue18531  opened by barry

#18532: hashlib.HASH objects should officially expose the hash name
http://bugs.python.org/issue18532  opened by jason.coombs

#18533: Avoid error from repr() of recursive dictview
http://bugs.python.org/issue18533  opened by bennorth

#18534: State clearly that open() 'file' param is "name" attr of the r
http://bugs.python.org/issue18534  opened by ncoghlan

#18535: termios.tcgetattr should return namedtuple
http://bugs.python.org/issue18535  opened by techtonik

#18538: `python -m dis ` relying on argparse
http://bugs.python.org/issue18538  opened by maker

#18539: Arguments tooltip wrong if def contains fractional default val
http://bugs.python.org/issue18539  opened by ariel_bruner

#18540: imaplib.IMAP4() ends with "Name or service not known" on Fedor
http://bugs.python.org/issue18540  opened by sYnfo

#18543: urllib.parse.urlopen shouldn't ignore installed opener when ca
http://bugs.python.org/issue18543  opened by moritzs

#18544: subprocess.Popen support for redirection of arbitrary file des
http://bugs.python.org/issue18544  opened by ropoctorix

#18546: ssl.get_server_certificate like addition for cert chain
http://bugs.python.org/issue18546  opened by underrun

#18548: In unittest doc change WidgetTestCase to SimpleWidgetTestCase 
http://bugs.python.org/issue18548  opened by py.user

#18550: internal_setblocking() doesn't check return value of fcntl()
http://bugs.python.org/issue18550  opened by christian.heimes

#18551: child_exec() doesn't check return value of fcntl()
http://bugs.python.org/issue18551  opened by christian.heimes

#18552: obj2ast_object() doesn't check return value of PyArena_AddPyOb
http://bugs.python.org/issue18552  opened by christian.heimes

#18553: os.isatty() is not Unix only
http://bugs.python.org/issue18553  opened by techtonik

#18554: os.__all__ is incomplete
http://bugs.python.org/issue18554  opened by ronaldoussoren

#18555: type_set_bases() doesn't check return value of  PyArg_UnpackTu
http://bugs.python.org/issue18555  opened by christian.heimes

#18558: Iterable glossary entry needs clarification
http://bugs.python.org/issue18558  opened by Zero

#18559: _pickle: NULL ptr dereference when PyLong_FromSsize_t() fails
http://bugs.python.org/issue18559  opened by christian.heimes

#18560: builtin_sum() doesn't check return value of PyLong_FromLong()
http://bugs.python.org/issue18560  opened by christian.heimes

#18561: ctypes _build_callargs() doesn't check name for NULL
http://bugs.python.org/issue18561  opened by christian.heimes



Most recent 15 issues with no replies (15)
==========================================

#18561: ctypes _build_callargs() doesn't check name for NULL
http://bugs.python.org/issue18561

#18560: builtin_sum() doesn't check return value of PyLong_FromLong()
http://bugs.python.org/issue18560

#18559: _pickle: NULL ptr dereference when PyLong_FromSsize_t() fails
http://bugs.python.org/issue18559

#18554: os.__all__ is incomplete
http://bugs.python.org/issue18554

#18553: os.isatty() is not Unix only
http://bugs.python.org/issue18553

#18552: obj2ast_object() doesn't check return value of PyArena_AddPyOb
http://bugs.python.org/issue18552

#18551: child_exec() doesn't check return value of fcntl()
http://bugs.python.org/issue18551

#18550: internal_setblocking() doesn't check return value of fcntl()
http://bugs.python.org/issue18550

#18548: In unittest doc change WidgetTestCase to SimpleWidgetTestCase 
http://bugs.python.org/issue18548

#18544: subprocess.Popen support for redirection of arbitrary file des
http://bugs.python.org/issue18544

#18543: urllib.parse.urlopen shouldn't ignore installed opener when ca
http://bugs.python.org/issue18543

#18539: Arguments tooltip wrong if def contains fractional default val
http://bugs.python.org/issue18539

#18527: Upgrade Modules/zlib to 1.2.8
http://bugs.python.org/issue18527

#18509: CJK decoders should return MBERR_EXCEPTION on PyUnicodeWriter 
http://bugs.python.org/issue18509

#18507: import_init() should not use Py_FatalError() but return an err
http://bugs.python.org/issue18507



Most recent 15 issues waiting for review (15)
=============================================

#18560: builtin_sum() doesn't check return value of PyLong_FromLong()
http://bugs.python.org/issue18560

#18559: _pickle: NULL ptr dereference when PyLong_FromSsize_t() fails
http://bugs.python.org/issue18559

#18554: os.__all__ is incomplete
http://bugs.python.org/issue18554

#18552: obj2ast_object() doesn't check return value of PyArena_AddPyOb
http://bugs.python.org/issue18552

#18548: In unittest doc change WidgetTestCase to SimpleWidgetTestCase 
http://bugs.python.org/issue18548

#18540: imaplib.IMAP4() ends with "Name or service not known" on Fedor
http://bugs.python.org/issue18540

#18538: `python -m dis ` relying on argparse
http://bugs.python.org/issue18538

#18533: Avoid error from repr() of recursive dictview
http://bugs.python.org/issue18533

#18530: posixpath.ismount performs extra lstat calls
http://bugs.python.org/issue18530

#18529: Use long dash
http://bugs.python.org/issue18529

#18528: Possible fd leak in socketmodule
http://bugs.python.org/issue18528

#18526: Add resource management/guarding to unittest
http://bugs.python.org/issue18526

#18519: test_sqlite crashes on OS X tiger 3.x buildbot
http://bugs.python.org/issue18519

#18517: "xxlimited" extension declared incorrectly in setup.py
http://bugs.python.org/issue18517

#18515: zipfile._ZipDecryptor generates wasteful crc32 table on import
http://bugs.python.org/issue18515



Top 10 most discussed issues (10)
=================================

#15805: Add stdout redirection tool to contextlib
http://bugs.python.org/issue15805  24 msgs

#17899: os.listdir() leaks FDs if invoked on FD pointing to a non-dire
http://bugs.python.org/issue17899  13 msgs

#18535: termios.tcgetattr should return namedtuple
http://bugs.python.org/issue18535   8 msgs

#18519: test_sqlite crashes on OS X tiger 3.x buildbot
http://bugs.python.org/issue18519   7 msgs

#18530: posixpath.ismount performs extra lstat calls
http://bugs.python.org/issue18530   7 msgs

#18441: Idle: Make test.support.requires('gui') skip when it should.
http://bugs.python.org/issue18441   6 msgs

#18533: Avoid error from repr() of recursive dictview
http://bugs.python.org/issue18533   6 msgs

#18078: threading.Condition to allow notify on a specific waiter
http://bugs.python.org/issue18078   5 msgs

#18324: set_payload does not handle binary payloads correctly
http://bugs.python.org/issue18324   5 msgs

#18408: Fixes crashes found by pyfailmalloc
http://bugs.python.org/issue18408   5 msgs



Issues closed (34)
==================

#8079: make install fails with -j8 with python2.6/config on FreeBSD
http://bugs.python.org/issue8079  closed by christian.heimes

#9177: ssl.read/write on closed socket raises AttributeError
http://bugs.python.org/issue9177  closed by pitrou

#14853: test_file.py depends on sys.stdin being unseekable
http://bugs.python.org/issue14853  closed by ezio.melotti

#15130: remove redundant paragraph in socket howto
http://bugs.python.org/issue15130  closed by r.david.murray

#15905: Copy to fixed size buffer w/o check in sys_update_path
http://bugs.python.org/issue15905  closed by christian.heimes

#16937: -u (unbuffered I/O) command line option documentation mismatch
http://bugs.python.org/issue16937  closed by ezio.melotti

#17818: aifc.Aifc_read/Aifc_write.getparams can return a namedtuple
http://bugs.python.org/issue17818  closed by r.david.murray

#17944: Refactor test_zipfile
http://bugs.python.org/issue17944  closed by serhiy.storchaka

#18327: swapped arguments in compatible_for_assignment()?
http://bugs.python.org/issue18327  closed by christian.heimes

#18439: Patchcheck for ACKS, NEWS does not work on Windows.
http://bugs.python.org/issue18439  closed by terry.reedy

#18460: .chm documentation files advertised in download.html but not t
http://bugs.python.org/issue18460  closed by loewis

#18488: sqlite: finalize() method of user function may be called with 
http://bugs.python.org/issue18488  closed by haypo

#18491: Add "exe wrapper" functionality to Windows launcher
http://bugs.python.org/issue18491  closed by python-dev

#18503: No assertion in test_del_param_on_nonexistent_header function 
http://bugs.python.org/issue18503  closed by r.david.murray

#18506: DISABLEd Tkinter.Listbox still fires <<ListboxSelect>> for mou
http://bugs.python.org/issue18506  closed by ned.deily

#18508: enum.Enum population of _value2member_map does not match fallb
http://bugs.python.org/issue18508  closed by python-dev

#18510: dict.__contains__ and dict.keys().__contains__ raises exceptio
http://bugs.python.org/issue18510  closed by ronaldoussoren

#18511: random.shuffle could be faster
http://bugs.python.org/issue18511  closed by rhettinger

#18513: test.cmath fails on OS/X with gcc-4.8 in non-debug build
http://bugs.python.org/issue18513  closed by mark.dickinson

#18514: Unreachable  Py_DECREF() in ctypes's PyCData_FromBaseObj()
http://bugs.python.org/issue18514  closed by christian.heimes

#18520: Fixes bugs found by pyfailmalloc during Python initialization
http://bugs.python.org/issue18520  closed by haypo

#18522: Error creating a raw string of r'\\?\'
http://bugs.python.org/issue18522  closed by pitrou

#18523: test_signal failure under Windows
http://bugs.python.org/issue18523  closed by haypo

#18524: BufferedReader.read1() documentation/implementation difference
http://bugs.python.org/issue18524  closed by pitrou

#18525: Shutil cannot import WindowsError on windows
http://bugs.python.org/issue18525  closed by ronaldoussoren

#18536: ????????? Google+ ????????????????????????
http://bugs.python.org/issue18536  closed by pitrou

#18537: bool.toggle()
http://bugs.python.org/issue18537  closed by eric.smith

#18541: LoggerAdapter example is counter-productive
http://bugs.python.org/issue18541  closed by python-dev

#18542: httpd memory consuption increasing continously due to mod_pyth
http://bugs.python.org/issue18542  closed by ronaldoussoren

#18545: enum always runs member_type when use_args is True
http://bugs.python.org/issue18545  closed by python-dev

#18547: os.path.dirname() does not behave as expected for path without
http://bugs.python.org/issue18547  closed by r.david.murray

#18549: Dead code in socket_ntohl()
http://bugs.python.org/issue18549  closed by christian.heimes

#18556: ctypes' U_set() doesn't check return value of PyUnicode_AsWide
http://bugs.python.org/issue18556  closed by brett.cannon

#18557: email address parser desired
http://bugs.python.org/issue18557  closed by r.david.murray

From solipsis at pitrou.net  Fri Jul 26 19:07:35 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 26 Jul 2013 19:07:35 +0200
Subject: [Python-Dev] Daemon creation code in the standard library
References: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
	<ksn359$q3a$1@ger.gmane.org>
	<CAP7+vJKGUS9cseDz+LgV0Z5DKJyYzEru6fAr1=Ct9GMBYB7i_w@mail.gmail.com>
	<7wy58vs86l.fsf_-_@benfinney.id.au>
	<7w1u6njsap.fsf@benfinney.id.au>
	<54C49643-17F7-4CAE-AAB4-1FD143DC22E5@mac.com>
Message-ID: <20130726190735.4c4582e2@fsol>

On Fri, 26 Jul 2013 09:38:10 +0200
Ronald Oussoren <ronaldoussoren at mac.com> wrote:
> 
> On 25 Jul, 2013, at 4:18, Ben Finney <ben+python at benfinney.id.au> wrote:
> 
> > Ben Finney <ben+python at benfinney.id.au> writes:
> > 
> >> Work continues on the PEP 3143-compatible ?python-daemon?, porting it to
> >> Python 3 and aiming for inclusion in the standard library.
> 
> At first glance the library appears to close all open files, with an option
> to exclude some specific file descriptors (that is, you need to pass a list
> of files that shouldn't be closed). 

Indeed, it's annoying when you want to setup logging before
daemonization starts. I had to hack my way through logging handlers to
find the fd I had to keep open.

> That makes it a lot harder to do some initialization before daemonizing.
> I prefer to perform at least some initialization early in program startup to
> be able to give sensible error messages. I've had too many initscripts that
> claimed to have started a daemon sucessfully, only to have that daemon stop
> right away because it noticed a problem right after it detached itself.

Agreed.

Regards

Antoine.



From christian at python.org  Fri Jul 26 22:09:44 2013
From: christian at python.org (Christian Heimes)
Date: Fri, 26 Jul 2013 22:09:44 +0200
Subject: [Python-Dev] Coverity Scan
In-Reply-To: <CAP1=2W7=_3==+oR5QKQML6KVX3+0GF3fPSEfh8+rV7mUC0bi3g@mail.gmail.com>
References: <ksrrsr$apk$1@ger.gmane.org> <kss76k$5l9$1@ger.gmane.org>
	<kss914$tar$1@ger.gmane.org> <51F1AD1E.70309@python.org>
	<CAP1=2W7=_3==+oR5QKQML6KVX3+0GF3fPSEfh8+rV7mUC0bi3g@mail.gmail.com>
Message-ID: <ksul21$j7g$1@ger.gmane.org>

Am 26.07.2013 16:29, schrieb Brett Cannon:
> Have you tried asking for help from Coverity? They have been rather nice
> so far and they may be willing to just give us free help in getting the
> modeling file set up properly.

Yes, I'm in contact with Dakshesh. I was able to figure out one model
for a false positive on my own. Dakshesh is helping me with another.

Christian


From victor.stinner at gmail.com  Fri Jul 26 22:17:47 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Fri, 26 Jul 2013 22:17:47 +0200
Subject: [Python-Dev] Inherance of file descriptor and handles on
 Windows (PEP 446)
In-Reply-To: <20130726142525.79448c82@pitrou.net>
References: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
	<ksn359$q3a$1@ger.gmane.org>
	<CAP7+vJKGUS9cseDz+LgV0Z5DKJyYzEru6fAr1=Ct9GMBYB7i_w@mail.gmail.com>
	<CAMpsgwZq9qR-BBSoXNDwZq=VYmYcv5AMgjv7bPnHBhPLfse6HA@mail.gmail.com>
	<20130726142525.79448c82@pitrou.net>
Message-ID: <CAMpsgwZXNeiZqbZUh+VvtSnNjBM69_b45wAYZf+7GfXg2X=R0Q@mail.gmail.com>

2013/7/26 Antoine Pitrou <solipsis at pitrou.net>:
>> The main drawback is the additionnal syscalls: on some platforms, 2
>> additional syscalls are need to make a file descriptor non-inheritable
>> for each creation of file descriptor. According to my benchmark on the
>> implementation of the PEP 433: the overhead of making a file
>> descriptor non-inheritable is between 1% and 3% (7.8 ?s => 7.9 or 8.0
>> ?s) on Linux 3.6.
>
> 1% and 3% of what?
> You're telling us there's a 0.1?s overhead. It's positively tiny.

Copy-paste of the link:

"""
On Linux, setting the close-on-flag has a low overhead on
performances. Results of bench_cloexec.py on Linux 3.6:

- close-on-flag not set: 7.8 us
- O_CLOEXEC: 1% slower (7.9 us)
- ioctl(): 3% slower (8.0 us)
- fcntl(): 3% slower (8.0 us)
"""

The overhead is between 0.1 and 0.2 ?s (100 and 200 ns) according to
my micro-benchmark.

"python -c pass" takes 19,000 ?s (0.019 sec) on my PC. It uses 207
syscalls creating file descriptors (open() and openat()): 67 are
successful, 140 are failing with ENOENT. The estimated overhead on
"python -c pass" is 0.2*67=13.4 ?s (0.07%).

Victor

From solipsis at pitrou.net  Fri Jul 26 23:24:32 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 26 Jul 2013 23:24:32 +0200
Subject: [Python-Dev] Inherance of file descriptor and handles on
	Windows (PEP 446)
In-Reply-To: <CAMpsgwZXNeiZqbZUh+VvtSnNjBM69_b45wAYZf+7GfXg2X=R0Q@mail.gmail.com>
References: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
	<ksn359$q3a$1@ger.gmane.org>
	<CAP7+vJKGUS9cseDz+LgV0Z5DKJyYzEru6fAr1=Ct9GMBYB7i_w@mail.gmail.com>
	<CAMpsgwZq9qR-BBSoXNDwZq=VYmYcv5AMgjv7bPnHBhPLfse6HA@mail.gmail.com>
	<20130726142525.79448c82@pitrou.net>
	<CAMpsgwZXNeiZqbZUh+VvtSnNjBM69_b45wAYZf+7GfXg2X=R0Q@mail.gmail.com>
Message-ID: <20130726232432.3651f1d5@fsol>

On Fri, 26 Jul 2013 22:17:47 +0200
Victor Stinner <victor.stinner at gmail.com> wrote:

> 2013/7/26 Antoine Pitrou <solipsis at pitrou.net>:
> >> The main drawback is the additionnal syscalls: on some platforms, 2
> >> additional syscalls are need to make a file descriptor non-inheritable
> >> for each creation of file descriptor. According to my benchmark on the
> >> implementation of the PEP 433: the overhead of making a file
> >> descriptor non-inheritable is between 1% and 3% (7.8 ?s => 7.9 or 8.0
> >> ?s) on Linux 3.6.
> >
> > 1% and 3% of what?
> > You're telling us there's a 0.1?s overhead. It's positively tiny.
> 
> Copy-paste of the link:
> 
> """
> On Linux, setting the close-on-flag has a low overhead on
> performances. Results of bench_cloexec.py on Linux 3.6:
> 
> - close-on-flag not set: 7.8 us
> - O_CLOEXEC: 1% slower (7.9 us)
> - ioctl(): 3% slower (8.0 us)
> - fcntl(): 3% slower (8.0 us)
> """

You aren't answering my question: slower than what?
Benchmarking is useless if you aren't telling us what exactly you are
benchmarking.

> The overhead is between 0.1 and 0.2 ?s (100 and 200 ns) according to
> my micro-benchmark.

That's what I figured out (see above). It's tiny.

Antoine.

From guido at python.org  Sat Jul 27 00:12:18 2013
From: guido at python.org (Guido van Rossum)
Date: Fri, 26 Jul 2013 15:12:18 -0700
Subject: [Python-Dev] Inherance of file descriptor and handles on
 Windows (PEP 446)
In-Reply-To: <CAMpsgwZq9qR-BBSoXNDwZq=VYmYcv5AMgjv7bPnHBhPLfse6HA@mail.gmail.com>
References: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
	<ksn359$q3a$1@ger.gmane.org>
	<CAP7+vJKGUS9cseDz+LgV0Z5DKJyYzEru6fAr1=Ct9GMBYB7i_w@mail.gmail.com>
	<CAMpsgwZq9qR-BBSoXNDwZq=VYmYcv5AMgjv7bPnHBhPLfse6HA@mail.gmail.com>
Message-ID: <CAP7+vJ+w7exQM+epLNGQK--zMrVjqtooAupxVv9O5wpSFYunwQ@mail.gmail.com>

On Fri, Jul 26, 2013 at 5:08 AM, Victor Stinner
<victor.stinner at gmail.com> wrote:
> After having written 2 PEP on the topic, I slowly agree that make all
> file descriptors non-inheritable is the best *compromise*. It solves
> most, or all, issues.

Right.

> The main drawback is the additionnal syscalls: on some platforms, 2
> additional syscalls are need to make a file descriptor non-inheritable
> for each creation of file descriptor. According to my benchmark on the
> implementation of the PEP 433: the overhead of making a file
> descriptor non-inheritable is between 1% and 3% (7.8 ?s => 7.9 or 8.0
> ?s) on Linux 3.6.
> http://www.python.org/dev/peps/pep-0433/#performances

Remember that this is going to be Python 3.4 and newer. AFAICT setting
O_CLOEXEC on open works on OSX (at least the man page on OSX 10.8 has
it), on newer Linuxes, and the equivalent on Windows. So even if it
does cost an extra syscall on older systems, those systems will be
obsolete before Python 3.4 becomes mainstream there. And it does look
like the syscalls are pretty cheap. I also don't think I'm not
particularly worried about the cost of syscalls for making a socket
(non)blocking -- although we should probably avoid the second fcntl()
call if the first call shows the flag is already set the way we want
it.

-- 
--Guido van Rossum (python.org/~guido)

From victor.stinner at gmail.com  Sat Jul 27 00:18:40 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sat, 27 Jul 2013 00:18:40 +0200
Subject: [Python-Dev] Inherance of file descriptor and handles on
 Windows (PEP 446)
In-Reply-To: <20130726232432.3651f1d5@fsol>
References: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
	<ksn359$q3a$1@ger.gmane.org>
	<CAP7+vJKGUS9cseDz+LgV0Z5DKJyYzEru6fAr1=Ct9GMBYB7i_w@mail.gmail.com>
	<CAMpsgwZq9qR-BBSoXNDwZq=VYmYcv5AMgjv7bPnHBhPLfse6HA@mail.gmail.com>
	<20130726142525.79448c82@pitrou.net>
	<CAMpsgwZXNeiZqbZUh+VvtSnNjBM69_b45wAYZf+7GfXg2X=R0Q@mail.gmail.com>
	<20130726232432.3651f1d5@fsol>
Message-ID: <CAMpsgwaSXugXKwU5AkBN0fHZ0WeJWfWkNhdnS9b9XeYwF3bONA@mail.gmail.com>

2013/7/26 Antoine Pitrou <solipsis at pitrou.net>:
> On Fri, 26 Jul 2013 22:17:47 +0200
>> """
>> On Linux, setting the close-on-flag has a low overhead on
>> performances. Results of bench_cloexec.py on Linux 3.6:
>>
>> - close-on-flag not set: 7.8 us
>> - O_CLOEXEC: 1% slower (7.9 us)
>> - ioctl(): 3% slower (8.0 us)
>> - fcntl(): 3% slower (8.0 us)
>> """
>
> You aren't answering my question: slower than what?

Ah, you didn't understand the labels. bench_cloexec.py runs a
benchmark on os.open(path, os.O_RDONLY, cloexec=False) and
os.open(path, os.O_RDONLY, cloexec=True) with different implementation
of making the file descriptor non-inheritable.

close-on-flag not set: 7.8 us
=> C code: open(path, O_RDONLY)

O_CLOEXEC: 1% slower (7.9 us)
=> C code: open(path, O_RDONLY|CLOEXEC)
=> 1% slower than open(path, O_RDONLY)

ioctl(): 3% slower (8.0 us)
=> C code: fd=open(path, O_RDONLY); ioctl(fd, FIOCLEX, 0)
=> 3% slower than open(path, O_RDONLY)

fcntl(): 3% slower (8.0 us)
=> C code: fd=open(path, O_RDONLY); flags = fcntl(fd, F_GETFD);
fcntl(fd, F_SETFD, flags | FD_CLOEXEC)
=> 3% slower than open(path, O_RDONLY)

Victor

From solipsis at pitrou.net  Sat Jul 27 00:23:59 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 27 Jul 2013 00:23:59 +0200
Subject: [Python-Dev] Inherance of file descriptor and handles on
	Windows (PEP 446)
In-Reply-To: <CAMpsgwaSXugXKwU5AkBN0fHZ0WeJWfWkNhdnS9b9XeYwF3bONA@mail.gmail.com>
References: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
	<ksn359$q3a$1@ger.gmane.org>
	<CAP7+vJKGUS9cseDz+LgV0Z5DKJyYzEru6fAr1=Ct9GMBYB7i_w@mail.gmail.com>
	<CAMpsgwZq9qR-BBSoXNDwZq=VYmYcv5AMgjv7bPnHBhPLfse6HA@mail.gmail.com>
	<20130726142525.79448c82@pitrou.net>
	<CAMpsgwZXNeiZqbZUh+VvtSnNjBM69_b45wAYZf+7GfXg2X=R0Q@mail.gmail.com>
	<20130726232432.3651f1d5@fsol>
	<CAMpsgwaSXugXKwU5AkBN0fHZ0WeJWfWkNhdnS9b9XeYwF3bONA@mail.gmail.com>
Message-ID: <20130727002359.2631ea8a@fsol>

On Sat, 27 Jul 2013 00:18:40 +0200
Victor Stinner <victor.stinner at gmail.com> wrote:
> 2013/7/26 Antoine Pitrou <solipsis at pitrou.net>:
> > On Fri, 26 Jul 2013 22:17:47 +0200
> >> """
> >> On Linux, setting the close-on-flag has a low overhead on
> >> performances. Results of bench_cloexec.py on Linux 3.6:
> >>
> >> - close-on-flag not set: 7.8 us
> >> - O_CLOEXEC: 1% slower (7.9 us)
> >> - ioctl(): 3% slower (8.0 us)
> >> - fcntl(): 3% slower (8.0 us)
> >> """
> >
> > You aren't answering my question: slower than what?
> 
> Ah, you didn't understand the labels. bench_cloexec.py runs a
> benchmark on os.open(path, os.O_RDONLY, cloexec=False) and
> os.open(path, os.O_RDONLY, cloexec=True) with different implementation
> of making the file descriptor non-inheritable.
> 
> close-on-flag not set: 7.8 us
> => C code: open(path, O_RDONLY)
> 
> O_CLOEXEC: 1% slower (7.9 us)
> => C code: open(path, O_RDONLY|CLOEXEC)
> => 1% slower than open(path, O_RDONLY)
> 
> ioctl(): 3% slower (8.0 us)
> => C code: fd=open(path, O_RDONLY); ioctl(fd, FIOCLEX, 0)
> => 3% slower than open(path, O_RDONLY)
> 
> fcntl(): 3% slower (8.0 us)
> => C code: fd=open(path, O_RDONLY); flags = fcntl(fd, F_GETFD);
> fcntl(fd, F_SETFD, flags | FD_CLOEXEC)
> => 3% slower than open(path, O_RDONLY)

Ok, so I think this it is a totally reasonable compromise.

People who bother about a 3% slowdown when calling os.open() can
optimize the hell out of their code using Cython for all I care :-)

Regards

Antoine.

From greg at krypto.org  Sat Jul 27 06:26:00 2013
From: greg at krypto.org (Gregory P. Smith)
Date: Fri, 26 Jul 2013 21:26:00 -0700
Subject: [Python-Dev] Inherance of file descriptor and handles on
 Windows (PEP 446)
In-Reply-To: <20130727002359.2631ea8a@fsol>
References: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
	<ksn359$q3a$1@ger.gmane.org>
	<CAP7+vJKGUS9cseDz+LgV0Z5DKJyYzEru6fAr1=Ct9GMBYB7i_w@mail.gmail.com>
	<CAMpsgwZq9qR-BBSoXNDwZq=VYmYcv5AMgjv7bPnHBhPLfse6HA@mail.gmail.com>
	<20130726142525.79448c82@pitrou.net>
	<CAMpsgwZXNeiZqbZUh+VvtSnNjBM69_b45wAYZf+7GfXg2X=R0Q@mail.gmail.com>
	<20130726232432.3651f1d5@fsol>
	<CAMpsgwaSXugXKwU5AkBN0fHZ0WeJWfWkNhdnS9b9XeYwF3bONA@mail.gmail.com>
	<20130727002359.2631ea8a@fsol>
Message-ID: <CAGE7PNLfZKEtXSNNUN=fq+gHXkQsp0iuyz2SgQDZJowFc2spnA@mail.gmail.com>

On Fri, Jul 26, 2013 at 3:23 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Sat, 27 Jul 2013 00:18:40 +0200
> Victor Stinner <victor.stinner at gmail.com> wrote:
> > 2013/7/26 Antoine Pitrou <solipsis at pitrou.net>:
> > > On Fri, 26 Jul 2013 22:17:47 +0200
> > >> """
> > >> On Linux, setting the close-on-flag has a low overhead on
> > >> performances. Results of bench_cloexec.py on Linux 3.6:
> > >>
> > >> - close-on-flag not set: 7.8 us
> > >> - O_CLOEXEC: 1% slower (7.9 us)
> > >> - ioctl(): 3% slower (8.0 us)
> > >> - fcntl(): 3% slower (8.0 us)
> > >> """
> > >
> > > You aren't answering my question: slower than what?
> >
> > Ah, you didn't understand the labels. bench_cloexec.py runs a
> > benchmark on os.open(path, os.O_RDONLY, cloexec=False) and
> > os.open(path, os.O_RDONLY, cloexec=True) with different implementation
> > of making the file descriptor non-inheritable.
> >
> > close-on-flag not set: 7.8 us
> > => C code: open(path, O_RDONLY)
> >
> > O_CLOEXEC: 1% slower (7.9 us)
> > => C code: open(path, O_RDONLY|CLOEXEC)
> > => 1% slower than open(path, O_RDONLY)
> >
> > ioctl(): 3% slower (8.0 us)
> > => C code: fd=open(path, O_RDONLY); ioctl(fd, FIOCLEX, 0)
> > => 3% slower than open(path, O_RDONLY)
> >
> > fcntl(): 3% slower (8.0 us)
> > => C code: fd=open(path, O_RDONLY); flags = fcntl(fd, F_GETFD);
> > fcntl(fd, F_SETFD, flags | FD_CLOEXEC)
> > => 3% slower than open(path, O_RDONLY)
>
> Ok, so I think this it is a totally reasonable compromise.
>
> People who bother about a 3% slowdown when calling os.open() can
> optimize the hell out of their code using Cython for all I care :-)
>
>
+1  ;)

and +1 for making the sane default of noinherit / cloexec /
whatever-others-call-it by default for all fds/handles ever opened by
Python. It stops ignoring the issue (ie: the status quo of matching the
default behavior of C as defined in the 1970s)... That is a GOOD thing. :)

-gps
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130726/1cd8e72f/attachment.html>

From guido at python.org  Sat Jul 27 06:36:01 2013
From: guido at python.org (Guido van Rossum)
Date: Fri, 26 Jul 2013 21:36:01 -0700
Subject: [Python-Dev] Inherance of file descriptor and handles on
 Windows (PEP 446)
In-Reply-To: <CAGE7PNLfZKEtXSNNUN=fq+gHXkQsp0iuyz2SgQDZJowFc2spnA@mail.gmail.com>
References: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
	<ksn359$q3a$1@ger.gmane.org>
	<CAP7+vJKGUS9cseDz+LgV0Z5DKJyYzEru6fAr1=Ct9GMBYB7i_w@mail.gmail.com>
	<CAMpsgwZq9qR-BBSoXNDwZq=VYmYcv5AMgjv7bPnHBhPLfse6HA@mail.gmail.com>
	<20130726142525.79448c82@pitrou.net>
	<CAMpsgwZXNeiZqbZUh+VvtSnNjBM69_b45wAYZf+7GfXg2X=R0Q@mail.gmail.com>
	<20130726232432.3651f1d5@fsol>
	<CAMpsgwaSXugXKwU5AkBN0fHZ0WeJWfWkNhdnS9b9XeYwF3bONA@mail.gmail.com>
	<20130727002359.2631ea8a@fsol>
	<CAGE7PNLfZKEtXSNNUN=fq+gHXkQsp0iuyz2SgQDZJowFc2spnA@mail.gmail.com>
Message-ID: <CAP7+vJ+mKg2sL8T8ecCOjRq7tJxCLqnNV=O=H78dNpuEak5yjw@mail.gmail.com>

On Fri, Jul 26, 2013 at 9:26 PM, Gregory P. Smith <greg at krypto.org> wrote:
>
> On Fri, Jul 26, 2013 at 3:23 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>>
>> On Sat, 27 Jul 2013 00:18:40 +0200
>> Victor Stinner <victor.stinner at gmail.com> wrote:
>> > 2013/7/26 Antoine Pitrou <solipsis at pitrou.net>:
>> > > On Fri, 26 Jul 2013 22:17:47 +0200
>> > >> """
>> > >> On Linux, setting the close-on-flag has a low overhead on
>> > >> performances. Results of bench_cloexec.py on Linux 3.6:
>> > >>
>> > >> - close-on-flag not set: 7.8 us
>> > >> - O_CLOEXEC: 1% slower (7.9 us)
>> > >> - ioctl(): 3% slower (8.0 us)
>> > >> - fcntl(): 3% slower (8.0 us)
>> > >> """
>> > >
>> > > You aren't answering my question: slower than what?
>> >
>> > Ah, you didn't understand the labels. bench_cloexec.py runs a
>> > benchmark on os.open(path, os.O_RDONLY, cloexec=False) and
>> > os.open(path, os.O_RDONLY, cloexec=True) with different implementation
>> > of making the file descriptor non-inheritable.
>> >
>> > close-on-flag not set: 7.8 us
>> > => C code: open(path, O_RDONLY)
>> >
>> > O_CLOEXEC: 1% slower (7.9 us)
>> > => C code: open(path, O_RDONLY|CLOEXEC)
>> > => 1% slower than open(path, O_RDONLY)
>> >
>> > ioctl(): 3% slower (8.0 us)
>> > => C code: fd=open(path, O_RDONLY); ioctl(fd, FIOCLEX, 0)
>> > => 3% slower than open(path, O_RDONLY)
>> >
>> > fcntl(): 3% slower (8.0 us)
>> > => C code: fd=open(path, O_RDONLY); flags = fcntl(fd, F_GETFD);
>> > fcntl(fd, F_SETFD, flags | FD_CLOEXEC)
>> > => 3% slower than open(path, O_RDONLY)
>>
>> Ok, so I think this it is a totally reasonable compromise.
>>
>> People who bother about a 3% slowdown when calling os.open() can
>> optimize the hell out of their code using Cython for all I care :-)
>>
>
> +1  ;)
>
> and +1 for making the sane default of noinherit / cloexec /
> whatever-others-call-it by default for all fds/handles ever opened by
> Python. It stops ignoring the issue (ie: the status quo of matching the
> default behavior of C as defined in the 1970s)... That is a GOOD thing. :)

Do we even need a new PEP, or should we just do it? Or can we adapt
Victor's PEP 446?

-- 
--Guido van Rossum (python.org/~guido)

From ncoghlan at gmail.com  Sat Jul 27 07:04:31 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 27 Jul 2013 15:04:31 +1000
Subject: [Python-Dev] Inherance of file descriptor and handles on
 Windows (PEP 446)
In-Reply-To: <CAP7+vJ+mKg2sL8T8ecCOjRq7tJxCLqnNV=O=H78dNpuEak5yjw@mail.gmail.com>
References: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
	<ksn359$q3a$1@ger.gmane.org>
	<CAP7+vJKGUS9cseDz+LgV0Z5DKJyYzEru6fAr1=Ct9GMBYB7i_w@mail.gmail.com>
	<CAMpsgwZq9qR-BBSoXNDwZq=VYmYcv5AMgjv7bPnHBhPLfse6HA@mail.gmail.com>
	<20130726142525.79448c82@pitrou.net>
	<CAMpsgwZXNeiZqbZUh+VvtSnNjBM69_b45wAYZf+7GfXg2X=R0Q@mail.gmail.com>
	<20130726232432.3651f1d5@fsol>
	<CAMpsgwaSXugXKwU5AkBN0fHZ0WeJWfWkNhdnS9b9XeYwF3bONA@mail.gmail.com>
	<20130727002359.2631ea8a@fsol>
	<CAGE7PNLfZKEtXSNNUN=fq+gHXkQsp0iuyz2SgQDZJowFc2spnA@mail.gmail.com>
	<CAP7+vJ+mKg2sL8T8ecCOjRq7tJxCLqnNV=O=H78dNpuEak5yjw@mail.gmail.com>
Message-ID: <CADiSq7dJ4dL=kTF6K6hbPJO-KvQM+6i1XvzE0NK4v=Xw4iwbzg@mail.gmail.com>

On 27 July 2013 14:36, Guido van Rossum <guido at python.org> wrote:
> On Fri, Jul 26, 2013 at 9:26 PM, Gregory P. Smith <greg at krypto.org> wrote:
>>
>> On Fri, Jul 26, 2013 at 3:23 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>>>
>>> On Sat, 27 Jul 2013 00:18:40 +0200
>>> Victor Stinner <victor.stinner at gmail.com> wrote:
>>> > 2013/7/26 Antoine Pitrou <solipsis at pitrou.net>:
>>> > > On Fri, 26 Jul 2013 22:17:47 +0200
>>> > >> """
>>> > >> On Linux, setting the close-on-flag has a low overhead on
>>> > >> performances. Results of bench_cloexec.py on Linux 3.6:
>>> > >>
>>> > >> - close-on-flag not set: 7.8 us
>>> > >> - O_CLOEXEC: 1% slower (7.9 us)
>>> > >> - ioctl(): 3% slower (8.0 us)
>>> > >> - fcntl(): 3% slower (8.0 us)
>>> > >> """
>>> > >
>>> > > You aren't answering my question: slower than what?
>>> >
>>> > Ah, you didn't understand the labels. bench_cloexec.py runs a
>>> > benchmark on os.open(path, os.O_RDONLY, cloexec=False) and
>>> > os.open(path, os.O_RDONLY, cloexec=True) with different implementation
>>> > of making the file descriptor non-inheritable.
>>> >
>>> > close-on-flag not set: 7.8 us
>>> > => C code: open(path, O_RDONLY)
>>> >
>>> > O_CLOEXEC: 1% slower (7.9 us)
>>> > => C code: open(path, O_RDONLY|CLOEXEC)
>>> > => 1% slower than open(path, O_RDONLY)
>>> >
>>> > ioctl(): 3% slower (8.0 us)
>>> > => C code: fd=open(path, O_RDONLY); ioctl(fd, FIOCLEX, 0)
>>> > => 3% slower than open(path, O_RDONLY)
>>> >
>>> > fcntl(): 3% slower (8.0 us)
>>> > => C code: fd=open(path, O_RDONLY); flags = fcntl(fd, F_GETFD);
>>> > fcntl(fd, F_SETFD, flags | FD_CLOEXEC)
>>> > => 3% slower than open(path, O_RDONLY)
>>>
>>> Ok, so I think this it is a totally reasonable compromise.
>>>
>>> People who bother about a 3% slowdown when calling os.open() can
>>> optimize the hell out of their code using Cython for all I care :-)
>>>
>>
>> +1  ;)
>>
>> and +1 for making the sane default of noinherit / cloexec /
>> whatever-others-call-it by default for all fds/handles ever opened by
>> Python. It stops ignoring the issue (ie: the status quo of matching the
>> default behavior of C as defined in the 1970s)... That is a GOOD thing. :)
>
> Do we even need a new PEP, or should we just do it? Or can we adapt
> Victor's PEP 446?

Adapting the PEP sounds good - while I agree with switching to a sane
default, I think the daemonisation thread suggests there may need to
be a supporting API to help force FDs created by nominated logging
handlers to be inherited.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From p.f.moore at gmail.com  Sat Jul 27 10:11:32 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 27 Jul 2013 09:11:32 +0100
Subject: [Python-Dev] Adding Python scripts to PATHEXT on Windows
In-Reply-To: <CACac1F9xLhfEPZuJ3-s8RaV-fi+UGaRMxp3SrzyupSkWWPG9OA@mail.gmail.com>
References: <CACac1F9WtTj69gBv2SKR3chxFdduhNtxNOmPqrt=mK5kHb_ZMA@mail.gmail.com>
	<51EEAB2E.9030804@v.loewis.de>
	<CACac1F9xLhfEPZuJ3-s8RaV-fi+UGaRMxp3SrzyupSkWWPG9OA@mail.gmail.com>
Message-ID: <CACac1F8nHjjTx-keO_E2cK8dTtWFawiqOo-gH0ZGHHPzEe81vA@mail.gmail.com>

On 23 July 2013 17:33, Paul Moore <p.f.moore at gmail.com> wrote:

> On 23 July 2013 17:11, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>
>> Am 15.07.13 10:26, schrieb Paul Moore:
>> > Does anyone have any objections to this? I could try to write a patch,
>> > but I know next to nothing about building MSIs, so if any of the
>> > installer experts could help that would be fantastic.
>>
>> It's fine with me. I could write the patch, but will likely forget - so
>> please remind me with every alpha that misses this. Also, contributions
>> are welcome.
>>
>
> Thanks.
>
> I think I have the basic idea of what's needed, so I'll write an initial
> patch. If you can check it that'd be great.
>

OK, see http://bugs.python.org/issue18569

As I mention in the patch, I have yet to successfully manage to build an
MSI installer (I tried a while ago, and the instructions didn't work - I'm
not 100% sure why and I'm afraid I don't have the time right now to try
again) so the patch will definitely need checking by someone who can build
the MSI.

When I get some time (probably not for a few weeks) I'll try the "how to
build the installer" instructions again and see if I can work out what went
wrong and either fix the issue or report back.

Paul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130727/d8a19b16/attachment.html>

From victor.stinner at gmail.com  Sat Jul 27 15:44:20 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sat, 27 Jul 2013 15:44:20 +0200
Subject: [Python-Dev] Inherance of file descriptor and handles on
 Windows (PEP 446)
In-Reply-To: <CAP7+vJ+mKg2sL8T8ecCOjRq7tJxCLqnNV=O=H78dNpuEak5yjw@mail.gmail.com>
References: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
	<ksn359$q3a$1@ger.gmane.org>
	<CAP7+vJKGUS9cseDz+LgV0Z5DKJyYzEru6fAr1=Ct9GMBYB7i_w@mail.gmail.com>
	<CAMpsgwZq9qR-BBSoXNDwZq=VYmYcv5AMgjv7bPnHBhPLfse6HA@mail.gmail.com>
	<20130726142525.79448c82@pitrou.net>
	<CAMpsgwZXNeiZqbZUh+VvtSnNjBM69_b45wAYZf+7GfXg2X=R0Q@mail.gmail.com>
	<20130726232432.3651f1d5@fsol>
	<CAMpsgwaSXugXKwU5AkBN0fHZ0WeJWfWkNhdnS9b9XeYwF3bONA@mail.gmail.com>
	<20130727002359.2631ea8a@fsol>
	<CAGE7PNLfZKEtXSNNUN=fq+gHXkQsp0iuyz2SgQDZJowFc2spnA@mail.gmail.com>
	<CAP7+vJ+mKg2sL8T8ecCOjRq7tJxCLqnNV=O=H78dNpuEak5yjw@mail.gmail.com>
Message-ID: <CAMpsgwYLThSGwQBX=OWz-0JVtuppRhcCK4S2T_ghie1Z8aNywA@mail.gmail.com>

2013/7/27 Guido van Rossum <guido at python.org>:
> Do we even need a new PEP, or should we just do it? Or can we adapt
> Victor's PEP 446?

I can rewrite the PEP 446 to:

* make all file descriptors and handles non-inheritable
* remove the cloexec parameter
* remove everything about non-blocking sockets (O_NONBLOCK), it should
be discussed in a new PEP (it's no more related to O_CLOEXEC /
HANDLE_INHERIT_FLAG)

Should I rename os.set_cloexec(fd, cloexec) to os.set_inheritable(fd,
inheritable), and os.get_cloexec(fd) to os.get_inheritable(fd)?

Or do you prefer a simple os.make_inheritable(fd) with no inheritable
parameter? I prefer an explicit parameter, so it's also possible to
force again non-inheritable, which also makes sense if the file
descriptor was not created by Python.

Victor

From guido at python.org  Sat Jul 27 16:36:19 2013
From: guido at python.org (Guido van Rossum)
Date: Sat, 27 Jul 2013 07:36:19 -0700
Subject: [Python-Dev] Inherance of file descriptor and handles on
 Windows (PEP 446)
In-Reply-To: <CAMpsgwYLThSGwQBX=OWz-0JVtuppRhcCK4S2T_ghie1Z8aNywA@mail.gmail.com>
References: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
	<ksn359$q3a$1@ger.gmane.org>
	<CAP7+vJKGUS9cseDz+LgV0Z5DKJyYzEru6fAr1=Ct9GMBYB7i_w@mail.gmail.com>
	<CAMpsgwZq9qR-BBSoXNDwZq=VYmYcv5AMgjv7bPnHBhPLfse6HA@mail.gmail.com>
	<20130726142525.79448c82@pitrou.net>
	<CAMpsgwZXNeiZqbZUh+VvtSnNjBM69_b45wAYZf+7GfXg2X=R0Q@mail.gmail.com>
	<20130726232432.3651f1d5@fsol>
	<CAMpsgwaSXugXKwU5AkBN0fHZ0WeJWfWkNhdnS9b9XeYwF3bONA@mail.gmail.com>
	<20130727002359.2631ea8a@fsol>
	<CAGE7PNLfZKEtXSNNUN=fq+gHXkQsp0iuyz2SgQDZJowFc2spnA@mail.gmail.com>
	<CAP7+vJ+mKg2sL8T8ecCOjRq7tJxCLqnNV=O=H78dNpuEak5yjw@mail.gmail.com>
	<CAMpsgwYLThSGwQBX=OWz-0JVtuppRhcCK4S2T_ghie1Z8aNywA@mail.gmail.com>
Message-ID: <CAP7+vJJ=4qUDHBeYWgScHki=ndzDxu9DiD47Up1yYPVgr_mtjQ@mail.gmail.com>

On Saturday, July 27, 2013, Victor Stinner wrote:

> 2013/7/27 Guido van Rossum <guido at python.org <javascript:;>>:
> > Do we even need a new PEP, or should we just do it? Or can we adapt
> > Victor's PEP 446?
>
> I can rewrite the PEP 446 to:
>
> * make all file descriptors and handles non-inheritable
> * remove the cloexec parameter
> * remove everything about non-blocking sockets (O_NONBLOCK), it should
> be discussed in a new PEP (it's no more related to O_CLOEXEC /
> HANDLE_INHERIT_FLAG)


Sounds good.

>
> Should I rename os.set_cloexec(fd, cloexec) to os.set_inheritable(fd,
> inheritable), and os.get_cloexec(fd) to os.get_inheritable(fd)?


Yes.

>
> Or do you prefer a simple os.make_inheritable(fd) with no inheritable
> parameter? I prefer an explicit parameter, so it's also possible to
> force again non-inheritable, which also makes sense if the file
> descriptor was not created by Python.


Agreed.

>
> Victor
>


-- 
--Guido van Rossum (on iPad)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130727/1ba77198/attachment.html>

From guido at python.org  Sat Jul 27 16:41:30 2013
From: guido at python.org (Guido van Rossum)
Date: Sat, 27 Jul 2013 07:41:30 -0700
Subject: [Python-Dev] Inherance of file descriptor and handles on
 Windows (PEP 446)
In-Reply-To: <CAP7+vJJ=4qUDHBeYWgScHki=ndzDxu9DiD47Up1yYPVgr_mtjQ@mail.gmail.com>
References: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
	<ksn359$q3a$1@ger.gmane.org>
	<CAP7+vJKGUS9cseDz+LgV0Z5DKJyYzEru6fAr1=Ct9GMBYB7i_w@mail.gmail.com>
	<CAMpsgwZq9qR-BBSoXNDwZq=VYmYcv5AMgjv7bPnHBhPLfse6HA@mail.gmail.com>
	<20130726142525.79448c82@pitrou.net>
	<CAMpsgwZXNeiZqbZUh+VvtSnNjBM69_b45wAYZf+7GfXg2X=R0Q@mail.gmail.com>
	<20130726232432.3651f1d5@fsol>
	<CAMpsgwaSXugXKwU5AkBN0fHZ0WeJWfWkNhdnS9b9XeYwF3bONA@mail.gmail.com>
	<20130727002359.2631ea8a@fsol>
	<CAGE7PNLfZKEtXSNNUN=fq+gHXkQsp0iuyz2SgQDZJowFc2spnA@mail.gmail.com>
	<CAP7+vJ+mKg2sL8T8ecCOjRq7tJxCLqnNV=O=H78dNpuEak5yjw@mail.gmail.com>
	<CAMpsgwYLThSGwQBX=OWz-0JVtuppRhcCK4S2T_ghie1Z8aNywA@mail.gmail.com>
	<CAP7+vJJ=4qUDHBeYWgScHki=ndzDxu9DiD47Up1yYPVgr_mtjQ@mail.gmail.com>
Message-ID: <CAP7+vJJ0DOM53sO+-O-A7w0CarHh9wNBMVi-cj1qzVMLjupRGw@mail.gmail.com>

P.S. perhaps more important than a PEP rewrite is a working patch to see
how realistic this is. Could you make the alpha 1 release?

On Saturday, July 27, 2013, Guido van Rossum wrote:

>
>
> On Saturday, July 27, 2013, Victor Stinner wrote:
>
>> 2013/7/27 Guido van Rossum <guido at python.org>:
>> > Do we even need a new PEP, or should we just do it? Or can we adapt
>> > Victor's PEP 446?
>>
>> I can rewrite the PEP 446 to:
>>
>> * make all file descriptors and handles non-inheritable
>> * remove the cloexec parameter
>> * remove everything about non-blocking sockets (O_NONBLOCK), it should
>> be discussed in a new PEP (it's no more related to O_CLOEXEC /
>> HANDLE_INHERIT_FLAG)
>
>
> Sounds good.
>
>>
>> Should I rename os.set_cloexec(fd, cloexec) to os.set_inheritable(fd,
>> inheritable), and os.get_cloexec(fd) to os.get_inheritable(fd)?
>
>
> Yes.
>
>>
>> Or do you prefer a simple os.make_inheritable(fd) with no inheritable
>> parameter? I prefer an explicit parameter, so it's also possible to
>> force again non-inheritable, which also makes sense if the file
>> descriptor was not created by Python.
>
>
> Agreed.
>
>>
>> Victor
>>
>
>
> --
> --Guido van Rossum (on iPad)
>


-- 
--Guido van Rossum (on iPad)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130727/089c46e2/attachment.html>

From victor.stinner at gmail.com  Sat Jul 27 19:56:10 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sat, 27 Jul 2013 19:56:10 +0200
Subject: [Python-Dev] Inherance of file descriptor and handles on
 Windows (PEP 446)
In-Reply-To: <CAP7+vJJ0DOM53sO+-O-A7w0CarHh9wNBMVi-cj1qzVMLjupRGw@mail.gmail.com>
References: <CAMpsgwZugy=R-UEjMtaOh-LgY89DD8=sT3tN3xjZPGkL_6_JKA@mail.gmail.com>
	<ksn359$q3a$1@ger.gmane.org>
	<CAP7+vJKGUS9cseDz+LgV0Z5DKJyYzEru6fAr1=Ct9GMBYB7i_w@mail.gmail.com>
	<CAMpsgwZq9qR-BBSoXNDwZq=VYmYcv5AMgjv7bPnHBhPLfse6HA@mail.gmail.com>
	<20130726142525.79448c82@pitrou.net>
	<CAMpsgwZXNeiZqbZUh+VvtSnNjBM69_b45wAYZf+7GfXg2X=R0Q@mail.gmail.com>
	<20130726232432.3651f1d5@fsol>
	<CAMpsgwaSXugXKwU5AkBN0fHZ0WeJWfWkNhdnS9b9XeYwF3bONA@mail.gmail.com>
	<20130727002359.2631ea8a@fsol>
	<CAGE7PNLfZKEtXSNNUN=fq+gHXkQsp0iuyz2SgQDZJowFc2spnA@mail.gmail.com>
	<CAP7+vJ+mKg2sL8T8ecCOjRq7tJxCLqnNV=O=H78dNpuEak5yjw@mail.gmail.com>
	<CAMpsgwYLThSGwQBX=OWz-0JVtuppRhcCK4S2T_ghie1Z8aNywA@mail.gmail.com>
	<CAP7+vJJ=4qUDHBeYWgScHki=ndzDxu9DiD47Up1yYPVgr_mtjQ@mail.gmail.com>
	<CAP7+vJJ0DOM53sO+-O-A7w0CarHh9wNBMVi-cj1qzVMLjupRGw@mail.gmail.com>
Message-ID: <CAMpsgwa5GUrmRtQn1o_zUz4daZTNriP8hzaJ8dxgDAf9JWz96A@mail.gmail.com>

2013/7/27 Guido van Rossum <guido at python.org>:
> P.S. perhaps more important than a PEP rewrite is a working patch to see how
> realistic this is. Could you make the alpha 1 release?

I already ran the whole Python test suite with non-inheritable file
descriptors when I developed the PEP 433: it just works. So I'm
confident :-)

I "just" had to fix the cgi module, and some tests. For example,
test_socket checks the exact type of sockets, whereas SOCK_CLOEXEC
flag is present in sockobj.type for non-inheritable sockets created
with this flag.

I implemented the *new* PEP 446 (not written yet :-)) in a new repository:
http://hg.python.org/features/pep-446

I had to invert the value of cloexec (inheritable value is just the opposite).

The implementation works but it is not completed:

* The doc should be reviewed
* test_swap_fds() of test_subprocess fails
* The implementation should be tested on Windows, FreeBSD and Solaris
* I have to check if _Py_try_set_inheritable() can/must be replaced
with _Py_set_inheritable()

The implementation can be seen as a patch and reviewed using the
following new issue:
http://bugs.python.org/issue18571

Victor

From Steve.Dower at microsoft.com  Sat Jul 27 22:14:49 2013
From: Steve.Dower at microsoft.com (Steve Dower)
Date: Sat, 27 Jul 2013 20:14:49 +0000
Subject: [Python-Dev] Adding Python scripts to PATHEXT on Windows
In-Reply-To: <CACac1F8nHjjTx-keO_E2cK8dTtWFawiqOo-gH0ZGHHPzEe81vA@mail.gmail.com>
References: <CACac1F9WtTj69gBv2SKR3chxFdduhNtxNOmPqrt=mK5kHb_ZMA@mail.gmail.com>
	<51EEAB2E.9030804@v.loewis.de>
	<CACac1F9xLhfEPZuJ3-s8RaV-fi+UGaRMxp3SrzyupSkWWPG9OA@mail.gmail.com>,
	<CACac1F8nHjjTx-keO_E2cK8dTtWFawiqOo-gH0ZGHHPzEe81vA@mail.gmail.com>
Message-ID: <e54095b20f004c65a76db155b17d042b@BLUPR03MB199.namprd03.prod.outlook.com>

Any chance of this being made optional when installing? It provides no benefit for people who prefer to associate scripts with an editor and may be a source of confusion/complaints.

Env var changes typically require a reboot to take full effect, which I've also seen cause confusion.

Also, it should include .pyw, .pyz and whatever was decided instead of .pywz (.pwz, I think?).

Steve

(Apologies for the top post.)
Sent from my Windows Phone
________________________________
From: Paul Moore<mailto:p.f.moore at gmail.com>
Sent: ?7/?27/?2013 1:13
To: Martin v. L?wis<mailto:martin at v.loewis.de>
Cc: Python Dev<mailto:python-dev at python.org>
Subject: Re: [Python-Dev] Adding Python scripts to PATHEXT on Windows

On 23 July 2013 17:33, Paul Moore <p.f.moore at gmail.com<mailto:p.f.moore at gmail.com>> wrote:
On 23 July 2013 17:11, "Martin v. L?wis" <martin at v.loewis.de<mailto:martin at v.loewis.de>> wrote:
Am 15.07.13 10:26, schrieb Paul Moore:
> Does anyone have any objections to this? I could try to write a patch,
> but I know next to nothing about building MSIs, so if any of the
> installer experts could help that would be fantastic.

It's fine with me. I could write the patch, but will likely forget - so
please remind me with every alpha that misses this. Also, contributions
are welcome.

Thanks.

I think I have the basic idea of what's needed, so I'll write an initial patch. If you can check it that'd be great.

OK, see http://bugs.python.org/issue18569

As I mention in the patch, I have yet to successfully manage to build an MSI installer (I tried a while ago, and the instructions didn't work - I'm not 100% sure why and I'm afraid I don't have the time right now to try again) so the patch will definitely need checking by someone who can build the MSI.

When I get some time (probably not for a few weeks) I'll try the "how to build the installer" instructions again and see if I can work out what went wrong and either fix the issue or report back.

Paul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130727/ab161ced/attachment.html>

From p.f.moore at gmail.com  Sun Jul 28 00:36:19 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 27 Jul 2013 23:36:19 +0100
Subject: [Python-Dev] Adding Python scripts to PATHEXT on Windows
In-Reply-To: <e54095b20f004c65a76db155b17d042b@BLUPR03MB199.namprd03.prod.outlook.com>
References: <CACac1F9WtTj69gBv2SKR3chxFdduhNtxNOmPqrt=mK5kHb_ZMA@mail.gmail.com>
	<51EEAB2E.9030804@v.loewis.de>
	<CACac1F9xLhfEPZuJ3-s8RaV-fi+UGaRMxp3SrzyupSkWWPG9OA@mail.gmail.com>
	<CACac1F8nHjjTx-keO_E2cK8dTtWFawiqOo-gH0ZGHHPzEe81vA@mail.gmail.com>
	<e54095b20f004c65a76db155b17d042b@BLUPR03MB199.namprd03.prod.outlook.com>
Message-ID: <CACac1F_Bt9eiciVb10m4JTpVKp7WxmBrvvs1oA6UcaJrcQiR=A@mail.gmail.com>

On 27 July 2013 21:14, Steve Dower <Steve.Dower at microsoft.com> wrote:

>  Any chance of this being made optional when installing? It provides no
> benefit for people who prefer to associate scripts with an editor and may
> be a source of confusion/complaints.
>

Personally, I don't know how to do this so someone else would have to - but
if someone does, I won't particularly object. I'd prefer it to be on by
default, though (just like associating py.exe with scripts is) because
otherwise it's not reasonable to assume that a .py script will work as a
command in the same way that a bat file does - which is really the point of
wanting this in the first place.

I'm a bit confused by your comment about "people who prefer to associate
scripts with an editor", though. This is only done in precisely those
situations when the installer associates .py scripts with py.exe (the
launcher). So if you switch that off, you don't get PATHEXT either. And if
you change the association after the fact, you're presumably just as
capable of changing PATHEXT.

The key point here is that I want the *default* behaviour of Python when
installed using the standard msi to be that if I put a script foo.py on
PATH somewhere, then typing "foo" at the command line will run that script.
For that to be the case, PATHEXT needs to be set, and I view that as a
(minor) omission which was made when the association with py.exe was
implemented.

(Which reminds me - it would be nice if the standalone installer for the py
launcher included setting PATHEXT as well).


> Env var changes typically require a reboot to take full effect, which I've
> also seen cause confusion.
>

Really? If I add python.exe to PATH in the installer do I need to reboot?
(I never do this, so I don't know - it's a genuine question).

Also, it should include .pyw, .pyz and whatever was decided instead of
> .pywz (.pwz, I think?).
>

It's basically a command line thing - I don't see why you'd want to launch
a GUI script from the command line in the same way. But again, if someone
wants to do it, I don't really mind (in this case I know what to do, so if
there's consensus that it's needed, I can add it to the patch myself).

The PEP for pyz and pwz hasn't been accepted yet. If it does, then yes pyz
should be treated the same way (and we can have the same debate about pwz
as we do about pyw :-))

Paul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130727/6c232ba8/attachment.html>

From Steve.Dower at microsoft.com  Sun Jul 28 01:30:47 2013
From: Steve.Dower at microsoft.com (Steve Dower)
Date: Sat, 27 Jul 2013 23:30:47 +0000
Subject: [Python-Dev] Adding Python scripts to PATHEXT on Windows
In-Reply-To: <CACac1F_Bt9eiciVb10m4JTpVKp7WxmBrvvs1oA6UcaJrcQiR=A@mail.gmail.com>
References: <CACac1F9WtTj69gBv2SKR3chxFdduhNtxNOmPqrt=mK5kHb_ZMA@mail.gmail.com>
	<51EEAB2E.9030804@v.loewis.de>
	<CACac1F9xLhfEPZuJ3-s8RaV-fi+UGaRMxp3SrzyupSkWWPG9OA@mail.gmail.com>
	<CACac1F8nHjjTx-keO_E2cK8dTtWFawiqOo-gH0ZGHHPzEe81vA@mail.gmail.com>
	<e54095b20f004c65a76db155b17d042b@BLUPR03MB199.namprd03.prod.outlook.com>,
	<CACac1F_Bt9eiciVb10m4JTpVKp7WxmBrvvs1oA6UcaJrcQiR=A@mail.gmail.com>
Message-ID: <70f1d786f4e8472c9cd29b38f20675fe@BLUPR03MB199.namprd03.prod.outlook.com>

From: Paul Moore <p.f.moore at gmail.com>
> I'm a bit confused by your comment about "people who prefer to associate scripts
> with an editor", though. This is only done in precisely those situations when
> the installer associates .py scripts with py.exe (the launcher). So if you
> switch that off, you don't get PATHEXT either.

That's optional enough for me - I always disable that (because I typically have 4-10 different versions installed at once).

> And if you change the association after the fact, you're presumably just as capable
> of changing PATHEXT.

Not if the association is changed by another installer (presumably with the user's explicit permission). It would be very easy for people to go a long time without noticing this until some program does ShellExecute(...) on a name that happens to be a Python script and Visual Studio pops open...

>> Env var changes typically require a reboot to take full effect, which I've also seen cause confusion.
> Really? If I add python.exe to PATH in the installer do I need to reboot? (I
> never do this, so I don't know - it's a genuine question).

Env vars are inherited by child processes, so until you create a process without inheriting them, you won't see the new value. Depending on the shortcut used, cmd.exe is likely to inherit the variables from explorer.exe, which won't see the new ones until you log off/on again. Certainly any running prompts will need to be closed and restarted after installing.

That said, on my Win8 machine, cmd.exe seems to be okay after changing the variables. However, PowerShell did not see the change immediately, so cmd.exe may be getting special treatment (or just ignoring the default environment).

>> Also, it should include .pyw, .pyz and whatever was decided instead
>> of .pywz (.pwz, I think?).
>
> It's basically a command line thing - I don't see why you'd want to launch
> a GUI script from the command line in the same way.

That's fair enough, though it may be seen as inconsistent. TBH I've never actually seen anything in the wild other than .py.

My suspicion is that the .exe wrappers would remain for tools like pip anyway, rather than breaking people who choose to deselect that option. Of course, it's easy enough to replace them.

Another issue to consider is that the modification to PATHEXT can't be undone when Python is uninstalled, unless each installation adds another ".PY" and each uninstall removes only one (so my PATHEXT would look like ...;.PY;.PY;.PY;.PY;.PY;.PY;.PY;.PY;.PY;.PY). I haven't checked, but this could seriously hurt performance of anything that uses PATHEXT to search PATH for anything ending in this extension. I hope that cmd.exe and PowerShell check for duplicate extensions, but I wouldn't be betting any money that they do.


Steve



From victor.stinner at gmail.com  Sun Jul 28 02:39:19 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sun, 28 Jul 2013 02:39:19 +0200
Subject: [Python-Dev] PEP 446: Open issues/questions
Message-ID: <CAMpsgwZJuAJaa=4dYh=q_fFvg51T4j+ZGQSuWGym+1q0kHX7jQ@mail.gmail.com>

Hi,

I have a few more questions on the PEP 446:

(A) How should we support support where os.set_inheritable() is not
supported? Can we announce that os.set_inheritable() is always
available or not? Does such platform exist?

(B) Should subprocess make the file descriptors of pass_fds
inheritable? If yes, should it be done before or after the fork? If it
is done after the fork and before exec, it only affects the child
process, at least on Linux (the file descriptor is still
non-inheritable in the parent process).

(C) Should we handle standard streams (0: stdin, 1: stdout, 2: stderr)
differently? For example, os.dup2(fd, 0) should make the file
descriptor 0 (stdin) inheritable or non-inheritable? On Windows,
os.set_inheritable(fd, False) fails (error 87, invalid argument) on
standard streams (0, 1, 2) and copies of the standard streams (created
by os.dup()).

Victor

From p.f.moore at gmail.com  Sun Jul 28 09:53:18 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Sun, 28 Jul 2013 08:53:18 +0100
Subject: [Python-Dev] Adding Python scripts to PATHEXT on Windows
In-Reply-To: <70f1d786f4e8472c9cd29b38f20675fe@BLUPR03MB199.namprd03.prod.outlook.com>
References: <CACac1F9WtTj69gBv2SKR3chxFdduhNtxNOmPqrt=mK5kHb_ZMA@mail.gmail.com>
	<51EEAB2E.9030804@v.loewis.de>
	<CACac1F9xLhfEPZuJ3-s8RaV-fi+UGaRMxp3SrzyupSkWWPG9OA@mail.gmail.com>
	<CACac1F8nHjjTx-keO_E2cK8dTtWFawiqOo-gH0ZGHHPzEe81vA@mail.gmail.com>
	<e54095b20f004c65a76db155b17d042b@BLUPR03MB199.namprd03.prod.outlook.com>
	<CACac1F_Bt9eiciVb10m4JTpVKp7WxmBrvvs1oA6UcaJrcQiR=A@mail.gmail.com>
	<70f1d786f4e8472c9cd29b38f20675fe@BLUPR03MB199.namprd03.prod.outlook.com>
Message-ID: <CACac1F9u=gDaOnchyruqZihvYsYPbLLdSSFVFVg63f7Vz-aZqw@mail.gmail.com>

On 28 July 2013 00:30, Steve Dower <Steve.Dower at microsoft.com> wrote:

> Another issue to consider is that the modification to PATHEXT can't be
> undone when Python is uninstalled, unless each installation adds another
> ".PY" and each uninstall removes only one (so my PATHEXT would look like
> ...;.PY;.PY;.PY;.PY;.PY;.PY;.PY;.PY;.PY;.PY). I haven't checked, but this
> could seriously hurt performance of anything that uses PATHEXT to search
> PATH for anything ending in this extension. I hope that cmd.exe and
> PowerShell check for duplicate extensions, but I wouldn't be betting any
> money that they do.
>

I used the same mechanism in msi.py (which is MSI functionality, I believe)
that is used to add Python's directory to PATH - this adds on install and
removes on ininstall. So it should work just as well as that does.

Paul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130728/0b59a3cc/attachment.html>

From p.f.moore at gmail.com  Sun Jul 28 09:58:51 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Sun, 28 Jul 2013 08:58:51 +0100
Subject: [Python-Dev] Adding Python scripts to PATHEXT on Windows
In-Reply-To: <70f1d786f4e8472c9cd29b38f20675fe@BLUPR03MB199.namprd03.prod.outlook.com>
References: <CACac1F9WtTj69gBv2SKR3chxFdduhNtxNOmPqrt=mK5kHb_ZMA@mail.gmail.com>
	<51EEAB2E.9030804@v.loewis.de>
	<CACac1F9xLhfEPZuJ3-s8RaV-fi+UGaRMxp3SrzyupSkWWPG9OA@mail.gmail.com>
	<CACac1F8nHjjTx-keO_E2cK8dTtWFawiqOo-gH0ZGHHPzEe81vA@mail.gmail.com>
	<e54095b20f004c65a76db155b17d042b@BLUPR03MB199.namprd03.prod.outlook.com>
	<CACac1F_Bt9eiciVb10m4JTpVKp7WxmBrvvs1oA6UcaJrcQiR=A@mail.gmail.com>
	<70f1d786f4e8472c9cd29b38f20675fe@BLUPR03MB199.namprd03.prod.outlook.com>
Message-ID: <CACac1F8focxnp5CXD8Hw4E8EE8giQs65RdYqK0ghHQzrG_9=5Q@mail.gmail.com>

On 28 July 2013 00:30, Steve Dower <Steve.Dower at microsoft.com> wrote:

> > And if you change the association after the fact, you're presumably just
> as capable
> > of changing PATHEXT.
>
> Not if the association is changed by another installer (presumably with
> the user's explicit permission). It would be very easy for people to go a
> long time without noticing this until some program does ShellExecute(...)
> on a name that happens to be a Python script and Visual Studio pops open...


Mph. I'd argue that as Python "owns" the .py extension, any other
application that changes the default action of .py scripts without both
making that behaviour optional and making it off by default, is at least
unfriendly and arguably even broken. But that's just a matter of opinion...

As regards ShellExecute, I don't know how that is affected by PATHEXT. I
looked quite hard and couldn't find complete "official" documentation on
how PATHEXT is supposed to work, so I'm working a bit on experiment and
best guesses here :-( I'll go and check what the ShellExecute documentation
says on the matter (but I'm surprised - I got the impression that PATHEXT
was purely a console mechanism, so I don't quite understand why
ShellExecute would use it).

Paul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130728/cc8e3dc6/attachment.html>

From ncoghlan at gmail.com  Sun Jul 28 10:11:21 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 28 Jul 2013 18:11:21 +1000
Subject: [Python-Dev] Adding Python scripts to PATHEXT on Windows
In-Reply-To: <CACac1F_Bt9eiciVb10m4JTpVKp7WxmBrvvs1oA6UcaJrcQiR=A@mail.gmail.com>
References: <CACac1F9WtTj69gBv2SKR3chxFdduhNtxNOmPqrt=mK5kHb_ZMA@mail.gmail.com>
	<51EEAB2E.9030804@v.loewis.de>
	<CACac1F9xLhfEPZuJ3-s8RaV-fi+UGaRMxp3SrzyupSkWWPG9OA@mail.gmail.com>
	<CACac1F8nHjjTx-keO_E2cK8dTtWFawiqOo-gH0ZGHHPzEe81vA@mail.gmail.com>
	<e54095b20f004c65a76db155b17d042b@BLUPR03MB199.namprd03.prod.outlook.com>
	<CACac1F_Bt9eiciVb10m4JTpVKp7WxmBrvvs1oA6UcaJrcQiR=A@mail.gmail.com>
Message-ID: <CADiSq7fLDUvm6mEyY2OfZ1ef9Yp4L=WOVeOmk_B8bQ0FQs6FXQ@mail.gmail.com>

On 28 July 2013 08:36, Paul Moore <p.f.moore at gmail.com> wrote:
> The PEP for pyz and pwz hasn't been accepted yet. If it does, then yes pyz
> should be treated the same way (and we can have the same debate about pwz as
> we do about pyw :-))

Oops, thanks for the reminder. I'll nudge Daniel about that :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From solipsis at pitrou.net  Sun Jul 28 11:43:41 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 28 Jul 2013 11:43:41 +0200
Subject: [Python-Dev] PEP 446: Open issues/questions
References: <CAMpsgwZJuAJaa=4dYh=q_fFvg51T4j+ZGQSuWGym+1q0kHX7jQ@mail.gmail.com>
Message-ID: <20130728114341.23ec940f@fsol>

On Sun, 28 Jul 2013 02:39:19 +0200
Victor Stinner <victor.stinner at gmail.com> wrote:
> Hi,
> 
> I have a few more questions on the PEP 446:
> 
> (A) How should we support support where os.set_inheritable() is not
> supported? Can we announce that os.set_inheritable() is always
> available or not? Does such platform exist?

FD_CLOEXEC is POSIX:
http://pubs.opengroup.org/onlinepubs/9699919799/functions/fcntl.html

> (B) Should subprocess make the file descriptors of pass_fds
> inheritable? If yes, should it be done before or after the fork? If it
> is done after the fork and before exec, it only affects the child
> process, at least on Linux (the file descriptor is still
> non-inheritable in the parent process).

If it is done, it should definitely be done after the fork, IMO.
subprocess shouldn't have any long-lasting effects on the process.

> (C) Should we handle standard streams (0: stdin, 1: stdout, 2: stderr)
> differently? For example, os.dup2(fd, 0) should make the file
> descriptor 0 (stdin) inheritable or non-inheritable? On Windows,
> os.set_inheritable(fd, False) fails (error 87, invalid argument) on
> standard streams (0, 1, 2) and copies of the standard streams (created
> by os.dup()).

I have been advocating for that, but I now realize that special-casing
these three descriptors in a myriad of fd-creating functions isn't very
attractive.
(if a standard stream fd has been closed, any fd-creating function can
re-create that fd: including socket.socket(), etc.)

So perhaps only the *original* standard streams should be left
inheritable?

The Windows error is a bit more worrying: will it bubble up to Python
code when set_inheritable(fd, False) is called implicitly by a
fd-creating function? Should set_inheritable() be best effort and
silence that error?

Regards

Antoine.



From cf.natali at gmail.com  Sun Jul 28 12:18:55 2013
From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=)
Date: Sun, 28 Jul 2013 12:18:55 +0200
Subject: [Python-Dev] PEP 446: Open issues/questions
In-Reply-To: <20130728114341.23ec940f@fsol>
References: <CAMpsgwZJuAJaa=4dYh=q_fFvg51T4j+ZGQSuWGym+1q0kHX7jQ@mail.gmail.com>
	<20130728114341.23ec940f@fsol>
Message-ID: <CAH_1eM0XMiFct_Ef3eZjvgCCLFyDsja0icrNvWnYNQS-_BNYKA@mail.gmail.com>

2013/7/28 Antoine Pitrou <solipsis at pitrou.net>:
>> (C) Should we handle standard streams (0: stdin, 1: stdout, 2: stderr)
>> differently? For example, os.dup2(fd, 0) should make the file
>> descriptor 0 (stdin) inheritable or non-inheritable? On Windows,
>> os.set_inheritable(fd, False) fails (error 87, invalid argument) on
>> standard streams (0, 1, 2) and copies of the standard streams (created
>> by os.dup()).
>
> I have been advocating for that, but I now realize that special-casing
> these three descriptors in a myriad of fd-creating functions isn't very
> attractive.
> (if a standard stream fd has been closed, any fd-creating function can
> re-create that fd: including socket.socket(), etc.)
>
> So perhaps only the *original* standard streams should be left
> inheritable?

Having stdin/stdout/stderr cloexec (e.g. after a dup() to redirect to
a log file, a socket...) will likely break a lot of code, e.g. code
using os.system(), or code calling exec manually (and I'm sure there's
a bunch of it).

Also, it'll be puzzling to have syscalls automatically set the cloexec
flag. I guess a lot of people doing system programming with Python
will get bitten, but that's a discussion we already had months ago...

cf

From ncoghlan at gmail.com  Sun Jul 28 12:20:46 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 28 Jul 2013 20:20:46 +1000
Subject: [Python-Dev] Refactoring test.support into a subpackage
In-Reply-To: <CADiSq7f3nxGaEk2nUztGOHim4sk6oG9ZyFT1maOxnX7H5Y=0ig@mail.gmail.com>
References: <CADiSq7f3nxGaEk2nUztGOHim4sk6oG9ZyFT1maOxnX7H5Y=0ig@mail.gmail.com>
Message-ID: <CADiSq7eFXOSumQZn-tfNrfxmpjKzkGMCuMQ+qnyxsUv-SD5e4g@mail.gmail.com>

On 15 July 2013 20:22, Nick Coghlan <ncoghlan at gmail.com> wrote:
> At the PyCon AU sprints, some of the sprinters worked on a plan Chris
> Jerdonek and I came up with months ago to convert test.support from a
> module into a subpackage.
>
> This plan arose from some nasty test suite hacks in the lead up to the
> release of Python 3.3, where some of the pkgutil tests ended up in
> test.test_runpy so they could use the package creation infrastructure
> I have in there without needing to extract a helper module with a
> release imminent.
>
> The scope of the plan covers a few distinct steps:
>
> 1. Move Lib/test/support.py to Lib/test/support/__init__.py
> (http://bugs.python.org/issue15494)
> 2. Add a new (documented) test.support.pkg_helper submodule with the
> package creation support code (http://bugs.python.org/issue15403 and
> http://bugs.python.org/issue15376)
> 3. Move the pkgutil tests to test.test_pkgutil where they belong
> (http://bugs.python.org/issue15358)

Just a heads up that I'm working on these first three steps this
evening, so the Lib/test/support.py -> Lib/test/support/__init__.py
move will happen shortly.

I'll also be landing the patch to add test.support.temp_dir() and
test.support.change_cwd()

This shouldn't affect any other patches, unless they're adding or
change things in test.support.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From chris at python.org  Sun Jul 28 13:23:16 2013
From: chris at python.org (Chris Withers)
Date: Sun, 28 Jul 2013 12:23:16 +0100
Subject: [Python-Dev] Official github mirror for CPython?
In-Reply-To: <CAP1=2W4rM8CVAn45Mz5L=eic+ivuJOGSwqQrEFucTUp=nop=UA@mail.gmail.com>
References: <CAF-Rda-=wFp_pWSa2gqFxet1M7pf68si=LKZ7FyGU7UrfRS3FA@mail.gmail.com>
	<51F1380E.9070608@python.org>
	<CAD+XWwrCD_tLM0b1OFea5ZZq7oH+1TYR_yNyp=mNDKJe7ZBALQ@mail.gmail.com>
	<CAF-Rda82reci5WoY7J_1eZ_pBv+58sxXBaZ_MKSHef3onTuuRA@mail.gmail.com>
	<CAP1=2W4rM8CVAn45Mz5L=eic+ivuJOGSwqQrEFucTUp=nop=UA@mail.gmail.com>
Message-ID: <51F4FF24.2040600@python.org>

On 25/07/2013 16:30, Brett Cannon wrote:
> Based on the list of people who are members of github.com/python
> <http://github.com/python> it's as official as it's going to get
> (depends on who of that group owns it).
>
> But assuming whomever owns it is okay with hosting a mirror, what
> exactly is going to be required to have it updated regularly? Someone is
> going to have to write the scripts to pull from the hg repo and then
> push up to github else it's just going to end up out-of-date on top of
> maintaining whatever setup is devised.

Mike Bayer has set up some pretty slick stuff in this area for 
SQLAlchemy and Alembic.

I'm sure he could give pointers to anyone interested in getting it going...

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From ncoghlan at gmail.com  Sun Jul 28 14:41:51 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 28 Jul 2013 22:41:51 +1000
Subject: [Python-Dev] Refactoring test.support into a subpackage
In-Reply-To: <CADiSq7eFXOSumQZn-tfNrfxmpjKzkGMCuMQ+qnyxsUv-SD5e4g@mail.gmail.com>
References: <CADiSq7f3nxGaEk2nUztGOHim4sk6oG9ZyFT1maOxnX7H5Y=0ig@mail.gmail.com>
	<CADiSq7eFXOSumQZn-tfNrfxmpjKzkGMCuMQ+qnyxsUv-SD5e4g@mail.gmail.com>
Message-ID: <CADiSq7dNF2qL2xCHegxi5rXLO4qB6RNn8+yw0xkpFc5Fjvbekg@mail.gmail.com>

On 28 July 2013 20:20, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> The scope of the plan covers a few distinct steps:
>>
>> 1. Move Lib/test/support.py to Lib/test/support/__init__.py
>> (http://bugs.python.org/issue15494)
>> 2. Add a new (documented) test.support.pkg_helper submodule with the
>> package creation support code (http://bugs.python.org/issue15403 and
>> http://bugs.python.org/issue15376)
>> 3. Move the pkgutil tests to test.test_pkgutil where they belong
>> (http://bugs.python.org/issue15358)
>
> Just a heads up that I'm working on these first three steps this
> evening, so the Lib/test/support.py -> Lib/test/support/__init__.py
> move will happen shortly.
>
> I'll also be landing the patch to add test.support.temp_dir() and
> test.support.change_cwd()

The test.support switch (#15494) and the factoring out of the
temporary directory helpers to test.support (#15415) are done for both
3.3 and default (so it should be OK to rely on them without causing
merge conflicts in the tests).

#15403, #15376 and #15358 (the package testing cleanups) are still
open, but are also simpler to deal with now that test.support is a
package.

I also created #18576 to cover moving and documenting script_helper,
and #18578 to cover moving and documenting bytecode_helper.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From Steve.Dower at microsoft.com  Sun Jul 28 16:54:54 2013
From: Steve.Dower at microsoft.com (Steve Dower)
Date: Sun, 28 Jul 2013 14:54:54 +0000
Subject: [Python-Dev] Adding Python scripts to PATHEXT on Windows
In-Reply-To: <CACac1F8focxnp5CXD8Hw4E8EE8giQs65RdYqK0ghHQzrG_9=5Q@mail.gmail.com>
References: <CACac1F9WtTj69gBv2SKR3chxFdduhNtxNOmPqrt=mK5kHb_ZMA@mail.gmail.com>
	<51EEAB2E.9030804@v.loewis.de>
	<CACac1F9xLhfEPZuJ3-s8RaV-fi+UGaRMxp3SrzyupSkWWPG9OA@mail.gmail.com>
	<CACac1F8nHjjTx-keO_E2cK8dTtWFawiqOo-gH0ZGHHPzEe81vA@mail.gmail.com>
	<e54095b20f004c65a76db155b17d042b@BLUPR03MB199.namprd03.prod.outlook.com>
	<CACac1F_Bt9eiciVb10m4JTpVKp7WxmBrvvs1oA6UcaJrcQiR=A@mail.gmail.com>
	<70f1d786f4e8472c9cd29b38f20675fe@BLUPR03MB199.namprd03.prod.outlook.com>,
	<CACac1F8focxnp5CXD8Hw4E8EE8giQs65RdYqK0ghHQzrG_9=5Q@mail.gmail.com>
Message-ID: <65bbfabbef24446db8be71d231ad9afe@BLUPR03MB199.namprd03.prod.outlook.com>

Windows fully supports *user* customization of file associations and there is no concept of ownership. The first program installed that sets up an association will normally get it, and as of Win8 the next application will be blocked (and suggested to the user next time they open a file). Whether the other application stole the extension is irrelevant: a user is allowed to associate .py with something other than py.exe.

I'm also working off experience with Windows here, I haven't done any special research. It may be that the APIs will assume .exe if no extension is specified, but I'd be surprised if backwards compatibility with with DOS, which I assume included PATHEXT, hadn't been a consideration. The Run dialog uses ShellExecute, so you can test things there easily.

Sent from my Windows Phone
________________________________
From: Paul Moore<mailto:p.f.moore at gmail.com>
Sent: ?7/?28/?2013 1:00
To: Steve Dower<mailto:Steve.Dower at microsoft.com>
Cc: Martin v. L?wis<mailto:martin at v.loewis.de>; Python Dev<mailto:python-dev at python.org>
Subject: Re: [Python-Dev] Adding Python scripts to PATHEXT on Windows

On 28 July 2013 00:30, Steve Dower <Steve.Dower at microsoft.com<mailto:Steve.Dower at microsoft.com>> wrote:
> And if you change the association after the fact, you're presumably just as capable
> of changing PATHEXT.

Not if the association is changed by another installer (presumably with the user's explicit permission). It would be very easy for people to go a long time without noticing this until some program does ShellExecute(...) on a name that happens to be a Python script and Visual Studio pops open...

Mph. I'd argue that as Python "owns" the .py extension, any other application that changes the default action of .py scripts without both making that behaviour optional and making it off by default, is at least unfriendly and arguably even broken. But that's just a matter of opinion...

As regards ShellExecute, I don't know how that is affected by PATHEXT. I looked quite hard and couldn't find complete "official" documentation on how PATHEXT is supposed to work, so I'm working a bit on experiment and best guesses here :-( I'll go and check what the ShellExecute documentation says on the matter (but I'm surprised - I got the impression that PATHEXT was purely a console mechanism, so I don't quite understand why ShellExecute would use it).

Paul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130728/c603783e/attachment.html>

From solipsis at pitrou.net  Mon Jul 29 10:11:30 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 29 Jul 2013 10:11:30 +0200
Subject: [Python-Dev] PEP 446: Open issues/questions
References: <CAMpsgwZJuAJaa=4dYh=q_fFvg51T4j+ZGQSuWGym+1q0kHX7jQ@mail.gmail.com>
	<20130728114341.23ec940f@fsol>
	<CAH_1eM0XMiFct_Ef3eZjvgCCLFyDsja0icrNvWnYNQS-_BNYKA@mail.gmail.com>
Message-ID: <20130729101130.3b12d0fb@pitrou.net>

Le Sun, 28 Jul 2013 12:18:55 +0200,
Charles-Fran?ois Natali <cf.natali at gmail.com> a ?crit :
> 2013/7/28 Antoine Pitrou <solipsis at pitrou.net>:
> >> (C) Should we handle standard streams (0: stdin, 1: stdout, 2:
> >> stderr) differently? For example, os.dup2(fd, 0) should make the
> >> file descriptor 0 (stdin) inheritable or non-inheritable? On
> >> Windows, os.set_inheritable(fd, False) fails (error 87, invalid
> >> argument) on standard streams (0, 1, 2) and copies of the standard
> >> streams (created by os.dup()).
> >
> > I have been advocating for that, but I now realize that
> > special-casing these three descriptors in a myriad of fd-creating
> > functions isn't very attractive.
> > (if a standard stream fd has been closed, any fd-creating function
> > can re-create that fd: including socket.socket(), etc.)
> >
> > So perhaps only the *original* standard streams should be left
> > inheritable?
> 
> Having stdin/stdout/stderr cloexec (e.g. after a dup() to redirect to
> a log file, a socket...) will likely break a lot of code, e.g. code
> using os.system(), or code calling exec manually (and I'm sure there's
> a bunch of it).

Hmm. os.exec*() could easily make standard streams non-CLOEXEC before
calling the underlying C library function. Things are more annoying for
os.system(), though.

> Also, it'll be puzzling to have syscalls automatically set the cloexec
> flag. I guess a lot of people doing system programming with Python
> will get bitten, but that's a discussion we already had months ago...

Perhaps this advocates for a global flag, e.g.
sys.set_default_fd_inheritance(), with False (non-inheritable) being
the default for sanity and security.

Regards

Antoine.



From ethan at stoneleaf.us  Mon Jul 29 09:55:53 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 29 Jul 2013 00:55:53 -0700
Subject: [Python-Dev] MyOpenID.com no longer supported
Message-ID: <51F62009.7080304@stoneleaf.us>

Excerpt from http://meta.stackoverflow.com/q/190442/176681:

Janrain no longer actively supports MyOpenID, and announced on Twitter that their users should proceed with caution.

This decision was made by Janrain, [snip]

I know the Python bug tracker allows MyOpenID logins; if that is your only login method you should add another that 
doesn't depend on MyOpenID.

--
~Ethan~

From ronaldoussoren at mac.com  Mon Jul 29 12:30:34 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Mon, 29 Jul 2013 12:30:34 +0200
Subject: [Python-Dev] PEP 446: cloexec flag on os.open
Message-ID: <B0F402A6-2427-4129-8559-0F6741E68813@mac.com>

Victor,

PEP 446 mentions that a cloexec flag gets added to os.open. This API already has a way to specify this: the O_CLOEXEC bit in the flags argument. A new cloexec parameter is nicely consistent with the other APIs, but introcudes a second way to set that flag.

What will the following calls do?:

    os.open(path, os.O_RDONLY|os.O_CLOEXEC, cloexec=False)
    os.open(path, os.O_RDONLY, cloexec=True)

The PEP doesn't specify this, but the implementation for PEP 443 in issue 17036 basicly ignores the cloexec argument in the first call and adds O_CLOEXEC in the second call. That can lead to confusing behavior when the flags argument to os.open is passed from elsewhere (e.g. a wrapper around os.open that passes in arguments and just overrides the cloexec argument).

It might be better to just drop the cloexec flag to os.open and make os.O_CLOEXEC an alias for os.O_NOINHERIT on Windows.

Ronald

From ronaldoussoren at mac.com  Mon Jul 29 14:49:18 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Mon, 29 Jul 2013 14:49:18 +0200
Subject: [Python-Dev] PEP 447: add type.__locallookup__
Message-ID: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com>

Hi,

This PEP proposed to add a __locallookup__ slot to type objects,
which is used by _PyType_Lookup and super_getattro instead of peeking
in the tp_dict of classes.  The PEP text explains why this is needed.

Differences with the previous version:

* Better explanation of why this is a useful addition

* type.__locallookup__ is no longer optional.

* I've added benchmarking results using pybench.
  (using the patch attached to issue 18181)

Ronald




PEP: 447
Title: Add __locallookup__ method to metaclass
Version: $Revision$
Last-Modified: $Date$
Author: Ronald Oussoren <ronaldoussoren at mac.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 12-Jun-2013
Post-History: 2-Jul-2013, 15-Jul-2013, 29-Jul-2013


Abstract
========

Currently ``object.__getattribute__`` and ``super.__getattribute__`` peek
in the ``__dict__`` of classes on the MRO for a class when looking for
an attribute. This PEP adds an optional ``__locallookup__`` method to
a metaclass that can be used to override this behavior.

Rationale
=========

It is currently not possible to influence how the `super class`_ looks
up attributes (that is, ``super.__getattribute__`` unconditionally
peeks in the class ``__dict__``), and that can be problematic for
dynamic classes that can grow new methods on demand.

The ``__locallookup__`` method makes it possible to dynamicly add
attributes even when looking them up using the `super class`_.

The new method affects ``object.__getattribute__`` (and
`PyObject_GenericGetAttr`_) as well for consistency.

Background
----------

The current behavior of ``super.__getattribute__`` causes problems for
classes that are dynamic proxies for other (non-Python) classes or types,
an example of which is `PyObjC`_. PyObjC creates a Python class for every
class in the Objective-C runtime, and looks up methods in the Objective-C
runtime when they are used. This works fine for normal access, but doesn't
work for access with ``super`` objects. Because of this PyObjC currently
includes a custom ``super`` that must be used with its classes.

The API in this PEP makes it possible to remove the custom ``super`` and
simplifies the implementation because the custom lookup behavior can be
added in a central location.


The superclass attribute lookup hook
====================================

Both ``super.__getattribute__`` and ``object.__getattribute__`` (or
`PyObject_GenericGetAttr`_ in C code) walk an object's MRO and peek in the
class' ``__dict__`` to look up attributes. A way to affect this lookup is
using a method on the meta class for the type, that by default looks up
the name in the class ``__dict__``.

In Python code
--------------

A meta type can define a method ``__locallookup__`` that is called during
attribute resolution by both ``super.__getattribute__`` and ``object.__getattribute``::

    class MetaType(type):
        def __locallookup__(cls, name):
            try:
                return cls.__dict__[name]
            except KeyError:
                raise AttributeError(name) from None

The ``__locallookup__`` method has as its arguments a class and the name of the attribute
that is looked up. It should return the value of the attribute without invoking descriptors,
or raise `AttributeError`_ when the name cannot be found.

The `type`_ class provides a default implementation for ``__locallookup__``, that
looks up the name in the class dictionary.

Example usage
.............

The code below implements a silly metaclass that redirects attribute lookup to uppercase
versions of names::

    class UpperCaseAccess (type):
        def __locallookup__(cls, name):
	    return cls.__dict__[name.upper()]

    class SillyObject (metaclass=UpperCaseAccess):
        def m(self):
	    return 42

	def M(self):
	    return "fourtytwo"

    obj = SillyObject()
    assert obj.m() == "fortytwo"


In C code
---------

A new slot ``tp_locallookup`` is added to the ``PyTypeObject`` struct, this slot
corresponds to the ``__locallookup__`` method on `type`_.

The slot has the following prototype::

    PyObject* (*locallookupfunc)(PyTypeObject* cls, PyObject* name);

This method should lookup *name* in the namespace of *cls*, without looking at superclasses,
and should not invoke descriptors. The method returns ``NULL`` without setting an exception
when the *name* cannot be found, and returns a new reference otherwise (not a borrowed reference).

Use of this hook by the interpreter
-----------------------------------

The new method is required for metatypes and as such is defined on `type_`.  Both
``super.__getattribute__`` and ``object.__getattribute__``/`PyObject_GenericGetAttr`_
(through ``_PyType_Lookup``) use the this ``__locallookup__`` method when walking
the MRO.

Other changes to the implementation
-----------------------------------

The change for `PyObject_GenericGetAttr`_ will be done by changing the private function
``_PyType_Lookup``. This currently returns a borrowed reference, but must return a new
reference when the ``__locallookup__`` method is present. Because of this ``_PyType_Lookup``
will be renamed to ``_PyType_LookupName``, this will cause compile-time errors for all out-of-tree
users of this private API.

The attribute lookup cache in ``Objects/typeobject.c`` is disabled for classes that have a
metaclass that overrides ``__locallookup__``, because using the cache might not be valid
for such classes.

Performance impact
------------------

The pybench output below compares an implementation of this PEP with the regular
source tree, both based on changeset a5681f50bae2, run on an idle machine an
Core i7 processor running Centos 6.4.

Even though the machine was idle there were clear differences between runs,
I've seen difference in "minimum time" vary from -0.1% to +1.5%, with simular
(but slightly smaller) differences in the "average time" difference.

.. ::

	-------------------------------------------------------------------------------
	PYBENCH 2.1
	-------------------------------------------------------------------------------
	* using CPython 3.4.0a0 (default, Jul 29 2013, 13:01:34) [GCC 4.4.7 20120313 (Red Hat 4.4.7-3)]
	* disabled garbage collection
	* system check interval set to maximum: 2147483647
	* using timer: time.perf_counter
	* timer: resolution=1e-09, implementation=clock_gettime(CLOCK_MONOTONIC)

	-------------------------------------------------------------------------------
	Benchmark: pep447.pybench
	-------------------------------------------------------------------------------

	    Rounds: 10
	    Warp:   10
	    Timer:  time.perf_counter

	    Machine Details:
	       Platform ID:    Linux-2.6.32-358.114.1.openstack.el6.x86_64-x86_64-with-centos-6.4-Final
	       Processor:      x86_64

	    Python:
	       Implementation: CPython
	       Executable:     /tmp/default-pep447/bin/python3
	       Version:        3.4.0a0
	       Compiler:       GCC 4.4.7 20120313 (Red Hat 4.4.7-3)
	       Bits:           64bit
	       Build:          Jul 29 2013 14:09:12 (#default)
	       Unicode:        UCS4


	-------------------------------------------------------------------------------
	Comparing with: default.pybench
	-------------------------------------------------------------------------------

	    Rounds: 10
	    Warp:   10
	    Timer:  time.perf_counter

	    Machine Details:
	       Platform ID:    Linux-2.6.32-358.114.1.openstack.el6.x86_64-x86_64-with-centos-6.4-Final
	       Processor:      x86_64

	    Python:
	       Implementation: CPython
	       Executable:     /tmp/default/bin/python3
	       Version:        3.4.0a0
	       Compiler:       GCC 4.4.7 20120313 (Red Hat 4.4.7-3)
	       Bits:           64bit
	       Build:          Jul 29 2013 13:01:34 (#default)
	       Unicode:        UCS4


	Test                             minimum run-time        average  run-time
					 this    other   diff    this    other   diff
	-------------------------------------------------------------------------------
		  BuiltinFunctionCalls:    45ms    44ms   +1.3%    45ms    44ms   +1.3%
		   BuiltinMethodLookup:    26ms    27ms   -2.4%    27ms    27ms   -2.2%
			 CompareFloats:    33ms    34ms   -0.7%    33ms    34ms   -1.1%
		 CompareFloatsIntegers:    66ms    67ms   -0.9%    66ms    67ms   -0.8%
		       CompareIntegers:    51ms    50ms   +0.9%    51ms    50ms   +0.8%
		CompareInternedStrings:    34ms    33ms   +0.4%    34ms    34ms   -0.4%
			  CompareLongs:    29ms    29ms   -0.1%    29ms    29ms   -0.0%
			CompareStrings:    43ms    44ms   -1.8%    44ms    44ms   -1.8%
	    ComplexPythonFunctionCalls:    44ms    42ms   +3.9%    44ms    42ms   +4.1%
			 ConcatStrings:    33ms    33ms   -0.4%    33ms    33ms   -1.0%
		       CreateInstances:    47ms    48ms   -2.9%    47ms    49ms   -3.4%
		    CreateNewInstances:    35ms    36ms   -2.5%    36ms    36ms   -2.5%
	       CreateStringsWithConcat:    69ms    70ms   -0.7%    69ms    70ms   -0.9%
			  DictCreation:    52ms    50ms   +3.1%    52ms    50ms   +3.0%
		     DictWithFloatKeys:    40ms    44ms  -10.1%    43ms    45ms   -5.8%
		   DictWithIntegerKeys:    32ms    36ms  -11.2%    35ms    37ms   -4.6%
		    DictWithStringKeys:    29ms    34ms  -15.7%    35ms    40ms  -11.0%
			      ForLoops:    30ms    29ms   +2.2%    30ms    29ms   +2.2%
			    IfThenElse:    38ms    41ms   -6.7%    38ms    41ms   -6.9%
			   ListSlicing:    36ms    36ms   -0.7%    36ms    37ms   -1.3%
			NestedForLoops:    43ms    45ms   -3.1%    43ms    45ms   -3.2%
	      NestedListComprehensions:    39ms    40ms   -1.7%    39ms    40ms   -2.1%
		  NormalClassAttribute:    86ms    82ms   +5.1%    86ms    82ms   +5.0%
	       NormalInstanceAttribute:    42ms    42ms   +0.3%    42ms    42ms   +0.0%
		   PythonFunctionCalls:    39ms    38ms   +3.5%    39ms    38ms   +2.8%
		     PythonMethodCalls:    51ms    49ms   +3.0%    51ms    50ms   +2.8%
			     Recursion:    67ms    68ms   -1.4%    67ms    68ms   -1.4%
			  SecondImport:    41ms    36ms  +12.5%    41ms    36ms  +12.6%
		   SecondPackageImport:    45ms    40ms  +13.1%    45ms    40ms  +13.2%
		 SecondSubmoduleImport:    92ms    95ms   -2.4%    95ms    98ms   -3.6%
	       SimpleComplexArithmetic:    28ms    28ms   -0.1%    28ms    28ms   -0.2%
		SimpleDictManipulation:    57ms    57ms   -1.0%    57ms    58ms   -1.0%
		 SimpleFloatArithmetic:    29ms    28ms   +4.7%    29ms    28ms   +4.9%
	      SimpleIntFloatArithmetic:    37ms    41ms   -8.5%    37ms    41ms   -8.7%
	       SimpleIntegerArithmetic:    37ms    41ms   -9.4%    37ms    42ms  -10.2%
	      SimpleListComprehensions:    33ms    33ms   -1.9%    33ms    34ms   -2.9%
		SimpleListManipulation:    28ms    30ms   -4.3%    29ms    30ms   -4.1%
		  SimpleLongArithmetic:    26ms    26ms   +0.5%    26ms    26ms   +0.5%
			    SmallLists:    40ms    40ms   +0.1%    40ms    40ms   +0.1%
			   SmallTuples:    46ms    47ms   -2.4%    46ms    48ms   -3.0%
		 SpecialClassAttribute:   126ms   120ms   +4.7%   126ms   121ms   +4.4%
	      SpecialInstanceAttribute:    42ms    42ms   +0.6%    42ms    42ms   +0.8%
			StringMappings:    94ms    91ms   +3.9%    94ms    91ms   +3.8%
		      StringPredicates:    48ms    49ms   -1.7%    48ms    49ms   -2.1%
			 StringSlicing:    45ms    45ms   +1.4%    46ms    45ms   +1.5%
			     TryExcept:    23ms    22ms   +4.9%    23ms    22ms   +4.8%
			    TryFinally:    32ms    32ms   -0.1%    32ms    32ms   +0.1%
			TryRaiseExcept:    17ms    17ms   +0.9%    17ms    17ms   +0.5%
			  TupleSlicing:    49ms    48ms   +1.1%    49ms    49ms   +1.0%
			   WithFinally:    48ms    47ms   +2.3%    48ms    47ms   +2.4%
		       WithRaiseExcept:    45ms    44ms   +0.8%    45ms    45ms   +0.5%
	-------------------------------------------------------------------------------
	Totals:                          2284ms  2287ms   -0.1%  2306ms  2308ms   -0.1%

	(this=pep447.pybench, other=default.pybench)


Alternative proposals
---------------------

``__getattribute_super__``
..........................

An earlier version of this PEP used the following static method on classes::

    def __getattribute_super__(cls, name, object, owner): pass

This method performed name lookup as well as invoking descriptors and was necessarily
limited to working only with ``super.__getattribute__``.


Reuse ``tp_getattro``
.....................

It would be nice to avoid adding a new slot, thus keeping the API simpler and
easier to understand.  A comment on `Issue 18181`_ asked about reusing the
``tp_getattro`` slot, that is super could call the ``tp_getattro`` slot of all
methods along the MRO.

That won't work because ``tp_getattro`` will look in the instance
``__dict__`` before it tries to resolve attributes using classes in the MRO.
This would mean that using ``tp_getattro`` instead of peeking the class
dictionaries changes the semantics of the `super class`_.


References
==========

* `Issue 18181`_ contains a prototype implementation

Copyright
=========

This document has been placed in the public domain.

.. _`Issue 18181`: http://bugs.python.org/issue18181

.. _`super class`: http://docs.python.org/3/library/functions.html#super

.. _`NotImplemented`: http://docs.python.org/3/library/constants.html#NotImplemented

.. _`PyObject_GenericGetAttr`: http://docs.python.org/3/c-api/object.html#PyObject_GenericGetAttr

.. _`type`: http://docs.python.org/3/library/functions.html#type

.. _`AttributeError`: http://docs.python.org/3/library/exceptions.html#AttributeError

.. _`PyObjC`: http://pyobjc.sourceforge.net/

.. _`classmethod`: http://docs.python.org/3/library/functions.html#classmethod

From solipsis at pitrou.net  Mon Jul 29 14:58:30 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 29 Jul 2013 14:58:30 +0200
Subject: [Python-Dev] PEP 447: add type.__locallookup__
References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com>
Message-ID: <20130729145830.2d0317e7@pitrou.net>


Hi,

Le Mon, 29 Jul 2013 14:49:18 +0200,
Ronald Oussoren <ronaldoussoren at mac.com> a ?crit :
> Hi,
> 
> This PEP proposed to add a __locallookup__ slot to type objects,
> which is used by _PyType_Lookup and super_getattro instead of peeking
> in the tp_dict of classes.  The PEP text explains why this is needed.
> 
> Differences with the previous version:
> 
> * Better explanation of why this is a useful addition
> 
> * type.__locallookup__ is no longer optional.
> 
> * I've added benchmarking results using pybench.
>   (using the patch attached to issue 18181)

Could you please run the whole benchmark suite?
http://hg.python.org/benchmarks/

Thanks

Antoine.



From ronaldoussoren at mac.com  Mon Jul 29 15:07:37 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Mon, 29 Jul 2013 15:07:37 +0200
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <20130729145830.2d0317e7@pitrou.net>
References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com>
	<20130729145830.2d0317e7@pitrou.net>
Message-ID: <16097E4E-CF34-4660-AF70-FF67C0BDC393@mac.com>


On 29 Jul, 2013, at 14:58, Antoine Pitrou <solipsis at pitrou.net> wrote:

> 
> Hi,
> 
> Le Mon, 29 Jul 2013 14:49:18 +0200,
> Ronald Oussoren <ronaldoussoren at mac.com> a ?crit :
>> Hi,
>> 
>> This PEP proposed to add a __locallookup__ slot to type objects,
>> which is used by _PyType_Lookup and super_getattro instead of peeking
>> in the tp_dict of classes.  The PEP text explains why this is needed.
>> 
>> Differences with the previous version:
>> 
>> * Better explanation of why this is a useful addition
>> 
>> * type.__locallookup__ is no longer optional.
>> 
>> * I've added benchmarking results using pybench.
>>  (using the patch attached to issue 18181)
> 
> Could you please run the whole benchmark suite?
> http://hg.python.org/benchmarks/

Sure.

Ronald


From ronaldoussoren at mac.com  Mon Jul 29 15:26:41 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Mon, 29 Jul 2013 15:26:41 +0200
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <16097E4E-CF34-4660-AF70-FF67C0BDC393@mac.com>
References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com>
	<20130729145830.2d0317e7@pitrou.net>
	<16097E4E-CF34-4660-AF70-FF67C0BDC393@mac.com>
Message-ID: <D1061604-1CE4-4C6A-B701-CD36C0E4E103@mac.com>


On 29 Jul, 2013, at 15:07, Ronald Oussoren <ronaldoussoren at mac.com> wrote:

> 
> On 29 Jul, 2013, at 14:58, Antoine Pitrou <solipsis at pitrou.net> wrote:
> 
>> 
>> Hi,
>> 
>> Le Mon, 29 Jul 2013 14:49:18 +0200,
>> Ronald Oussoren <ronaldoussoren at mac.com> a ?crit :
>>> Hi,
>>> 
>>> This PEP proposed to add a __locallookup__ slot to type objects,
>>> which is used by _PyType_Lookup and super_getattro instead of peeking
>>> in the tp_dict of classes.  The PEP text explains why this is needed.
>>> 
>>> Differences with the previous version:
>>> 
>>> * Better explanation of why this is a useful addition
>>> 
>>> * type.__locallookup__ is no longer optional.
>>> 
>>> * I've added benchmarking results using pybench.
>>> (using the patch attached to issue 18181)
>> 
>> Could you please run the whole benchmark suite?
>> http://hg.python.org/benchmarks/
> 
> Sure.

That's harder than I had expected, when I use the "make_perf3.sh" to create
a python 3 compatible version of the benchmark suite and then run the suite
it craps out because it cannnot find "spitfire", which isn't translated
(as are several other benchmarks).

I'll have to investigate why the suite doesn't work.

Ronald

> 
> Ronald
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com


From solipsis at pitrou.net  Mon Jul 29 15:41:25 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 29 Jul 2013 15:41:25 +0200
Subject: [Python-Dev] PEP 447: add type.__locallookup__
References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com>
	<20130729145830.2d0317e7@pitrou.net>
	<16097E4E-CF34-4660-AF70-FF67C0BDC393@mac.com>
	<D1061604-1CE4-4C6A-B701-CD36C0E4E103@mac.com>
Message-ID: <20130729154125.28c3e7c2@pitrou.net>

Le Mon, 29 Jul 2013 15:26:41 +0200,
Ronald Oussoren <ronaldoussoren at mac.com> a ?crit :
> 
> On 29 Jul, 2013, at 15:07, Ronald Oussoren <ronaldoussoren at mac.com>
> wrote:
> 
> > 
> > On 29 Jul, 2013, at 14:58, Antoine Pitrou <solipsis at pitrou.net>
> > wrote:
> > 
> >> 
> >> Hi,
> >> 
> >> Le Mon, 29 Jul 2013 14:49:18 +0200,
> >> Ronald Oussoren <ronaldoussoren at mac.com> a ?crit :
> >>> Hi,
> >>> 
> >>> This PEP proposed to add a __locallookup__ slot to type objects,
> >>> which is used by _PyType_Lookup and super_getattro instead of
> >>> peeking in the tp_dict of classes.  The PEP text explains why
> >>> this is needed.
> >>> 
> >>> Differences with the previous version:
> >>> 
> >>> * Better explanation of why this is a useful addition
> >>> 
> >>> * type.__locallookup__ is no longer optional.
> >>> 
> >>> * I've added benchmarking results using pybench.
> >>> (using the patch attached to issue 18181)
> >> 
> >> Could you please run the whole benchmark suite?
> >> http://hg.python.org/benchmarks/
> > 
> > Sure.
> 
> That's harder than I had expected, when I use the "make_perf3.sh" to
> create a python 3 compatible version of the benchmark suite and then
> run the suite it craps out because it cannnot find "spitfire", which
> isn't translated (as are several other benchmarks).

Ah, sorry. I think it's enough to run the "2n3" subsuite here, though.

Regards

Antoine.



From guido at python.org  Mon Jul 29 17:08:30 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 29 Jul 2013 08:08:30 -0700
Subject: [Python-Dev] MyOpenID.com no longer supported
In-Reply-To: <51F62009.7080304@stoneleaf.us>
References: <51F62009.7080304@stoneleaf.us>
Message-ID: <CAP7+vJLm-W7an6d5QVvGnjL4MRt7XzQ5qWOtt1FNTrJUV2oQng@mail.gmail.com>

On Mon, Jul 29, 2013 at 12:55 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
> Excerpt from http://meta.stackoverflow.com/q/190442/176681:
>
> Janrain no longer actively supports MyOpenID, and announced on Twitter that
> their users should proceed with caution.
>
> This decision was made by Janrain, [snip]
>
> I know the Python bug tracker allows MyOpenID logins; if that is your only
> login method you should add another that doesn't depend on MyOpenID.

I imagine plenty of people don't read either of the two lists you
posted to. Would it be possible to dig through the database and find
out which users use MyOpenID and email them individually?

-- 
--Guido van Rossum (python.org/~guido)

From rdmurray at bitdance.com  Mon Jul 29 18:56:27 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Mon, 29 Jul 2013 12:56:27 -0400
Subject: [Python-Dev] MyOpenID.com no longer supported
In-Reply-To: <CAP7+vJLm-W7an6d5QVvGnjL4MRt7XzQ5qWOtt1FNTrJUV2oQng@mail.gmail.com>
References: <51F62009.7080304@stoneleaf.us>
	<CAP7+vJLm-W7an6d5QVvGnjL4MRt7XzQ5qWOtt1FNTrJUV2oQng@mail.gmail.com>
Message-ID: <20130729165628.4CBB02502CD@webabinitio.net>

On Mon, 29 Jul 2013 08:08:30 -0700, Guido van Rossum <guido at python.org> wrote:
> On Mon, Jul 29, 2013 at 12:55 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
> > Excerpt from http://meta.stackoverflow.com/q/190442/176681:
> >
> > Janrain no longer actively supports MyOpenID, and announced on Twitter that
> > their users should proceed with caution.
> >
> > This decision was made by Janrain, [snip]
> >
> > I know the Python bug tracker allows MyOpenID logins; if that is your only
> > login method you should add another that doesn't depend on MyOpenID.
> 
> I imagine plenty of people don't read either of the two lists you
> posted to. Would it be possible to dig through the database and find
> out which users use MyOpenID and email them individually?

I don't know for sure that it is possible, but I presume that it is.
However, when I tried to do it, I couldn't figure out how to get from
the _openid_discovery table (which has 22 entries that have myopenid as
the endpoint server) to the corresponding user record.

Either Martin needs to clue me in, or I'll have to find time to read
his openid code :)

--David

From christian at python.org  Mon Jul 29 19:15:40 2013
From: christian at python.org (Christian Heimes)
Date: Mon, 29 Jul 2013 19:15:40 +0200
Subject: [Python-Dev] C code coverage report with lcov
Message-ID: <kt67vl$ebr$1@ger.gmane.org>

Hi,

I have done some experiments with GCC's gcov and lcov to get the C code
coverage of our unit test suite. You may find today's report at

  http://tiran.bitbucket.org/python-lcov/

I'm working on a patch for our Makefile to include all steps in one
simple make tag. http://bugs.python.org/issue18481

Christian


From brett at python.org  Mon Jul 29 19:58:55 2013
From: brett at python.org (Brett Cannon)
Date: Mon, 29 Jul 2013 13:58:55 -0400
Subject: [Python-Dev] C code coverage report with lcov
In-Reply-To: <kt67vl$ebr$1@ger.gmane.org>
References: <kt67vl$ebr$1@ger.gmane.org>
Message-ID: <CAP1=2W40hA=WFjQ8nn4OXaTs2HpKcXHmNkfdepuLO5ZUgRKg5g@mail.gmail.com>

On Mon, Jul 29, 2013 at 1:15 PM, Christian Heimes <christian at python.org>wrote:

> Hi,
>
> I have done some experiments with GCC's gcov and lcov to get the C code
> coverage of our unit test suite. You may find today's report at
>
>   http://tiran.bitbucket.org/python-lcov/


Thanks!

I took a quick poke around and it seems some things are legitimately not
being executed, while others are error conditions that we wouldn't expect
to occur (e.g. memory exhaustion). If we ever decide to get serious about
code coverage (both C and Python code) we may need to have a discussion as
a group about our feelings related to pragmas dictating when code should be
left out of coverage reports.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130729/dbf704ce/attachment.html>

From rdmurray at bitdance.com  Mon Jul 29 20:11:17 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Mon, 29 Jul 2013 14:11:17 -0400
Subject: [Python-Dev] C code coverage report with lcov
In-Reply-To: <CAP1=2W40hA=WFjQ8nn4OXaTs2HpKcXHmNkfdepuLO5ZUgRKg5g@mail.gmail.com>
References: <kt67vl$ebr$1@ger.gmane.org>
	<CAP1=2W40hA=WFjQ8nn4OXaTs2HpKcXHmNkfdepuLO5ZUgRKg5g@mail.gmail.com>
Message-ID: <20130729181118.8450025014C@webabinitio.net>

On Mon, 29 Jul 2013 13:58:55 -0400, Brett Cannon <brett at python.org> wrote:
> On Mon, Jul 29, 2013 at 1:15 PM, Christian Heimes <christian at python.org>wrote:
> 
> > Hi,
> >
> > I have done some experiments with GCC's gcov and lcov to get the C code
> > coverage of our unit test suite. You may find today's report at
> >
> >   http://tiran.bitbucket.org/python-lcov/
> 
> 
> Thanks!
> 
> I took a quick poke around and it seems some things are legitimately not
> being executed, while others are error conditions that we wouldn't expect
> to occur (e.g. memory exhaustion). If we ever decide to get serious about
> code coverage (both C and Python code) we may need to have a discussion as
> a group about our feelings related to pragmas dictating when code should be
> left out of coverage reports.

I suspect Victor will eventually have something for us for exercising the
memory exhaustion code :)

--David

From christian at python.org  Mon Jul 29 21:31:02 2013
From: christian at python.org (Christian Heimes)
Date: Mon, 29 Jul 2013 21:31:02 +0200
Subject: [Python-Dev] C code coverage report with lcov
In-Reply-To: <CAP1=2W40hA=WFjQ8nn4OXaTs2HpKcXHmNkfdepuLO5ZUgRKg5g@mail.gmail.com>
References: <kt67vl$ebr$1@ger.gmane.org>
	<CAP1=2W40hA=WFjQ8nn4OXaTs2HpKcXHmNkfdepuLO5ZUgRKg5g@mail.gmail.com>
Message-ID: <51F6C2F6.8040704@python.org>

Am 29.07.2013 19:58, schrieb Brett Cannon:
> I took a quick poke around and it seems some things are legitimately not
> being executed, while others are error conditions that we wouldn't
> expect to occur (e.g. memory exhaustion). If we ever decide to get
> serious about code coverage (both C and Python code) we may need to have
> a discussion as a group about our feelings related to pragmas dictating
> when code should be left out of coverage reports.

Yeah, object allocation and creation checks as well as verification of
operating system API call are reason for lots branch and line misses.
I'm not sure what we can do about that. I don't want to plaster the code
with LCOV_EXCL_LINE comments.

As first action we should look into function coverage. We may not be
able to execute every branch but at least we should be able to call and
verify each function.

Christian

From brett at python.org  Mon Jul 29 21:38:54 2013
From: brett at python.org (Brett Cannon)
Date: Mon, 29 Jul 2013 15:38:54 -0400
Subject: [Python-Dev] C code coverage report with lcov
In-Reply-To: <51F6C2F6.8040704@python.org>
References: <kt67vl$ebr$1@ger.gmane.org>
	<CAP1=2W40hA=WFjQ8nn4OXaTs2HpKcXHmNkfdepuLO5ZUgRKg5g@mail.gmail.com>
	<51F6C2F6.8040704@python.org>
Message-ID: <CAP1=2W7PT6-PTKsYo9kxpHV37zLq9f3em5w+1w2sdMWRJkkQDQ@mail.gmail.com>

On Mon, Jul 29, 2013 at 3:31 PM, Christian Heimes <christian at python.org>wrote:

> Am 29.07.2013 19:58, schrieb Brett Cannon:
> > I took a quick poke around and it seems some things are legitimately not
> > being executed, while others are error conditions that we wouldn't
> > expect to occur (e.g. memory exhaustion). If we ever decide to get
> > serious about code coverage (both C and Python code) we may need to have
> > a discussion as a group about our feelings related to pragmas dictating
> > when code should be left out of coverage reports.
>
> Yeah, object allocation and creation checks as well as verification of
> operating system API call are reason for lots branch and line misses.
> I'm not sure what we can do about that. I don't want to plaster the code
> with LCOV_EXCL_LINE comments.
>
> As first action we should look into function coverage. We may not be
> able to execute every branch but at least we should be able to call and
> verify each function.
>

If there's a way to report just function coverage then I think that's a
great place to start.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130729/9dda47d0/attachment.html>

From solipsis at pitrou.net  Mon Jul 29 21:48:33 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 29 Jul 2013 21:48:33 +0200
Subject: [Python-Dev] C code coverage report with lcov
References: <kt67vl$ebr$1@ger.gmane.org>
	<CAP1=2W40hA=WFjQ8nn4OXaTs2HpKcXHmNkfdepuLO5ZUgRKg5g@mail.gmail.com>
	<51F6C2F6.8040704@python.org>
Message-ID: <20130729214833.5c20e9aa@fsol>

On Mon, 29 Jul 2013 21:31:02 +0200
Christian Heimes <christian at python.org> wrote:
> Am 29.07.2013 19:58, schrieb Brett Cannon:
> > I took a quick poke around and it seems some things are legitimately not
> > being executed, while others are error conditions that we wouldn't
> > expect to occur (e.g. memory exhaustion). If we ever decide to get
> > serious about code coverage (both C and Python code) we may need to have
> > a discussion as a group about our feelings related to pragmas dictating
> > when code should be left out of coverage reports.
> 
> Yeah, object allocation and creation checks as well as verification of
> operating system API call are reason for lots branch and line misses.
> I'm not sure what we can do about that. I don't want to plaster the code
> with LCOV_EXCL_LINE comments.

Ideally, we should run coverage runs on different systems (Unices,
Windows...) and merge the results together, so that we know which paths
are really uncovered.

Regards

Antoine.



From christian at python.org  Mon Jul 29 22:05:36 2013
From: christian at python.org (Christian Heimes)
Date: Mon, 29 Jul 2013 22:05:36 +0200
Subject: [Python-Dev] C code coverage report with lcov
In-Reply-To: <CAP1=2W7PT6-PTKsYo9kxpHV37zLq9f3em5w+1w2sdMWRJkkQDQ@mail.gmail.com>
References: <kt67vl$ebr$1@ger.gmane.org>
	<CAP1=2W40hA=WFjQ8nn4OXaTs2HpKcXHmNkfdepuLO5ZUgRKg5g@mail.gmail.com>
	<51F6C2F6.8040704@python.org>
	<CAP1=2W7PT6-PTKsYo9kxpHV37zLq9f3em5w+1w2sdMWRJkkQDQ@mail.gmail.com>
Message-ID: <51F6CB10.6020204@python.org>

Am 29.07.2013 21:38, schrieb Brett Cannon:
> If there's a way to report just function coverage then I think
> that's a great place to start.

lcov's genhtml command doesn't support just function coverage. But I
have removed branch coverage. It makes the report a little bit more
readable.

Christian

From victor.stinner at gmail.com  Mon Jul 29 23:42:36 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 29 Jul 2013 23:42:36 +0200
Subject: [Python-Dev] PEP 446: Open issues/questions
In-Reply-To: <CAH_1eM0XMiFct_Ef3eZjvgCCLFyDsja0icrNvWnYNQS-_BNYKA@mail.gmail.com>
References: <CAMpsgwZJuAJaa=4dYh=q_fFvg51T4j+ZGQSuWGym+1q0kHX7jQ@mail.gmail.com>
	<20130728114341.23ec940f@fsol>
	<CAH_1eM0XMiFct_Ef3eZjvgCCLFyDsja0icrNvWnYNQS-_BNYKA@mail.gmail.com>
Message-ID: <CAMpsgwbVGhuY_uYwEJhkJNALgEc1+etvD7NefA-FVtOVAs+Rpw@mail.gmail.com>

2013/7/28 Charles-Fran?ois Natali <cf.natali at gmail.com>:
> 2013/7/28 Antoine Pitrou <solipsis at pitrou.net>:
>>> (C) Should we handle standard streams (0: stdin, 1: stdout, 2: stderr)
>>> differently? For example, os.dup2(fd, 0) should make the file
>>> descriptor 0 (stdin) inheritable or non-inheritable? On Windows,
>>> os.set_inheritable(fd, False) fails (error 87, invalid argument) on
>>> standard streams (0, 1, 2) and copies of the standard streams (created
>>> by os.dup()).
>>
>> I have been advocating for that, but I now realize that special-casing
>> these three descriptors in a myriad of fd-creating functions isn't very
>> attractive.
>> (if a standard stream fd has been closed, any fd-creating function can
>> re-create that fd: including socket.socket(), etc.)

>> So perhaps only the *original* standard streams should be left
>> inheritable?

I plan to only change functions *creating* (and replacing) new file
descriptors. Existing file descriptors (like 0, 1, 2) are unchanged.

> Having stdin/stdout/stderr cloexec (e.g. after a dup() to redirect to
> a log file, a socket...) will likely break a lot of code, e.g. code
> using os.system(), or code calling exec manually (and I'm sure there's
> a bunch of it).

I propose to add just one special case os.dup2(fd, fd2): if fd2 < 3,
fd2 is set to inheritable (otherwise, fd2 is set to non-inheritable).
os.dup2() is commonly used to replace stdin, stdout and stderr between
fork and exec. The http.server (cgi server) and pty modules use dup2()
for example

http://hg.python.org/features/pep-446/rev/f8a52518be4c

Victor

From solipsis at pitrou.net  Mon Jul 29 23:49:49 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 29 Jul 2013 23:49:49 +0200
Subject: [Python-Dev] PEP 446: Open issues/questions
In-Reply-To: <CAMpsgwbVGhuY_uYwEJhkJNALgEc1+etvD7NefA-FVtOVAs+Rpw@mail.gmail.com>
References: <CAMpsgwZJuAJaa=4dYh=q_fFvg51T4j+ZGQSuWGym+1q0kHX7jQ@mail.gmail.com>
	<20130728114341.23ec940f@fsol>
	<CAH_1eM0XMiFct_Ef3eZjvgCCLFyDsja0icrNvWnYNQS-_BNYKA@mail.gmail.com>
	<CAMpsgwbVGhuY_uYwEJhkJNALgEc1+etvD7NefA-FVtOVAs+Rpw@mail.gmail.com>
Message-ID: <20130729234949.3eb52577@fsol>

On Mon, 29 Jul 2013 23:42:36 +0200
Victor Stinner <victor.stinner at gmail.com> wrote:
> 
> >> So perhaps only the *original* standard streams should be left
> >> inheritable?
> 
> I plan to only change functions *creating* (and replacing) new file
> descriptors. Existing file descriptors (like 0, 1, 2) are unchanged.

You can create fds 0, 1 and 2 if they were closed before.

> > Having stdin/stdout/stderr cloexec (e.g. after a dup() to redirect to
> > a log file, a socket...) will likely break a lot of code, e.g. code
> > using os.system(), or code calling exec manually (and I'm sure there's
> > a bunch of it).
> 
> I propose to add just one special case os.dup2(fd, fd2): if fd2 < 3,
> fd2 is set to inheritable (otherwise, fd2 is set to non-inheritable).
> os.dup2() is commonly used to replace stdin, stdout and stderr between
> fork and exec. The http.server (cgi server) and pty modules use dup2()
> for example

Mmh. Doing it only in dup2() sounds like too much special casing for me.
I would prefer adding a new explicit parameter:
  os.dup2(fd, fd2, inheritable=False)

(as a bonus, it can be implemented using dup3() under (GNU/)Linux)

Regards

Antoine.

From syedk at pacificloud.com  Mon Jul 29 23:57:08 2013
From: syedk at pacificloud.com (syed khalid)
Date: Mon, 29 Jul 2013 11:57:08 -1000
Subject: [Python-Dev] importing modules
Message-ID: <CAHc+NX2j9Y0ipgw5vV8_i7cguhYWjh0MH-+CDM4a-SXJqhB6LQ@mail.gmail.com>

I am attempting to import modules from Shogun to python from a non-standard
python directory ie from my /home/xxx directory. is there a way on ubuntu
to selectively some modules, scripts, data  from one directory and others
modules, scripts from another directory. In other words, is there a file(s)
that provide pointers to where these modules are located.

regards

Sy
--
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130729/00ee4ca8/attachment.html>

From victor.stinner at gmail.com  Tue Jul 30 00:34:01 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 30 Jul 2013 00:34:01 +0200
Subject: [Python-Dev] PEP 446: Open issues/questions
In-Reply-To: <20130728114341.23ec940f@fsol>
References: <CAMpsgwZJuAJaa=4dYh=q_fFvg51T4j+ZGQSuWGym+1q0kHX7jQ@mail.gmail.com>
	<20130728114341.23ec940f@fsol>
Message-ID: <CAMpsgwYX81ud8SnuMBw2t5HudeF4F5WsFe6VE4SXv3bKcrSSfA@mail.gmail.com>

2013/7/28 Antoine Pitrou <solipsis at pitrou.net>:
>> (B) Should subprocess make the file descriptors of pass_fds
>> inheritable? If yes, should it be done before or after the fork? If it
>> is done after the fork and before exec, it only affects the child
>> process, at least on Linux (the file descriptor is still
>> non-inheritable in the parent process).
>
> If it is done, it should definitely be done after the fork, IMO.
> subprocess shouldn't have any long-lasting effects on the process.

I modified the subprocess module to make fds of pass_fds inheritable.
http://hg.python.org/features/pep-446/rev/75e5d34898aa

If we don't do that, it will probably break all applications using
pass_fds (and so the backward compatibility).

Victor

From steve at pearwood.info  Tue Jul 30 00:53:13 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 30 Jul 2013 08:53:13 +1000
Subject: [Python-Dev] importing modules
In-Reply-To: <CAHc+NX2j9Y0ipgw5vV8_i7cguhYWjh0MH-+CDM4a-SXJqhB6LQ@mail.gmail.com>
References: <CAHc+NX2j9Y0ipgw5vV8_i7cguhYWjh0MH-+CDM4a-SXJqhB6LQ@mail.gmail.com>
Message-ID: <51F6F259.106@pearwood.info>

Hi Syed,

On 30/07/13 07:57, syed khalid wrote:
> I am attempting to import modules from Shogun to python from a non-standard
> python directory ie from my /home/xxx directory. is there a way on ubuntu
> to selectively some modules, scripts, data  from one directory and others
> modules, scripts from another directory. In other words, is there a file(s)
> that provide pointers to where these modules are located.

This mailing list is for development OF Python, not development WITH Python. Questions like yours are best asked on the python-list at python.org mailing list, or if you prefer Usenet, comp.lang.python.

http://mail.python.org/mailman/listinfo/python-list

My (brief) answer to your question, is you probably should set sys.path appropriately. If that's not enough to point you in the right direction to solve the problem, please take any further discussion to python-list rather than ask here.



Regards,



-- 
Steven

From cf.natali at gmail.com  Tue Jul 30 09:09:38 2013
From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=)
Date: Tue, 30 Jul 2013 09:09:38 +0200
Subject: [Python-Dev] PEP 446: Open issues/questions
In-Reply-To: <20130729101130.3b12d0fb@pitrou.net>
References: <CAMpsgwZJuAJaa=4dYh=q_fFvg51T4j+ZGQSuWGym+1q0kHX7jQ@mail.gmail.com>
	<20130728114341.23ec940f@fsol>
	<CAH_1eM0XMiFct_Ef3eZjvgCCLFyDsja0icrNvWnYNQS-_BNYKA@mail.gmail.com>
	<20130729101130.3b12d0fb@pitrou.net>
Message-ID: <CAH_1eM0LAWTvkS40oixpMEiwEjgPsyJ+uEZbL5VweojrSxXd_A@mail.gmail.com>

>> Having stdin/stdout/stderr cloexec (e.g. after a dup() to redirect to
>> a log file, a socket...) will likely break a lot of code, e.g. code
>> using os.system(), or code calling exec manually (and I'm sure there's
>> a bunch of it).
>
> Hmm. os.exec*() could easily make standard streams non-CLOEXEC before
> calling the underlying C library function. Things are more annoying for
> os.system(), though.
>
>> Also, it'll be puzzling to have syscalls automatically set the cloexec
>> flag. I guess a lot of people doing system programming with Python
>> will get bitten, but that's a discussion we already had months ago...
>
> Perhaps this advocates for a global flag, e.g.
> sys.set_default_fd_inheritance(), with False (non-inheritable) being
> the default for sanity and security.

This looks more and more like PEP 433 :-)

And honestly, when I think about it, I think that this whole mess is a
solution looking for a problem.
If we don't want to inherit file descriptors in child processes, the
answer is simple: the subprocess module (this fact is not even
mentioned in the PEP).
If a user wants to use the execve() syscall directly, then he should
be aware of the implications. I don't think that patching half the
stdlib and complicating the API of many functions is the right way to
do this.

The stdlib should be updated to replace the handful of places where
exec() is called explicitly by subprocess (the only one I can think on
top of my head is http.server.CGIHTTPRequestHandler (issue #16945)),
otherwise that's about it.

cf



>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/cf.natali%40gmail.com

From victor.stinner at gmail.com  Tue Jul 30 11:24:42 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 30 Jul 2013 11:24:42 +0200
Subject: [Python-Dev] PEP 446: Open issues/questions
In-Reply-To: <CAH_1eM0XMiFct_Ef3eZjvgCCLFyDsja0icrNvWnYNQS-_BNYKA@mail.gmail.com>
References: <CAMpsgwZJuAJaa=4dYh=q_fFvg51T4j+ZGQSuWGym+1q0kHX7jQ@mail.gmail.com>
	<20130728114341.23ec940f@fsol>
	<CAH_1eM0XMiFct_Ef3eZjvgCCLFyDsja0icrNvWnYNQS-_BNYKA@mail.gmail.com>
Message-ID: <CAMpsgwb3ZoJgY41Jg4+pnSs2cxwdMTWVYhnJrsRE784OMouuxQ@mail.gmail.com>

2013/7/28 Charles-Fran?ois Natali <cf.natali at gmail.com>:
> Also, it'll be puzzling to have syscalls automatically set the cloexec
> flag. I guess a lot of people doing system programming with Python
> will get bitten, but that's a discussion we already had months ago...

The inheritance of file descriptors (and Windows handles) is discussed
since january 2013. Thanks to all exchanges on python-dev, we now know well
the perimeter of such changes. Each option has been discussed, and
advantages and drawbacks of each option were listed.

Going against the POSIX standard (clear inheritable flag when creating a
fd) is a drawback, but it's less important than issues fixed by such change
(don't "leak" fd or handle to child processes which fixes many other
issues).

Victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130730/42b69e77/attachment.html>

From solipsis at pitrou.net  Tue Jul 30 11:29:12 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 30 Jul 2013 11:29:12 +0200
Subject: [Python-Dev] PEP 446: Open issues/questions
References: <CAMpsgwZJuAJaa=4dYh=q_fFvg51T4j+ZGQSuWGym+1q0kHX7jQ@mail.gmail.com>
	<20130728114341.23ec940f@fsol>
	<CAH_1eM0XMiFct_Ef3eZjvgCCLFyDsja0icrNvWnYNQS-_BNYKA@mail.gmail.com>
	<20130729101130.3b12d0fb@pitrou.net>
	<CAH_1eM0LAWTvkS40oixpMEiwEjgPsyJ+uEZbL5VweojrSxXd_A@mail.gmail.com>
Message-ID: <20130730112912.53e108a7@pitrou.net>

Le Tue, 30 Jul 2013 09:09:38 +0200,
Charles-Fran?ois Natali <cf.natali at gmail.com> a ?crit :
> >
> > Perhaps this advocates for a global flag, e.g.
> > sys.set_default_fd_inheritance(), with False (non-inheritable) being
> > the default for sanity and security.
> 
> This looks more and more like PEP 433 :-)
> 
> And honestly, when I think about it, I think that this whole mess is a
> solution looking for a problem.
> If we don't want to inherit file descriptors in child processes, the
> answer is simple: the subprocess module (this fact is not even
> mentioned in the PEP).

This is a good point. Are there any reasons (other than fd inheritance)
not to use subprocess? If there are, perhaps we should try to eliminate
them by improving subprocess.

Regards

Antoine.



From victor.stinner at gmail.com  Tue Jul 30 11:37:56 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 30 Jul 2013 11:37:56 +0200
Subject: [Python-Dev] PEP 446: Open issues/questions
In-Reply-To: <CAH_1eM0LAWTvkS40oixpMEiwEjgPsyJ+uEZbL5VweojrSxXd_A@mail.gmail.com>
References: <CAMpsgwZJuAJaa=4dYh=q_fFvg51T4j+ZGQSuWGym+1q0kHX7jQ@mail.gmail.com>
	<20130728114341.23ec940f@fsol>
	<CAH_1eM0XMiFct_Ef3eZjvgCCLFyDsja0icrNvWnYNQS-_BNYKA@mail.gmail.com>
	<20130729101130.3b12d0fb@pitrou.net>
	<CAH_1eM0LAWTvkS40oixpMEiwEjgPsyJ+uEZbL5VweojrSxXd_A@mail.gmail.com>
Message-ID: <CAMpsgwZa1pbECH1M4nC=1Kqg8fnx5orvmkgtKOz-kQsCfGv_gg@mail.gmail.com>

Le 30 juil. 2013 09:11, "Charles-Fran?ois Natali" <cf.natali at gmail.com> a
?crit :
> > Perhaps this advocates for a global flag, e.g.
> > sys.set_default_fd_inheritance(), with False (non-inheritable) being
> > the default for sanity and security.
>
> This looks more and more like PEP 433 :-)

I don't like the global modifiable default, for the reasons Charles
Francois gave some months ago.

I did have time yet to rewrite the PEP 446, I'm writing first its
implementation. Basically, the new PEP 446 is:

- functions creating file descriptors and Windows handle now clear the
inheritable flag (disable inheritance by default)
- add new functions os.get/set_inheritable()
- subprocess makes fds of pass_fds inheritable
- (maybe) os.dup2(fd, fd2) makes fd2 inheritable for fd 0, 1, 2

That's all! No more new cloexec parameter to +15 functions and classes, no
more global variable (default inheritavle values).

Victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130730/71782725/attachment-0001.html>

From shibturn at gmail.com  Tue Jul 30 11:52:38 2013
From: shibturn at gmail.com (Richard Oudkerk)
Date: Tue, 30 Jul 2013 10:52:38 +0100
Subject: [Python-Dev] PEP 446: Open issues/questions
In-Reply-To: <CAH_1eM0LAWTvkS40oixpMEiwEjgPsyJ+uEZbL5VweojrSxXd_A@mail.gmail.com>
References: <CAMpsgwZJuAJaa=4dYh=q_fFvg51T4j+ZGQSuWGym+1q0kHX7jQ@mail.gmail.com>
	<20130728114341.23ec940f@fsol>
	<CAH_1eM0XMiFct_Ef3eZjvgCCLFyDsja0icrNvWnYNQS-_BNYKA@mail.gmail.com>
	<20130729101130.3b12d0fb@pitrou.net>
	<CAH_1eM0LAWTvkS40oixpMEiwEjgPsyJ+uEZbL5VweojrSxXd_A@mail.gmail.com>
Message-ID: <kt82d3$6hg$1@ger.gmane.org>

On 30/07/2013 8:09am, Charles-Fran?ois Natali wrote:
> If we don't want to inherit file descriptors in child processes, the
> answer is simple: the subprocess module (this fact is not even
> mentioned in the PEP).

Note that on Windows subprocess has no equivalent of a passfds argument, 
and if you redirect the standard streams then you are forced to inherit 
all inheritable handles.

-- 
Richard


From victor.stinner at gmail.com  Tue Jul 30 12:52:55 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 30 Jul 2013 12:52:55 +0200
Subject: [Python-Dev] PEP 446: Open issues/questions
In-Reply-To: <kt82d3$6hg$1@ger.gmane.org>
References: <CAMpsgwZJuAJaa=4dYh=q_fFvg51T4j+ZGQSuWGym+1q0kHX7jQ@mail.gmail.com>
	<20130728114341.23ec940f@fsol>
	<CAH_1eM0XMiFct_Ef3eZjvgCCLFyDsja0icrNvWnYNQS-_BNYKA@mail.gmail.com>
	<20130729101130.3b12d0fb@pitrou.net>
	<CAH_1eM0LAWTvkS40oixpMEiwEjgPsyJ+uEZbL5VweojrSxXd_A@mail.gmail.com>
	<kt82d3$6hg$1@ger.gmane.org>
Message-ID: <CAMpsgwbauJ2O6zjD8E93CeoA-jYVrVEyqJkGnL8eLVLcLgu4eQ@mail.gmail.com>

2013/7/30 Richard Oudkerk <shibturn at gmail.com>:
> Note that on Windows subprocess has no equivalent of a passfds argument, and
> if you redirect the standard streams then you are forced to inherit all
> inheritable handles.

You can redirect standard streams (stdin, stdout, stderr) using the
startup info structure:

startupinfo.dwFlags |= _winapi.STARTF_USESTDHANDLES
startupinfo.hStdInput = p2cread
startupinfo.hStdOutput = c2pwrite
startupinfo.hStdError = errwrite

But I don't know how to add another streams (ex: pipe). I would be
nice to have a "pass_handles" on Windows.

Victor

From shibturn at gmail.com  Tue Jul 30 14:12:12 2013
From: shibturn at gmail.com (Richard Oudkerk)
Date: Tue, 30 Jul 2013 13:12:12 +0100
Subject: [Python-Dev] PEP 446: Open issues/questions
In-Reply-To: <CAMpsgwbauJ2O6zjD8E93CeoA-jYVrVEyqJkGnL8eLVLcLgu4eQ@mail.gmail.com>
References: <CAMpsgwZJuAJaa=4dYh=q_fFvg51T4j+ZGQSuWGym+1q0kHX7jQ@mail.gmail.com>
	<20130728114341.23ec940f@fsol>
	<CAH_1eM0XMiFct_Ef3eZjvgCCLFyDsja0icrNvWnYNQS-_BNYKA@mail.gmail.com>
	<20130729101130.3b12d0fb@pitrou.net>
	<CAH_1eM0LAWTvkS40oixpMEiwEjgPsyJ+uEZbL5VweojrSxXd_A@mail.gmail.com>
	<kt82d3$6hg$1@ger.gmane.org>
	<CAMpsgwbauJ2O6zjD8E93CeoA-jYVrVEyqJkGnL8eLVLcLgu4eQ@mail.gmail.com>
Message-ID: <kt8aio$1qb$1@ger.gmane.org>

On 30/07/2013 11:52am, Victor Stinner wrote:
 > You can redirect standard streams (stdin, stdout, stderr) using the
 > startup info structure:
 >
 > startupinfo.dwFlags |= _winapi.STARTF_USESTDHANDLES
 > startupinfo.hStdInput = p2cread
 > startupinfo.hStdOutput = c2pwrite
 > startupinfo.hStdError = errwrite

The documentation for STARTUPINFO says this about STARTF_USESTDHANDLES:

   If this flag is specified when calling one of the process creation
   functions, the handles must be inheritable and the function's
   bInheritHandles parameter must be set to TRUE.

So, as I said, if you redirect the streams then you inherit all 
inheritable handles.

-- 
Richard


From martin at v.loewis.de  Tue Jul 30 19:02:44 2013
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 30 Jul 2013 19:02:44 +0200
Subject: [Python-Dev] MyOpenID.com no longer supported
In-Reply-To: <20130729165628.4CBB02502CD@webabinitio.net>
References: <51F62009.7080304@stoneleaf.us>
	<CAP7+vJLm-W7an6d5QVvGnjL4MRt7XzQ5qWOtt1FNTrJUV2oQng@mail.gmail.com>
	<20130729165628.4CBB02502CD@webabinitio.net>
Message-ID: <51F7F1B4.50908@v.loewis.de>

Am 29.07.13 18:56, schrieb R. David Murray:
> Either Martin needs to clue me in, or I'll have to find time to read
> his openid code :)

If you want to read the code, this issue can be discovered from
schema.py. The openids are stored per user in a single field;
multiple openids are simply space-separated.

To find all myopenid users, do

select id,_username,_address,_openids from _user where _openids like
'%myopenid.com/%';

You could filter out those with multiple openids (and not _openids
like '% %').

I'd appreciate if you could contact them; if anybody loses access and
registered a working email address, the password reset procedure
should still work for them.

FWIW, 204 out of 13581 users only have a myopenid registered;
how many of these also have a working password, I cannot tell.
49 more have a second OpenID registered (which atleast must
have worked when they registered it - but it could be an alias
for the myopenid one).

Regards,
Martin


From martin at v.loewis.de  Tue Jul 30 19:24:49 2013
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 30 Jul 2013 19:24:49 +0200
Subject: [Python-Dev] Adding Python scripts to PATHEXT on Windows
In-Reply-To: <CACac1F_Bt9eiciVb10m4JTpVKp7WxmBrvvs1oA6UcaJrcQiR=A@mail.gmail.com>
References: <CACac1F9WtTj69gBv2SKR3chxFdduhNtxNOmPqrt=mK5kHb_ZMA@mail.gmail.com>
	<51EEAB2E.9030804@v.loewis.de>
	<CACac1F9xLhfEPZuJ3-s8RaV-fi+UGaRMxp3SrzyupSkWWPG9OA@mail.gmail.com>
	<CACac1F8nHjjTx-keO_E2cK8dTtWFawiqOo-gH0ZGHHPzEe81vA@mail.gmail.com>
	<e54095b20f004c65a76db155b17d042b@BLUPR03MB199.namprd03.prod.outlook.com>
	<CACac1F_Bt9eiciVb10m4JTpVKp7WxmBrvvs1oA6UcaJrcQiR=A@mail.gmail.com>
Message-ID: <51F7F6E1.4040002@v.loewis.de>

Am 28.07.13 00:36, schrieb Paul Moore:
> On 27 July 2013 21:14, Steve Dower <Steve.Dower at microsoft.com
> <mailto:Steve.Dower at microsoft.com>> wrote:
> 
>     Any chance of this being made optional when installing? It provides
>     no benefit for people who prefer to associate scripts with an editor
>     and may be a source of confusion/complaints.
> 
> 
> Personally, I don't know how to do this so someone else would have to -

It seems that this was settled as fine as-is; if you ever wanted to
allow this to be specifically enabled, you'd have to do three things

1. create a new "feature" in the feature tree. Your patch currently
   uses the "Extensions" feature (child of the default feature);
   you could make your feature a subfeature of Extensions.
2. create a new "component". This should work similar to REGISTRY.path.
3. Add a "FeatureComponents" entry saying that the "REGISTRY.pathext"
   component (say) belongs to the "PathExt" feature (say).

Instead of hooking the environment variable to the REGISTRY.def
component, it should then go to the REGISTRY.pathext component.

The nice thing about this feature tree is that it allows convenient
user configuration yet powerful scripted installation, see

http://msdn.microsoft.com/en-us/library/windows/desktop/aa368585(v=vs.85).aspx

For example, you could set the msidbFeatureAttributesFollowParent
flag, which would make it be selected/unselected if the parent
feature is, yet still allow it to be independent of the parent
for people who really want to.

Regards,
Martin


From rdmurray at bitdance.com  Tue Jul 30 19:31:01 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Tue, 30 Jul 2013 13:31:01 -0400
Subject: [Python-Dev]
	=?utf-8?q?=5BPython-checkins=5D_cpython_=282=2E7=29?=
	=?utf-8?q?=3A_Issue_=2318441=3A_Make_test=2Esupport=2Erequires=28?=
	=?utf-8?q?=27gui=27=29_skip_when_it_should=2E?=
In-Reply-To: <3c2jY31SQ0z7LjW@mail.python.org>
References: <3c2jY31SQ0z7LjW@mail.python.org>
Message-ID: <20130730173102.1195025007D@webabinitio.net>

On Sun, 28 Jul 2013 01:09:43 +0200, terry.reedy <python-checkins at python.org> wrote:
> http://hg.python.org/cpython/rev/dd9941f5fcda
> changeset:   84870:dd9941f5fcda
> branch:      2.7
> parent:      84865:c0788ee86a65
> user:        Terry Jan Reedy <tjreedy at udel.edu>
> date:        Sun Jul 21 20:13:24 2013 -0400
> summary:
>   Issue #18441: Make test.support.requires('gui') skip when it should.
> (Consolidating this check and various checks in tkinter files and moving them
> to test.support and test.regrtest will be another issue.)
> 
> files:
>   Lib/idlelib/idle_test/test_text.py |   5 +---
>   Lib/test/test_idle.py              |  20 ++++++++++++++---
>   2 files changed, 17 insertions(+), 8 deletions(-)
> 
> 
> diff --git a/Lib/idlelib/idle_test/test_text.py b/Lib/idlelib/idle_test/test_text.py
> --- a/Lib/idlelib/idle_test/test_text.py
> +++ b/Lib/idlelib/idle_test/test_text.py
> @@ -216,10 +216,7 @@
>          requires('gui')
>          from Tkinter import Tk, Text
>          cls.Text = Text
> -        try:
> -            cls.root = Tk()
> -        except TclError as msg:
> -            raise unittest.SkipTest('TclError: %s' % msg)
> +        cls.root = Tk()
>  
>      @classmethod
>      def tearDownClass(cls):
> diff --git a/Lib/test/test_idle.py b/Lib/test/test_idle.py
> --- a/Lib/test/test_idle.py
> +++ b/Lib/test/test_idle.py
> @@ -1,9 +1,21 @@
> -# Skip test if _tkinter or _thread wasn't built or idlelib was deleted.
> -from test.test_support import import_module
> -import_module('Tkinter')
> -import_module('threading')  # imported by PyShell, imports _thread
> +# Skip test if _thread or _tkinter wasn't built or idlelib was deleted.
> +from test.test_support import import_module, use_resources
> +import_module('threading')  # imported by idlelib.PyShell, imports _thread
> +tk = import_module('Tkinter')
>  idletest = import_module('idlelib.idle_test')
>  
> +# If buildbot improperly sets gui resource (#18365, #18441), remove it
> +# so requires('gui') tests are skipped while non-gui tests still run.
> +if use_resources and 'gui' in use_resources:
> +    try:
> +        root = tk.Tk()
> +        root.destroy()
> +    except TclError:
> +        while True:
> +            use_resources.delete('gui')
> +            if 'gui' not in use_resources:
> +                break

I believe that this logic is incorrect.  If regrtest is run with "-u
gui", given this code the skip message will be "requires gui
resource"...but the caller specified the gui resource, which will make
the skip message completely confusing.

Instead, if it is true that 'gui' always means 'tk must work', then the
_is_gui_available function should do the Tk() check.  Currently as far
as I can see it is indeed the case that requires('gui') always means tk
must work.  So, I believe that the correct fix is to move
check_tk_availability to test.support, and call it from
_is_gui_available.  If we ever add another gui toolkit, we can deal with
moving the tk check out into a separate 'tk' resource at that time.

> +
>  # Without test_main present, regrtest.runtest_inner (line1219) calls
>  # unittest.TestLoader().loadTestsFromModule(this_module) which calls
>  # load_tests() if it finds it. (Unittest.main does the same.)
> 
> -- 
> Repository URL: http://hg.python.org/cpython
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> http://mail.python.org/mailman/listinfo/python-checkins

From solipsis at pitrou.net  Tue Jul 30 20:42:00 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 30 Jul 2013 20:42:00 +0200
Subject: [Python-Dev] PEP 442 aftermath: module globals at shutdown
Message-ID: <20130730204200.6d77df8b@fsol>


Hello,

PEP 442 has now been committed in time for testing in Alpha 1.

This paves the way for the removal of another well-known annoyance: the
behaviour of module globals at shutdown. Now that reference cycles
aren't a barrier to object finalization anymore, we shouldn't need
to set module globals to None before trying to reclaim modules.
(and then, we don't need to cache global functions for use in
finalizers, either)

I have a patch to suppress the hack in
http://bugs.python.org/issue18214
Once I get to add some tests, I would like to commit it soon too!

Regards

Antoine.



From p.f.moore at gmail.com  Tue Jul 30 20:56:46 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 30 Jul 2013 19:56:46 +0100
Subject: [Python-Dev] Adding Python scripts to PATHEXT on Windows
In-Reply-To: <51F7F6E1.4040002@v.loewis.de>
References: <CACac1F9WtTj69gBv2SKR3chxFdduhNtxNOmPqrt=mK5kHb_ZMA@mail.gmail.com>
	<51EEAB2E.9030804@v.loewis.de>
	<CACac1F9xLhfEPZuJ3-s8RaV-fi+UGaRMxp3SrzyupSkWWPG9OA@mail.gmail.com>
	<CACac1F8nHjjTx-keO_E2cK8dTtWFawiqOo-gH0ZGHHPzEe81vA@mail.gmail.com>
	<e54095b20f004c65a76db155b17d042b@BLUPR03MB199.namprd03.prod.outlook.com>
	<CACac1F_Bt9eiciVb10m4JTpVKp7WxmBrvvs1oA6UcaJrcQiR=A@mail.gmail.com>
	<51F7F6E1.4040002@v.loewis.de>
Message-ID: <CACac1F-C5-BqTcJLvSxvmsFj4V7Ex75XhowcccrB_fbH11+LtQ@mail.gmail.com>

On 30 July 2013 18:24, "Martin v. L?wis" <martin at v.loewis.de> wrote:

> > Personally, I don't know how to do this so someone else would have to -
>
> It seems that this was settled as fine as-is; if you ever wanted to
> allow this to be specifically enabled, you'd have to do three things
>

Thanks, Martin - MSI is a mystery to me, so this is very useful. Saved for
future reference.

Paul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130730/5b14bd1f/attachment.html>

From guido at python.org  Tue Jul 30 21:58:58 2013
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Jul 2013 12:58:58 -0700
Subject: [Python-Dev] PEP 442 aftermath: module globals at shutdown
In-Reply-To: <20130730204200.6d77df8b@fsol>
References: <20130730204200.6d77df8b@fsol>
Message-ID: <CAP7+vJ+rtxbmY=Fr626OHJKCcRwZ8eNc2re9P+3_7uX__O03Qg@mail.gmail.com>

I'm very excited to see this happening! It's been a constant pain and
one of the things I've always regretted. Thanks Antoine!

--Guido

On Tue, Jul 30, 2013 at 11:42 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>
> Hello,
>
> PEP 442 has now been committed in time for testing in Alpha 1.
>
> This paves the way for the removal of another well-known annoyance: the
> behaviour of module globals at shutdown. Now that reference cycles
> aren't a barrier to object finalization anymore, we shouldn't need
> to set module globals to None before trying to reclaim modules.
> (and then, we don't need to cache global functions for use in
> finalizers, either)
>
> I have a patch to suppress the hack in
> http://bugs.python.org/issue18214
> Once I get to add some tests, I would like to commit it soon too!
>
> Regards
>
> Antoine.

-- 
--Guido van Rossum (python.org/~guido)

From pjenvey at underboss.org  Tue Jul 30 22:39:02 2013
From: pjenvey at underboss.org (Philip Jenvey)
Date: Tue, 30 Jul 2013 13:39:02 -0700
Subject: [Python-Dev] PyPy3 2.1 beta 1 released
Message-ID: <F8242E6D-1A20-48CF-ADA6-6CB0F831E556@underboss.org>

================
PyPy3 2.1 beta 1
================

We're pleased to announce the first beta of the upcoming 2.1 release of
PyPy3. This is the first release of PyPy which targets Python 3 (3.2.3)
compatibility.

We would like to thank all of the people who donated_ to the `py3k proposal`_
for supporting the work that went into this and future releases.

You can download the PyPy3 2.1 beta 1 release here:

    http://pypy.org/download.html#pypy3-2-1-beta-1

Highlights
==========

* The first release of PyPy3: support for Python 3, targetting CPython 3.2.3!

  - There are some `known issues`_ including performance regressions (issues
    `#1540`_ & `#1541`_) slated to be resolved before the final release.

What is PyPy?
==============

PyPy is a very compliant Python interpreter, almost a drop-in replacement for
CPython 2.7.3 or 3.2.3. It's fast due to its integrated tracing JIT compiler.

This release supports x86 machines running Linux 32/64, Mac OS X 64 or Windows
32. Also this release supports ARM machines running Linux 32bit - anything with
``ARMv6`` (like the Raspberry Pi) or ``ARMv7`` (like Beagleboard,
Chromebook, Cubieboard, etc.) that supports ``VFPv3`` should work.


Cheers,
the PyPy team

From tseaver at palladion.com  Tue Jul 30 23:18:44 2013
From: tseaver at palladion.com (Tres Seaver)
Date: Tue, 30 Jul 2013 17:18:44 -0400
Subject: [Python-Dev] PyPy3 2.1 beta 1 released
In-Reply-To: <F8242E6D-1A20-48CF-ADA6-6CB0F831E556@underboss.org>
References: <F8242E6D-1A20-48CF-ADA6-6CB0F831E556@underboss.org>
Message-ID: <kt9aje$khh$1@ger.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 07/30/2013 04:39 PM, Philip Jenvey wrote:
> ================ PyPy3 2.1 beta 1 ================
> 
> We're pleased to announce the first beta of the upcoming 2.1 release
> of PyPy3. This is the first release of PyPy which targets Python 3
> (3.2.3) compatibility.
> 
> We would like to thank all of the people who donated_ to the `py3k
> proposal`_ for supporting the work that went into this and future
> releases.
> 
> You can download the PyPy3 2.1 beta 1 release here:
> 
> http://pypy.org/download.html#pypy3-2-1-beta-1
> 
> Highlights ==========
> 
> * The first release of PyPy3: support for Python 3, targetting CPython
> 3.2.3!
> 
> - There are some `known issues`_ including performance regressions
> (issues `#1540`_ & `#1541`_) slated to be resolved before the final
> release.
> 
> What is PyPy? ==============
> 
> PyPy is a very compliant Python interpreter, almost a drop-in
> replacement for CPython 2.7.3 or 3.2.3. It's fast due to its
> integrated tracing JIT compiler.
> 
> This release supports x86 machines running Linux 32/64, Mac OS X 64 or
> Windows 32. Also this release supports ARM machines running Linux
> 32bit - anything with ``ARMv6`` (like the Raspberry Pi) or ``ARMv7``
> (like Beagleboard, Chromebook, Cubieboard, etc.) that supports
> ``VFPv3`` should work.

Wow -- congratulations!


Tres.
- -- 
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlH4LbQACgkQ+gerLs4ltQ655ACfXIlTNAldCJdCSjY0Os8xIhLu
vl4AoIi8eXqB6Ef/RO2W46iIbQvlX41j
=R2vL
-----END PGP SIGNATURE-----


From solipsis at pitrou.net  Tue Jul 30 23:32:23 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 30 Jul 2013 23:32:23 +0200
Subject: [Python-Dev] PEP 442 aftermath: module globals at shutdown
In-Reply-To: <CAP7+vJ+rtxbmY=Fr626OHJKCcRwZ8eNc2re9P+3_7uX__O03Qg@mail.gmail.com>
References: <20130730204200.6d77df8b@fsol>
	<CAP7+vJ+rtxbmY=Fr626OHJKCcRwZ8eNc2re9P+3_7uX__O03Qg@mail.gmail.com>
Message-ID: <20130730233223.1e472996@fsol>

On Tue, 30 Jul 2013 12:58:58 -0700
Guido van Rossum <guido at python.org> wrote:
> I'm very excited to see this happening! It's been a constant pain and
> one of the things I've always regretted. Thanks Antoine!

Note this is currently imperfect. I've identified two reasons why a
pure Python module could stay alive even after being removed from
sys.modules:

- it is held alive by a C extension: the main example is the locale
  module, which is held alive by _io and in turn keeps alive other
  Python modules (such as collections or re). But there is also the
  readline module, whose completer can keep alive other stuff (this
  is aggravated by the fact the readline module currently doesn't
  have a proper dealloc routine).

- it is held alive through builtins: the site module patches builtins
  with additional objects, which themselves keep references to the site
  module's globals, and with it other modules

Regards

Antoine.



> 
> --Guido
> 
> On Tue, Jul 30, 2013 at 11:42 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> >
> > Hello,
> >
> > PEP 442 has now been committed in time for testing in Alpha 1.
> >
> > This paves the way for the removal of another well-known annoyance: the
> > behaviour of module globals at shutdown. Now that reference cycles
> > aren't a barrier to object finalization anymore, we shouldn't need
> > to set module globals to None before trying to reclaim modules.
> > (and then, we don't need to cache global functions for use in
> > finalizers, either)
> >
> > I have a patch to suppress the hack in
> > http://bugs.python.org/issue18214
> > Once I get to add some tests, I would like to commit it soon too!
> >
> > Regards
> >
> > Antoine.
> 


From christian at python.org  Wed Jul 31 02:20:52 2013
From: christian at python.org (Christian Heimes)
Date: Wed, 31 Jul 2013 02:20:52 +0200
Subject: [Python-Dev] C code coverage report with lcov
In-Reply-To: <kt67vl$ebr$1@ger.gmane.org>
References: <kt67vl$ebr$1@ger.gmane.org>
Message-ID: <kt9l8t$q1p$1@ger.gmane.org>

Am 29.07.2013 19:15, schrieb Christian Heimes:
> Hi,
> 
> I have done some experiments with GCC's gcov and lcov to get the C code
> coverage of our unit test suite. You may find today's report at
> 
>   http://tiran.bitbucket.org/python-lcov/
> 
> I'm working on a patch for our Makefile to include all steps in one
> simple make tag. http://bugs.python.org/issue18481

Have fun with

   make coverage-report

as documented in the devguide


http://docs.python.org/devguide/coverage.html#measuring-coverage-of-c-code-with-gcov-and-lcov

Christian



From christian at python.org  Wed Jul 31 02:28:39 2013
From: christian at python.org (Christian Heimes)
Date: Wed, 31 Jul 2013 02:28:39 +0200
Subject: [Python-Dev] C code coverage report with lcov
In-Reply-To: <20130729214833.5c20e9aa@fsol>
References: <kt67vl$ebr$1@ger.gmane.org>
	<CAP1=2W40hA=WFjQ8nn4OXaTs2HpKcXHmNkfdepuLO5ZUgRKg5g@mail.gmail.com>
	<51F6C2F6.8040704@python.org> <20130729214833.5c20e9aa@fsol>
Message-ID: <51F85A37.3040903@python.org>

Am 29.07.2013 21:48, schrieb Antoine Pitrou:
> Ideally, we should run coverage runs on different systems (Unices,
> Windows...) and merge the results together, so that we know which paths
> are really uncovered.

I don't that is easily possible. The coverage report depends on GCC and
its gcov extension -- so much for Windows. I also don't know if gcov
supports cross-profiling on varying platforms and operating systems.

By the way gcov understands preprocessor output. It doesn't report lines
as uncovered when the lines or functions are #ifdef-ed out.

Christian


From tjreedy at udel.edu  Wed Jul 31 02:52:27 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 30 Jul 2013 20:52:27 -0400
Subject: [Python-Dev] cpython (2.7): Issue #18441: Make
 test.support.requires('gui') skip when it should.
In-Reply-To: <20130730173102.1195025007D@webabinitio.net>
References: <3c2jY31SQ0z7LjW@mail.python.org>
	<20130730173102.1195025007D@webabinitio.net>
Message-ID: <51F85FCB.8090008@udel.edu>

On 7/30/2013 1:31 PM, R. David Murray wrote:
> On Sun, 28 Jul 2013 01:09:43 +0200, terry.reedy <python-checkins at python.org> wrote:
>
>>    Issue #18441: Make test.support.requires('gui') skip when it should.
>> (Consolidating this check and various checks in tkinter files and moving them
>> to test.support and test.regrtest will be another issue.)

>> +# Skip test if _thread or _tkinter wasn't built or idlelib was deleted.
>> +from test.test_support import import_module, use_resources
>> +import_module('threading')  # imported by idlelib.PyShell, imports _thread
>> +tk = import_module('Tkinter')
>>   idletest = import_module('idlelib.idle_test')
>>
>> +# If buildbot improperly sets gui resource (#18365, #18441), remove it
>> +# so requires('gui') tests are skipped while non-gui tests still run.
>> +if use_resources and 'gui' in use_resources:
>> +    try:
>> +        root = tk.Tk()
>> +        root.destroy()
>> +    except TclError:
>> +        while True:
>> +            use_resources.delete('gui')
>> +            if 'gui' not in use_resources:
>> +                break
>
> I believe that this logic is incorrect.

It works for the intended purpose.

> If regrtest is run with "-u gui", given this code the skip message will
 > be "requires gui resource" but the caller specified the gui resource,
 > which will make the skip message completely confusing.

The message is accurate; string 'gui' does not create or enable the 
required physical graphics subsystem. Anyway, the message is not in any 
of the current buildbot reports I looked at. This patch has no effect on 
normal machines with graphics and nor on most of the buildbots (at least 
the stable one). On the 3 stable buildbots it is written for, the 
message will only appear if test_idle fails (in one of the text tests 
that do run) and is rerun or displayed in verbose mode. I doubt that 
such a rare occurrence will seriously confuse any of the few developers 
who will read the verbose idle_test output.

> Instead, if it is true that 'gui' always means 'tk must work', then the
> _is_gui_available function should do the Tk() check.  Currently as far
> as I can see it is indeed the case that requires('gui') always means tk
> must work.  So, I believe that the correct fix is to move
> check_tk_availability to test.support, and call it from
> _is_gui_available.

Ned and I agreed in the issue discussion that gui checks (also in 
tkinter files) should be consolidated into test.support. See
http://bugs.python.org/issue18604
The check in this patch will be modified or simply removed once that is 
done. But I need it here now to continue pushing new idle tests.

A consolidated gui check could report to the user something like "'gui' 
is being ignored because graphics are not usable".

 > If we ever add another gui toolkit, we can deal with
> moving the tk check out into a separate 'tk' resource at that time.

The above check is intended mostly as a gui check, not a tk check, 
because it is only performed after successfully importing tkinter, which 
imports _tkinter, which initializes tcl/tk.

-- 
Terry Jan Reedy

From greg.ewing at canterbury.ac.nz  Wed Jul 31 03:30:58 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 31 Jul 2013 13:30:58 +1200
Subject: [Python-Dev] PEP 442 aftermath: module globals at shutdown
In-Reply-To: <20130730233223.1e472996@fsol>
References: <20130730204200.6d77df8b@fsol>
	<CAP7+vJ+rtxbmY=Fr626OHJKCcRwZ8eNc2re9P+3_7uX__O03Qg@mail.gmail.com>
	<20130730233223.1e472996@fsol>
Message-ID: <51F868D2.7040601@canterbury.ac.nz>

Antoine Pitrou wrote:
> - it is held alive through builtins: the site module patches builtins
>   with additional objects, which themselves keep references to the site
>   module's globals,

The module probably *should* stay alive in that case, since
it's still accessible via those patched builtins.

-- 
Greg

From rdmurray at bitdance.com  Wed Jul 31 04:08:31 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Tue, 30 Jul 2013 22:08:31 -0400
Subject: [Python-Dev] cpython (2.7): Issue #18441: Make
	test.support.requires('gui') skip when it should.
In-Reply-To: <51F85FCB.8090008@udel.edu>
References: <3c2jY31SQ0z7LjW@mail.python.org>
	<20130730173102.1195025007D@webabinitio.net>
	<51F85FCB.8090008@udel.edu>
Message-ID: <20130731020832.07CF62500B9@webabinitio.net>

On Tue, 30 Jul 2013 20:52:27 -0400, Terry Reedy <tjreedy at udel.edu> wrote:
> On 7/30/2013 1:31 PM, R. David Murray wrote:
> > On Sun, 28 Jul 2013 01:09:43 +0200, terry.reedy <python-checkins at python.org> wrote:
> > I believe that this logic is incorrect.
> 
> It works for the intended purpose.
> 
> > If regrtest is run with "-u gui", given this code the skip message will
>  > be "requires gui resource" but the caller specified the gui resource,
>  > which will make the skip message completely confusing.
> 
> The message is accurate; string 'gui' does not create or enable the 
> required physical graphics subsystem. Anyway, the message is not in any 
> of the current buildbot reports I looked at. This patch has no effect on 
> normal machines with graphics and nor on most of the buildbots (at least 
> the stable one). On the 3 stable buildbots it is written for, the 
> message will only appear if test_idle fails (in one of the text tests 
> that do run) and is rerun or displayed in verbose mode. I doubt that 
> such a rare occurrence will seriously confuse any of the few developers 
> who will read the verbose idle_test output.

OK, I see what you are saying, but I really do think the skip message
should differentiate between the flag not being specified and the
physical resource not existing.

> > Instead, if it is true that 'gui' always means 'tk must work', then the
> > _is_gui_available function should do the Tk() check.  Currently as far
> > as I can see it is indeed the case that requires('gui') always means tk
> > must work.  So, I believe that the correct fix is to move
> > check_tk_availability to test.support, and call it from
> > _is_gui_available.
> 
> Ned and I agreed in the issue discussion that gui checks (also in 
> tkinter files) should be consolidated into test.support. See
> http://bugs.python.org/issue18604
> The check in this patch will be modified or simply removed once that is 
> done. But I need it here now to continue pushing new idle tests.
> 
> A consolidated gui check could report to the user something like "'gui' 
> is being ignored because graphics are not usable".

As long as this is the end result, I'm fine with it.

--David

From ronaldoussoren at mac.com  Wed Jul 31 08:15:50 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Wed, 31 Jul 2013 08:15:50 +0200
Subject: [Python-Dev] to rename or not...
Message-ID: <28A29D72-0EFC-4B77-98B9-B416E522549B@mac.com>

Hi,

The relevant issue for this question is <http://bugs.python.org/issue14455>

Plistlib contains object serialization logic that is in spirit simular to json and pickle, but has a different output format and different limitations. The functions in the module don't reflect that though, they are "readPlist" and "writePlist" instead of "load" and "dump".

While working on the issue I noticed something uglier than that: plistlib in py3k inherits a design decision that was necessary in py2 and feels decidedly odd in py3k. It represents binary data objects of type plistlib.Data instead of bytes. Those objects have an attribute with the actual data. The distinction was necessary in Python 2 to make it possible to keep binary data and strings apart, but is no longer necessary in Python 3.

Because of this I'd like to introduce a new API in plistlib that fixes both problems. In particular:

* Add 'load', 'loads', 'dump' and 'dumps', those use "bytes" for binary data by default

* Keep and deprecate "readPlist", "writePlist" and the their string equivalents, those still use Data objects (and call the new API to do the actual work).

I'd like some feedback on this change. On the one hand the new APIs make it possible to clean up the API of plistlib, on the other hand this is a big API change.

Ronald

P.S. The issue itself is about adding support for binary plist files, I got a bit carried away while testing and refactoring the original patch :-(

From ronaldoussoren at mac.com  Wed Jul 31 08:17:55 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Wed, 31 Jul 2013 08:17:55 +0200
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com>
References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com>
Message-ID: <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>


On 29 Jul, 2013, at 14:49, Ronald Oussoren <ronaldoussoren at mac.com> wrote:

> Hi,
> 
> This PEP proposed to add a __locallookup__ slot to type objects,
> which is used by _PyType_Lookup and super_getattro instead of peeking
> in the tp_dict of classes.  The PEP text explains why this is needed.
> 
> Differences with the previous version:
> 
> * Better explanation of why this is a useful addition
> 
> * type.__locallookup__ is no longer optional.
> 
> * I've added benchmarking results using pybench.
>  (using the patch attached to issue 18181)
> 
> Ronald

And something I forgot to ask: is anyone willing to be the BDFL-Delegate for
PEP 447?

Ronald

From solipsis at pitrou.net  Wed Jul 31 08:21:50 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 31 Jul 2013 08:21:50 +0200
Subject: [Python-Dev] PEP 442 aftermath: module globals at shutdown
References: <20130730204200.6d77df8b@fsol>
	<CAP7+vJ+rtxbmY=Fr626OHJKCcRwZ8eNc2re9P+3_7uX__O03Qg@mail.gmail.com>
	<20130730233223.1e472996@fsol> <51F868D2.7040601@canterbury.ac.nz>
Message-ID: <20130731082150.293a5a74@fsol>

On Wed, 31 Jul 2013 13:30:58 +1200
Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Antoine Pitrou wrote:
> > - it is held alive through builtins: the site module patches builtins
> >   with additional objects, which themselves keep references to the site
> >   module's globals,
> 
> The module probably *should* stay alive in that case, since
> it's still accessible via those patched builtins.

Of course. I'm not gonna kill reachable modules like a cowboy on the
loose :-)
The problem is that its global objects will get finalized very late.

Anyway, the solution is to unpatch builtins through an atexit handler
in the site module.

(my patch still has a last step where surviving modules get their
globals wiped, though)

Regards

Antoine.



From solipsis at pitrou.net  Wed Jul 31 10:09:46 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 31 Jul 2013 10:09:46 +0200
Subject: [Python-Dev] to rename or not...
References: <28A29D72-0EFC-4B77-98B9-B416E522549B@mac.com>
Message-ID: <20130731100946.752a3c43@pitrou.net>

Le Wed, 31 Jul 2013 08:15:50 +0200,
Ronald Oussoren <ronaldoussoren at mac.com> a ?crit :
> 
> Because of this I'd like to introduce a new API in plistlib that
> fixes both problems. In particular:
> 
> * Add 'load', 'loads', 'dump' and 'dumps', those use "bytes" for
> binary data by default
> 
> * Keep and deprecate "readPlist", "writePlist" and the their string
> equivalents, those still use Data objects (and call the new API to do
> the actual work).
> 
> I'd like some feedback on this change. On the one hand the new APIs
> make it possible to clean up the API of plistlib, on the other hand
> this is a big API change.

+1 on the principle. I can't say more, not being a Mac user.

Regards

Antoine.



From greg.ewing at canterbury.ac.nz  Wed Jul 31 10:39:10 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 31 Jul 2013 20:39:10 +1200
Subject: [Python-Dev] to rename or not...
In-Reply-To: <28A29D72-0EFC-4B77-98B9-B416E522549B@mac.com>
References: <28A29D72-0EFC-4B77-98B9-B416E522549B@mac.com>
Message-ID: <51F8CD2E.4010703@canterbury.ac.nz>

Ronald Oussoren wrote:
> * Add 'load', 'loads', 'dump' and 'dumps', those use "bytes" for binary data
> by default
> 
> * Keep and deprecate "readPlist", "writePlist" and the their string
> equivalents, those still use Data objects

+1, makes sense to me.

-- 
Greg

From rdmurray at bitdance.com  Wed Jul 31 12:52:25 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Wed, 31 Jul 2013 06:52:25 -0400
Subject: [Python-Dev] to rename or not...
In-Reply-To: <51F8CD2E.4010703@canterbury.ac.nz>
References: <28A29D72-0EFC-4B77-98B9-B416E522549B@mac.com>
	<51F8CD2E.4010703@canterbury.ac.nz>
Message-ID: <20130731105226.E842025007D@webabinitio.net>

On Wed, 31 Jul 2013 20:39:10 +1200, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Ronald Oussoren wrote:
> > * Add 'load', 'loads', 'dump' and 'dumps', those use "bytes" for binary data
> > by default
> > 
> > * Keep and deprecate "readPlist", "writePlist" and the their string
> > equivalents, those still use Data objects
> 
> +1, makes sense to me.

Yes, +1.  Presumably the plistlib module was "ported" to py3k just enough
to make it pass its tests, without reconsidering the design (and API) the
way you have now done.  So I think it is appropriate to do so.  But, then,
I would, considering I'm doing the same thing to the email module for
much the same reason :).

--David

From ronaldoussoren at mac.com  Wed Jul 31 13:12:42 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Wed, 31 Jul 2013 13:12:42 +0200
Subject: [Python-Dev] to rename or not...
In-Reply-To: <20130731100946.752a3c43@pitrou.net>
References: <28A29D72-0EFC-4B77-98B9-B416E522549B@mac.com>
	<20130731100946.752a3c43@pitrou.net>
Message-ID: <9F88D258-8B93-40DA-95F6-2BD7909E2CF0@mac.com>


On 31 Jul, 2013, at 10:09, Antoine Pitrou <solipsis at pitrou.net> wrote:

> Le Wed, 31 Jul 2013 08:15:50 +0200,
> Ronald Oussoren <ronaldoussoren at mac.com> a ?crit :
>> 
>> Because of this I'd like to introduce a new API in plistlib that
>> fixes both problems. In particular:
>> 
>> * Add 'load', 'loads', 'dump' and 'dumps', those use "bytes" for
>> binary data by default
>> 
>> * Keep and deprecate "readPlist", "writePlist" and the their string
>> equivalents, those still use Data objects (and call the new API to do
>> the actual work).
>> 
>> I'd like some feedback on this change. On the one hand the new APIs
>> make it possible to clean up the API of plistlib, on the other hand
>> this is a big API change.
> 
> +1 on the principle. I can't say more, not being a Mac user.

plistlib is the portable part of the stdlib :-)

But anyway, I've got enough positive feedback that I'll introduce the
new API, although it probably won't make the first alpha release.

Ronald


From brett at python.org  Wed Jul 31 15:37:04 2013
From: brett at python.org (Brett Cannon)
Date: Wed, 31 Jul 2013 09:37:04 -0400
Subject: [Python-Dev] C code coverage report with lcov
In-Reply-To: <51F85A37.3040903@python.org>
References: <kt67vl$ebr$1@ger.gmane.org>
	<CAP1=2W40hA=WFjQ8nn4OXaTs2HpKcXHmNkfdepuLO5ZUgRKg5g@mail.gmail.com>
	<51F6C2F6.8040704@python.org> <20130729214833.5c20e9aa@fsol>
	<51F85A37.3040903@python.org>
Message-ID: <CAP1=2W4CYxrkHUNYp-SPVR6ULS_aSjnJVJdFsXW2u4SB3hDBiA@mail.gmail.com>

On Tue, Jul 30, 2013 at 8:28 PM, Christian Heimes <christian at python.org>wrote:

> Am 29.07.2013 21:48, schrieb Antoine Pitrou:
> > Ideally, we should run coverage runs on different systems (Unices,
> > Windows...) and merge the results together, so that we know which paths
> > are really uncovered.
>
> I don't that is easily possible. The coverage report depends on GCC and
> its gcov extension -- so much for Windows. I also don't know if gcov
> supports cross-profiling on varying platforms and operating systems.
>
> By the way gcov understands preprocessor output. It doesn't report lines
> as uncovered when the lines or functions are #ifdef-ed out.
>

In case are curious to try and make this work using clang, I found the
flags to use at
http://clang-developers.42468.n3.nabble.com/Code-coverage-on-clang-td4033066.html
and
how to make it work with lcov. I also discovered clang itself uses
https://github.com/ddunbar/zcov instead of lcov. This is all untested, but
I didn't want to forget the links and in case someone has clang installed
with compiler-rt and wants to see how it works.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130731/0bc3afe4/attachment.html>