From stefan at bytereef.org  Sun Mar  1 20:45:31 2015
From: stefan at bytereef.org (s.krah)
Date: Sun, 01 Mar 2015 19:45:31 +0000
Subject: [Python-Dev] Wg: Re: [Python-checkins] cpython (3.4): Issue #23446:
 Use PyMem_New instead of PyMem_Malloc to avoid possible integer
In-Reply-To: <20150301185759.GA23016@bytereef.org>
References: <20150216113449.60738.84196@psf.io>
 <20150216163424.GA3056@bytereef.org>
 <CAMpsgwYuBt9HBj6VBhgU-chPT7GECmCDJqgc0XfBQsEZ2K-ZCA@mail.gmail.com>
 <20150301185759.GA23016@bytereef.org>
Message-ID: <14bd6df46d2.c371a38e217388.9204368102411012987@bytereef.org>



 == == == == == == Weitergeleitete Nachricht == == == == == == 
Absender : Stefan Krah&lt;stefan at bytereef.org&gt;
Empf?nger : "Victor Stinner"&lt;victor.stinner at gmail.com&gt;
Datum : So, 01 Mrz 2015 18:58:43 +0000
Betreff : Re: [Python-Dev] [Python-checkins] cpython (3.4): Issue #23446: Use PyMem_New instead of PyMem_Malloc to avoid possible integer
 == == == == == == Weitergeleitete Nachricht == == == == == == 

On Mon, Feb 16, 2015 at 10:14:59PM +0100, Victor Stinner wrote: 
&gt; 2015-02-16 17:34 GMT+01:00 Stefan Krah &lt;stefan at bytereef.org&gt;: 
&gt; &gt; 
&gt; &gt; On Mon, Feb 16, 2015 at 11:35:52AM +0000, serhiy.storchaka wrote: 
&gt; &gt;&gt; diff --git a/Modules/_testbuffer.c b/Modules/_testbuffer.c 
&gt; &gt;&gt; --- a/Modules/_testbuffer.c 
&gt; &gt;&gt; +++ b/Modules/_testbuffer.c 
&gt; &gt;&gt; @@ -850,7 +850,7 @@ 
&gt; &gt;&gt; Py_ssize_t *dest; 
&gt; &gt;&gt; Py_ssize_t x, i; 
&gt; &gt;&gt; 
&gt; &gt;&gt; - dest = PyMem_Malloc(len * (sizeof *dest)); 
&gt; &gt;&gt; + dest = PyMem_New(Py_ssize_t, len); 
&gt; &gt;&gt; if (dest == NULL) { 
&gt; &gt;&gt; PyErr_NoMemory(); 
&gt; &gt;&gt; return NULL; 
&gt; &gt; 
&gt; &gt; This, too, was already protected by len == ndim &lt;= 64. 
&gt; 
&gt; I don't understand why you don't want to use PyMem_New() even if it 
&gt; cannot overflow. PyMem_New() is more readable no? 
 
It's readable, but I don't see a reason to change code that already has an 
overflow analysis, especially in 3.4. 
 
As I wrote in http://bugs.python.org/issue23446#msg235770 , people need to 
see that one *can* make certain assumptions about PEP-3118 buffers (otherwise 
one would go insane with overflow checks when doing arithmetic on the buffers. 
 
 
So, in a sense, this commit removes information for the reader. I would 
prefer an "assert(len &lt;= 64)" for documentation purposes while keeping the 
multiplication. 
 
 
 
Stefan Krah 
 
 




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150301/1fd3fa5d/attachment.html>

From mistersheik at gmail.com  Mon Mar  2 19:35:52 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Mon, 2 Mar 2015 13:35:52 -0500
Subject: [Python-Dev] PEP 448 review
In-Reply-To: <CAP7+vJK58SYgBExknnSO2xwiQm5+AO5_=f0yAarNJUkbmczcPg@mail.gmail.com>
References: <CAP7+vJK58SYgBExknnSO2xwiQm5+AO5_=f0yAarNJUkbmczcPg@mail.gmail.com>
Message-ID: <CAA68w_m1cB4MmMeNX=zA5MjfaxytgU9+=8fVqEK173DGy3K5Zw@mail.gmail.com>

Hi everyone,

The patch is ready for review now, and I should have time this week to make
changes and respond to comments.

Best,

Neil

On Wed, Feb 25, 2015 at 2:42 PM, Guido van Rossum <guido at python.org> wrote:

> I'm back, I've re-read the PEP, and I've re-read the long thread with "(no
> subject)".
>
> I think Georg Brandl nailed it:
>
> """
>
>
>
>
>
>
>
>
> *I like the "sequence and dict flattening" part of the PEP, mostly because
> itis consistent and should be easy to understand, but the comprehension
> syntaxenhancements seem to be bad for readability and "comprehending" what
> the codedoes.The call syntax part is a mixed bag on the one hand it is nice
> to be consistent with the extended possibilities in literals (flattening),
> but on the other hand there would be small but annoying inconsistencies
> anyways (e.g. the duplicate kwarg case above).*
> """
>
> Greg Ewing followed up explaining that the inconsistency between dict
> flattening and call syntax is inherent in the pre-existing different rules
> for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while f(a=1,
> a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1,
> **{'a': 1}) is a TypeError.)
>
> For me, allowing f(*a, *b) and f(**d, **e) and all the other combinations
> for function calls proposed by the PEP is an easy +1 -- it's a
> straightforward extension of the existing pattern, and anybody who knows
> what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d,
> **e) means shouldn't be hard either. Understanding the edge case for
> duplicate keys with f(**d, **e) is a little harder, but the error messages
> are pretty clear, and it is not a new edge case.
>
> The sequence and dict flattening syntax proposals are also clean and
> logical -- we already have *-unpacking on the receiving side, so allowing
> *x in tuple expressions reads pretty naturally (and the similarity with *a
> in argument lists certainly helps). From here, having [a, *x, b, *y] is
> also natural, and then the extension to other displays is natural: {a, *x,
> b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me.
>
> So that leaves comprehensions. IIRC, during the development of the patch
> we realized that f(*x for x in xs) is sufficiently ambiguous that we
> decided to disallow it -- note that f(x for x in xs) is already somewhat of
> a special case because an argument can only be a "bare" generator
> expression if it is the only argument. The same reasoning doesn't apply (in
> that form) to list, set and dict comprehensions -- while f(x for x in xs)
> is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the
> same as [(x for x in xs)] (that's a list of one element, and the element is
> a generator expression).
>
> The basic premise of this part of the proposal is that if you have a few
> iterables, the new proposal (without comprehensions) lets you create a list
> or generator expression that iterates over all of them, essentially
> flattening them:
>
>     >>> xs = [1, 2, 3]
>     >>> ys = ['abc', 'def']
>     >>> zs = [99]
>     >>> [*xs, *ys, *zs]
>     [1, 2, 3, 'abc', 'def', 99]
>     >>>
>
> But now suppose you have a list of iterables:
>
>     >>> xss = [[1, 2, 3], ['abc', 'def'], [99]]
>     >>> [*xss[0], *xss[1], *xss[2]]
>     [1, 2, 3, 'abc', 'def', 99]
>     >>>
>
> Wouldn't it be nice if you could write the latter using a comprehension?
>
>     >>> xss = [[1, 2, 3], ['abc', 'def'], [99]]
>     >>> [*xs for xs in xss]
>     [1, 2, 3, 'abc', 'def', 99]
>     >>>
>
> This is somewhat seductive, and the following is even nicer: the *xs
> position may be an expression, e.g.:
>
>     >>> xss = [[1, 2, 3], ['abc', 'def'], [99]]
>     >>> [*xs[:2] for xs in xss]
>     [1, 2, 'abc', 'def', 99]
>     >>>
>
> On the other hand, I had to explore the possibilities here by
> experimenting in the interpreter, and I discovered some odd edge cases
> (e.g. you can parenthesize the starred expression, but that seems a
> syntactic accident).
>
> All in all I am personally +0 on the comprehension part of the PEP, and I
> like that it provides a way to "flatten" a sequence of sequences, but I
> think very few people in the thread have supported this part. Therefore I
> would like to ask Neil to update the PEP and the patch to take out the
> comprehension part, so that the two "easy wins" can make it into Python 3.5
> (basically, I am accepting two-thirds of the PEP :-). There is some time
> yet until alpha 2.
>
> I would also like code reviewers (Benjamin?) to start reviewing the patch
> <http://bugs.python.org/issue2292>, taking into account that the
> comprehension part needs to be removed.
>
> --
> --Guido van Rossum (python.org/~guido)
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150302/a4261c51/attachment.html>

From victor.stinner at gmail.com  Mon Mar  2 21:17:52 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 2 Mar 2015 21:17:52 +0100
Subject: [Python-Dev] PEP 448 review
In-Reply-To: <CAA68w_m1cB4MmMeNX=zA5MjfaxytgU9+=8fVqEK173DGy3K5Zw@mail.gmail.com>
References: <CAP7+vJK58SYgBExknnSO2xwiQm5+AO5_=f0yAarNJUkbmczcPg@mail.gmail.com>
 <CAA68w_m1cB4MmMeNX=zA5MjfaxytgU9+=8fVqEK173DGy3K5Zw@mail.gmail.com>
Message-ID: <CAMpsgwa8JY3rCobEPOe2CN2ovcQJE7AkHfRPtw5SZE83eF1+fQ@mail.gmail.com>

Where is the patch?

Victor

Le lundi 2 mars 2015, Neil Girdhar <mistersheik at gmail.com> a ?crit :

> Hi everyone,
>
> The patch is ready for review now, and I should have time this week to
> make changes and respond to comments.
>
> Best,
>
> Neil
>
> On Wed, Feb 25, 2015 at 2:42 PM, Guido van Rossum <guido at python.org
> <javascript:_e(%7B%7D,'cvml','guido at python.org');>> wrote:
>
>> I'm back, I've re-read the PEP, and I've re-read the long thread with
>> "(no subject)".
>>
>> I think Georg Brandl nailed it:
>>
>> """
>>
>>
>>
>>
>>
>>
>>
>>
>> *I like the "sequence and dict flattening" part of the PEP, mostly
>> because itis consistent and should be easy to understand, but the
>> comprehension syntaxenhancements seem to be bad for readability and
>> "comprehending" what the codedoes.The call syntax part is a mixed bag on
>> the one hand it is nice to be consistent with the extended possibilities in
>> literals (flattening), but on the other hand there would be small but
>> annoying inconsistencies anyways (e.g. the duplicate kwarg case above).*
>> """
>>
>> Greg Ewing followed up explaining that the inconsistency between dict
>> flattening and call syntax is inherent in the pre-existing different rules
>> for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while f(a=1,
>> a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1,
>> **{'a': 1}) is a TypeError.)
>>
>> For me, allowing f(*a, *b) and f(**d, **e) and all the other combinations
>> for function calls proposed by the PEP is an easy +1 -- it's a
>> straightforward extension of the existing pattern, and anybody who knows
>> what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d,
>> **e) means shouldn't be hard either. Understanding the edge case for
>> duplicate keys with f(**d, **e) is a little harder, but the error messages
>> are pretty clear, and it is not a new edge case.
>>
>> The sequence and dict flattening syntax proposals are also clean and
>> logical -- we already have *-unpacking on the receiving side, so allowing
>> *x in tuple expressions reads pretty naturally (and the similarity with *a
>> in argument lists certainly helps). From here, having [a, *x, b, *y] is
>> also natural, and then the extension to other displays is natural: {a, *x,
>> b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me.
>>
>> So that leaves comprehensions. IIRC, during the development of the patch
>> we realized that f(*x for x in xs) is sufficiently ambiguous that we
>> decided to disallow it -- note that f(x for x in xs) is already somewhat of
>> a special case because an argument can only be a "bare" generator
>> expression if it is the only argument. The same reasoning doesn't apply (in
>> that form) to list, set and dict comprehensions -- while f(x for x in xs)
>> is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the
>> same as [(x for x in xs)] (that's a list of one element, and the element is
>> a generator expression).
>>
>> The basic premise of this part of the proposal is that if you have a few
>> iterables, the new proposal (without comprehensions) lets you create a list
>> or generator expression that iterates over all of them, essentially
>> flattening them:
>>
>>     >>> xs = [1, 2, 3]
>>     >>> ys = ['abc', 'def']
>>     >>> zs = [99]
>>     >>> [*xs, *ys, *zs]
>>     [1, 2, 3, 'abc', 'def', 99]
>>     >>>
>>
>> But now suppose you have a list of iterables:
>>
>>     >>> xss = [[1, 2, 3], ['abc', 'def'], [99]]
>>     >>> [*xss[0], *xss[1], *xss[2]]
>>     [1, 2, 3, 'abc', 'def', 99]
>>     >>>
>>
>> Wouldn't it be nice if you could write the latter using a comprehension?
>>
>>     >>> xss = [[1, 2, 3], ['abc', 'def'], [99]]
>>     >>> [*xs for xs in xss]
>>     [1, 2, 3, 'abc', 'def', 99]
>>     >>>
>>
>> This is somewhat seductive, and the following is even nicer: the *xs
>> position may be an expression, e.g.:
>>
>>     >>> xss = [[1, 2, 3], ['abc', 'def'], [99]]
>>     >>> [*xs[:2] for xs in xss]
>>     [1, 2, 'abc', 'def', 99]
>>     >>>
>>
>> On the other hand, I had to explore the possibilities here by
>> experimenting in the interpreter, and I discovered some odd edge cases
>> (e.g. you can parenthesize the starred expression, but that seems a
>> syntactic accident).
>>
>> All in all I am personally +0 on the comprehension part of the PEP, and I
>> like that it provides a way to "flatten" a sequence of sequences, but I
>> think very few people in the thread have supported this part. Therefore I
>> would like to ask Neil to update the PEP and the patch to take out the
>> comprehension part, so that the two "easy wins" can make it into Python 3.5
>> (basically, I am accepting two-thirds of the PEP :-). There is some time
>> yet until alpha 2.
>>
>> I would also like code reviewers (Benjamin?) to start reviewing the patch
>> <http://bugs.python.org/issue2292>, taking into account that the
>> comprehension part needs to be removed.
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150302/c5f9d362/attachment.html>

From mistersheik at gmail.com  Mon Mar  2 21:18:12 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Mon, 2 Mar 2015 15:18:12 -0500
Subject: [Python-Dev] PEP 448 review
In-Reply-To: <CAMpsgwa8JY3rCobEPOe2CN2ovcQJE7AkHfRPtw5SZE83eF1+fQ@mail.gmail.com>
References: <CAP7+vJK58SYgBExknnSO2xwiQm5+AO5_=f0yAarNJUkbmczcPg@mail.gmail.com>
 <CAA68w_m1cB4MmMeNX=zA5MjfaxytgU9+=8fVqEK173DGy3K5Zw@mail.gmail.com>
 <CAMpsgwa8JY3rCobEPOe2CN2ovcQJE7AkHfRPtw5SZE83eF1+fQ@mail.gmail.com>
Message-ID: <CAA68w_kFVmMQJsk+hdgnKtOui2rqYdecQmaNEjk_UPk7N7Q_xg@mail.gmail.com>

http://bugs.python.org/issue2292

On Mon, Mar 2, 2015 at 3:17 PM, Victor Stinner <victor.stinner at gmail.com>
wrote:

> Where is the patch?
>
> Victor
>
> Le lundi 2 mars 2015, Neil Girdhar <mistersheik at gmail.com> a ?crit :
>
> Hi everyone,
>>
>> The patch is ready for review now, and I should have time this week to
>> make changes and respond to comments.
>>
>> Best,
>>
>> Neil
>>
>> On Wed, Feb 25, 2015 at 2:42 PM, Guido van Rossum <guido at python.org>
>> wrote:
>>
>>> I'm back, I've re-read the PEP, and I've re-read the long thread with
>>> "(no subject)".
>>>
>>> I think Georg Brandl nailed it:
>>>
>>> """
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *I like the "sequence and dict flattening" part of the PEP, mostly
>>> because itis consistent and should be easy to understand, but the
>>> comprehension syntaxenhancements seem to be bad for readability and
>>> "comprehending" what the codedoes.The call syntax part is a mixed bag on
>>> the one hand it is nice to be consistent with the extended possibilities in
>>> literals (flattening), but on the other hand there would be small but
>>> annoying inconsistencies anyways (e.g. the duplicate kwarg case above).*
>>> """
>>>
>>> Greg Ewing followed up explaining that the inconsistency between dict
>>> flattening and call syntax is inherent in the pre-existing different rules
>>> for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while f(a=1,
>>> a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1,
>>> **{'a': 1}) is a TypeError.)
>>>
>>> For me, allowing f(*a, *b) and f(**d, **e) and all the other
>>> combinations for function calls proposed by the PEP is an easy +1 -- it's a
>>> straightforward extension of the existing pattern, and anybody who knows
>>> what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d,
>>> **e) means shouldn't be hard either. Understanding the edge case for
>>> duplicate keys with f(**d, **e) is a little harder, but the error messages
>>> are pretty clear, and it is not a new edge case.
>>>
>>> The sequence and dict flattening syntax proposals are also clean and
>>> logical -- we already have *-unpacking on the receiving side, so allowing
>>> *x in tuple expressions reads pretty naturally (and the similarity with *a
>>> in argument lists certainly helps). From here, having [a, *x, b, *y] is
>>> also natural, and then the extension to other displays is natural: {a, *x,
>>> b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me.
>>>
>>> So that leaves comprehensions. IIRC, during the development of the patch
>>> we realized that f(*x for x in xs) is sufficiently ambiguous that we
>>> decided to disallow it -- note that f(x for x in xs) is already somewhat of
>>> a special case because an argument can only be a "bare" generator
>>> expression if it is the only argument. The same reasoning doesn't apply (in
>>> that form) to list, set and dict comprehensions -- while f(x for x in xs)
>>> is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the
>>> same as [(x for x in xs)] (that's a list of one element, and the element is
>>> a generator expression).
>>>
>>> The basic premise of this part of the proposal is that if you have a few
>>> iterables, the new proposal (without comprehensions) lets you create a list
>>> or generator expression that iterates over all of them, essentially
>>> flattening them:
>>>
>>>     >>> xs = [1, 2, 3]
>>>     >>> ys = ['abc', 'def']
>>>     >>> zs = [99]
>>>     >>> [*xs, *ys, *zs]
>>>     [1, 2, 3, 'abc', 'def', 99]
>>>     >>>
>>>
>>> But now suppose you have a list of iterables:
>>>
>>>     >>> xss = [[1, 2, 3], ['abc', 'def'], [99]]
>>>     >>> [*xss[0], *xss[1], *xss[2]]
>>>     [1, 2, 3, 'abc', 'def', 99]
>>>     >>>
>>>
>>> Wouldn't it be nice if you could write the latter using a comprehension?
>>>
>>>     >>> xss = [[1, 2, 3], ['abc', 'def'], [99]]
>>>     >>> [*xs for xs in xss]
>>>     [1, 2, 3, 'abc', 'def', 99]
>>>     >>>
>>>
>>> This is somewhat seductive, and the following is even nicer: the *xs
>>> position may be an expression, e.g.:
>>>
>>>     >>> xss = [[1, 2, 3], ['abc', 'def'], [99]]
>>>     >>> [*xs[:2] for xs in xss]
>>>     [1, 2, 'abc', 'def', 99]
>>>     >>>
>>>
>>> On the other hand, I had to explore the possibilities here by
>>> experimenting in the interpreter, and I discovered some odd edge cases
>>> (e.g. you can parenthesize the starred expression, but that seems a
>>> syntactic accident).
>>>
>>> All in all I am personally +0 on the comprehension part of the PEP, and
>>> I like that it provides a way to "flatten" a sequence of sequences, but I
>>> think very few people in the thread have supported this part. Therefore I
>>> would like to ask Neil to update the PEP and the patch to take out the
>>> comprehension part, so that the two "easy wins" can make it into Python 3.5
>>> (basically, I am accepting two-thirds of the PEP :-). There is some time
>>> yet until alpha 2.
>>>
>>> I would also like code reviewers (Benjamin?) to start reviewing the
>>> patch <http://bugs.python.org/issue2292>, taking into account that the
>>> comprehension part needs to be removed.
>>>
>>> --
>>> --Guido van Rossum (python.org/~guido)
>>>
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150302/b1c4a2c0/attachment.html>

From ethan at stoneleaf.us  Mon Mar  2 22:03:33 2015
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 02 Mar 2015 13:03:33 -0800
Subject: [Python-Dev] PEP 448 review
In-Reply-To: <CAA68w_kFVmMQJsk+hdgnKtOui2rqYdecQmaNEjk_UPk7N7Q_xg@mail.gmail.com>
References: <CAP7+vJK58SYgBExknnSO2xwiQm5+AO5_=f0yAarNJUkbmczcPg@mail.gmail.com>
 <CAA68w_m1cB4MmMeNX=zA5MjfaxytgU9+=8fVqEK173DGy3K5Zw@mail.gmail.com>
 <CAMpsgwa8JY3rCobEPOe2CN2ovcQJE7AkHfRPtw5SZE83eF1+fQ@mail.gmail.com>
 <CAA68w_kFVmMQJsk+hdgnKtOui2rqYdecQmaNEjk_UPk7N7Q_xg@mail.gmail.com>
Message-ID: <54F4D025.3010000@stoneleaf.us>

On 03/02/2015 12:18 PM, Neil Girdhar wrote:
> On Mon, Mar 2, 2015 at 3:17 PM, Victor Stinner wrote:
>> Le lundi 2 mars 2015, Neil Girdhar a ?crit :
>>>
>>>  The patch is ready for review now, and I should have time this week to make changes and respond to comments.
>> 
>> Where is the patch?
>
> http://bugs.python.org/issue2292

The latest file there is from Feb 26, while your message that the patch was ready for review is from today -- so is the
patch from five days ago the most recent?

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150302/e7a05a83/attachment.sig>

From mistersheik at gmail.com  Mon Mar  2 22:54:38 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Mon, 2 Mar 2015 16:54:38 -0500
Subject: [Python-Dev] PEP 448 review
In-Reply-To: <CAA68w_kFVmMQJsk+hdgnKtOui2rqYdecQmaNEjk_UPk7N7Q_xg@mail.gmail.com>
References: <CAP7+vJK58SYgBExknnSO2xwiQm5+AO5_=f0yAarNJUkbmczcPg@mail.gmail.com>
 <CAA68w_m1cB4MmMeNX=zA5MjfaxytgU9+=8fVqEK173DGy3K5Zw@mail.gmail.com>
 <CAMpsgwa8JY3rCobEPOe2CN2ovcQJE7AkHfRPtw5SZE83eF1+fQ@mail.gmail.com>
 <CAA68w_kFVmMQJsk+hdgnKtOui2rqYdecQmaNEjk_UPk7N7Q_xg@mail.gmail.com>
Message-ID: <CAA68w_ngnkX0KPnDni8Vvi9iG=RgiN4B_0dp9NuEMvaNO6=5OA@mail.gmail.com>

It's from five days ago.  I asked Joshua to take a look at something, but I
guess he is busy.

Best,

Neil

?

The latest file there is from Feb 26, while your message that the patch was
ready for review is from today -- so is the
patch from five days ago the most recent?

-- 
~Ethan~

On Mon, Mar 2, 2015 at 3:18 PM, Neil Girdhar <mistersheik at gmail.com> wrote:

> http://bugs.python.org/issue2292
>
> On Mon, Mar 2, 2015 at 3:17 PM, Victor Stinner <victor.stinner at gmail.com>
> wrote:
>
>> Where is the patch?
>>
>> Victor
>>
>> Le lundi 2 mars 2015, Neil Girdhar <mistersheik at gmail.com> a ?crit :
>>
>> Hi everyone,
>>>
>>> The patch is ready for review now, and I should have time this week to
>>> make changes and respond to comments.
>>>
>>> Best,
>>>
>>> Neil
>>>
>>> On Wed, Feb 25, 2015 at 2:42 PM, Guido van Rossum <guido at python.org>
>>> wrote:
>>>
>>>> I'm back, I've re-read the PEP, and I've re-read the long thread with
>>>> "(no subject)".
>>>>
>>>> I think Georg Brandl nailed it:
>>>>
>>>> """
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *I like the "sequence and dict flattening" part of the PEP, mostly
>>>> because itis consistent and should be easy to understand, but the
>>>> comprehension syntaxenhancements seem to be bad for readability and
>>>> "comprehending" what the codedoes.The call syntax part is a mixed bag on
>>>> the one hand it is nice to be consistent with the extended possibilities in
>>>> literals (flattening), but on the other hand there would be small but
>>>> annoying inconsistencies anyways (e.g. the duplicate kwarg case above).*
>>>> """
>>>>
>>>> Greg Ewing followed up explaining that the inconsistency between dict
>>>> flattening and call syntax is inherent in the pre-existing different rules
>>>> for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while f(a=1,
>>>> a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1,
>>>> **{'a': 1}) is a TypeError.)
>>>>
>>>> For me, allowing f(*a, *b) and f(**d, **e) and all the other
>>>> combinations for function calls proposed by the PEP is an easy +1 -- it's a
>>>> straightforward extension of the existing pattern, and anybody who knows
>>>> what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d,
>>>> **e) means shouldn't be hard either. Understanding the edge case for
>>>> duplicate keys with f(**d, **e) is a little harder, but the error messages
>>>> are pretty clear, and it is not a new edge case.
>>>>
>>>> The sequence and dict flattening syntax proposals are also clean and
>>>> logical -- we already have *-unpacking on the receiving side, so allowing
>>>> *x in tuple expressions reads pretty naturally (and the similarity with *a
>>>> in argument lists certainly helps). From here, having [a, *x, b, *y] is
>>>> also natural, and then the extension to other displays is natural: {a, *x,
>>>> b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me.
>>>>
>>>> So that leaves comprehensions. IIRC, during the development of the
>>>> patch we realized that f(*x for x in xs) is sufficiently ambiguous that we
>>>> decided to disallow it -- note that f(x for x in xs) is already somewhat of
>>>> a special case because an argument can only be a "bare" generator
>>>> expression if it is the only argument. The same reasoning doesn't apply (in
>>>> that form) to list, set and dict comprehensions -- while f(x for x in xs)
>>>> is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the
>>>> same as [(x for x in xs)] (that's a list of one element, and the element is
>>>> a generator expression).
>>>>
>>>> The basic premise of this part of the proposal is that if you have a
>>>> few iterables, the new proposal (without comprehensions) lets you create a
>>>> list or generator expression that iterates over all of them, essentially
>>>> flattening them:
>>>>
>>>>     >>> xs = [1, 2, 3]
>>>>     >>> ys = ['abc', 'def']
>>>>     >>> zs = [99]
>>>>     >>> [*xs, *ys, *zs]
>>>>     [1, 2, 3, 'abc', 'def', 99]
>>>>     >>>
>>>>
>>>> But now suppose you have a list of iterables:
>>>>
>>>>     >>> xss = [[1, 2, 3], ['abc', 'def'], [99]]
>>>>     >>> [*xss[0], *xss[1], *xss[2]]
>>>>     [1, 2, 3, 'abc', 'def', 99]
>>>>     >>>
>>>>
>>>> Wouldn't it be nice if you could write the latter using a comprehension?
>>>>
>>>>     >>> xss = [[1, 2, 3], ['abc', 'def'], [99]]
>>>>     >>> [*xs for xs in xss]
>>>>     [1, 2, 3, 'abc', 'def', 99]
>>>>     >>>
>>>>
>>>> This is somewhat seductive, and the following is even nicer: the *xs
>>>> position may be an expression, e.g.:
>>>>
>>>>     >>> xss = [[1, 2, 3], ['abc', 'def'], [99]]
>>>>     >>> [*xs[:2] for xs in xss]
>>>>     [1, 2, 'abc', 'def', 99]
>>>>     >>>
>>>>
>>>> On the other hand, I had to explore the possibilities here by
>>>> experimenting in the interpreter, and I discovered some odd edge cases
>>>> (e.g. you can parenthesize the starred expression, but that seems a
>>>> syntactic accident).
>>>>
>>>> All in all I am personally +0 on the comprehension part of the PEP, and
>>>> I like that it provides a way to "flatten" a sequence of sequences, but I
>>>> think very few people in the thread have supported this part. Therefore I
>>>> would like to ask Neil to update the PEP and the patch to take out the
>>>> comprehension part, so that the two "easy wins" can make it into Python 3.5
>>>> (basically, I am accepting two-thirds of the PEP :-). There is some time
>>>> yet until alpha 2.
>>>>
>>>> I would also like code reviewers (Benjamin?) to start reviewing the
>>>> patch <http://bugs.python.org/issue2292>, taking into account that the
>>>> comprehension part needs to be removed.
>>>>
>>>> --
>>>> --Guido van Rossum (python.org/~guido)
>>>>
>>>>
>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150302/0aa3025b/attachment-0001.html>

From chris.barker at noaa.gov  Tue Mar  3 06:25:21 2015
From: chris.barker at noaa.gov (Chris Barker)
Date: Mon, 2 Mar 2015 21:25:21 -0800
Subject: [Python-Dev] PEP 485 review (isclose())
In-Reply-To: <CALGmxEJQEGSqYKQp4dzJBLM2xB7J8dz_7hFx4gBvOLL+Ra9Bhw@mail.gmail.com>
References: <CAP7+vJJZY+8LNJ+-130ot8TSiC25p3gkwFGxgF6vuF2GEKo_3A@mail.gmail.com>
 <CALGmxEJQEGSqYKQp4dzJBLM2xB7J8dz_7hFx4gBvOLL+Ra9Bhw@mail.gmail.com>
Message-ID: <CALGmxELG2iBbD4q41QSW+f6oGL6zM1YAeQwrmDOi-TMJ2E8q6A@mail.gmail.com>

On Fri, Feb 27, 2015 at 12:07 PM, Chris Barker <chris.barker at noaa.gov>
wrote:

> I'll edit the text as you suggest,
>

Done.


> and then work on a patch -- I'm sure I'll have questions for Python-dev
> when I actually do that, but I'll get started on my  own and see how far I
> get.
>

OK --  big question 1:

As far as I can tell, the math module is entirely a C extension. So I can:

1) write isclose() in C -- I could probably do that (It's been a while --
I'm a big Cython fan these days...) and I'd probably punt on having it work
with anything other than floats if I did that -- which would fit how most
of the match module works anyway.

2) Write it in Python, and monkey-patch it in to the math module -- I
honestly have no idea how to do that, but as I can add a new name to the
math module after importing it, it should be doable --but I have no idea
where the code would go.

Suggestions??

Thanks,
  -Chris








> -Chris
>
>
>
>
>
> On Fri, Feb 27, 2015 at 11:38 AM, Guido van Rossum <guido at python.org>
> wrote:
>
>> I think it's time to accept PEP 485. I've re-read it once more, and it
>> looks like the text is in great shape. (My only recommendation would be to
>> update the Abstract to state that we're specifically adding math.isclose().)
>>
>> A wording question: "This implementation has a flag that lets the user
>> select which relative tolerance test to apply -- this PEP does not suggest
>> that that be retained, but rather than the weak test be selected." -- I
>> think this was meant to say "... rather *that* the weak test be selected",
>> right? (It would be nice if the sample implementation defaulted to the
>> choice in the PEP.)
>>
>> However, those are just minor edits, and I hereby approve the PEP. Thanks
>> Chris and everyone else for the fruitful discussion (and thanks especially
>> to Chris for eventually ending the bikeshedding and writing a PEP that
>> explains each of the choices). Congrats!
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>>
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150302/fa0ad8f2/attachment.html>

From guido at python.org  Tue Mar  3 07:39:21 2015
From: guido at python.org (Guido van Rossum)
Date: Mon, 2 Mar 2015 22:39:21 -0800
Subject: [Python-Dev] PEP 485 review (isclose())
In-Reply-To: <CALGmxELG2iBbD4q41QSW+f6oGL6zM1YAeQwrmDOi-TMJ2E8q6A@mail.gmail.com>
References: <CAP7+vJJZY+8LNJ+-130ot8TSiC25p3gkwFGxgF6vuF2GEKo_3A@mail.gmail.com>
 <CALGmxEJQEGSqYKQp4dzJBLM2xB7J8dz_7hFx4gBvOLL+Ra9Bhw@mail.gmail.com>
 <CALGmxELG2iBbD4q41QSW+f6oGL6zM1YAeQwrmDOi-TMJ2E8q6A@mail.gmail.com>
Message-ID: <CAP7+vJ+USHx1kFFk3cvE672+Ariq-aWKXu7gFBk4qK19kOvw6g@mail.gmail.com>

It should be written in C. I thought you knew. :-)

I guess it's okay to drop Decimal support (*). Complex support probably
should be in cmath.

But this is an amendmend to the PEP as accepted. If anyone cares about
these changes they should speak up, otherwise the PEP should be updated.

(*) Adding it to the decimal module would require a discussion with Raymond
Hettinger, but Decimal users can probably copy and paste the formula from
the PEP.

On Monday, March 2, 2015, Chris Barker <chris.barker at noaa.gov> wrote:

> On Fri, Feb 27, 2015 at 12:07 PM, Chris Barker <chris.barker at noaa.gov
> <javascript:_e(%7B%7D,'cvml','chris.barker at noaa.gov');>> wrote:
>
>> I'll edit the text as you suggest,
>>
>
> Done.
>
>
>> and then work on a patch -- I'm sure I'll have questions for Python-dev
>> when I actually do that, but I'll get started on my  own and see how far I
>> get.
>>
>
> OK --  big question 1:
>
> As far as I can tell, the math module is entirely a C extension. So I can:
>
> 1) write isclose() in C -- I could probably do that (It's been a while --
> I'm a big Cython fan these days...) and I'd probably punt on having it work
> with anything other than floats if I did that -- which would fit how most
> of the match module works anyway.
>
> 2) Write it in Python, and monkey-patch it in to the math module -- I
> honestly have no idea how to do that, but as I can add a new name to the
> math module after importing it, it should be doable --but I have no idea
> where the code would go.
>
> Suggestions??
>
> Thanks,
>   -Chris
>
>
>
>
>
>
>
>
>> -Chris
>>
>>
>>
>>
>>
>> On Fri, Feb 27, 2015 at 11:38 AM, Guido van Rossum <guido at python.org
>> <javascript:_e(%7B%7D,'cvml','guido at python.org');>> wrote:
>>
>>> I think it's time to accept PEP 485. I've re-read it once more, and it
>>> looks like the text is in great shape. (My only recommendation would be to
>>> update the Abstract to state that we're specifically adding math.isclose().)
>>>
>>> A wording question: "This implementation has a flag that lets the user
>>> select which relative tolerance test to apply -- this PEP does not suggest
>>> that that be retained, but rather than the weak test be selected." -- I
>>> think this was meant to say "... rather *that* the weak test be selected",
>>> right? (It would be nice if the sample implementation defaulted to the
>>> choice in the PEP.)
>>>
>>> However, those are just minor edits, and I hereby approve the PEP.
>>> Thanks Chris and everyone else for the fruitful discussion (and thanks
>>> especially to Chris for eventually ending the bikeshedding and writing a
>>> PEP that explains each of the choices). Congrats!
>>>
>>> --
>>> --Guido van Rossum (python.org/~guido)
>>>
>>
>>
>>
>> --
>>
>> Christopher Barker, Ph.D.
>> Oceanographer
>>
>> Emergency Response Division
>> NOAA/NOS/OR&R            (206) 526-6959   voice
>> 7600 Sand Point Way NE   (206) 526-6329   fax
>> Seattle, WA  98115       (206) 526-6317   main reception
>>
>> Chris.Barker at noaa.gov
>> <javascript:_e(%7B%7D,'cvml','Chris.Barker at noaa.gov');>
>>
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
> <javascript:_e(%7B%7D,'cvml','Chris.Barker at noaa.gov');>
>


-- 
--Guido van Rossum (on iPad)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150302/9b5c1998/attachment.html>

From chris.barker at noaa.gov  Tue Mar  3 07:48:31 2015
From: chris.barker at noaa.gov (Chris Barker)
Date: Mon, 2 Mar 2015 22:48:31 -0800
Subject: [Python-Dev] PEP 485 review (isclose())
In-Reply-To: <CAP7+vJ+USHx1kFFk3cvE672+Ariq-aWKXu7gFBk4qK19kOvw6g@mail.gmail.com>
References: <CAP7+vJJZY+8LNJ+-130ot8TSiC25p3gkwFGxgF6vuF2GEKo_3A@mail.gmail.com>
 <CALGmxEJQEGSqYKQp4dzJBLM2xB7J8dz_7hFx4gBvOLL+Ra9Bhw@mail.gmail.com>
 <CALGmxELG2iBbD4q41QSW+f6oGL6zM1YAeQwrmDOi-TMJ2E8q6A@mail.gmail.com>
 <CAP7+vJ+USHx1kFFk3cvE672+Ariq-aWKXu7gFBk4qK19kOvw6g@mail.gmail.com>
Message-ID: <CALGmxEL1Nq-JAwm4J1qUkASdjz_qSMY0CaGZ4v1Lpjn4imqEjA@mail.gmail.com>

On Mon, Mar 2, 2015 at 10:39 PM, Guido van Rossum <guido at python.org> wrote:

> It should be written in C. I thought you knew. :-)
>

It crossed my mind when we talked about the math module -- but I didn't
really think about it...

I guess it's okay to drop Decimal support (*). Complex support probably
> should be in cmath.
>

fair enough -- and easy enough to do if we do float-only in math, and
complex-only in cmath.


But this is an amendmend to the PEP as accepted. If anyone cares about
> these changes they should speak up, otherwise the PEP should be updated.
>

will do.


> (*) Adding it to the decimal module would require a discussion with
> Raymond Hettinger, but Decimal users can probably copy and paste the
> formula from the PEP.
>
>
yup -- but maybe worth putting in there while we're at it. though as
Decimal is arbitrary precision, maybe it's not needed....

-Chris



> On Monday, March 2, 2015, Chris Barker <chris.barker at noaa.gov> wrote:
>
>> On Fri, Feb 27, 2015 at 12:07 PM, Chris Barker <chris.barker at noaa.gov>
>> wrote:
>>
>>> I'll edit the text as you suggest,
>>>
>>
>> Done.
>>
>>
>>> and then work on a patch -- I'm sure I'll have questions for Python-dev
>>> when I actually do that, but I'll get started on my  own and see how far I
>>> get.
>>>
>>
>> OK --  big question 1:
>>
>> As far as I can tell, the math module is entirely a C extension. So I can:
>>
>> 1) write isclose() in C -- I could probably do that (It's been a while --
>> I'm a big Cython fan these days...) and I'd probably punt on having it work
>> with anything other than floats if I did that -- which would fit how most
>> of the match module works anyway.
>>
>> 2) Write it in Python, and monkey-patch it in to the math module -- I
>> honestly have no idea how to do that, but as I can add a new name to the
>> math module after importing it, it should be doable --but I have no idea
>> where the code would go.
>>
>> Suggestions??
>>
>> Thanks,
>>   -Chris
>>
>>
>>
>>
>>
>>
>>
>>
>>> -Chris
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Feb 27, 2015 at 11:38 AM, Guido van Rossum <guido at python.org>
>>> wrote:
>>>
>>>> I think it's time to accept PEP 485. I've re-read it once more, and it
>>>> looks like the text is in great shape. (My only recommendation would be to
>>>> update the Abstract to state that we're specifically adding math.isclose().)
>>>>
>>>> A wording question: "This implementation has a flag that lets the user
>>>> select which relative tolerance test to apply -- this PEP does not suggest
>>>> that that be retained, but rather than the weak test be selected." -- I
>>>> think this was meant to say "... rather *that* the weak test be selected",
>>>> right? (It would be nice if the sample implementation defaulted to the
>>>> choice in the PEP.)
>>>>
>>>> However, those are just minor edits, and I hereby approve the PEP.
>>>> Thanks Chris and everyone else for the fruitful discussion (and thanks
>>>> especially to Chris for eventually ending the bikeshedding and writing a
>>>> PEP that explains each of the choices). Congrats!
>>>>
>>>> --
>>>> --Guido van Rossum (python.org/~guido)
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Christopher Barker, Ph.D.
>>> Oceanographer
>>>
>>> Emergency Response Division
>>> NOAA/NOS/OR&R            (206) 526-6959   voice
>>> 7600 Sand Point Way NE   (206) 526-6329   fax
>>> Seattle, WA  98115       (206) 526-6317   main reception
>>>
>>> Chris.Barker at noaa.gov
>>>
>>
>>
>>
>> --
>>
>> Christopher Barker, Ph.D.
>> Oceanographer
>>
>> Emergency Response Division
>> NOAA/NOS/OR&R            (206) 526-6959   voice
>> 7600 Sand Point Way NE   (206) 526-6329   fax
>> Seattle, WA  98115       (206) 526-6317   main reception
>>
>> Chris.Barker at noaa.gov
>>
>
>
> --
> --Guido van Rossum (on iPad)
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150302/b9c89801/attachment.html>

From victor.stinner at gmail.com  Tue Mar  3 10:17:40 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 3 Mar 2015 10:17:40 +0100
Subject: [Python-Dev] PEP 485 review (isclose())
In-Reply-To: <CALGmxELG2iBbD4q41QSW+f6oGL6zM1YAeQwrmDOi-TMJ2E8q6A@mail.gmail.com>
References: <CAP7+vJJZY+8LNJ+-130ot8TSiC25p3gkwFGxgF6vuF2GEKo_3A@mail.gmail.com>
 <CALGmxEJQEGSqYKQp4dzJBLM2xB7J8dz_7hFx4gBvOLL+Ra9Bhw@mail.gmail.com>
 <CALGmxELG2iBbD4q41QSW+f6oGL6zM1YAeQwrmDOi-TMJ2E8q6A@mail.gmail.com>
Message-ID: <CAMpsgwbabEpkaPSykHr0TFHvWKycCAR2RGdj9eSm3sYMH5nTOg@mail.gmail.com>

2015-03-03 6:25 GMT+01:00 Chris Barker <chris.barker at noaa.gov>:
> As far as I can tell, the math module is entirely a C extension. So I can:
> (...)
> 2) Write it in Python, and monkey-patch it in to the math module -- I
> honestly have no idea how to do that, but as I can add a new name to the
> math module after importing it, it should be doable --but I have no idea
> where the code would go.

Maybe it's time to rename the math module to _math and create a
math.py module, like _decimal/decimal? math.py should end with "from
_math import *".

It may be interesting to have Python implementation of math.fsum(),
math.factorial(), math.degrees() and math.radians().

Extract of fsum() comment:

   Full precision summation of a sequence of floats.

   def msum(iterable):
       partials = []  # sorted, non-overlapping partial sums
       for x in iterable:
           i = 0
           for y in partials:
               if abs(x) < abs(y):
                   x, y = y, x
               hi = x + y
               lo = y - (hi - x)
               if lo:
                   partials[i] = lo
                   i += 1
               x = hi
           partials[i:] = [x]
       return sum_exact(partials)

The C implementation of factorial is not naive: "Divide-and-conquer
factorial algorithm" (see the C code).

Obvisouly, a expect lower performances from Python code manipulating
numbers (because of the cost of boxing-unboxing, cost of functions
calls, etc.). But it might help other Python implementations to
implement the math module.

Victor

From guido at python.org  Tue Mar  3 17:14:39 2015
From: guido at python.org (Guido van Rossum)
Date: Tue, 3 Mar 2015 08:14:39 -0800
Subject: [Python-Dev] PEP 485 review (isclose())
In-Reply-To: <CALGmxEL1Nq-JAwm4J1qUkASdjz_qSMY0CaGZ4v1Lpjn4imqEjA@mail.gmail.com>
References: <CAP7+vJJZY+8LNJ+-130ot8TSiC25p3gkwFGxgF6vuF2GEKo_3A@mail.gmail.com>
 <CALGmxEJQEGSqYKQp4dzJBLM2xB7J8dz_7hFx4gBvOLL+Ra9Bhw@mail.gmail.com>
 <CALGmxELG2iBbD4q41QSW+f6oGL6zM1YAeQwrmDOi-TMJ2E8q6A@mail.gmail.com>
 <CAP7+vJ+USHx1kFFk3cvE672+Ariq-aWKXu7gFBk4qK19kOvw6g@mail.gmail.com>
 <CALGmxEL1Nq-JAwm4J1qUkASdjz_qSMY0CaGZ4v1Lpjn4imqEjA@mail.gmail.com>
Message-ID: <CAP7+vJKYO4yzOHswCqugqtNOL4EE3hZSSjUYxQShShm3QV51tQ@mail.gmail.com>

On Mon, Mar 2, 2015 at 10:48 PM, Chris Barker <chris.barker at noaa.gov> wrote:

> On Mon, Mar 2, 2015 at 10:39 PM, Guido van Rossum <guido at python.org>
> wrote:
>
> (*) Adding it to the decimal module would require a discussion with
>> Raymond Hettinger, but Decimal users can probably copy and paste the
>> formula from the PEP.
>>
>
> yup -- but maybe worth putting in there while we're at it. though as
> Decimal is arbitrary precision, maybe it's not needed....
>

It's not really arbitrary precision, it's decimal *floating* point with a
large but finite precision, so all the same arguments apply.

But my reasoning was more that (at least when I was last involved in the
project) the decimal module tries to stick pretty close to the IEEE 754
standard (maybe nowadays a later version?) and random Python-only additions
are controversial.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150303/19cd361a/attachment.html>

From ethan at stoneleaf.us  Tue Mar  3 17:43:06 2015
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 03 Mar 2015 08:43:06 -0800
Subject: [Python-Dev] PEP 485 review (isclose())
In-Reply-To: <CAMpsgwbabEpkaPSykHr0TFHvWKycCAR2RGdj9eSm3sYMH5nTOg@mail.gmail.com>
References: <CAP7+vJJZY+8LNJ+-130ot8TSiC25p3gkwFGxgF6vuF2GEKo_3A@mail.gmail.com>
 <CALGmxEJQEGSqYKQp4dzJBLM2xB7J8dz_7hFx4gBvOLL+Ra9Bhw@mail.gmail.com>
 <CALGmxELG2iBbD4q41QSW+f6oGL6zM1YAeQwrmDOi-TMJ2E8q6A@mail.gmail.com>
 <CAMpsgwbabEpkaPSykHr0TFHvWKycCAR2RGdj9eSm3sYMH5nTOg@mail.gmail.com>
Message-ID: <54F5E49A.8080803@stoneleaf.us>

On 03/03/2015 01:17 AM, Victor Stinner wrote:
> 2015-03-03 6:25 GMT+01:00 Chris Barker :
>>
>> As far as I can tell, the math module is entirely a C extension. So I can:

>> 2) Write it in Python, and monkey-patch it in to the math module -- I
>> honestly have no idea how to do that, but as I can add a new name to the
>> math module after importing it, it should be doable --but I have no idea
>> where the code would go.

> Maybe it's time to rename the math module to _math and create a
> math.py module, like _decimal/decimal? math.py should end with "from
> _math import *".

+1

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150303/ee0ad859/attachment.sig>

From chris.barker at noaa.gov  Wed Mar  4 21:12:34 2015
From: chris.barker at noaa.gov (Chris Barker)
Date: Wed, 4 Mar 2015 12:12:34 -0800
Subject: [Python-Dev] PEP 485 review (isclose())
In-Reply-To: <54F5E49A.8080803@stoneleaf.us>
References: <CAP7+vJJZY+8LNJ+-130ot8TSiC25p3gkwFGxgF6vuF2GEKo_3A@mail.gmail.com>
 <CALGmxEJQEGSqYKQp4dzJBLM2xB7J8dz_7hFx4gBvOLL+Ra9Bhw@mail.gmail.com>
 <CALGmxELG2iBbD4q41QSW+f6oGL6zM1YAeQwrmDOi-TMJ2E8q6A@mail.gmail.com>
 <CAMpsgwbabEpkaPSykHr0TFHvWKycCAR2RGdj9eSm3sYMH5nTOg@mail.gmail.com>
 <54F5E49A.8080803@stoneleaf.us>
Message-ID: <CALGmxEKSz6wBzq7NyfN9B0S5EBdg3Y9wJttBoBpoebu=8i3jFA@mail.gmail.com>

On Tue, Mar 3, 2015 at 8:43 AM, Ethan Furman <ethan at stoneleaf.us> wrote:

> On 03/03/2015 01:17 AM, Victor Stinner wrote:
>


> > Maybe it's time to rename the math module to _math and create a
> > math.py module, like _decimal/decimal? math.py should end with "from
> > _math import *".
>
> +1
>

What do folks think? If we're going to do this, I'll write isclose() in
python. And I could do the work to split it out, too, I suppose.

-Chris





-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150304/a6ef308b/attachment.html>

From guido at python.org  Wed Mar  4 21:22:41 2015
From: guido at python.org (Guido van Rossum)
Date: Wed, 4 Mar 2015 12:22:41 -0800
Subject: [Python-Dev] PEP 485 review (isclose())
In-Reply-To: <CALGmxEKSz6wBzq7NyfN9B0S5EBdg3Y9wJttBoBpoebu=8i3jFA@mail.gmail.com>
References: <CAP7+vJJZY+8LNJ+-130ot8TSiC25p3gkwFGxgF6vuF2GEKo_3A@mail.gmail.com>
 <CALGmxEJQEGSqYKQp4dzJBLM2xB7J8dz_7hFx4gBvOLL+Ra9Bhw@mail.gmail.com>
 <CALGmxELG2iBbD4q41QSW+f6oGL6zM1YAeQwrmDOi-TMJ2E8q6A@mail.gmail.com>
 <CAMpsgwbabEpkaPSykHr0TFHvWKycCAR2RGdj9eSm3sYMH5nTOg@mail.gmail.com>
 <54F5E49A.8080803@stoneleaf.us>
 <CALGmxEKSz6wBzq7NyfN9B0S5EBdg3Y9wJttBoBpoebu=8i3jFA@mail.gmail.com>
Message-ID: <CAP7+vJJMxAqG2nuXAXnsFn4+m7N9-0aC4XDe6M07F2-fTNFifw@mail.gmail.com>

No.

On Wed, Mar 4, 2015 at 12:12 PM, Chris Barker <chris.barker at noaa.gov> wrote:

> On Tue, Mar 3, 2015 at 8:43 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
>
>> On 03/03/2015 01:17 AM, Victor Stinner wrote:
>>
>
>
>> > Maybe it's time to rename the math module to _math and create a
>> > math.py module, like _decimal/decimal? math.py should end with "from
>> > _math import *".
>>
>> +1
>>
>
> What do folks think? If we're going to do this, I'll write isclose() in
> python. And I could do the work to split it out, too, I suppose.
>
> -Chris
>
>
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150304/e108818c/attachment.html>

From brett at python.org  Wed Mar  4 21:23:24 2015
From: brett at python.org (Brett Cannon)
Date: Wed, 04 Mar 2015 20:23:24 +0000
Subject: [Python-Dev] PEP 485 review (isclose())
References: <CAP7+vJJZY+8LNJ+-130ot8TSiC25p3gkwFGxgF6vuF2GEKo_3A@mail.gmail.com>
 <CALGmxEJQEGSqYKQp4dzJBLM2xB7J8dz_7hFx4gBvOLL+Ra9Bhw@mail.gmail.com>
 <CALGmxELG2iBbD4q41QSW+f6oGL6zM1YAeQwrmDOi-TMJ2E8q6A@mail.gmail.com>
 <CAMpsgwbabEpkaPSykHr0TFHvWKycCAR2RGdj9eSm3sYMH5nTOg@mail.gmail.com>
 <54F5E49A.8080803@stoneleaf.us>
 <CALGmxEKSz6wBzq7NyfN9B0S5EBdg3Y9wJttBoBpoebu=8i3jFA@mail.gmail.com>
Message-ID: <CAP1=2W4PZo25prx+U7m4CZX83QD6Qw_K2QEZTkn-WJyaQHevFA@mail.gmail.com>

On Wed, Mar 4, 2015 at 3:14 PM Chris Barker <chris.barker at noaa.gov> wrote:

> On Tue, Mar 3, 2015 at 8:43 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
>
>> On 03/03/2015 01:17 AM, Victor Stinner wrote:
>>
>
>
>> > Maybe it's time to rename the math module to _math and create a
>> > math.py module, like _decimal/decimal? math.py should end with "from
>> > _math import *".
>>
>> +1
>>
>
> What do folks think? If we're going to do this, I'll write isclose() in
> python. And I could do the work to split it out, too, I suppose.
>

My vote -- as always -- is to do it in Python. If someone is sufficiently
motivated to re-implement in C then that's great, but I don't think it
should be required to be in C.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150304/d2643082/attachment.html>

From guido at python.org  Wed Mar  4 21:33:48 2015
From: guido at python.org (Guido van Rossum)
Date: Wed, 4 Mar 2015 12:33:48 -0800
Subject: [Python-Dev] PEP 485 review (isclose())
In-Reply-To: <CAP1=2W4PZo25prx+U7m4CZX83QD6Qw_K2QEZTkn-WJyaQHevFA@mail.gmail.com>
References: <CAP7+vJJZY+8LNJ+-130ot8TSiC25p3gkwFGxgF6vuF2GEKo_3A@mail.gmail.com>
 <CALGmxEJQEGSqYKQp4dzJBLM2xB7J8dz_7hFx4gBvOLL+Ra9Bhw@mail.gmail.com>
 <CALGmxELG2iBbD4q41QSW+f6oGL6zM1YAeQwrmDOi-TMJ2E8q6A@mail.gmail.com>
 <CAMpsgwbabEpkaPSykHr0TFHvWKycCAR2RGdj9eSm3sYMH5nTOg@mail.gmail.com>
 <54F5E49A.8080803@stoneleaf.us>
 <CALGmxEKSz6wBzq7NyfN9B0S5EBdg3Y9wJttBoBpoebu=8i3jFA@mail.gmail.com>
 <CAP1=2W4PZo25prx+U7m4CZX83QD6Qw_K2QEZTkn-WJyaQHevFA@mail.gmail.com>
Message-ID: <CAP7+vJK0h0f=J-DA7hCgom41V+fC=jFc6h0E3-1WoUBeQH-3Qg@mail.gmail.com>

In this case I disagree. The math module mostly wraps the C math library
and the bar should remain high for things to be added to it. Let this be
someone's opportunity to learn C (I guess not Chris's :-).

On Wed, Mar 4, 2015 at 12:23 PM, Brett Cannon <brett at python.org> wrote:

>
>
> On Wed, Mar 4, 2015 at 3:14 PM Chris Barker <chris.barker at noaa.gov> wrote:
>
>> On Tue, Mar 3, 2015 at 8:43 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
>>
>>> On 03/03/2015 01:17 AM, Victor Stinner wrote:
>>>
>>
>>
>>> > Maybe it's time to rename the math module to _math and create a
>>> > math.py module, like _decimal/decimal? math.py should end with "from
>>> > _math import *".
>>>
>>> +1
>>>
>>
>> What do folks think? If we're going to do this, I'll write isclose() in
>> python. And I could do the work to split it out, too, I suppose.
>>
>
> My vote -- as always -- is to do it in Python. If someone is sufficiently
> motivated to re-implement in C then that's great, but I don't think it
> should be required to be in C.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150304/8a086610/attachment.html>

From chris.barker at noaa.gov  Thu Mar  5 00:26:40 2015
From: chris.barker at noaa.gov (Chris Barker)
Date: Wed, 4 Mar 2015 15:26:40 -0800
Subject: [Python-Dev] PEP 485 review (isclose())
In-Reply-To: <CAP7+vJK0h0f=J-DA7hCgom41V+fC=jFc6h0E3-1WoUBeQH-3Qg@mail.gmail.com>
References: <CAP7+vJJZY+8LNJ+-130ot8TSiC25p3gkwFGxgF6vuF2GEKo_3A@mail.gmail.com>
 <CALGmxEJQEGSqYKQp4dzJBLM2xB7J8dz_7hFx4gBvOLL+Ra9Bhw@mail.gmail.com>
 <CALGmxELG2iBbD4q41QSW+f6oGL6zM1YAeQwrmDOi-TMJ2E8q6A@mail.gmail.com>
 <CAMpsgwbabEpkaPSykHr0TFHvWKycCAR2RGdj9eSm3sYMH5nTOg@mail.gmail.com>
 <54F5E49A.8080803@stoneleaf.us>
 <CALGmxEKSz6wBzq7NyfN9B0S5EBdg3Y9wJttBoBpoebu=8i3jFA@mail.gmail.com>
 <CAP1=2W4PZo25prx+U7m4CZX83QD6Qw_K2QEZTkn-WJyaQHevFA@mail.gmail.com>
 <CAP7+vJK0h0f=J-DA7hCgom41V+fC=jFc6h0E3-1WoUBeQH-3Qg@mail.gmail.com>
Message-ID: <CALGmxE+So8SnybKaEBRpZjos8szrBpsCa3N7HcRusweHsgE_2A@mail.gmail.com>

On Wed, Mar 4, 2015 at 12:33 PM, Guido van Rossum <guido at python.org> wrote:

> In this case I disagree. The math module mostly wraps the C math library
> and the bar should remain high for things to be added to it. Let this be
> someone's opportunity to learn C (I guess not Chris's :-).
>

As it happens, the first C code I ever wrote, beyond toy examples, was
python extensions. I dropped that for Cython a good while ago, but  I can
remember enough for this simple function ;-)

I agree that it's not worth going to python for the math module for just
this -- but others were proposing other reasons to do it.

Back to look at K&R ;-)

-Chris




> On Wed, Mar 4, 2015 at 12:23 PM, Brett Cannon <brett at python.org> wrote:
>
>>
>>
>> On Wed, Mar 4, 2015 at 3:14 PM Chris Barker <chris.barker at noaa.gov>
>> wrote:
>>
>>> On Tue, Mar 3, 2015 at 8:43 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
>>>
>>>> On 03/03/2015 01:17 AM, Victor Stinner wrote:
>>>>
>>>
>>>
>>>> > Maybe it's time to rename the math module to _math and create a
>>>> > math.py module, like _decimal/decimal? math.py should end with "from
>>>> > _math import *".
>>>>
>>>> +1
>>>>
>>>
>>> What do folks think? If we're going to do this, I'll write isclose() in
>>> python. And I could do the work to split it out, too, I suppose.
>>>
>>
>> My vote -- as always -- is to do it in Python. If someone is sufficiently
>> motivated to re-implement in C then that's great, but I don't think it
>> should be required to be in C.
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150304/10af4471/attachment.html>

From guido at python.org  Thu Mar  5 01:24:39 2015
From: guido at python.org (Guido van Rossum)
Date: Wed, 4 Mar 2015 16:24:39 -0800
Subject: [Python-Dev] Any volunteers to implement PEP 479?
In-Reply-To: <CAPTjJmr87QOb4A5GwO_qsPtkxGo2ikiMF2e6uaf7n0qkPW1GDg@mail.gmail.com>
References: <CAP7+vJJW630SKE6jw+yLm6R1wzEzj3b=ObNFWFm0a=pUOkmJnA@mail.gmail.com>
 <CAPTjJmr87QOb4A5GwO_qsPtkxGo2ikiMF2e6uaf7n0qkPW1GDg@mail.gmail.com>
Message-ID: <CAP7+vJJxwJYwCLk4soVbHEi4EC3v1+TfCwP6_b1nWLx==ncUXA@mail.gmail.com>

Ping? Anyone? http://bugs.python.org/issue22906

On Fri, Feb 20, 2015 at 9:56 PM, Chris Angelico <rosuav at gmail.com> wrote:

> On Wed, Jan 7, 2015 at 2:48 PM, Guido van Rossum <guido at python.org> wrote:
> > There's a proof of concept patch in http://bugs.python.org/issue22906,
> but
> > it doesn't have the __future__ import and probably gets other details
> wrong.
> >
> > Reference:
> > PEP 479 -- Change StopIteration handling inside generators
> >
> > --
> > --Guido van Rossum (python.org/~guido)
>
> For the benefit of people who haven't been following the tracker
> issue: There is now a patch which *does* create the __future__
> directive and so on. It applies cleanly to current tip (I just tested
> it again today), and the test suite passes on Debian AMD64. Can other
> people please try this, on other platforms, and see how it goes?
>
> ChrisA
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150304/57d420cc/attachment.html>

From ethan at stoneleaf.us  Thu Mar  5 01:42:15 2015
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 04 Mar 2015 16:42:15 -0800
Subject: [Python-Dev] Any volunteers to implement PEP 479?
In-Reply-To: <CAP7+vJJxwJYwCLk4soVbHEi4EC3v1+TfCwP6_b1nWLx==ncUXA@mail.gmail.com>
References: <CAP7+vJJW630SKE6jw+yLm6R1wzEzj3b=ObNFWFm0a=pUOkmJnA@mail.gmail.com>
 <CAPTjJmr87QOb4A5GwO_qsPtkxGo2ikiMF2e6uaf7n0qkPW1GDg@mail.gmail.com>
 <CAP7+vJJxwJYwCLk4soVbHEi4EC3v1+TfCwP6_b1nWLx==ncUXA@mail.gmail.com>
Message-ID: <54F7A667.4050205@stoneleaf.us>

On 03/04/2015 04:24 PM, Guido van Rossum wrote:
>
> Ping? Anyone? http://bugs.python.org/issue22906

Running tests on an ubuntu system...

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150304/96c32fd5/attachment.sig>

From shahrukhkhan201994 at gmail.com  Thu Mar  5 12:13:39 2015
From: shahrukhkhan201994 at gmail.com (SHAHRUKH KHAN)
Date: Thu, 5 Mar 2015 16:43:39 +0530
Subject: [Python-Dev] Python-Dev Digest, Vol 140, Issue 6
In-Reply-To: <mailman.71.1425553204.8868.python-dev@python.org>
References: <mailman.71.1425553204.8868.python-dev@python.org>
Message-ID: <CAP_KOd1uARLWitWNPH43_akJbQu64+EUMrTfKb7amg-nH7GZag@mail.gmail.com>

i think it's time to rename math module to math.py module

On Thu, Mar 5, 2015 at 4:30 PM, <python-dev-request at python.org> wrote:

> Send Python-Dev mailing list submissions to
>         python-dev at python.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         https://mail.python.org/mailman/listinfo/python-dev
> or, via email, send a message with subject or body 'help' to
>         python-dev-request at python.org
>
> You can reach the person managing the list at
>         python-dev-owner at python.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Python-Dev digest..."
>
>
> Today's Topics:
>
>    1. Re: PEP 485 review (isclose()) (Chris Barker)
>    2. Re: PEP 485 review (isclose()) (Guido van Rossum)
>    3. Re: PEP 485 review (isclose()) (Brett Cannon)
>    4. Re: PEP 485 review (isclose()) (Guido van Rossum)
>    5. Re: PEP 485 review (isclose()) (Chris Barker)
>    6. Re: Any volunteers to implement PEP 479? (Guido van Rossum)
>    7. Re: Any volunteers to implement PEP 479? (Ethan Furman)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Wed, 4 Mar 2015 12:12:34 -0800
> From: Chris Barker <chris.barker at noaa.gov>
> To: Ethan Furman <ethan at stoneleaf.us>
> Cc: Python Dev <python-dev at python.org>
> Subject: Re: [Python-Dev] PEP 485 review (isclose())
> Message-ID:
>         <CALGmxEKSz6wBzq7NyfN9B0S5EBdg3Y9wJttBoBpoebu=
> 8i3jFA at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> On Tue, Mar 3, 2015 at 8:43 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
>
> > On 03/03/2015 01:17 AM, Victor Stinner wrote:
> >
>
>
> > > Maybe it's time to rename the math module to _math and create a
> > > math.py module, like _decimal/decimal? math.py should end with "from
> > > _math import *".
> >
> > +1
> >
>
> What do folks think? If we're going to do this, I'll write isclose() in
> python. And I could do the work to split it out, too, I suppose.
>
> -Chris
>
>
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://mail.python.org/pipermail/python-dev/attachments/20150304/a6ef308b/attachment-0001.html
> >
>
> ------------------------------
>
> Message: 2
> Date: Wed, 4 Mar 2015 12:22:41 -0800
> From: Guido van Rossum <guido at python.org>
> To: Chris Barker <chris.barker at noaa.gov>
> Cc: Ethan Furman <ethan at stoneleaf.us>, Python Dev
>         <python-dev at python.org>
> Subject: Re: [Python-Dev] PEP 485 review (isclose())
> Message-ID:
>         <
> CAP7+vJJMxAqG2nuXAXnsFn4+m7N9-0aC4XDe6M07F2-fTNFifw at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> No.
>
> On Wed, Mar 4, 2015 at 12:12 PM, Chris Barker <chris.barker at noaa.gov>
> wrote:
>
> > On Tue, Mar 3, 2015 at 8:43 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
> >
> >> On 03/03/2015 01:17 AM, Victor Stinner wrote:
> >>
> >
> >
> >> > Maybe it's time to rename the math module to _math and create a
> >> > math.py module, like _decimal/decimal? math.py should end with "from
> >> > _math import *".
> >>
> >> +1
> >>
> >
> > What do folks think? If we're going to do this, I'll write isclose() in
> > python. And I could do the work to split it out, too, I suppose.
> >
> > -Chris
> >
> >
> >
> >
> >
> > --
> >
> > Christopher Barker, Ph.D.
> > Oceanographer
> >
> > Emergency Response Division
> > NOAA/NOS/OR&R            (206) 526-6959   voice
> > 7600 Sand Point Way NE   (206) 526-6329   fax
> > Seattle, WA  98115       (206) 526-6317   main reception
> >
> > Chris.Barker at noaa.gov
> >
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > https://mail.python.org/mailman/options/python-dev/guido%40python.org
> >
> >
>
>
> --
> --Guido van Rossum (python.org/~guido)
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://mail.python.org/pipermail/python-dev/attachments/20150304/e108818c/attachment-0001.html
> >
>
> ------------------------------
>
> Message: 3
> Date: Wed, 04 Mar 2015 20:23:24 +0000
> From: Brett Cannon <brett at python.org>
> To: Chris Barker <chris.barker at noaa.gov>, Ethan Furman
>         <ethan at stoneleaf.us>
> Cc: Python Dev <python-dev at python.org>
> Subject: Re: [Python-Dev] PEP 485 review (isclose())
> Message-ID:
>         <CAP1=
> 2W4PZo25prx+U7m4CZX83QD6Qw_K2QEZTkn-WJyaQHevFA at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> On Wed, Mar 4, 2015 at 3:14 PM Chris Barker <chris.barker at noaa.gov> wrote:
>
> > On Tue, Mar 3, 2015 at 8:43 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
> >
> >> On 03/03/2015 01:17 AM, Victor Stinner wrote:
> >>
> >
> >
> >> > Maybe it's time to rename the math module to _math and create a
> >> > math.py module, like _decimal/decimal? math.py should end with "from
> >> > _math import *".
> >>
> >> +1
> >>
> >
> > What do folks think? If we're going to do this, I'll write isclose() in
> > python. And I could do the work to split it out, too, I suppose.
> >
>
> My vote -- as always -- is to do it in Python. If someone is sufficiently
> motivated to re-implement in C then that's great, but I don't think it
> should be required to be in C.
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://mail.python.org/pipermail/python-dev/attachments/20150304/d2643082/attachment-0001.html
> >
>
> ------------------------------
>
> Message: 4
> Date: Wed, 4 Mar 2015 12:33:48 -0800
> From: Guido van Rossum <guido at python.org>
> To: Brett Cannon <brett at python.org>
> Cc: Chris Barker <chris.barker at noaa.gov>, Ethan Furman
>         <ethan at stoneleaf.us>,  Python Dev <python-dev at python.org>
> Subject: Re: [Python-Dev] PEP 485 review (isclose())
> Message-ID:
>         <CAP7+vJK0h0f=J-DA7hCgom41V+fC=
> jFc6h0E3-1WoUBeQH-3Qg at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> In this case I disagree. The math module mostly wraps the C math library
> and the bar should remain high for things to be added to it. Let this be
> someone's opportunity to learn C (I guess not Chris's :-).
>
> On Wed, Mar 4, 2015 at 12:23 PM, Brett Cannon <brett at python.org> wrote:
>
> >
> >
> > On Wed, Mar 4, 2015 at 3:14 PM Chris Barker <chris.barker at noaa.gov>
> wrote:
> >
> >> On Tue, Mar 3, 2015 at 8:43 AM, Ethan Furman <ethan at stoneleaf.us>
> wrote:
> >>
> >>> On 03/03/2015 01:17 AM, Victor Stinner wrote:
> >>>
> >>
> >>
> >>> > Maybe it's time to rename the math module to _math and create a
> >>> > math.py module, like _decimal/decimal? math.py should end with "from
> >>> > _math import *".
> >>>
> >>> +1
> >>>
> >>
> >> What do folks think? If we're going to do this, I'll write isclose() in
> >> python. And I could do the work to split it out, too, I suppose.
> >>
> >
> > My vote -- as always -- is to do it in Python. If someone is sufficiently
> > motivated to re-implement in C then that's great, but I don't think it
> > should be required to be in C.
> >
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > https://mail.python.org/mailman/options/python-dev/guido%40python.org
> >
> >
>
>
> --
> --Guido van Rossum (python.org/~guido)
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://mail.python.org/pipermail/python-dev/attachments/20150304/8a086610/attachment-0001.html
> >
>
> ------------------------------
>
> Message: 5
> Date: Wed, 4 Mar 2015 15:26:40 -0800
> From: Chris Barker <chris.barker at noaa.gov>
> To: Guido van Rossum <guido at python.org>
> Cc: Brett Cannon <brett at python.org>, Ethan Furman
>         <ethan at stoneleaf.us>,  Python Dev <python-dev at python.org>
> Subject: Re: [Python-Dev] PEP 485 review (isclose())
> Message-ID:
>         <
> CALGmxE+So8SnybKaEBRpZjos8szrBpsCa3N7HcRusweHsgE_2A at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> On Wed, Mar 4, 2015 at 12:33 PM, Guido van Rossum <guido at python.org>
> wrote:
>
> > In this case I disagree. The math module mostly wraps the C math library
> > and the bar should remain high for things to be added to it. Let this be
> > someone's opportunity to learn C (I guess not Chris's :-).
> >
>
> As it happens, the first C code I ever wrote, beyond toy examples, was
> python extensions. I dropped that for Cython a good while ago, but  I can
> remember enough for this simple function ;-)
>
> I agree that it's not worth going to python for the math module for just
> this -- but others were proposing other reasons to do it.
>
> Back to look at K&R ;-)
>
> -Chris
>
>
>
>
> > On Wed, Mar 4, 2015 at 12:23 PM, Brett Cannon <brett at python.org> wrote:
> >
> >>
> >>
> >> On Wed, Mar 4, 2015 at 3:14 PM Chris Barker <chris.barker at noaa.gov>
> >> wrote:
> >>
> >>> On Tue, Mar 3, 2015 at 8:43 AM, Ethan Furman <ethan at stoneleaf.us>
> wrote:
> >>>
> >>>> On 03/03/2015 01:17 AM, Victor Stinner wrote:
> >>>>
> >>>
> >>>
> >>>> > Maybe it's time to rename the math module to _math and create a
> >>>> > math.py module, like _decimal/decimal? math.py should end with "from
> >>>> > _math import *".
> >>>>
> >>>> +1
> >>>>
> >>>
> >>> What do folks think? If we're going to do this, I'll write isclose() in
> >>> python. And I could do the work to split it out, too, I suppose.
> >>>
> >>
> >> My vote -- as always -- is to do it in Python. If someone is
> sufficiently
> >> motivated to re-implement in C then that's great, but I don't think it
> >> should be required to be in C.
> >>
> >> _______________________________________________
> >> Python-Dev mailing list
> >> Python-Dev at python.org
> >> https://mail.python.org/mailman/listinfo/python-dev
> >> Unsubscribe:
> >> https://mail.python.org/mailman/options/python-dev/guido%40python.org
> >>
> >>
> >
> >
> > --
> > --Guido van Rossum (python.org/~guido)
> >
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://mail.python.org/pipermail/python-dev/attachments/20150304/10af4471/attachment-0001.html
> >
>
> ------------------------------
>
> Message: 6
> Date: Wed, 4 Mar 2015 16:24:39 -0800
> From: Guido van Rossum <guido at python.org>
> To: Chris Angelico <rosuav at gmail.com>
> Cc: Python-Dev <python-dev at python.org>
> Subject: Re: [Python-Dev] Any volunteers to implement PEP 479?
> Message-ID:
>         <CAP7+vJJxwJYwCLk4soVbHEi4EC3v1+TfCwP6_b1nWLx==
> ncUXA at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Ping? Anyone? http://bugs.python.org/issue22906
>
> On Fri, Feb 20, 2015 at 9:56 PM, Chris Angelico <rosuav at gmail.com> wrote:
>
> > On Wed, Jan 7, 2015 at 2:48 PM, Guido van Rossum <guido at python.org>
> wrote:
> > > There's a proof of concept patch in http://bugs.python.org/issue22906,
> > but
> > > it doesn't have the __future__ import and probably gets other details
> > wrong.
> > >
> > > Reference:
> > > PEP 479 -- Change StopIteration handling inside generators
> > >
> > > --
> > > --Guido van Rossum (python.org/~guido)
> >
> > For the benefit of people who haven't been following the tracker
> > issue: There is now a patch which *does* create the __future__
> > directive and so on. It applies cleanly to current tip (I just tested
> > it again today), and the test suite passes on Debian AMD64. Can other
> > people please try this, on other platforms, and see how it goes?
> >
> > ChrisA
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > https://mail.python.org/mailman/options/python-dev/guido%40python.org
> >
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://mail.python.org/pipermail/python-dev/attachments/20150304/57d420cc/attachment-0001.html
> >
>
> ------------------------------
>
> Message: 7
> Date: Wed, 04 Mar 2015 16:42:15 -0800
> From: Ethan Furman <ethan at stoneleaf.us>
> To: python-dev at python.org
> Subject: Re: [Python-Dev] Any volunteers to implement PEP 479?
> Message-ID: <54F7A667.4050205 at stoneleaf.us>
> Content-Type: text/plain; charset="iso-8859-1"
>
> On 03/04/2015 04:24 PM, Guido van Rossum wrote:
> >
> > Ping? Anyone? http://bugs.python.org/issue22906
>
> Running tests on an ubuntu system...
>
> --
> ~Ethan~
>
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: signature.asc
> Type: application/pgp-signature
> Size: 836 bytes
> Desc: OpenPGP digital signature
> URL: <
> http://mail.python.org/pipermail/python-dev/attachments/20150304/96c32fd5/attachment-0001.sig
> >
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
>
>
> ------------------------------
>
> End of Python-Dev Digest, Vol 140, Issue 6
> ******************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150305/bb6afa7c/attachment.html>

From storchaka at gmail.com  Thu Mar  5 23:36:08 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Fri, 06 Mar 2015 00:36:08 +0200
Subject: [Python-Dev] Make review
Message-ID: <mdalop$rck$1@ger.gmane.org>

If you have ready patches that wait for review and committing, tell me. 
Send me no more than 5 links to issues per person (for first time) in 
private and I'll try to make reviews if I'm acquainted with affected 
modules or areas.


From victor.stinner at gmail.com  Fri Mar  6 00:03:48 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Fri, 6 Mar 2015 00:03:48 +0100
Subject: [Python-Dev] PEP 485 review (isclose())
In-Reply-To: <CAMpsgwbabEpkaPSykHr0TFHvWKycCAR2RGdj9eSm3sYMH5nTOg@mail.gmail.com>
References: <CAP7+vJJZY+8LNJ+-130ot8TSiC25p3gkwFGxgF6vuF2GEKo_3A@mail.gmail.com>
 <CALGmxEJQEGSqYKQp4dzJBLM2xB7J8dz_7hFx4gBvOLL+Ra9Bhw@mail.gmail.com>
 <CALGmxELG2iBbD4q41QSW+f6oGL6zM1YAeQwrmDOi-TMJ2E8q6A@mail.gmail.com>
 <CAMpsgwbabEpkaPSykHr0TFHvWKycCAR2RGdj9eSm3sYMH5nTOg@mail.gmail.com>
Message-ID: <CAMpsgwa9xiuo+KL8m4OW7x-FoNeBfNT6sCgnPLhotJC5dk6fbQ@mail.gmail.com>

2015-03-03 10:17 GMT+01:00 Victor Stinner <victor.stinner at gmail.com>:
> Maybe it's time to rename the math module to _math and create a
> math.py module, like _decimal/decimal? math.py should end with "from
> _math import *".

Since the idea looks to be well accepted, I proposed a patch to implement it:
http://bugs.python.org/issue23595?

math.py contains 4 constants (e, pi, inf, nan; nan is optionnal) and 4
functions: degrees(), radians(), factorial(), fsum().

See the issue for details.

Victor

From victor.stinner at gmail.com  Fri Mar  6 13:53:07 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Fri, 6 Mar 2015 13:53:07 +0100
Subject: [Python-Dev] subprocess, buffered files,
	pipes and broken pipe errors
Message-ID: <CAMpsgwb45=w7ig2OY0=VmHgkGTMi9MB-5tTnM7HkVOF6-q4cNw@mail.gmail.com>

Hi,

=> I propose to ignore BrokenPipeError on Popen.__exit__(), what do you think?

A recent issue fixed subprocess.Popen.__exit__() to read the exit
status of the child process, even if stdin.close() raised a
BrokenPipeError:
http://bugs.python.org/issue21619

When I saw the issue, I was surprised that Popen.__exit__() doesn't
ignore BrokenPipeError.

Popen.communicate() is designed to handle corner cases and it ignores
BrokenPipeError.

Popen.__exit__() only returns when the process exited. The method
starts by closing all pipes. Usually the child process notices that
stdin was closed and exit (programs used as "pipes" like grep, cat,
wc, etc.).

The problem is that Popen uses a buffered writer for Popen.stdin with
a default buffer size of 8 KB. We may buffer some data and so
stdin.close() will call stdin.write(). If the child process exited or
closed stdin, the pipe is broken, and the write in the parent process
(stdin.close() in Popen.__exit__) will raise BrokenPipeError.

I propose to ignore BrokenPipeError in Popen.__exit__, as done in
communicate(), for convinience:
http://bugs.python.org/issue23570

Serhiy wants to keep BrokenPipeError, he wrote that file.close()
should not ignore write errors (read the issue for details).

I consider that BrokenPipeError on a pipe is different than a write
error on a regular file.

EPIPE and SIGPIPE are designed to notify that the pipe is closed and
that it's now inefficient to continue to write into this pipe.

Ignoring BrokenPipeError in Popen.__exit__() respects this constrain
because the method closes stdin and only returns when the process
exited. So the caller will not write anything into stdin anymore.

What do you think?

Victor

From storchaka at gmail.com  Fri Mar  6 16:10:44 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Fri, 06 Mar 2015 17:10:44 +0200
Subject: [Python-Dev] subprocess, buffered files,
	pipes and broken pipe errors
In-Reply-To: <CAMpsgwb45=w7ig2OY0=VmHgkGTMi9MB-5tTnM7HkVOF6-q4cNw@mail.gmail.com>
References: <CAMpsgwb45=w7ig2OY0=VmHgkGTMi9MB-5tTnM7HkVOF6-q4cNw@mail.gmail.com>
Message-ID: <mdcg1k$nqk$1@ger.gmane.org>

On 06.03.15 14:53, Victor Stinner wrote:
> I propose to ignore BrokenPipeError in Popen.__exit__, as done in
> communicate(), for convinience:
> http://bugs.python.org/issue23570
>
> Serhiy wants to keep BrokenPipeError, he wrote that file.close()
> should not ignore write errors (read the issue for details).

I rather said about file.__exit__.

> I consider that BrokenPipeError on a pipe is different than a write
> error on a regular file.
>
> EPIPE and SIGPIPE are designed to notify that the pipe is closed and
> that it's now inefficient to continue to write into this pipe.

And into the file like open('/dev/stdout', 'wb').

> Ignoring BrokenPipeError in Popen.__exit__() respects this constrain
> because the method closes stdin and only returns when the process
> exited. So the caller will not write anything into stdin anymore.

And the caller will not write anything into the file after calling 
file.__exit__.

I don't see large difference between open('file', 'wb') and Popen('cat 
 >file', stdin=PIPE), between sys.stdout with redirecting stdout and 
running external program with Pipe().


From bcannon at gmail.com  Fri Mar  6 17:34:45 2015
From: bcannon at gmail.com (Brett Cannon)
Date: Fri, 06 Mar 2015 16:34:45 +0000
Subject: [Python-Dev] PEP 488: elimination of PYO files
Message-ID: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>

Over on the import-sig I proposed eliminating the concept of .pyo files
since they only signify that *some* optimization took place, not
*what* optimizations
took place. Everyone on the SIG was positive with the idea so I wrote a
PEP, got positive feedback from the SIG again, and so now I present to you
PEP 488 for discussion.

There is no patch yet, but this is not a complicated change and I could get
it done at the sprints at PyCon if necessary (I suspect updating the test
suite will take the most work).

There are currently two open issues, although one is purely a bikeshed
topic on formatting of file names so I don't really consider it open for
change from what is proposed in the PEP without Guido saying he hates my
preference or someone having a really good argument for some alternative.
The second open issue on the common case file name is something to
reasonably debate and come to consensus on.

=======================

PEP: 488
Title: Elimination of PYO files
Version: $Revision$
Last-Modified: $Date$
Author: Brett Cannon <brett at python.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 20-Feb-2015
Post-History:
    2015-03-06

Abstract
========

This PEP proposes eliminating the concept of PYO files from Python.
To continue the support of the separation of bytecode files based on
their optimization level, this PEP proposes extending the PYC file
name to include the optimization level in bytecode repository
directory (i.e., the ``__pycache__`` directory).


Rationale
=========

As of today, bytecode files come in two flavours: PYC and PYO. A PYC
file is the bytecode file generated and read from when no
optimization level is specified at interpreter startup (i.e., ``-O``
is not specified). A PYO file represents the bytecode file that is
read/written when **any** optimization level is specified (i.e., when
``-O`` is specified, including ``-OO``). This means that while PYC
files clearly delineate the optimization level used when they were
generated -- namely no optimizations beyond the peepholer -- the same
is not true for PYO files. Put in terms of optimization levels and
the file extension:

  - 0: ``.pyc``
  - 1 (``-O``): ``.pyo``
  - 2 (``-OO``): ``.pyo``

The reuse of the ``.pyo`` file extension for both level 1 and 2
optimizations means that there is no clear way to tell what
optimization level was used to generate the bytecode file. In terms
of reading PYO files, this can lead to an interpreter using a mixture
of optimization levels with its code if the user was not careful to
make sure all PYO files were generated using the same optimization
level (typically done by blindly deleting all PYO files and then
using the `compileall` module to compile all-new PYO files [1]_).
This issue is only compounded when people optimize Python code beyond
what the interpreter natively supports, e.g., using the astoptimizer
project [2]_.

In terms of writing PYO files, the need to delete all PYO files
every time one either changes the optimization level they want to use
or are unsure of what optimization was used the last time PYO files
were generated leads to unnecessary file churn. The change proposed
by this PEP also allows for **all** optimization levels to be
pre-compiled for bytecode files ahead of time, something that is
currently impossible thanks to the reuse of the ``.pyo`` file
extension for multiple optimization levels.

As for distributing bytecode-only modules, having to distribute both
``.pyc`` and ``.pyo`` files is unnecessary for the common use-case
of code obfuscation and smaller file deployments.


Proposal
========

To eliminate the ambiguity that PYO files present, this PEP proposes
eliminating the concept of PYO files and their accompanying ``.pyo``
file extension. To allow for the optimization level to be unambiguous
as well as to avoid having to regenerate optimized bytecode files
needlessly in the `__pycache__` directory, the optimization level
used to generate a PYC file will be incorporated into the bytecode
file name. Currently bytecode file names are created by
``importlib.util.cache_from_source()``, approximately using the
following expression defined by PEP 3147 [3]_, [4]_, [5]_::

    '{name}.{cache_tag}.pyc'.format(name=module_name,
                                    cache_tag=sys.implementation.cache_tag)

This PEP proposes to change the expression to::

    '{name}.{cache_tag}.opt-{optimization}.pyc'.format(
            name=module_name,
            cache_tag=sys.implementation.cache_tag,
            optimization=str(sys.flags.optimize))

The "opt-" prefix was chosen so as to provide a visual separator
from the cache tag. The placement of the optimization level after
the cache tag was chosen to preserve lexicographic sort order of
bytecode file names based on module name and cache tag which will
not vary for a single interpreter. The "opt-" prefix was chosen over
"o" so as to be somewhat self-documenting. The "opt-" prefix was
chosen over "O" so as to not have any confusion with "0" while being
so close to the interpreter version number.

A period was chosen over a hyphen as a separator so as to distinguish
clearly that the optimization level is not part of the interpreter
version as specified by the cache tag. It also lends to the use of
the period in the file name to delineate semantically different
concepts.

For example, the bytecode file name of ``importlib.cpython-35.pyc``
would become ``importlib.cpython-35.opt-0.pyc``. If ``-OO`` had been
passed to the interpreter then instead of
``importlib.cpython-35.pyo`` the file name would be
``importlib.cpython-35.opt-2.pyc``.


Implementation
==============

importlib
---------

As ``importlib.util.cache_from_source()`` is the API that exposes
bytecode file paths as while as being directly used by importlib, it
requires the most critical change. As of Python 3.4, the function's
signature is::

  importlib.util.cache_from_source(path, debug_override=None)

This PEP proposes changing the signature in Python 3.5 to::

  importlib.util.cache_from_source(path, debug_override=None, *,
optimization=None)

The introduced ``optimization`` keyword-only parameter will control
what optimization level is specified in the file name. If the
argument is ``None`` then the current optimization level of the
interpreter will be assumed. Any argument given for ``optimization``
will be passed to ``str()`` and must have ``str.isalnum()`` be true,
else ``ValueError`` will be raised (this prevents invalid characters
being used in the file name). If the empty string is passed in for
``optimization`` then the addition of the optimization will be
suppressed, reverting to the file name format which predates this
PEP.

It is expected that beyond Python's own
0-2 optimization levels, third-party code will use a hash of
optimization names to specify the optimization level, e.g.
``hashlib.sha256(','.join(['dead code elimination', 'constant
folding'])).hexdigest()``.
While this might lead to long file names, it is assumed that most
users never look at the contents of the __pycache__ directory and so
this won't be an issue.

The ``debug_override`` parameter will be deprecated. As the parameter
expects a boolean, the integer value of the boolean will be used as
if it had been provided as the argument to ``optimization`` (a
``None`` argument will mean the same as for ``optimization``). A
deprecation warning will be raised when ``debug_override`` is given a
value other than ``None``, but there are no plans for the complete
removal of the parameter as this time (but removal will be no later
than Python 4).

The various module attributes for importlib.machinery which relate to
bytecode file suffixes will be updated [7]_. The
``DEBUG_BYTECODE_SUFFIXES`` and ``OPTIMIZED_BYTECODE_SUFFIXES`` will
both be documented as deprecated and set to the same value as
``BYTECODE_SUFFIXES`` (removal of ``DEBUG_BYTECODE_SUFFIXES`` and
``OPTIMIZED_BYTECODE_SUFFIXES`` is not currently planned, but will be
not later than Python 4).

All various finders and loaders will also be updated as necessary,
but updating the previous mentioned parts of importlib should be all
that is required.


Rest of the standard library
----------------------------

The various functions exposed by the ``py_compile`` and
``compileall`` functions will be updated as necessary to make sure
they follow the new bytecode file name semantics [6]_, [1]_. The CLI
for the ``compileall`` module will not be directly affected (the
``-b`` flag will be implicitly as it will no longer generate ``.pyo``
files when ``-O`` is specified).


Compatibility Considerations
============================

Any code directly manipulating bytecode files from Python 3.2 on
will need to consider the impact of this change on their code (prior
to Python 3.2 -- including all of Python 2 -- there was no
__pycache__ which already necessitates bifurcating bytecode file
handling support). If code was setting the ``debug_override``
argument to ``importlib.util.cache_from_source()`` then care will be
needed if they want the path to a bytecode file with an optimization
level of 2. Otherwise only code **not** using
``importlib.util.cache_from_source()`` will need updating.

As for people who distribute bytecode-only modules (i.e., use a
bytecode file instead of a source file), they will have to choose
which optimization level they want their bytecode files to be since
distributing a ``.pyo`` file with a ``.pyc`` file will no longer be
of any use. Since people typically only distribute bytecode files for
code obfuscation purposes or smaller distribution size then only
having to distribute a single ``.pyc`` should actually be beneficial
to these use-cases. And since the magic number for bytecode files
changed in Python 3.5 to support PEP 465 there is no need to support
pre-existing ``.pyo`` files [8]_.


Rejected Ideas
==============

N/A


Open Issues
===========

Formatting of the optimization level in the file name
-----------------------------------------------------

Using the "opt-" prefix and placing the optimization level between
the cache tag and file extension is not critical. All options which
have been considered are:

* ``importlib.cpython-35.opt-0.pyc``
* ``importlib.cpython-35.opt0.pyc``
* ``importlib.cpython-35.o0.pyc``
* ``importlib.cpython-35.O0.pyc``
* ``importlib.cpython-35.0.pyc``
* ``importlib.cpython-35-O0.pyc``
* ``importlib.O0.cpython-35.pyc``
* ``importlib.o0.cpython-35.pyc``
* ``importlib.0.cpython-35.pyc``

These were initially rejected either because they would change the
sort order of bytecode files, possible ambiguity with the cache tag,
or were not self-documenting enough.


Not specifying the optimization level when it is at 0
-----------------------------------------------------

It has been suggested that for the common case of when the
optimizations are at level 0 that the entire part of the file name
relating to the optimization level be left out. This would allow for
file names of ``.pyc`` files to go unchanged, potentially leading to
less backwards-compatibility issues (although Python 3.5 introduces a
new magic number for bytecode so all bytecode files will have to be
regenerated regardless of the outcome of this PEP).

It would also allow a potentially redundant bit of information to be
left out of the file name if an implementation of Python did not
allow for optimizing bytecode. This would only occur, though, if the
interpreter didn't support ``-O`` **and** didn't implement the ast
module, else users could implement their own optimizations.

Arguments against allow this special case is "explicit is better than
implicit" and "special cases aren't special enough to break the
rules". There are also currently no Python 3 interpreters that don't
support ``-O``, so a potential Python 3 implementation which doesn't
allow bytecode optimization is entirely theoretical at the moment.


References
==========

.. [1] The compileall module
   (https://docs.python.org/3/library/compileall.html#module-compileall)

.. [2] The astoptimizer project
   (https://pypi.python.org/pypi/astoptimizer)

.. [3] ``importlib.util.cache_from_source()``
   (
https://docs.python.org/3.5/library/importlib.html#importlib.util.cache_from_source
)

.. [4] Implementation of ``importlib.util.cache_from_source()`` from
CPython 3.4.3rc1
   (
https://hg.python.org/cpython/file/038297948389/Lib/importlib/_bootstrap.py#l437
)

.. [5] PEP 3147, PYC Repository Directories, Warsaw
   (http://www.python.org/dev/peps/pep-3147)

.. [6] The py_compile module
   (https://docs.python.org/3/library/compileall.html#module-compileall)

.. [7] The importlib.machinery module
   (
https://docs.python.org/3/library/importlib.html#module-importlib.machinery)

.. [8] ``importlib.util.MAGIC_NUMBER``
   (
https://docs.python.org/3/library/importlib.html#importlib.util.MAGIC_NUMBER
)


Copyright
=========

This document has been placed in the public domain.


..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End:
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150306/b6318dcd/attachment.html>

From status at bugs.python.org  Fri Mar  6 18:08:19 2015
From: status at bugs.python.org (Python tracker)
Date: Fri,  6 Mar 2015 18:08:19 +0100 (CET)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20150306170819.68DE05663B@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2015-02-27 - 2015-03-06)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    4804 (+16)
  closed 30555 (+45)
  total  35359 (+61)

Open issues with patches: 2260 


Issues opened (47)
==================

#22079: Ensure in PyType_Ready() that base class of static type is sta
http://bugs.python.org/issue22079  reopened by doko

#23538: New Windows installer in 3.5.0a1 breaks compatibility with Win
http://bugs.python.org/issue23538  opened by Link Mauve

#23539: Content-length not set for HTTP methods expecting body when bo
http://bugs.python.org/issue23539  opened by demian.brecht

#23540: Proposal for asyncio: SubprocessTransport.detach() to detach a
http://bugs.python.org/issue23540  opened by martius

#23542: Update PEP 476 for using urllib2.build_opener()
http://bugs.python.org/issue23542  opened by Shakeel Mohamed

#23543: encoding error trying to save string to file
http://bugs.python.org/issue23543  opened by Gravitania

#23544: IDLE hangs when selecting Stack View with debug active
http://bugs.python.org/issue23544  opened by Andrew.Harrington

#23545: Turn on extra warnings on GCC
http://bugs.python.org/issue23545  opened by serhiy.storchaka

#23546: windows, IDLE and pep 397
http://bugs.python.org/issue23546  opened by Liam.Marsh

#23549: heapq docs should be more precise about how to access the smal
http://bugs.python.org/issue23549  opened by eli.bendersky

#23550: Add to unicodedata a function to query the "Quick_Check" prope
http://bugs.python.org/issue23550  opened by Hammerite

#23551: IDLE to provide menu options for using PIP
http://bugs.python.org/issue23551  opened by rhettinger

#23552: Have timeit warn about runs that are not independent of each o
http://bugs.python.org/issue23552  opened by rhettinger

#23553: Reduce the number of comparisons for range checking.
http://bugs.python.org/issue23553  opened by rhettinger

#23554: EchoServerClientProtocol class should be called EchoServerProt
http://bugs.python.org/issue23554  opened by gc

#23555: behavioural change in argparse set_defaults in python 2.7.9
http://bugs.python.org/issue23555  opened by doko

#23556: Scope for raise without argument is different in Python 2 and 
http://bugs.python.org/issue23556  opened by a3nm

#23557: Misc/SpecialBuilds.txt contains outdated information about PYM
http://bugs.python.org/issue23557  opened by Jakub Klama

#23560: Group the docs of similar methods in stdtypes.rst
http://bugs.python.org/issue23560  opened by ezio.melotti

#23564: Patch fixing sanity check for ordered fd sequence in _posixsub
http://bugs.python.org/issue23564  opened by Hisham Muhammad

#23565: local_clear walks the list of threads without holding head_loc
http://bugs.python.org/issue23565  opened by stutzbach

#23566: RFE: faulthandler.register() should support file descriptors
http://bugs.python.org/issue23566  opened by haypo

#23567: os.stat() tuple access vs named attribute access int vs float
http://bugs.python.org/issue23567  opened by gregory.p.smith

#23568: unittest.mock.MagicMock doesn't support __rdivmod__
http://bugs.python.org/issue23568  opened by zkrynicki

#23570: Change "with subprocess.Popen():" (context manager) to ignore 
http://bugs.python.org/issue23570  opened by haypo

#23571: Raise SystemError if a function returns a result with an excep
http://bugs.python.org/issue23571  opened by haypo

#23572: functools.singledispatch fails when "not BaseClass" is True
http://bugs.python.org/issue23572  opened by Sergio Pascual

#23573: Avoid redundant allocations in str.find and like
http://bugs.python.org/issue23573  opened by serhiy.storchaka

#23574: datetime: support leap seconds
http://bugs.python.org/issue23574  opened by haypo

#23575: MIPS64 needs ffi's n32.S
http://bugs.python.org/issue23575  opened by Simon Hoinkis

#23577: Add tests for wsgiref.validate
http://bugs.python.org/issue23577  opened by ashkop

#23578: struct.pack error messages do not indicate which argument was 
http://bugs.python.org/issue23578  opened by alynn

#23579: Amazon.com links
http://bugs.python.org/issue23579  opened by Edrie Ddrie

#23581: unittest.mock.MagicMock doesn't support matmul (@) operator
http://bugs.python.org/issue23581  opened by zkrynicki

#23583: IDLE: printing unicode subclasses broken (again)
http://bugs.python.org/issue23583  opened by mjpieters

#23584: test_doctest lineendings fails in verbose mode
http://bugs.python.org/issue23584  opened by jbeck

#23585: patchcheck doesn't depend on all
http://bugs.python.org/issue23585  opened by rbcollins

#23587: asyncio: use the new traceback.TracebackException class
http://bugs.python.org/issue23587  opened by haypo

#23588: Errno conflicts in ssl.SSLError
http://bugs.python.org/issue23588  opened by Ben.Darnell

#23589: Redundant sentence in FAQ
http://bugs.python.org/issue23589  opened by fossilet

#23591: Add IntFlags
http://bugs.python.org/issue23591  opened by serhiy.storchaka

#23592: SIGSEGV on interpreter shutdown, with daemon threads running w
http://bugs.python.org/issue23592  opened by A. Skrobov

#23593: Update Windows and OS X installer OpenSSL to 1.0.2
http://bugs.python.org/issue23593  opened by ned.deily

#23595: Split the math module into _math (C) + math (py)
http://bugs.python.org/issue23595  opened by haypo

#23596: gzip argparse interface
http://bugs.python.org/issue23596  opened by Antony.Lee

#23597: Allow easy display of local variables in log messages?
http://bugs.python.org/issue23597  opened by ncoghlan

#23598: No backspace on curses on iTerm (mac)
http://bugs.python.org/issue23598  opened by ragreener



Most recent 15 issues with no replies (15)
==========================================

#23598: No backspace on curses on iTerm (mac)
http://bugs.python.org/issue23598

#23591: Add IntFlags
http://bugs.python.org/issue23591

#23588: Errno conflicts in ssl.SSLError
http://bugs.python.org/issue23588

#23585: patchcheck doesn't depend on all
http://bugs.python.org/issue23585

#23584: test_doctest lineendings fails in verbose mode
http://bugs.python.org/issue23584

#23581: unittest.mock.MagicMock doesn't support matmul (@) operator
http://bugs.python.org/issue23581

#23578: struct.pack error messages do not indicate which argument was 
http://bugs.python.org/issue23578

#23577: Add tests for wsgiref.validate
http://bugs.python.org/issue23577

#23575: MIPS64 needs ffi's n32.S
http://bugs.python.org/issue23575

#23573: Avoid redundant allocations in str.find and like
http://bugs.python.org/issue23573

#23572: functools.singledispatch fails when "not BaseClass" is True
http://bugs.python.org/issue23572

#23568: unittest.mock.MagicMock doesn't support __rdivmod__
http://bugs.python.org/issue23568

#23565: local_clear walks the list of threads without holding head_loc
http://bugs.python.org/issue23565

#23560: Group the docs of similar methods in stdtypes.rst
http://bugs.python.org/issue23560

#23557: Misc/SpecialBuilds.txt contains outdated information about PYM
http://bugs.python.org/issue23557



Most recent 15 issues waiting for review (15)
=============================================

#23598: No backspace on curses on iTerm (mac)
http://bugs.python.org/issue23598

#23596: gzip argparse interface
http://bugs.python.org/issue23596

#23595: Split the math module into _math (C) + math (py)
http://bugs.python.org/issue23595

#23593: Update Windows and OS X installer OpenSSL to 1.0.2
http://bugs.python.org/issue23593

#23591: Add IntFlags
http://bugs.python.org/issue23591

#23589: Redundant sentence in FAQ
http://bugs.python.org/issue23589

#23585: patchcheck doesn't depend on all
http://bugs.python.org/issue23585

#23584: test_doctest lineendings fails in verbose mode
http://bugs.python.org/issue23584

#23583: IDLE: printing unicode subclasses broken (again)
http://bugs.python.org/issue23583

#23577: Add tests for wsgiref.validate
http://bugs.python.org/issue23577

#23575: MIPS64 needs ffi's n32.S
http://bugs.python.org/issue23575

#23574: datetime: support leap seconds
http://bugs.python.org/issue23574

#23573: Avoid redundant allocations in str.find and like
http://bugs.python.org/issue23573

#23571: Raise SystemError if a function returns a result with an excep
http://bugs.python.org/issue23571

#23570: Change "with subprocess.Popen():" (context manager) to ignore 
http://bugs.python.org/issue23570



Top 10 most discussed issues (10)
=================================

#23496: Steps for Android Native Build of Python 3.4.2
http://bugs.python.org/issue23496  26 msgs

#23539: Content-length not set for HTTP methods expecting body when bo
http://bugs.python.org/issue23539  26 msgs

#23595: Split the math module into _math (C) + math (py)
http://bugs.python.org/issue23595  14 msgs

#17911: traceback: add a new thin class storing a traceback without st
http://bugs.python.org/issue17911  13 msgs

#23553: Reduce the number of comparisons for range checking.
http://bugs.python.org/issue23553  12 msgs

#23571: Raise SystemError if a function returns a result with an excep
http://bugs.python.org/issue23571  12 msgs

#22936: traceback module has no way to show locals
http://bugs.python.org/issue22936  11 msgs

#23512: The list of built-in functions is not alphabetical on https://
http://bugs.python.org/issue23512  10 msgs

#23524: Use _set_thread_local_invalid_parameter_handler in posixmodule
http://bugs.python.org/issue23524  10 msgs

#23505: Urlparse insufficient validation leads to open redirect
http://bugs.python.org/issue23505   9 msgs



Issues closed (43)
==================

#3111: multiprocessing ppc Debian/ ia64 Ubuntu compilation error
http://bugs.python.org/issue3111  closed by davin

#6461: multiprocessing: freezing apps on Windows
http://bugs.python.org/issue6461  closed by davin

#11097: MSI: Remove win32com dependency from installer generator
http://bugs.python.org/issue11097  closed by steve.dower

#13697: python RLock implementation unsafe with signals
http://bugs.python.org/issue13697  closed by pitrou

#18382: multiprocessing's overlapped PipeConnection issues on Windows 
http://bugs.python.org/issue18382  closed by steve.dower

#19075: Add sorting algorithm visualization to turtledemo
http://bugs.python.org/issue19075  closed by ethan.furman

#19931: namedtuple docstrings are verbose for no added benefit
http://bugs.python.org/issue19931  closed by rhettinger

#19980: Improve help('non-topic') response
http://bugs.python.org/issue19980  closed by serhiy.storchaka

#20147: multiprocessing.Queue.get() raises queue.Empty exception if ev
http://bugs.python.org/issue20147  closed by davin

#20521: Cleanup for "dis" module documentation
http://bugs.python.org/issue20521  closed by berker.peksag

#21293: Remove "capsule hack" from object.c?
http://bugs.python.org/issue21293  closed by larry

#21619: Cleaning up a subprocess with a broken pipe
http://bugs.python.org/issue21619  closed by serhiy.storchaka

#22791: datetime.utcfromtimestamp() shoud have option for create tz aw
http://bugs.python.org/issue22791  closed by belopolsky

#22801: collections.Counter, when empty, doesn't raise an error with &
http://bugs.python.org/issue22801  closed by rhettinger

#22817: re.split fails with lookahead/behind
http://bugs.python.org/issue22817  closed by serhiy.storchaka

#22840: strpdate('20141110', '%Y%m%d%H%S') returns wrong date
http://bugs.python.org/issue22840  closed by belopolsky

#23263: Python 3 gives misleading errors when validating unicode ident
http://bugs.python.org/issue23263  closed by haypo

#23304: Unused Superclass in calendar.py
http://bugs.python.org/issue23304  closed by serhiy.storchaka

#23362: integer overflow in string translate
http://bugs.python.org/issue23362  closed by python-dev

#23367: integer overflow in unicodedata.normalize
http://bugs.python.org/issue23367  closed by python-dev

#23368: integer overflow in _PyUnicode_AsKind
http://bugs.python.org/issue23368  closed by serhiy.storchaka

#23387: test_urllib2 fails with HTTP Error 502: Bad Gateway
http://bugs.python.org/issue23387  closed by berker.peksag

#23471: 404 Not Found when downloading Python 3.4.3rc1 Documentation
http://bugs.python.org/issue23471  closed by larry

#23477: Increase coverage for wsgiref module
http://bugs.python.org/issue23477  closed by berker.peksag

#23494: adding timedelta to datetime object is not timezone aware
http://bugs.python.org/issue23494  closed by belopolsky

#23504: Add __all__ into types
http://bugs.python.org/issue23504  closed by serhiy.storchaka

#23521: OverflowError from timedelta * float in datetime.py
http://bugs.python.org/issue23521  closed by belopolsky

#23527: test_smtpnet uses incorrect port for STARTTLS
http://bugs.python.org/issue23527  closed by berker.peksag

#23541: Re module's match() fails to halt on a particular input
http://bugs.python.org/issue23541  closed by serhiy.storchaka

#23547: Misdirected
http://bugs.python.org/issue23547  closed by benjamin.peterson

#23548: TypeError in event loop finalizer, new in Python 3.4.3
http://bugs.python.org/issue23548  closed by haypo

#23558: Unable to install Python 3.4.3 amd64 on Windows 8.1
http://bugs.python.org/issue23558  closed by steve.dower

#23559: [doc] inconsistent range example output
http://bugs.python.org/issue23559  closed by georg.brandl

#23561: a little typo in dis.rst; need a non-trivial preceding dot
http://bugs.python.org/issue23561  closed by python-dev

#23562: file.read() followed by file.write() result in incorrect behav
http://bugs.python.org/issue23562  closed by r.david.murray

#23563: Faster urllib.urlparse utility functions
http://bugs.python.org/issue23563  closed by serhiy.storchaka

#23569: unittest.mock.MagicMock.__div__ works  but  __truediv__ doesn'
http://bugs.python.org/issue23569  closed by ned.deily

#23576: HTTPS reads can block when content length not available and ti
http://bugs.python.org/issue23576  closed by pitrou

#23580: Amazon.com links
http://bugs.python.org/issue23580  closed by ethan.furman

#23582: multiprocessing.Queue.get is not getting all the items in the 
http://bugs.python.org/issue23582  closed by r.david.murray

#23586: Add classproperty to the builtin functions
http://bugs.python.org/issue23586  closed by rhettinger

#23590: Potential leak in PyFloat_AsDouble.  Refcount error.
http://bugs.python.org/issue23590  closed by python-dev

#23594: Wrong variable name in traceback
http://bugs.python.org/issue23594  closed by ned.deily

From ericsnowcurrently at gmail.com  Fri Mar  6 18:10:56 2015
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Fri, 6 Mar 2015 10:10:56 -0700
Subject: [Python-Dev] PEP 488: elimination of PYO files
In-Reply-To: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
References: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
Message-ID: <CALFfu7CZhCsfbSFTeLS4yPE4T2QAD_f0010w2gvk+T3kLLniow@mail.gmail.com>

On Fri, Mar 6, 2015 at 9:34 AM, Brett Cannon <bcannon at gmail.com> wrote:
> Not specifying the optimization level when it is at 0
> -----------------------------------------------------
>
> It has been suggested that for the common case of when the
> optimizations are at level 0 that the entire part of the file name
> relating to the optimization level be left out. This would allow for
> file names of ``.pyc`` files to go unchanged, potentially leading to
> less backwards-compatibility issues (although Python 3.5 introduces a
> new magic number for bytecode so all bytecode files will have to be
> regenerated regardless of the outcome of this PEP).
>
> It would also allow a potentially redundant bit of information to be
> left out of the file name if an implementation of Python did not
> allow for optimizing bytecode. This would only occur, though, if the
> interpreter didn't support ``-O`` **and** didn't implement the ast
> module, else users could implement their own optimizations.

The presence of the "opt-0" part in the filename could imply that
there are other supported optimization levels, even when there aren't.
So leaving that out when optimizations aren't supported may be a good
idea.  Perhaps add sys.implementation.supports_optimization or
something like that?  Then only leave "opt-0" off if the
implementation does not support any optimization.

--eric

From mark at hotpy.org  Fri Mar  6 19:02:23 2015
From: mark at hotpy.org (Mark Shannon)
Date: Fri, 06 Mar 2015 18:02:23 +0000
Subject: [Python-Dev] PEP 488: elimination of PYO files
In-Reply-To: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
References: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
Message-ID: <54F9EBAF.1080400@hotpy.org>


On 06/03/15 16:34, Brett Cannon wrote:
> Over on the import-sig I proposed eliminating the concept of .pyo files
> since they only signify that /some/ optimization took place, not
> /what/ optimizations took place. Everyone on the SIG was positive with
> the idea so I wrote a PEP, got positive feedback from the SIG again, and
> so now I present to you PEP 488 for discussion.
>
[snip]

Historically -O and -OO have been the antithesis of optimisation, they 
change the behaviour of the program with no noticeable effect on 
performance.
If a change is to be made, why not just drop .pyo files and be done with it?

Any worthwhile optimisation needs to be done at runtime or involve much 
more than tweaking bytecode.

Cheers,
Mark.

From brett at python.org  Fri Mar  6 19:11:19 2015
From: brett at python.org (Brett Cannon)
Date: Fri, 06 Mar 2015 18:11:19 +0000
Subject: [Python-Dev] PEP 488: elimination of PYO files
References: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
 <54F9EBAF.1080400@hotpy.org>
Message-ID: <CAP1=2W6JUnDKpQFcr1Ud_b9rCBayk3RCp8XQQhd3zSQDYwmxtQ@mail.gmail.com>

On Fri, Mar 6, 2015 at 1:03 PM Mark Shannon <mark at hotpy.org> wrote:

>
> On 06/03/15 16:34, Brett Cannon wrote:
> > Over on the import-sig I proposed eliminating the concept of .pyo files
> > since they only signify that /some/ optimization took place, not
> > /what/ optimizations took place. Everyone on the SIG was positive with
> > the idea so I wrote a PEP, got positive feedback from the SIG again, and
> > so now I present to you PEP 488 for discussion.
> >
> [snip]
>
> Historically -O and -OO have been the antithesis of optimisation, they
> change the behaviour of the program with no noticeable effect on
> performance.
> If a change is to be made, why not just drop .pyo files and be done with
> it?
>

I disagree with your premise that .pyo files don't have a noticeable effect
on performance. If you don't use asserts a lot then there is no effect, but
if you use them heavily or have them perform expensive calculations then
there is an impact. And the dropping of docstrings does have an impact on
memory usage when you use Python at scale.

You're also assuming that we will never develop an AST optimizer that will
go beyond what the peepholer can do based on raw bytecode, or something
that involves a bit of calculation and thus something you wouldn't want to
do at startup.


>
> Any worthwhile optimisation needs to be done at runtime or involve much
> more than tweaking bytecode.
>

I disagree again. If you do something like whole program analysis and want
to use that to optimize something, you will surface that through bytecode
and not editing the source. So while you are doing "much more than tweaking
bytecode" externally to Python, you still have to surface to the
interpreter through bytecode.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150306/9d3d5562/attachment.html>

From mistersheik at gmail.com  Fri Mar  6 19:27:05 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Fri, 6 Mar 2015 13:27:05 -0500
Subject: [Python-Dev] PEP 488: elimination of PYO files
In-Reply-To: <CAP1=2W6JUnDKpQFcr1Ud_b9rCBayk3RCp8XQQhd3zSQDYwmxtQ@mail.gmail.com>
References: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
 <54F9EBAF.1080400@hotpy.org>
 <CAP1=2W6JUnDKpQFcr1Ud_b9rCBayk3RCp8XQQhd3zSQDYwmxtQ@mail.gmail.com>
Message-ID: <CAA68w_nJW9=_=U=fwfTWN+h5vdaWT-Si9YpqxddULJLduPA-ew@mail.gmail.com>

On Fri, Mar 6, 2015 at 1:11 PM, Brett Cannon <brett at python.org> wrote:

>
>
> On Fri, Mar 6, 2015 at 1:03 PM Mark Shannon <mark at hotpy.org> wrote:
>
>>
>> On 06/03/15 16:34, Brett Cannon wrote:
>> > Over on the import-sig I proposed eliminating the concept of .pyo files
>> > since they only signify that /some/ optimization took place, not
>> > /what/ optimizations took place. Everyone on the SIG was positive with
>> > the idea so I wrote a PEP, got positive feedback from the SIG again, and
>> > so now I present to you PEP 488 for discussion.
>> >
>> [snip]
>>
>> Historically -O and -OO have been the antithesis of optimisation, they
>> change the behaviour of the program with no noticeable effect on
>> performance.
>> If a change is to be made, why not just drop .pyo files and be done with
>> it?
>>
>
> I disagree with your premise that .pyo files don't have a noticeable
> effect on performance. If you don't use asserts a lot then there is no
> effect, but if you use them heavily or have them perform expensive
> calculations then there is an impact. And the dropping of docstrings does
> have an impact on memory usage when you use Python at scale.
>
> You're also assuming that we will never develop an AST optimizer that will
> go beyond what the peepholer can do based on raw bytecode, or something
> that involves a bit of calculation and thus something you wouldn't want to
> do at startup.
>

I don't want to speak for him, but you're going to get the best results
optimizing ASTs at runtime, which is what I thought he was suggesting.
Trying to optimize Python at compile time is setting your sights really
low.   You have so little information then.


>
>
>>
>> Any worthwhile optimisation needs to be done at runtime or involve much
>> more than tweaking bytecode.
>>
>
> I disagree again. If you do something like whole program analysis and want
> to use that to optimize something, you will surface that through bytecode
> and not editing the source. So while you are doing "much more than tweaking
> bytecode" externally to Python, you still have to surface to the
> interpreter through bytecode.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/mistersheik%40gmail.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150306/b2d07dc6/attachment.html>

From brett at python.org  Fri Mar  6 19:34:48 2015
From: brett at python.org (Brett Cannon)
Date: Fri, 06 Mar 2015 18:34:48 +0000
Subject: [Python-Dev] PEP 488: elimination of PYO files
References: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
 <54F9EBAF.1080400@hotpy.org>
 <CAP1=2W6JUnDKpQFcr1Ud_b9rCBayk3RCp8XQQhd3zSQDYwmxtQ@mail.gmail.com>
 <CAA68w_nJW9=_=U=fwfTWN+h5vdaWT-Si9YpqxddULJLduPA-ew@mail.gmail.com>
Message-ID: <CAP1=2W4g-sUwh74Zdu4rORqjOF8O24eD=HUatVinQ_HnVdKbVQ@mail.gmail.com>

On Fri, Mar 6, 2015 at 1:27 PM Neil Girdhar <mistersheik at gmail.com> wrote:

> On Fri, Mar 6, 2015 at 1:11 PM, Brett Cannon <brett at python.org> wrote:
>
>>
>>
>> On Fri, Mar 6, 2015 at 1:03 PM Mark Shannon <mark at hotpy.org> wrote:
>>
>>>
>>> On 06/03/15 16:34, Brett Cannon wrote:
>>> > Over on the import-sig I proposed eliminating the concept of .pyo files
>>> > since they only signify that /some/ optimization took place, not
>>> > /what/ optimizations took place. Everyone on the SIG was positive with
>>> > the idea so I wrote a PEP, got positive feedback from the SIG again,
>>> and
>>> > so now I present to you PEP 488 for discussion.
>>> >
>>> [snip]
>>>
>>> Historically -O and -OO have been the antithesis of optimisation, they
>>> change the behaviour of the program with no noticeable effect on
>>> performance.
>>> If a change is to be made, why not just drop .pyo files and be done with
>>> it?
>>>
>>
>> I disagree with your premise that .pyo files don't have a noticeable
>> effect on performance. If you don't use asserts a lot then there is no
>> effect, but if you use them heavily or have them perform expensive
>> calculations then there is an impact. And the dropping of docstrings does
>> have an impact on memory usage when you use Python at scale.
>>
>> You're also assuming that we will never develop an AST optimizer that
>> will go beyond what the peepholer can do based on raw bytecode, or
>> something that involves a bit of calculation and thus something you
>> wouldn't want to do at startup.
>>
>
> I don't want to speak for him, but you're going to get the best results
> optimizing ASTs at runtime, which is what I thought he was suggesting.
> Trying to optimize Python at compile time is setting your sights really
> low.   You have so little information then.
>

OK, I don't want to derail the discussion of the PEP into one over how best
to optimize CPython's performance relative to bytecode vs. runtime like
PyPy. The point is that we have -O and -OO and people do have uses for
those flags. People can also do custom optimizations thanks to the
flexibility of loaders.

All of this leads to wanting different bytecode files for different
optimization levels to make sure you're actually executing your code with
the optimizations you expect. If people think that optimizing code and
surfacing it in bytecode files is a waste and want to suggest either
dropping .pyo files entirely or dropping -O and only having -OO that's
fine, but that is not what this PEP is proposing nor a PEP I want to bother
writing.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150306/9b76c9e2/attachment.html>

From ethan at stoneleaf.us  Fri Mar  6 19:53:40 2015
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 06 Mar 2015 10:53:40 -0800
Subject: [Python-Dev] PEP 488: elimination of PYO files
In-Reply-To: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
References: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
Message-ID: <54F9F7B4.9050709@stoneleaf.us>

On 03/06/2015 08:34 AM, Brett Cannon wrote:
> Over on the import-sig I proposed eliminating the concept of .pyo files since they only signify that /some/ optimization
> took place, not /what/ optimizations took place. Everyone on the SIG was positive with the idea so I wrote a PEP, got
> positive feedback from the SIG again, and so now I present to you PEP 488 for discussion.

+1 overall, comments in-line.

> Implementation
> ==============
> 
> importlib
> ---------
> 
> As ``importlib.util.cache_from_source()`` is the API that exposes
> bytecode file paths as while as being directly used by importlib, it
> requires the most critical change.

Not sure what that sentence is supposed to say -- maybe "as well as" and not "as while as" ?


> The ``debug_override`` parameter will be deprecated. As the parameter
> expects a boolean, the integer value of the boolean will be used as
> if it had been provided as the argument to ``optimization`` (a
> ``None`` argument will mean the same as for ``optimization``). A
> deprecation warning will be raised when ``debug_override`` is given a
> value other than ``None``, but there are no plans for the complete
> removal of the parameter as this time (but removal will be no later
> than Python 4).

"at this time" not "as this time"


> Rest of the standard library
> ----------------------------
> 
> The various functions exposed by the ``py_compile`` and
> ``compileall`` functions will be updated as necessary to make sure
> they follow the new bytecode file name semantics [6]_, [1]_. The CLI
> for the ``compileall`` module will not be directly affected (the
> ``-b`` flag will be implicitly as it will no longer generate ``.pyo``
> files when ``-O`` is specified).

"will be implicit" not "will be implicitly"

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150306/eae47449/attachment.sig>

From brett at python.org  Fri Mar  6 19:59:02 2015
From: brett at python.org (Brett Cannon)
Date: Fri, 06 Mar 2015 18:59:02 +0000
Subject: [Python-Dev] PEP 488: elimination of PYO files
References: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
 <54F9F7B4.9050709@stoneleaf.us>
Message-ID: <CAP1=2W5RdXCK0Q7rA5j1rO+ZTjUjoUfr7Sj6xtyOgGg2HEfyKw@mail.gmail.com>

Thanks! All suggestions applied to my local copy.

On Fri, Mar 6, 2015 at 1:55 PM Ethan Furman <ethan at stoneleaf.us> wrote:

> On 03/06/2015 08:34 AM, Brett Cannon wrote:
> > Over on the import-sig I proposed eliminating the concept of .pyo files
> since they only signify that /some/ optimization
> > took place, not /what/ optimizations took place. Everyone on the SIG was
> positive with the idea so I wrote a PEP, got
> > positive feedback from the SIG again, and so now I present to you PEP
> 488 for discussion.
>
> +1 overall, comments in-line.
>
> > Implementation
> > ==============
> >
> > importlib
> > ---------
> >
> > As ``importlib.util.cache_from_source()`` is the API that exposes
> > bytecode file paths as while as being directly used by importlib, it
> > requires the most critical change.
>
> Not sure what that sentence is supposed to say -- maybe "as well as" and
> not "as while as" ?
>
>
> > The ``debug_override`` parameter will be deprecated. As the parameter
> > expects a boolean, the integer value of the boolean will be used as
> > if it had been provided as the argument to ``optimization`` (a
> > ``None`` argument will mean the same as for ``optimization``). A
> > deprecation warning will be raised when ``debug_override`` is given a
> > value other than ``None``, but there are no plans for the complete
> > removal of the parameter as this time (but removal will be no later
> > than Python 4).
>
> "at this time" not "as this time"
>
>
> > Rest of the standard library
> > ----------------------------
> >
> > The various functions exposed by the ``py_compile`` and
> > ``compileall`` functions will be updated as necessary to make sure
> > they follow the new bytecode file name semantics [6]_, [1]_. The CLI
> > for the ``compileall`` module will not be directly affected (the
> > ``-b`` flag will be implicitly as it will no longer generate ``.pyo``
> > files when ``-O`` is specified).
>
> "will be implicit" not "will be implicitly"
>
> --
> ~Ethan~
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150306/0060bc19/attachment.html>

From benjamin at python.org  Fri Mar  6 20:09:15 2015
From: benjamin at python.org (Benjamin Peterson)
Date: Fri, 06 Mar 2015 14:09:15 -0500
Subject: [Python-Dev] PEP 488: elimination of PYO files
In-Reply-To: <CAP1=2W4g-sUwh74Zdu4rORqjOF8O24eD=HUatVinQ_HnVdKbVQ@mail.gmail.com>
References: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
 <54F9EBAF.1080400@hotpy.org>
 <CAP1=2W6JUnDKpQFcr1Ud_b9rCBayk3RCp8XQQhd3zSQDYwmxtQ@mail.gmail.com>
 <CAA68w_nJW9=_=U=fwfTWN+h5vdaWT-Si9YpqxddULJLduPA-ew@mail.gmail.com>
 <CAP1=2W4g-sUwh74Zdu4rORqjOF8O24eD=HUatVinQ_HnVdKbVQ@mail.gmail.com>
Message-ID: <1425668955.3364036.236612829.494092A7@webmail.messagingengine.com>



On Fri, Mar 6, 2015, at 13:34, Brett Cannon wrote:
> On Fri, Mar 6, 2015 at 1:27 PM Neil Girdhar <mistersheik at gmail.com>
> wrote:
> 
> > On Fri, Mar 6, 2015 at 1:11 PM, Brett Cannon <brett at python.org> wrote:
> >
> >>
> >>
> >> On Fri, Mar 6, 2015 at 1:03 PM Mark Shannon <mark at hotpy.org> wrote:
> >>
> >>>
> >>> On 06/03/15 16:34, Brett Cannon wrote:
> >>> > Over on the import-sig I proposed eliminating the concept of .pyo files
> >>> > since they only signify that /some/ optimization took place, not
> >>> > /what/ optimizations took place. Everyone on the SIG was positive with
> >>> > the idea so I wrote a PEP, got positive feedback from the SIG again,
> >>> and
> >>> > so now I present to you PEP 488 for discussion.
> >>> >
> >>> [snip]
> >>>
> >>> Historically -O and -OO have been the antithesis of optimisation, they
> >>> change the behaviour of the program with no noticeable effect on
> >>> performance.
> >>> If a change is to be made, why not just drop .pyo files and be done with
> >>> it?
> >>>
> >>
> >> I disagree with your premise that .pyo files don't have a noticeable
> >> effect on performance. If you don't use asserts a lot then there is no
> >> effect, but if you use them heavily or have them perform expensive
> >> calculations then there is an impact. And the dropping of docstrings does
> >> have an impact on memory usage when you use Python at scale.
> >>
> >> You're also assuming that we will never develop an AST optimizer that
> >> will go beyond what the peepholer can do based on raw bytecode, or
> >> something that involves a bit of calculation and thus something you
> >> wouldn't want to do at startup.
> >>
> >
> > I don't want to speak for him, but you're going to get the best results
> > optimizing ASTs at runtime, which is what I thought he was suggesting.
> > Trying to optimize Python at compile time is setting your sights really
> > low.   You have so little information then.
> >
> 
> OK, I don't want to derail the discussion of the PEP into one over how
> best
> to optimize CPython's performance relative to bytecode vs. runtime like
> PyPy. The point is that we have -O and -OO and people do have uses for
> those flags. People can also do custom optimizations thanks to the
> flexibility of loaders.

I think it would be preferable deprecate -O and -OO and replace them
with flags like --no-docstrings or --no-asserts. Ideally, "optimization"
levels shouldn't change program semantics.

From rosuav at gmail.com  Fri Mar  6 21:13:05 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 7 Mar 2015 07:13:05 +1100
Subject: [Python-Dev] PEP 488: elimination of PYO files
In-Reply-To: <1425668955.3364036.236612829.494092A7@webmail.messagingengine.com>
References: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
 <54F9EBAF.1080400@hotpy.org>
 <CAP1=2W6JUnDKpQFcr1Ud_b9rCBayk3RCp8XQQhd3zSQDYwmxtQ@mail.gmail.com>
 <CAA68w_nJW9=_=U=fwfTWN+h5vdaWT-Si9YpqxddULJLduPA-ew@mail.gmail.com>
 <CAP1=2W4g-sUwh74Zdu4rORqjOF8O24eD=HUatVinQ_HnVdKbVQ@mail.gmail.com>
 <1425668955.3364036.236612829.494092A7@webmail.messagingengine.com>
Message-ID: <CAPTjJmr5Z+Fa2wz97fDcP-rOiBxP-KhjE3Ku25nPxVKr1X-9LA@mail.gmail.com>

On Sat, Mar 7, 2015 at 6:09 AM, Benjamin Peterson <benjamin at python.org> wrote:
> I think it would be preferable deprecate -O and -OO and replace them
> with flags like --no-docstrings or --no-asserts. Ideally, "optimization"
> levels shouldn't change program semantics.

Plenty of C compilers have optimization levels that can change
behaviour (replacing division with mult-by-reciprocal and such), so I
don't see any reason for Python to object. The removal of docstrings
will be a problem to only a handful of programs (eg [1]) which use
them for more than introspection. And the removal of assert shouldn't
be a semantic change to a well-written program.

[1] https://github.com/Rosuav/LetMeKnow/blob/master/letmeknow.py

ChrisA

From brett at python.org  Fri Mar  6 21:11:33 2015
From: brett at python.org (Brett Cannon)
Date: Fri, 06 Mar 2015 20:11:33 +0000
Subject: [Python-Dev] PEP 488: elimination of PYO files
In-Reply-To: <1425668955.3364036.236612829.494092A7@webmail.messagingengine.com>
References: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
 <54F9EBAF.1080400@hotpy.org>
 <CAP1=2W6JUnDKpQFcr1Ud_b9rCBayk3RCp8XQQhd3zSQDYwmxtQ@mail.gmail.com>
 <CAA68w_nJW9=_=U=fwfTWN+h5vdaWT-Si9YpqxddULJLduPA-ew@mail.gmail.com>
 <CAP1=2W4g-sUwh74Zdu4rORqjOF8O24eD=HUatVinQ_HnVdKbVQ@mail.gmail.com>
 <1425668955.3364036.236612829.494092A7@webmail.messagingengine.com>
Message-ID: <CAP1=2W4FbUQjaK5a-FgsfS1c3kP5qbqi=iZHE5h1+CxDYrJqtA@mail.gmail.com>

On Fri, Mar 6, 2015 at 2:09 PM Benjamin Peterson <benjamin at python.org>
wrote:

>
>
> On Fri, Mar 6, 2015, at 13:34, Brett Cannon wrote:
> > On Fri, Mar 6, 2015 at 1:27 PM Neil Girdhar <mistersheik at gmail.com>
> > wrote:
> >
> > > On Fri, Mar 6, 2015 at 1:11 PM, Brett Cannon <brett at python.org> wrote:
> > >
> > >>
> > >>
> > >> On Fri, Mar 6, 2015 at 1:03 PM Mark Shannon <mark at hotpy.org> wrote:
> > >>
> > >>>
> > >>> On 06/03/15 16:34, Brett Cannon wrote:
> > >>> > Over on the import-sig I proposed eliminating the concept of .pyo
> files
> > >>> > since they only signify that /some/ optimization took place, not
> > >>> > /what/ optimizations took place. Everyone on the SIG was positive
> with
> > >>> > the idea so I wrote a PEP, got positive feedback from the SIG
> again,
> > >>> and
> > >>> > so now I present to you PEP 488 for discussion.
> > >>> >
> > >>> [snip]
> > >>>
> > >>> Historically -O and -OO have been the antithesis of optimisation,
> they
> > >>> change the behaviour of the program with no noticeable effect on
> > >>> performance.
> > >>> If a change is to be made, why not just drop .pyo files and be done
> with
> > >>> it?
> > >>>
> > >>
> > >> I disagree with your premise that .pyo files don't have a noticeable
> > >> effect on performance. If you don't use asserts a lot then there is no
> > >> effect, but if you use them heavily or have them perform expensive
> > >> calculations then there is an impact. And the dropping of docstrings
> does
> > >> have an impact on memory usage when you use Python at scale.
> > >>
> > >> You're also assuming that we will never develop an AST optimizer that
> > >> will go beyond what the peepholer can do based on raw bytecode, or
> > >> something that involves a bit of calculation and thus something you
> > >> wouldn't want to do at startup.
> > >>
> > >
> > > I don't want to speak for him, but you're going to get the best results
> > > optimizing ASTs at runtime, which is what I thought he was suggesting.
> > > Trying to optimize Python at compile time is setting your sights really
> > > low.   You have so little information then.
> > >
> >
> > OK, I don't want to derail the discussion of the PEP into one over how
> > best
> > to optimize CPython's performance relative to bytecode vs. runtime like
> > PyPy. The point is that we have -O and -OO and people do have uses for
> > those flags. People can also do custom optimizations thanks to the
> > flexibility of loaders.
>
> I think it would be preferable deprecate -O and -OO and replace them
> with flags like --no-docstrings or --no-asserts. Ideally, "optimization"
> levels shouldn't change program semantics.
>

OK, but that doesn't influence the PEP's goal of dropping .pyo files.

Are you suggesting that the tag be changed to be less specific to
optimizations and more free-form? Like
`importlib.cpython-35.__no-asserts_no-docstrings__.pyc`? Otherwise stuff
like this gets baked into the .pyc file itself instead of the file name,
but I don't think we should just drop the ability to switch off asserts and
docstrings like Mark seemed to be suggesting.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150306/0ee93b9b/attachment.html>

From solipsis at pitrou.net  Fri Mar  6 21:37:05 2015
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 6 Mar 2015 21:37:05 +0100
Subject: [Python-Dev] PEP 488: elimination of PYO files
References: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
 <54F9EBAF.1080400@hotpy.org>
 <CAP1=2W6JUnDKpQFcr1Ud_b9rCBayk3RCp8XQQhd3zSQDYwmxtQ@mail.gmail.com>
Message-ID: <20150306213705.5a9f2c9c@fsol>

On Fri, 06 Mar 2015 18:11:19 +0000
Brett Cannon <brett at python.org> wrote:
> And the dropping of docstrings does have an impact on
> memory usage when you use Python at scale.

What kind of "scale" are you talking about? Do you have any numbers
about such impact?

> You're also assuming that we will never develop an AST optimizer

No, the assumption is that we don't have such an optimizer *right now*.
Having command-line options because they might be useful some day is
silly.

Regards

Antoine.



From steve at pearwood.info  Fri Mar  6 23:34:20 2015
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 7 Mar 2015 09:34:20 +1100
Subject: [Python-Dev] PEP 488: elimination of PYO files
In-Reply-To: <20150306213705.5a9f2c9c@fsol>
References: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
 <54F9EBAF.1080400@hotpy.org>
 <CAP1=2W6JUnDKpQFcr1Ud_b9rCBayk3RCp8XQQhd3zSQDYwmxtQ@mail.gmail.com>
 <20150306213705.5a9f2c9c@fsol>
Message-ID: <20150306223419.GN7655@ando.pearwood.info>

On Fri, Mar 06, 2015 at 09:37:05PM +0100, Antoine Pitrou wrote:
> On Fri, 06 Mar 2015 18:11:19 +0000
> Brett Cannon <brett at python.org> wrote:
> > And the dropping of docstrings does have an impact on
> > memory usage when you use Python at scale.
> 
> What kind of "scale" are you talking about? Do you have any numbers
> about such impact?
> 
> > You're also assuming that we will never develop an AST optimizer
> 
> No, the assumption is that we don't have such an optimizer *right now*.
> Having command-line options because they might be useful some day is
> silly.

Quoting the PEP:

    This issue is only compounded when people optimize Python 
    code beyond what the interpreter natively supports, e.g., 
    using the astoptimizer project [2]_.


Brett, I'm a very strong +1 on the PEP. It's well-written and gives a 
good explanation for why such a thing is needed. The current behaviour 
of re-using the same .pyo file for two distinct sets of bytecode is 
out-and-out buggy:

[steve at ando ~]$ python3.3 -O -c "import dis; print(dis.__doc__[:32])"
Disassembler of Python byte code
[steve at ando ~]$ python3.3 -OO -c "import dis; print(dis.__doc__[:32])"
Disassembler of Python byte code

The second should fail, since doc strings should be removed under -OO 
optimization, but because the .pyo file already exists it doesn't.

Even if CPython drops -O and -OO altogether, this PEP should still be 
accepted to allow third party optimizers like astoptimizer to interact 
without getting in each other's way.

(And for the record, I'm an equally strong -1 on dropping -O and -OO.)

Thank you.


-- 
Steve

From solipsis at pitrou.net  Fri Mar  6 23:47:08 2015
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 6 Mar 2015 23:47:08 +0100
Subject: [Python-Dev] PEP 488: elimination of PYO files
References: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
 <54F9EBAF.1080400@hotpy.org>
 <CAP1=2W6JUnDKpQFcr1Ud_b9rCBayk3RCp8XQQhd3zSQDYwmxtQ@mail.gmail.com>
 <20150306213705.5a9f2c9c@fsol>
 <20150306223419.GN7655@ando.pearwood.info>
Message-ID: <20150306234708.704a2fdf@fsol>

On Sat, 7 Mar 2015 09:34:20 +1100
Steven D'Aprano <steve at pearwood.info> wrote:

> On Fri, Mar 06, 2015 at 09:37:05PM +0100, Antoine Pitrou wrote:
> > On Fri, 06 Mar 2015 18:11:19 +0000
> > Brett Cannon <brett at python.org> wrote:
> > > And the dropping of docstrings does have an impact on
> > > memory usage when you use Python at scale.
> > 
> > What kind of "scale" are you talking about? Do you have any numbers
> > about such impact?
> > 
> > > You're also assuming that we will never develop an AST optimizer
> > 
> > No, the assumption is that we don't have such an optimizer *right now*.
> > Having command-line options because they might be useful some day is
> > silly.
> 
> Quoting the PEP:
> 
>     This issue is only compounded when people optimize Python 
>     code beyond what the interpreter natively supports, e.g., 
>     using the astoptimizer project [2]_.

The astoptimizer project is not part of Python. It's third-party
software that has no relationship to .pyo files.

Regards

Antoine.



From benjamin at python.org  Sat Mar  7 00:48:03 2015
From: benjamin at python.org (Benjamin Peterson)
Date: Fri, 06 Mar 2015 18:48:03 -0500
Subject: [Python-Dev] PEP 488: elimination of PYO files
In-Reply-To: <CAPTjJmr5Z+Fa2wz97fDcP-rOiBxP-KhjE3Ku25nPxVKr1X-9LA@mail.gmail.com>
References: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
 <54F9EBAF.1080400@hotpy.org>
 <CAP1=2W6JUnDKpQFcr1Ud_b9rCBayk3RCp8XQQhd3zSQDYwmxtQ@mail.gmail.com>
 <CAA68w_nJW9=_=U=fwfTWN+h5vdaWT-Si9YpqxddULJLduPA-ew@mail.gmail.com>
 <CAP1=2W4g-sUwh74Zdu4rORqjOF8O24eD=HUatVinQ_HnVdKbVQ@mail.gmail.com>
 <1425668955.3364036.236612829.494092A7@webmail.messagingengine.com>
 <CAPTjJmr5Z+Fa2wz97fDcP-rOiBxP-KhjE3Ku25nPxVKr1X-9LA@mail.gmail.com>
Message-ID: <1425685683.3874448.236704113.06D01977@webmail.messagingengine.com>



On Fri, Mar 6, 2015, at 15:13, Chris Angelico wrote:
> On Sat, Mar 7, 2015 at 6:09 AM, Benjamin Peterson <benjamin at python.org>
> wrote:
> > I think it would be preferable deprecate -O and -OO and replace them
> > with flags like --no-docstrings or --no-asserts. Ideally, "optimization"
> > levels shouldn't change program semantics.
> 
> Plenty of C compilers have optimization levels that can change
> behaviour (replacing division with mult-by-reciprocal and such), so I
> don't see any reason for Python to object.

Yes, but in the C world, you have pass scary flags like -ffast-math.
Just -O2 generally won't reach outside of standard semantics.

From benjamin at python.org  Sat Mar  7 00:49:11 2015
From: benjamin at python.org (Benjamin Peterson)
Date: Fri, 06 Mar 2015 18:49:11 -0500
Subject: [Python-Dev] PEP 488: elimination of PYO files
In-Reply-To: <CAP1=2W4FbUQjaK5a-FgsfS1c3kP5qbqi=iZHE5h1+CxDYrJqtA@mail.gmail.com>
References: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
 <54F9EBAF.1080400@hotpy.org>
 <CAP1=2W6JUnDKpQFcr1Ud_b9rCBayk3RCp8XQQhd3zSQDYwmxtQ@mail.gmail.com>
 <CAA68w_nJW9=_=U=fwfTWN+h5vdaWT-Si9YpqxddULJLduPA-ew@mail.gmail.com>
 <CAP1=2W4g-sUwh74Zdu4rORqjOF8O24eD=HUatVinQ_HnVdKbVQ@mail.gmail.com>
 <1425668955.3364036.236612829.494092A7@webmail.messagingengine.com>
 <CAP1=2W4FbUQjaK5a-FgsfS1c3kP5qbqi=iZHE5h1+CxDYrJqtA@mail.gmail.com>
Message-ID: <1425685751.3874542.236704353.11A94396@webmail.messagingengine.com>



On Fri, Mar 6, 2015, at 15:11, Brett Cannon wrote:
> 
> OK, but that doesn't influence the PEP's goal of dropping .pyo files.

Correct.

> 
> Are you suggesting that the tag be changed to be less specific to
> optimizations and more free-form? Like
> `importlib.cpython-35.__no-asserts_no-docstrings__.pyc`? Otherwise stuff
> like this gets baked into the .pyc file itself instead of the file name,
> but I don't think we should just drop the ability to switch off asserts
> and
> docstrings like Mark seemed to be suggesting.

Basically, though the filename strings could perhaps be more compact.

From brett at python.org  Sat Mar  7 01:01:39 2015
From: brett at python.org (Brett Cannon)
Date: Sat, 07 Mar 2015 00:01:39 +0000
Subject: [Python-Dev] PEP 488: elimination of PYO files
In-Reply-To: <20150306213705.5a9f2c9c@fsol>
References: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
 <54F9EBAF.1080400@hotpy.org>
 <CAP1=2W6JUnDKpQFcr1Ud_b9rCBayk3RCp8XQQhd3zSQDYwmxtQ@mail.gmail.com>
 <20150306213705.5a9f2c9c@fsol>
Message-ID: <CAP1=2W5S557NrU1uyc2N9tuXTcOvxHaW3Ei80fpXWroXSMEbLw@mail.gmail.com>

On Fri, Mar 6, 2015 at 3:37 PM Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Fri, 06 Mar 2015 18:11:19 +0000
> Brett Cannon <brett at python.org> wrote:
> > And the dropping of docstrings does have an impact on
> > memory usage when you use Python at scale.
>
> What kind of "scale" are you talking about? Do you have any numbers
> about such impact?
>

I know YouTube at least uses -OO and I don't have numbers to share (numbers
I were last shown were years ago and I wouldn't be authorized to share
anyway, but I do know they still use -OO).


>
> > You're also assuming that we will never develop an AST optimizer
>
> No, the assumption is that we don't have such an optimizer *right now*.
> Having command-line options because they might be useful some day is
> silly.
>

I'm not talking about changing any command-line option in the PEP so I
don't know what you're referring to.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150307/c28dfbfd/attachment.html>

From brett at python.org  Sat Mar  7 01:03:14 2015
From: brett at python.org (Brett Cannon)
Date: Sat, 07 Mar 2015 00:03:14 +0000
Subject: [Python-Dev] PEP 488: elimination of PYO files
In-Reply-To: <20150306234708.704a2fdf@fsol>
References: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
 <54F9EBAF.1080400@hotpy.org>
 <CAP1=2W6JUnDKpQFcr1Ud_b9rCBayk3RCp8XQQhd3zSQDYwmxtQ@mail.gmail.com>
 <20150306213705.5a9f2c9c@fsol> <20150306223419.GN7655@ando.pearwood.info>
 <20150306234708.704a2fdf@fsol>
Message-ID: <CAP1=2W4vr3NOZFHywxf4F6QHqEknfCA0PQePR84ugSKvYpWy8g@mail.gmail.com>

On Fri, Mar 6, 2015 at 5:47 PM Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Sat, 7 Mar 2015 09:34:20 +1100
> Steven D'Aprano <steve at pearwood.info> wrote:
>
> > On Fri, Mar 06, 2015 at 09:37:05PM +0100, Antoine Pitrou wrote:
> > > On Fri, 06 Mar 2015 18:11:19 +0000
> > > Brett Cannon <brett at python.org> wrote:
> > > > And the dropping of docstrings does have an impact on
> > > > memory usage when you use Python at scale.
> > >
> > > What kind of "scale" are you talking about? Do you have any numbers
> > > about such impact?
> > >
> > > > You're also assuming that we will never develop an AST optimizer
> > >
> > > No, the assumption is that we don't have such an optimizer *right now*.
> > > Having command-line options because they might be useful some day is
> > > silly.
> >
> > Quoting the PEP:
> >
> >     This issue is only compounded when people optimize Python
> >     code beyond what the interpreter natively supports, e.g.,
> >     using the astoptimizer project [2]_.
>
> The astoptimizer project is not part of Python. It's third-party
> software that has no relationship to .pyo files.
>

Directly, no. But the point is that the PEP enables the astoptimizer
project to write out .pyc files specifying different optimizations that
won't clash with -O or -OO .pyc files.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150307/ba0fe147/attachment.html>

From brett at python.org  Sat Mar  7 01:04:52 2015
From: brett at python.org (Brett Cannon)
Date: Sat, 07 Mar 2015 00:04:52 +0000
Subject: [Python-Dev] PEP 488: elimination of PYO files
In-Reply-To: <1425685751.3874542.236704353.11A94396@webmail.messagingengine.com>
References: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
 <54F9EBAF.1080400@hotpy.org>
 <CAP1=2W6JUnDKpQFcr1Ud_b9rCBayk3RCp8XQQhd3zSQDYwmxtQ@mail.gmail.com>
 <CAA68w_nJW9=_=U=fwfTWN+h5vdaWT-Si9YpqxddULJLduPA-ew@mail.gmail.com>
 <CAP1=2W4g-sUwh74Zdu4rORqjOF8O24eD=HUatVinQ_HnVdKbVQ@mail.gmail.com>
 <1425668955.3364036.236612829.494092A7@webmail.messagingengine.com>
 <CAP1=2W4FbUQjaK5a-FgsfS1c3kP5qbqi=iZHE5h1+CxDYrJqtA@mail.gmail.com>
 <1425685751.3874542.236704353.11A94396@webmail.messagingengine.com>
Message-ID: <CAP1=2W51WsR8jZZppQAM6g8VUfz2gZxu9H_XX2q5G=JwoJ0LPQ@mail.gmail.com>

On Fri, Mar 6, 2015 at 6:49 PM Benjamin Peterson <benjamin at python.org>
wrote:

>
>
> On Fri, Mar 6, 2015, at 15:11, Brett Cannon wrote:
> >
> > OK, but that doesn't influence the PEP's goal of dropping .pyo files.
>
> Correct.
>
> >
> > Are you suggesting that the tag be changed to be less specific to
> > optimizations and more free-form? Like
> > `importlib.cpython-35.__no-asserts_no-docstrings__.pyc`? Otherwise stuff
> > like this gets baked into the .pyc file itself instead of the file name,
> > but I don't think we should just drop the ability to switch off asserts
> > and
> > docstrings like Mark seemed to be suggesting.
>
> Basically, though the filename strings could perhaps be more compact.
>

That's fine. Do you have a file name format you want to propose then
instead of "opt-{}" (which is what I'm assuming your "basically" is
referring to)?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150307/0707cd34/attachment-0001.html>

From ron3200 at gmail.com  Sat Mar  7 02:00:20 2015
From: ron3200 at gmail.com (Ron Adam)
Date: Fri, 06 Mar 2015 20:00:20 -0500
Subject: [Python-Dev] PEP 488: elimination of PYO files
In-Reply-To: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
References: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
Message-ID: <mddij5$v2k$1@ger.gmane.org>



On 03/06/2015 11:34 AM, Brett Cannon wrote:

> There are currently two open issues, although one is purely a bikeshed
> topic on formatting of file names so I don't really consider it open for
> change from what is proposed in the PEP without Guido saying he hates my
> preference or someone having a really good argument for some alternative.
> The second open issue on the common case file name is something to
> reasonably debate and come to consensus on.
>
> =======================
>
> PEP: 488
> Title: Elimination of PYO files

+1

And...

Have you considered doing this by having different magic numbers in the 
.pyc file for standard, -O, and -O0 compiled bytecode files?  Python 
already checks that number and recompiles the files if it's not what it's 
expected to be.  And it wouldn't require any naming conventions or new 
cache directories.  It seems to me it would be much easier to do as well.

Cheers,
    Ron


From steve at pearwood.info  Sat Mar  7 10:58:59 2015
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 7 Mar 2015 20:58:59 +1100
Subject: [Python-Dev] PEP 488: elimination of PYO files
In-Reply-To: <mddij5$v2k$1@ger.gmane.org>
References: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
 <mddij5$v2k$1@ger.gmane.org>
Message-ID: <20150307095858.GO7655@ando.pearwood.info>

On Fri, Mar 06, 2015 at 08:00:20PM -0500, Ron Adam wrote:

> Have you considered doing this by having different magic numbers in the 
> .pyc file for standard, -O, and -O0 compiled bytecode files?  Python 
> already checks that number and recompiles the files if it's not what it's 
> expected to be.  And it wouldn't require any naming conventions or new 
> cache directories.  It seems to me it would be much easier to do as well.

And it would fail to solve the problem. The problem isn't just that the 
.pyo file can contain the wrong byte-code for the optimization level, 
that's only part of the problem. Another issue is that you cannot have 
pre-compiled byte-code for multiple different optimization levels. You 
can have a "no optimization" byte-code file, the .pyc file, but only one 
"optimized" byte-code file at the same time.

Brett's proposal will allow -O optimized and -OO optimized byte-code 
files to co-exist, as well as setting up a clear naming convention for 
future optimizers in either the Python compiler or third-party 
optimizers.

No new cache directories are needed. The __pycache__ directory has been 
used since Python 3.3 (or was it 3.2? I forget which). 



-- 
Steve

From ron3200 at gmail.com  Sat Mar  7 15:29:11 2015
From: ron3200 at gmail.com (Ron Adam)
Date: Sat, 07 Mar 2015 09:29:11 -0500
Subject: [Python-Dev] PEP 488: elimination of PYO files
In-Reply-To: <20150307095858.GO7655@ando.pearwood.info>
References: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
 <mddij5$v2k$1@ger.gmane.org> <20150307095858.GO7655@ando.pearwood.info>
Message-ID: <mdf1vo$f1r$1@ger.gmane.org>



On 03/07/2015 04:58 AM, Steven D'Aprano wrote:
> On Fri, Mar 06, 2015 at 08:00:20PM -0500, Ron Adam wrote:
>
>> >Have you considered doing this by having different magic numbers in the
>> >.pyc file for standard, -O, and -O0 compiled bytecode files?  Python
>> >already checks that number and recompiles the files if it's not what it's
>> >expected to be.  And it wouldn't require any naming conventions or new
>> >cache directories.  It seems to me it would be much easier to do as well.
> And it would fail to solve the problem. The problem isn't just that the
> .pyo file can contain the wrong byte-code for the optimization level,
> that's only part of the problem. Another issue is that you cannot have
> pre-compiled byte-code for multiple different optimization levels. You
> can have a "no optimization" byte-code file, the .pyc file, but only one
> "optimized" byte-code file at the same time.
>
> Brett's proposal will allow -O optimized and -OO optimized byte-code
> files to co-exist, as well as setting up a clear naming convention for
> future optimizers in either the Python compiler or third-party
> optimizers.

So all the different versions can be generated ahead of time. I think that 
is the main difference.

My suggestion would cause a recompile of all dependent python files when 
different optimisation levels are used in different projects. Which may be 
worse than not generating bytecode files at all.  OK


A few questions...

Can a submodule use an optimazation level that is different from the file 
that imports it?   (Other than the case this is trying to solve.)

Is there way to specify that an imported module not use any optimisation 
level, or to always use a specific optimisation level?

Is there a way to run tests with all the different optimisation levels?


Cheers,
    Ron


From brett at python.org  Sat Mar  7 15:55:11 2015
From: brett at python.org (Brett Cannon)
Date: Sat, 07 Mar 2015 14:55:11 +0000
Subject: [Python-Dev] PEP 488: elimination of PYO files
In-Reply-To: <mdf1vo$f1r$1@ger.gmane.org>
References: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
 <mddij5$v2k$1@ger.gmane.org> <20150307095858.GO7655@ando.pearwood.info>
 <mdf1vo$f1r$1@ger.gmane.org>
Message-ID: <CAP1=2W57Gqo7TmUO25hPeU_1++DoiFdVbtStpsOuVhj6RLWjmA@mail.gmail.com>

On Sat, Mar 7, 2015 at 9:29 AM Ron Adam <ron3200 at gmail.com> wrote:

>
>
> On 03/07/2015 04:58 AM, Steven D'Aprano wrote:
> > On Fri, Mar 06, 2015 at 08:00:20PM -0500, Ron Adam wrote:
> >
> >> >Have you considered doing this by having different magic numbers in the
> >> >.pyc file for standard, -O, and -O0 compiled bytecode files?  Python
> >> >already checks that number and recompiles the files if it's not what
> it's
> >> >expected to be.  And it wouldn't require any naming conventions or new
> >> >cache directories.  It seems to me it would be much easier to do as
> well.
> > And it would fail to solve the problem. The problem isn't just that the
> > .pyo file can contain the wrong byte-code for the optimization level,
> > that's only part of the problem. Another issue is that you cannot have
> > pre-compiled byte-code for multiple different optimization levels. You
> > can have a "no optimization" byte-code file, the .pyc file, but only one
> > "optimized" byte-code file at the same time.
> >
> > Brett's proposal will allow -O optimized and -OO optimized byte-code
> > files to co-exist, as well as setting up a clear naming convention for
> > future optimizers in either the Python compiler or third-party
> > optimizers.
>
> So all the different versions can be generated ahead of time. I think that
> is the main difference.
>
> My suggestion would cause a recompile of all dependent python files when
> different optimisation levels are used in different projects. Which may be
> worse than not generating bytecode files at all.  OK
>
>
> A few questions...
>
> Can a submodule use an optimazation level that is different from the file
> that imports it?   (Other than the case this is trying to solve.)
>

Currently yes, with this PEP no (without purposefully doing it with some
custom loader).


>
> Is there way to specify that an imported module not use any optimisation
> level, or to always use a specific optimisation level?
>

Not without a custom loader.


>
> Is there a way to run tests with all the different optimisation levels?
>

You have to remember you can't change the optimization levels of the
interpreter once you have started it up. The change in semantics is handled
deep in the AST compiler and there is no exposed way to flip-flop the
setting once the interpreter starts. So to test the different optimization
levels would require either (a) implementing the optimizations are part of
some AST optimizer and doing the right thing in terms of reloading the
modules, or (b) simply running the tests again by running the interpreter
again with different flags (this is when something like tox is useful).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150307/2b0328c1/attachment.html>

From scott+python-dev at scottdial.com  Sat Mar  7 18:30:59 2015
From: scott+python-dev at scottdial.com (Scott Dial)
Date: Sat, 07 Mar 2015 12:30:59 -0500
Subject: [Python-Dev] PEP 488: elimination of PYO files
In-Reply-To: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
References: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
Message-ID: <54FB35D3.7070104@scottdial.com>

On 2015-03-06 11:34 AM, Brett Cannon wrote:
> This PEP proposes eliminating the concept of PYO files from Python.
> To continue the support of the separation of bytecode files based on
> their optimization level, this PEP proposes extending the PYC file
> name to include the optimization level in bytecode repository
> directory (i.e., the ``__pycache__`` directory).

As a packager, this PEP is a bit silent on it's expectations about what
will happen with (for instance) Debian and Fedora packages for Python.
My familiarity is with Fedora, and on that platform, we ship .pyc and
.pyo files (using -O for the .pyo). Is it your expectation that such
platforms will still distribute -O only? Or also -OO? In my world, all
of the __pycache__ directories are owned by root.

-- 
Scott Dial
scott at scottdial.com

From pcmanticore at gmail.com  Sat Mar  7 22:46:28 2015
From: pcmanticore at gmail.com (Claudiu Popa)
Date: Sat, 7 Mar 2015 23:46:28 +0200
Subject: [Python-Dev] Asking for review for Windows issues 21518 and 22080
Message-ID: <CAMy=CLpQYyHnSN1fRVbO6MNd4dcSoWQ8OJZZUd0t5ZN2msmVxQ@mail.gmail.com>

Hello,


The winreg module has a function for loading a registry key under
another registry key, called winreg.LoadKey. Unfortunately, the module
doesn't provide a way to unload that key after the user finishes
operating with it. There's a patch [1] for exporting the RegUnloadKey
[2] API in winreg module as winreg.UnloadKey, similar to how
RegLoadKey is exported as winreg.LoadKey. The patch is helped by
another one [3],  which provides a new module,
test.support.windows_helper, for handling various issues on the
Windows platform, such as acquiring or releasing a privilege.
Unfortunately, it seems there's a dearth of reviewers for this
platform. Could someone knowledgeable with Windows be so kind to
review these patches?
They could make a good addition for Python 3.5.

Thank you very much.


[1] http://bugs.python.org/issue21518 - Expose RegUnloadKey in winreg
[2] https://msdn.microsoft.com/en-us/library/windows/desktop/ms724924%28v=vs.85%29.aspx
[3] http://bugs.python.org/issue22080 - Add windows_helper module helper


/Claudiu

From phqnha at gmail.com  Sat Mar  7 21:14:23 2015
From: phqnha at gmail.com (nha pham)
Date: Sat, 7 Mar 2015 13:14:23 -0700
Subject: [Python-Dev] Optimize binary insertion sort algorithm in Timsort.
Message-ID: <CAMLcfj1DmNvHxLER_HrChY_S9Oy0Jk5L_uEC7YWqbW17fwHkjA@mail.gmail.com>

This describes an optimization for "binary insertion sort" (BINSORT for
short).

BINSORT has been implemented in Python, CyThon, and Timsort (the default
Array.sort() in JAVA SE 7 and JAVA SE 8)
I have read the BINSORT in Timsort

http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/java/util/TimSort.java#TimSort.binarySort%28java.lang.Object%5B%5D%2Cint%2Cint%2Cint%2Cjava.util.Comparator%29
and I think that I can make a little more optimization.

=================
The old BINSORT:
The basic idea is to use binary search on the sorted list to find a final
position
for a new element X, then insert X to the sorted list.
 [SORTED_LIST], [X in UNSORTED_LIST]  // pick X in UNSORTED_LIST
index = binarySeach([SORTED_LIST]) // use binary search to find appropriate
location for X in SORTED_LIST
[SORTED_LIST].add(index, X) // insert X to index location

==================
New BINSORT:
[SORTED_LIST], [A]  // A is an UNSORTED_LIST
j = compare(A[i], A[i+1]) // pick A[i], compare to next element
index = binarySeach([SORTED_LIST], A[i])  // use binary search to find
  // appropriate location for A[i] in SORTED_LIST, and remember the index
[SORTED_LIST].add(index, A[i]) // insert A[i] to index location
 // Now for A[+1], where already know where it should be, compare to A[i]
if j >= 0:
// A[i] > A[i+1], so A[i+1] will be in the right side of A[i]
// We only have to search on a reduced Array:
index = binarySearch(SORTED_LIST[index : length(SORTED_LIST)], A[i+1])
else:
// we search on the left side of of A[i]
index = binarySearch(SORTED_LIST[0 : index], A[i+1])
[SORTED_LIST].add(index, A[i+1]) // insert A[i+1] to index location
 //repeat the loop
==================
Run test.
Intuitively, new BINSORT will be better if the Array become bigger, because
it reduce the array to search with the cost of only 1 more comparison.

I only care about small array, with the length < 100 (we have known that in
Timsort, the
list is divided to chunks with length 64, then apply BINSORT to them).

So I make a big Array, divide them, and apply new BINSORT in each chunk,
then compare to OLD BINSORT.
The source code is in the bottom of this document. Here is the results:

cpuinfo:
model name      : Intel(R) Core(TM) i3 CPU       M 350  @ 2.27GHz
stepping        : 2
microcode       : 0xc
cpu MHz         : 933.000
cache size      : 3072 KB
-----

random array:
ARRAY_SIZE: 1000000
CHUNK_SIZE: 100
DATA: randint(0, 1000000)

OLD BINSORT: 81.45754
new BINSORT: 5.26754
RATIO: (OLD - new) / new = 14.464
---
incremental array:
ARRAY_SIZE: 1000000
CHUNK_SIZE: 100
DATA: range(0, 1000000)

OLD BINSORT: 81.87927
new BINSORT: 5.41651
RATIO: (OLD - new) / new = 14.11659
---
decremental array:
ARRAY_SIZE: 1000000
CHUNK_SIZE: 100
DATA: range(0, 1000000)

OLD BINSORT: 81.45723
new BINSORT: 5.09823
RATIO: (OLD - new) / new = 14.97753
----
all equal array:
ARRAY_SIZE: 1000000
CHUNK_SIZE: 100
DATA: 50000

OLD BINSORT: 40.46027
new BINSORT: 5.41221
RATIO: (OLD - new) / new = 6.47573

====================
What should we do next:
- Tuning my test code (I have just graphed it).
- Test other cases, and bigger array (my laptop cannot handle array more
than 10^6.)
- Modifying Timsort in java.util. and test if it is better.

====================
My test code, written by Python.

from timeit import Timer

setup ="""\
import bisect
from random import randint
from timeit import Timer
SIZE = 1000000
CHUNK = 100
NUM_CHUNK = SIZE/CHUNK
data = []
data2 = []
data3 = []
for i in range(0,SIZE):
    data.append(randint(0,1000000))
    #data.append(i)
#data = data[::-1]
"""
sample ="""\
for j in range(0,NUM_CHUNK):
    low =  CHUNK*j
    high=  low + CHUNK
    data2.append(data[low])
    index = low
    for i in range(low,high):
        x = data[i]
        index = bisect.bisect_right(data2[low:], x, low, len(data2) - low-1)
        data2.insert(index, x)
"""

new ="""\
for j in range(0,NUM_CHUNK):
    low =  CHUNK*j
    high=  low + CHUNK
    data3.append(data[low])
    index = low
    for i in range(low,high):
        x = data[i]
        if x >= data[i-1]:
            index = bisect.bisect_right(data3[low:len(data3) - low-1], x,
index, len(data3) - low-1)
        else:
            index = bisect.bisect_right(data3[low:index], x, low, index)
        data3.insert(index, x)
"""
t2 = Timer(stmt = sample, setup=setup)
a = t2.timeit(1)
print a
t3 = Timer(stmt = new, setup=setup)
b = t3.timeit(1)
print b
print (str((a - b)/b))

====================================

Nha Pham
Mar 07 2015
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150307/8ae35952/attachment.html>

From brett at python.org  Sat Mar  7 23:03:18 2015
From: brett at python.org (Brett Cannon)
Date: Sat, 07 Mar 2015 22:03:18 +0000
Subject: [Python-Dev] PEP 488: elimination of PYO files
In-Reply-To: <54FB35D3.7070104@scottdial.com>
References: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
 <54FB35D3.7070104@scottdial.com>
Message-ID: <CAP1=2W7dTc88qCdUxB-cCGY34pM2mmMu2H1czWTpou-8dhMiYg@mail.gmail.com>

On Sat, Mar 7, 2015 at 12:39 PM Scott Dial <scott+python-dev at scottdial.com>
wrote:

> On 2015-03-06 11:34 AM, Brett Cannon wrote:
> > This PEP proposes eliminating the concept of PYO files from Python.
> > To continue the support of the separation of bytecode files based on
> > their optimization level, this PEP proposes extending the PYC file
> > name to include the optimization level in bytecode repository
> > directory (i.e., the ``__pycache__`` directory).
>
> As a packager, this PEP is a bit silent on it's expectations about what
> will happen with (for instance) Debian and Fedora packages for Python.
> My familiarity is with Fedora, and on that platform, we ship .pyc and
> .pyo files (using -O for the .pyo). Is it your expectation that such
> platforms will still distribute -O only? Or also -OO? In my world, all
> of the __pycache__ directories are owned by root.
>

I assume they will generate all .pyc files at all levels, but I don't if
it's my place to dictate such a thing since bytecode files are an
optimization to Python itself and do not influence how people interact with
the interpreter like with PEP 394 (The "python" Command on Unix-Like
Systems).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150307/b520306b/attachment.html>

From victor.stinner at gmail.com  Sun Mar  8 03:13:01 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sun, 8 Mar 2015 03:13:01 +0100
Subject: [Python-Dev] PEP 471 Final: os.scandir() merged into Python 3.5
Message-ID: <CAMpsgwYg=RZeyqTzLGn+Nr7xEiiff0EUDhEUgGS-So=B-DjJ1g@mail.gmail.com>

Hi,

FYI I commited the implementation of os.scandir() written by Ben Hoyt.
I hope that it will be part of Python 3.5 alpha 2 (Ben just sent the
final patch today).

Please test this new feature. You may benchmark here.
http://bugs.python.org/issue22524 contains some benchmark tools and
benchmark results of older versions of the patch.

The implementation was tested on Windows and Linux. I'm now watching
for buildbots to see how other platforms like os.scandir().

Bad news: OpenIndiana doesn't support d_type: the dirent structure has
no d_type field. I already fixed the implementation to support this
case. os.scandir() is still useful on OpenIndiana, because the stat
result is cached in a DirEntry, so only one syscall is required,
instead of multiple, when multiple DirEntry methods are called (ex:
entry.is_dir() and not entry.is_symlink()).

Victor

From benhoyt at gmail.com  Sun Mar  8 03:31:35 2015
From: benhoyt at gmail.com (Ben Hoyt)
Date: Sat, 7 Mar 2015 21:31:35 -0500
Subject: [Python-Dev] PEP 471 Final: os.scandir() merged into Python 3.5
In-Reply-To: <CAMpsgwYg=RZeyqTzLGn+Nr7xEiiff0EUDhEUgGS-So=B-DjJ1g@mail.gmail.com>
References: <CAMpsgwYg=RZeyqTzLGn+Nr7xEiiff0EUDhEUgGS-So=B-DjJ1g@mail.gmail.com>
Message-ID: <CAL9jXCEUOkR_EvGzA6kdodTk3S6+AfnWpRkTmJ27ZRtUaXyAYw@mail.gmail.com>

Thanks for committing this, Victor! And fixing the d_type issue on funky
platforms.

Others: if you want to benchmark this, the simplest way is to use my
os.walk() benchmark.py test program here: https://github.com/benhoyt/scandir
-- it compares the built-in os.walk() implemented with os.listdir() with a
version of walk() implemented with os.scandir(). I see huge gains on
Windows (12-50x) and modest gains on my Linux VM (3-5x).

Note that the actual CPython version of os.walk() doesn't yet use
os.scandir(). I intend to open a separate issue for that shortly (or Victor
can). But that part should be fairly straight-forward, as I already have a
version available in my GitHub project.

-Ben


On Sat, Mar 7, 2015 at 9:13 PM, Victor Stinner <victor.stinner at gmail.com>
wrote:

> Hi,
>
> FYI I commited the implementation of os.scandir() written by Ben Hoyt.
> I hope that it will be part of Python 3.5 alpha 2 (Ben just sent the
> final patch today).
>
> Please test this new feature. You may benchmark here.
> http://bugs.python.org/issue22524 contains some benchmark tools and
> benchmark results of older versions of the patch.
>
> The implementation was tested on Windows and Linux. I'm now watching
> for buildbots to see how other platforms like os.scandir().
>
> Bad news: OpenIndiana doesn't support d_type: the dirent structure has
> no d_type field. I already fixed the implementation to support this
> case. os.scandir() is still useful on OpenIndiana, because the stat
> result is cached in a DirEntry, so only one syscall is required,
> instead of multiple, when multiple DirEntry methods are called (ex:
> entry.is_dir() and not entry.is_symlink()).
>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/benhoyt%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150307/a9a4387d/attachment-0001.html>

From victor.stinner at gmail.com  Sun Mar  8 03:39:03 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sun, 8 Mar 2015 03:39:03 +0100
Subject: [Python-Dev] PEP 471 Final: os.scandir() merged into Python 3.5
In-Reply-To: <CAL9jXCEUOkR_EvGzA6kdodTk3S6+AfnWpRkTmJ27ZRtUaXyAYw@mail.gmail.com>
References: <CAMpsgwYg=RZeyqTzLGn+Nr7xEiiff0EUDhEUgGS-So=B-DjJ1g@mail.gmail.com>
 <CAL9jXCEUOkR_EvGzA6kdodTk3S6+AfnWpRkTmJ27ZRtUaXyAYw@mail.gmail.com>
Message-ID: <CAMpsgwbR1T5HVqS1Ceef9qL19n5nY90y-Gmcm7BookkLjp3O7Q@mail.gmail.com>

2015-03-08 3:31 GMT+01:00 Ben Hoyt <benhoyt at gmail.com>:
> Thanks for committing this, Victor! And fixing the d_type issue on funky
> platforms.

You're welcome.

> Note that the actual CPython version of os.walk() doesn't yet use
> os.scandir(). I intend to open a separate issue for that shortly (or Victor
> can). But that part should be fairly straight-forward, as I already have a
> version available in my GitHub project.

Yes, I just opened an issue for os.walk():
http://bugs.python.org/issue23605

We need a patch and benchmarks on Linux and Windows for that
(including benchmarks on a NFS share for the Linux case).

I changed the status of the PEP 471 to Final even if os.walk() was not
modified yet. IMO the most important part was os.scandir() since
"os.scandir()" is in the title of the PEP 471.

Victor

From ryan.stuart.85 at gmail.com  Sun Mar  8 04:48:44 2015
From: ryan.stuart.85 at gmail.com (Ryan Stuart)
Date: Sun, 08 Mar 2015 03:48:44 +0000
Subject: [Python-Dev] PEP 471 Final: os.scandir() merged into Python 3.5
In-Reply-To: <CAL9jXCEUOkR_EvGzA6kdodTk3S6+AfnWpRkTmJ27ZRtUaXyAYw@mail.gmail.com>
References: <CAMpsgwYg=RZeyqTzLGn+Nr7xEiiff0EUDhEUgGS-So=B-DjJ1g@mail.gmail.com>
 <CAL9jXCEUOkR_EvGzA6kdodTk3S6+AfnWpRkTmJ27ZRtUaXyAYw@mail.gmail.com>
Message-ID: <CAM8kh5tL4QgJa4SaaxRsGzgV4RVpZxpdBC0VZ1f1ZKth4nF7HQ@mail.gmail.com>

Hi,

On Sun, 8 Mar 2015 at 12:33 Ben Hoyt <benhoyt at gmail.com> wrote:

> Others: if you want to benchmark this, the simplest way is to use my
> os.walk() benchmark.py test program here:
> https://github.com/benhoyt/scandir -- it compares the built-in os.walk()
> implemented with os.listdir() with a version of walk() implemented with
> os.scandir(). I see huge gains on Windows (12-50x) and modest gains on my
> Linux VM (3-5x).
>

I have a MacBook Pro Laptop running OS X 10.10.2. I did the following:

   - hg update -r 8ef4f75a8018
   - patch -p1 < scandir-8.patch
   - ./configure --with-pydebug && make -j7

I then ran ./python.exe ~/Workspace/python/scandir/benchmark.py and I got:

*Creating tree at /Users/rstuart/Workspace/python/scandir/benchtree:
depth=4, num_dirs=5, num_files=50*
*Using slower ctypes version of scandir*
*Comparing against builtin version of os.walk()*
*Priming the system's cache...*
*Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree,
repeat 1/3...*
*Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree,
repeat 2/3...*
*Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree,
repeat 3/3...*
*os.walk took 0.184s, scandir.walk took 0.158s -- 1.2x as fast*

I then did ./python.exe ~/Workspace/python/scandir/benchmark.py -s and got:

*Using slower ctypes version of scandir*
*Comparing against builtin version of os.walk()*
*Priming the system's cache...*
*Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree,
repeat 1/3...*
*Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree,
repeat 2/3...*
*Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree,
repeat 3/3...*
*os.walk size 23400, scandir.walk size 23400 -- equal*
*os.walk took 0.483s, scandir.walk took 0.463s -- 1.0x as fast*

Hope this helps.

Cheers

Note that the actual CPython version of os.walk() doesn't yet use
> os.scandir(). I intend to open a separate issue for that shortly (or Victor
> can). But that part should be fairly straight-forward, as I already have a
> version available in my GitHub project.
>
> -Ben
>
>
> On Sat, Mar 7, 2015 at 9:13 PM, Victor Stinner <victor.stinner at gmail.com>
> wrote:
>
>> Hi,
>>
>> FYI I commited the implementation of os.scandir() written by Ben Hoyt.
>> I hope that it will be part of Python 3.5 alpha 2 (Ben just sent the
>> final patch today).
>>
>> Please test this new feature. You may benchmark here.
>> http://bugs.python.org/issue22524 contains some benchmark tools and
>> benchmark results of older versions of the patch.
>>
>> The implementation was tested on Windows and Linux. I'm now watching
>> for buildbots to see how other platforms like os.scandir().
>>
>> Bad news: OpenIndiana doesn't support d_type: the dirent structure has
>> no d_type field. I already fixed the implementation to support this
>> case. os.scandir() is still useful on OpenIndiana, because the stat
>> result is cached in a DirEntry, so only one syscall is required,
>> instead of multiple, when multiple DirEntry methods are called (ex:
>> entry.is_dir() and not entry.is_symlink()).
>>
>> Victor
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>>
> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/benhoyt%40gmail.com
>>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> ryan.stuart.85%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150308/1c1238d9/attachment.html>

From barry at python.org  Sun Mar  8 12:54:26 2015
From: barry at python.org (Barry Warsaw)
Date: Sun, 8 Mar 2015 07:54:26 -0400
Subject: [Python-Dev] PEP 488: elimination of PYO files
In-Reply-To: <54FB35D3.7070104@scottdial.com>
References: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
 <54FB35D3.7070104@scottdial.com>
Message-ID: <20150308075426.1019f87d@limelight.wooz.org>

On Mar 07, 2015, at 12:30 PM, Scott Dial wrote:

>As a packager, this PEP is a bit silent on it's expectations about what
>will happen with (for instance) Debian and Fedora packages for Python.
>My familiarity is with Fedora, and on that platform, we ship .pyc and
>.pyo files (using -O for the .pyo). Is it your expectation that such
>platforms will still distribute -O only? Or also -OO? In my world, all
>of the __pycache__ directories are owned by root.

On Debuntu, we don't create pyo files by default, although IIRC it's an
option.  The problem has been the collision between -O and -OO, so this PEP
may open the door to us generating both optimization levels automatically.
It's not something that's very high on my todo list, but if there's enough
interest it probably wouldn't be difficult to add.

Cheers,
-Barry

From ncoghlan at gmail.com  Sun Mar  8 13:17:16 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 8 Mar 2015 22:17:16 +1000
Subject: [Python-Dev] PEP 488: elimination of PYO files
In-Reply-To: <CAP1=2W7dTc88qCdUxB-cCGY34pM2mmMu2H1czWTpou-8dhMiYg@mail.gmail.com>
References: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
 <54FB35D3.7070104@scottdial.com>
 <CAP1=2W7dTc88qCdUxB-cCGY34pM2mmMu2H1czWTpou-8dhMiYg@mail.gmail.com>
Message-ID: <CADiSq7cevTp3cXUfx7Seq4pr5U8NT8JFtE4_4TdhhmHwnQQEmw@mail.gmail.com>

For the record here: +1 on the PEP from me (the comments I made on
import-sig have already incorporated into this version of the PEP)

On 8 March 2015 at 08:03, Brett Cannon <brett at python.org> wrote:
>
>
> On Sat, Mar 7, 2015 at 12:39 PM Scott Dial <scott+python-dev at scottdial.com>
> wrote:
>>
>> On 2015-03-06 11:34 AM, Brett Cannon wrote:
>> > This PEP proposes eliminating the concept of PYO files from Python.
>> > To continue the support of the separation of bytecode files based on
>> > their optimization level, this PEP proposes extending the PYC file
>> > name to include the optimization level in bytecode repository
>> > directory (i.e., the ``__pycache__`` directory).
>>
>> As a packager, this PEP is a bit silent on it's expectations about what
>> will happen with (for instance) Debian and Fedora packages for Python.
>> My familiarity is with Fedora, and on that platform, we ship .pyc and
>> .pyo files (using -O for the .pyo). Is it your expectation that such
>> platforms will still distribute -O only? Or also -OO? In my world, all
>> of the __pycache__ directories are owned by root.
>
>
> I assume they will generate all .pyc files at all levels, but I don't if
> it's my place to dictate such a thing since bytecode files are an
> optimization to Python itself and do not influence how people interact with
> the interpreter like with PEP 394 (The "python" Command on Unix-Like
> Systems).

= Fedora near term =

Nothing significant will change for RPMs with policy compliant
(https://fedoraproject.org/wiki/Packaging:Python#Byte_compiling) spec
files, even after switching to Python 3.5. The only different will be
that the current .pyo files will be replaced with files using the new
PEP 488 naming scheme. Folks using custom spec files with their own
pattern matches that assume "pyo" may need to adjust them (e.g. to
grab everything in __pycache__, as recommended in the packaging
guidelines)

= Fedora long term =

After the switch to Python 3.5 (which, given the release schedule, I
currently expect will be in Fedora 24 early next year), the RPM Python
build macros *might* be changed to generate -OO files in addition to
-O ones, making it easier to run Python applications in -OO mode to
reduce memory usage. The PEP doesn't mandate such a change, but it
does enable it. This would have the downside of making every Python
package in the distro slightly larger (all 15k+ of them), so there'd
need to be a clear and compelling pay-off to justify that cost.

While folks may be tempted to say "disk space is cheap", VM and
container provisioning latency are performance critical in a number of
use cases, so making the Fedora Cloud and Atomic Host images even
slightly larger isn't likely to win us any friends, so actually making
such a change would require compelling benchmarks demonstrating
reductions in runtime memory usage that justify the larger image size,
as well as confirming that the change doesn't place undue pressure on
the DVD ISO image contents.

This isn't really the right list for more in-depth discussion of what
Fedora *might* do though - any discussion will be entirely theoretical
until after Python 3.5 comes out, and even then the appropriate list
would be python-devel at lists.fedoraproject.org rather than here.

Regards,
Nick.

P.S. For those that aren't already aware, I became a voting member of
Fedora's Environments & Stacks Working Group several months ago and
now work closely with Slavek Kabrda (the Fedora Python maintainer,
amongst other things) on Python's integration into Fedora and its
derivatives.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sun Mar  8 13:20:28 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 8 Mar 2015 22:20:28 +1000
Subject: [Python-Dev] PEP 471 Final: os.scandir() merged into Python 3.5
In-Reply-To: <CAMpsgwYg=RZeyqTzLGn+Nr7xEiiff0EUDhEUgGS-So=B-DjJ1g@mail.gmail.com>
References: <CAMpsgwYg=RZeyqTzLGn+Nr7xEiiff0EUDhEUgGS-So=B-DjJ1g@mail.gmail.com>
Message-ID: <CADiSq7fTtJc6D6AgnOCMYVUiU7GiQopnWw7aYhLcCidifVLW7A@mail.gmail.com>

On 8 March 2015 at 12:13, Victor Stinner <victor.stinner at gmail.com> wrote:
> Hi,
>
> FYI I commited the implementation of os.scandir() written by Ben Hoyt.
> I hope that it will be part of Python 3.5 alpha 2 (Ben just sent the
> final patch today).

Thanks to everyone that worked on getting this PEP through to Final status!

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From p.f.moore at gmail.com  Sun Mar  8 13:36:45 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Sun, 8 Mar 2015 12:36:45 +0000
Subject: [Python-Dev] PEP 471 Final: os.scandir() merged into Python 3.5
In-Reply-To: <CAMpsgwYg=RZeyqTzLGn+Nr7xEiiff0EUDhEUgGS-So=B-DjJ1g@mail.gmail.com>
References: <CAMpsgwYg=RZeyqTzLGn+Nr7xEiiff0EUDhEUgGS-So=B-DjJ1g@mail.gmail.com>
Message-ID: <CACac1F9TvFPPnwQvDqdKhbgsHSnCC4TdM2c-G+vDMA=qPy0cpg@mail.gmail.com>

On 8 March 2015 at 02:13, Victor Stinner <victor.stinner at gmail.com> wrote:
> FYI I commited the implementation of os.scandir() written by Ben Hoyt.
> I hope that it will be part of Python 3.5 alpha 2 (Ben just sent the
> final patch today).

Yay! Great news.
Paul

From p.f.moore at gmail.com  Sun Mar  8 13:39:57 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Sun, 8 Mar 2015 12:39:57 +0000
Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving
 Python ZIP Application Support
In-Reply-To: <CACac1F96rbrc4gZbV9fkxKiAEMv7uVxPF0pPD60WwZHWZfyBEw@mail.gmail.com>
References: <CACac1F_E_a9hB818EMG4nbDTqqoOEc9B0_ZPZUzop6JKjk-Y3A@mail.gmail.com>
 <CAP1=2W4APXunO=wSYWpEyuYmYmkYjuSgEumvT_9=s6QdKobUgQ@mail.gmail.com>
 <CAG8k2+4JbRarMZmVg0dT18u1CnfWnS9f44=JvF++TVmMUJEgnw@mail.gmail.com>
 <CAP1=2W6GkDNvH9y2+gbh-WBpWvfM0tTX1BE92mniu74En1r7fA@mail.gmail.com>
 <CACac1F_t3uj8r5gTV1Y0ynGKWUF3oH3OHbX6Tf-H3inkxQOwvA@mail.gmail.com>
 <CAG8k2+4Mn9f1oujLr3mVY2DnNjfn52L5+Mc2vyJNJTvOtk1O5w@mail.gmail.com>
 <54EB7DDE.9030405@stoneleaf.us>
 <CAPdQG2pvRgeTiA6T-Lb6E7Th0eM1DZAzwrS9f85mPPWkPqAH4w@mail.gmail.com>
 <CAP7+vJ+=AzrnGdQfjCkq7Tc3tqHz2qq=jOYjQkyiuuDht5Hn-Q@mail.gmail.com>
 <CACac1F9W9vQ2TckokfGqEscUZMk-CLbt+ODEgxxkGz_oOnAg3g@mail.gmail.com>
 <CADiSq7fdPPQ=wRDhwZEv8Rh7iUm_Vt+k9MBgFhvo54svV0L2xA@mail.gmail.com>
 <CAP7+vJJ45esj8bbfHpBsitUhoxHHLN91yZyX72nLTHBkFe+6Cw@mail.gmail.com>
 <CACac1F_LsMGqBzF=ONe2D=AO44xZBZYkPXz8e=FHZhkUkj51QQ@mail.gmail.com>
 <CAP7+vJKXX=Ybz0rnfkQgH1+kgY-uVmvjJcOOhVR17Nhafp7N=w@mail.gmail.com>
 <CAP7+vJL_1danWZefJivqpDbNYM3DzNAeN_+q5PeqgC-ZMae-xw@mail.gmail.com>
 <CACac1F_tYTecaK0NursqPp4y4HwFDuhi4RQQ4Ggp9JKhC5ZUnA@mail.gmail.com>
 <CAP7+vJK-V=ipy4ALW_MYSENSVH=Mvm4jeTxey_B+iNuFppieCg@mail.gmail.com>
 <CACac1F96rbrc4gZbV9fkxKiAEMv7uVxPF0pPD60WwZHWZfyBEw@mail.gmail.com>
Message-ID: <CACac1F9yQdOA5zKN31hzkX+Efsn5W-k=xngB0vi8WmnnkQioeg@mail.gmail.com>

On 26 February 2015 at 21:48, Paul Moore <p.f.moore at gmail.com> wrote:
> On 26 February 2015 at 21:34, Guido van Rossum <guido at python.org> wrote:
>> Accepted!
>>
>> Thanks for your patience, Paul, and thanks everyone for their feedback.
>>
>> I know there are still a few small edits to the PEP, but those don't affect
>> my acceptance. Congrats!
>
> Excellent, thanks to everyone for the helpful comments throughout.

I'm still looking for someone who has the time to commit the final
patch on http://bugs.python.org/issue23491. Python 3.5a2 is due out
today (at least according to the PEP) so I've probably missed that,
but it would be nice if it could be in for alpha 3 at least.

Thanks,
Paul

From brett at python.org  Sun Mar  8 15:35:12 2015
From: brett at python.org (Brett Cannon)
Date: Sun, 08 Mar 2015 14:35:12 +0000
Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving
 Python ZIP Application Support
References: <CACac1F_E_a9hB818EMG4nbDTqqoOEc9B0_ZPZUzop6JKjk-Y3A@mail.gmail.com>
 <CAP1=2W4APXunO=wSYWpEyuYmYmkYjuSgEumvT_9=s6QdKobUgQ@mail.gmail.com>
 <CAG8k2+4JbRarMZmVg0dT18u1CnfWnS9f44=JvF++TVmMUJEgnw@mail.gmail.com>
 <CAP1=2W6GkDNvH9y2+gbh-WBpWvfM0tTX1BE92mniu74En1r7fA@mail.gmail.com>
 <CACac1F_t3uj8r5gTV1Y0ynGKWUF3oH3OHbX6Tf-H3inkxQOwvA@mail.gmail.com>
 <CAG8k2+4Mn9f1oujLr3mVY2DnNjfn52L5+Mc2vyJNJTvOtk1O5w@mail.gmail.com>
 <54EB7DDE.9030405@stoneleaf.us>
 <CAPdQG2pvRgeTiA6T-Lb6E7Th0eM1DZAzwrS9f85mPPWkPqAH4w@mail.gmail.com>
 <CAP7+vJ+=AzrnGdQfjCkq7Tc3tqHz2qq=jOYjQkyiuuDht5Hn-Q@mail.gmail.com>
 <CACac1F9W9vQ2TckokfGqEscUZMk-CLbt+ODEgxxkGz_oOnAg3g@mail.gmail.com>
 <CADiSq7fdPPQ=wRDhwZEv8Rh7iUm_Vt+k9MBgFhvo54svV0L2xA@mail.gmail.com>
 <CAP7+vJJ45esj8bbfHpBsitUhoxHHLN91yZyX72nLTHBkFe+6Cw@mail.gmail.com>
 <CACac1F_LsMGqBzF=ONe2D=AO44xZBZYkPXz8e=FHZhkUkj51QQ@mail.gmail.com>
 <CAP7+vJKXX=Ybz0rnfkQgH1+kgY-uVmvjJcOOhVR17Nhafp7N=w@mail.gmail.com>
 <CAP7+vJL_1danWZefJivqpDbNYM3DzNAeN_+q5PeqgC-ZMae-xw@mail.gmail.com>
 <CACac1F_tYTecaK0NursqPp4y4HwFDuhi4RQQ4Ggp9JKhC5ZUnA@mail.gmail.com>
 <CAP7+vJK-V=ipy4ALW_MYSENSVH=Mvm4jeTxey_B+iNuFppieCg@mail.gmail.com>
 <CACac1F96rbrc4gZbV9fkxKiAEMv7uVxPF0pPD60WwZHWZfyBEw@mail.gmail.com>
 <CACac1F9yQdOA5zKN31hzkX+Efsn5W-k=xngB0vi8WmnnkQioeg@mail.gmail.com>
Message-ID: <CAP1=2W7ma-O8Z9Qj37PRuF73aiv-bOUbpPbXCpH5psJ5B_ouRg@mail.gmail.com>

 On Sun, Mar 8, 2015, 08:40 Paul Moore <p.f.moore at gmail.com> wrote:

On 26 February 2015 at 21:48, Paul Moore <p.f.moore at gmail.com> wrote:
> On 26 February 2015 at 21:34, Guido van Rossum <guido at python.org> wrote:
>> Accepted!
>>
>> Thanks for your patience, Paul, and thanks everyone for their feedback.
>>
>> I know there are still a few small edits to the PEP, but those don't
affect
>> my acceptance. Congrats!
>
> Excellent, thanks to everyone for the helpful comments throughout.

I'm still looking for someone who has the time to commit the final
patch on http://bugs.python.org/issue23491. Python 3.5a2 is due out
today (at least according to the PEP) so I've probably missed that,
but it would be nice if it could be in for alpha 3 at least.

 If no one gets to it I can submit on Friday.

-brett


Thanks,
Paul
_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/brett%40python.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150308/f99c47f3/attachment.html>

From mistersheik at gmail.com  Sun Mar  8 21:38:44 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Sun, 8 Mar 2015 16:38:44 -0400
Subject: [Python-Dev] PEP 448 review
In-Reply-To: <CAA68w_ngnkX0KPnDni8Vvi9iG=RgiN4B_0dp9NuEMvaNO6=5OA@mail.gmail.com>
References: <CAP7+vJK58SYgBExknnSO2xwiQm5+AO5_=f0yAarNJUkbmczcPg@mail.gmail.com>
 <CAA68w_m1cB4MmMeNX=zA5MjfaxytgU9+=8fVqEK173DGy3K5Zw@mail.gmail.com>
 <CAMpsgwa8JY3rCobEPOe2CN2ovcQJE7AkHfRPtw5SZE83eF1+fQ@mail.gmail.com>
 <CAA68w_kFVmMQJsk+hdgnKtOui2rqYdecQmaNEjk_UPk7N7Q_xg@mail.gmail.com>
 <CAA68w_ngnkX0KPnDni8Vvi9iG=RgiN4B_0dp9NuEMvaNO6=5OA@mail.gmail.com>
Message-ID: <CAA68w_ki+sAti_WX1a+ACUCbRexh38a=aQpSG7BgEgbhWmsiig@mail.gmail.com>

Anyone have time to do a code review?

http://bugs.python.org/issue2292

On Mon, Mar 2, 2015 at 4:54 PM, Neil Girdhar <mistersheik at gmail.com> wrote:

> It's from five days ago.  I asked Joshua to take a look at something, but
> I guess he is busy.
>
> Best,
>
> Neil
>
> ?
>
> The latest file there is from Feb 26, while your message that the patch
> was ready for review is from today -- so is the
> patch from five days ago the most recent?
>
> --
> ~Ethan~
>
> On Mon, Mar 2, 2015 at 3:18 PM, Neil Girdhar <mistersheik at gmail.com>
> wrote:
>
>> http://bugs.python.org/issue2292
>>
>> On Mon, Mar 2, 2015 at 3:17 PM, Victor Stinner <victor.stinner at gmail.com>
>> wrote:
>>
>>> Where is the patch?
>>>
>>> Victor
>>>
>>> Le lundi 2 mars 2015, Neil Girdhar <mistersheik at gmail.com> a ?crit :
>>>
>>> Hi everyone,
>>>>
>>>> The patch is ready for review now, and I should have time this week to
>>>> make changes and respond to comments.
>>>>
>>>> Best,
>>>>
>>>> Neil
>>>>
>>>> On Wed, Feb 25, 2015 at 2:42 PM, Guido van Rossum <guido at python.org>
>>>> wrote:
>>>>
>>>>> I'm back, I've re-read the PEP, and I've re-read the long thread with
>>>>> "(no subject)".
>>>>>
>>>>> I think Georg Brandl nailed it:
>>>>>
>>>>> """
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *I like the "sequence and dict flattening" part of the PEP, mostly
>>>>> because itis consistent and should be easy to understand, but the
>>>>> comprehension syntaxenhancements seem to be bad for readability and
>>>>> "comprehending" what the codedoes.The call syntax part is a mixed bag on
>>>>> the one hand it is nice to be consistent with the extended possibilities in
>>>>> literals (flattening), but on the other hand there would be small but
>>>>> annoying inconsistencies anyways (e.g. the duplicate kwarg case above).*
>>>>> """
>>>>>
>>>>> Greg Ewing followed up explaining that the inconsistency between dict
>>>>> flattening and call syntax is inherent in the pre-existing different rules
>>>>> for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while f(a=1,
>>>>> a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1,
>>>>> **{'a': 1}) is a TypeError.)
>>>>>
>>>>> For me, allowing f(*a, *b) and f(**d, **e) and all the other
>>>>> combinations for function calls proposed by the PEP is an easy +1 -- it's a
>>>>> straightforward extension of the existing pattern, and anybody who knows
>>>>> what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d,
>>>>> **e) means shouldn't be hard either. Understanding the edge case for
>>>>> duplicate keys with f(**d, **e) is a little harder, but the error messages
>>>>> are pretty clear, and it is not a new edge case.
>>>>>
>>>>> The sequence and dict flattening syntax proposals are also clean and
>>>>> logical -- we already have *-unpacking on the receiving side, so allowing
>>>>> *x in tuple expressions reads pretty naturally (and the similarity with *a
>>>>> in argument lists certainly helps). From here, having [a, *x, b, *y] is
>>>>> also natural, and then the extension to other displays is natural: {a, *x,
>>>>> b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me.
>>>>>
>>>>> So that leaves comprehensions. IIRC, during the development of the
>>>>> patch we realized that f(*x for x in xs) is sufficiently ambiguous that we
>>>>> decided to disallow it -- note that f(x for x in xs) is already somewhat of
>>>>> a special case because an argument can only be a "bare" generator
>>>>> expression if it is the only argument. The same reasoning doesn't apply (in
>>>>> that form) to list, set and dict comprehensions -- while f(x for x in xs)
>>>>> is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the
>>>>> same as [(x for x in xs)] (that's a list of one element, and the element is
>>>>> a generator expression).
>>>>>
>>>>> The basic premise of this part of the proposal is that if you have a
>>>>> few iterables, the new proposal (without comprehensions) lets you create a
>>>>> list or generator expression that iterates over all of them, essentially
>>>>> flattening them:
>>>>>
>>>>>     >>> xs = [1, 2, 3]
>>>>>     >>> ys = ['abc', 'def']
>>>>>     >>> zs = [99]
>>>>>     >>> [*xs, *ys, *zs]
>>>>>     [1, 2, 3, 'abc', 'def', 99]
>>>>>     >>>
>>>>>
>>>>> But now suppose you have a list of iterables:
>>>>>
>>>>>     >>> xss = [[1, 2, 3], ['abc', 'def'], [99]]
>>>>>     >>> [*xss[0], *xss[1], *xss[2]]
>>>>>     [1, 2, 3, 'abc', 'def', 99]
>>>>>     >>>
>>>>>
>>>>> Wouldn't it be nice if you could write the latter using a
>>>>> comprehension?
>>>>>
>>>>>     >>> xss = [[1, 2, 3], ['abc', 'def'], [99]]
>>>>>     >>> [*xs for xs in xss]
>>>>>     [1, 2, 3, 'abc', 'def', 99]
>>>>>     >>>
>>>>>
>>>>> This is somewhat seductive, and the following is even nicer: the *xs
>>>>> position may be an expression, e.g.:
>>>>>
>>>>>     >>> xss = [[1, 2, 3], ['abc', 'def'], [99]]
>>>>>     >>> [*xs[:2] for xs in xss]
>>>>>     [1, 2, 'abc', 'def', 99]
>>>>>     >>>
>>>>>
>>>>> On the other hand, I had to explore the possibilities here by
>>>>> experimenting in the interpreter, and I discovered some odd edge cases
>>>>> (e.g. you can parenthesize the starred expression, but that seems a
>>>>> syntactic accident).
>>>>>
>>>>> All in all I am personally +0 on the comprehension part of the PEP,
>>>>> and I like that it provides a way to "flatten" a sequence of sequences, but
>>>>> I think very few people in the thread have supported this part. Therefore I
>>>>> would like to ask Neil to update the PEP and the patch to take out the
>>>>> comprehension part, so that the two "easy wins" can make it into Python 3.5
>>>>> (basically, I am accepting two-thirds of the PEP :-). There is some time
>>>>> yet until alpha 2.
>>>>>
>>>>> I would also like code reviewers (Benjamin?) to start reviewing the
>>>>> patch <http://bugs.python.org/issue2292>, taking into account that
>>>>> the comprehension part needs to be removed.
>>>>>
>>>>> --
>>>>> --Guido van Rossum (python.org/~guido)
>>>>>
>>>>>
>>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150308/6b8d7356/attachment.html>

From timothy.c.delaney at gmail.com  Sun Mar  8 23:00:31 2015
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Mon, 9 Mar 2015 09:00:31 +1100
Subject: [Python-Dev] PEP 471 Final: os.scandir() merged into Python 3.5
In-Reply-To: <CAL9jXCEUOkR_EvGzA6kdodTk3S6+AfnWpRkTmJ27ZRtUaXyAYw@mail.gmail.com>
References: <CAMpsgwYg=RZeyqTzLGn+Nr7xEiiff0EUDhEUgGS-So=B-DjJ1g@mail.gmail.com>
 <CAL9jXCEUOkR_EvGzA6kdodTk3S6+AfnWpRkTmJ27ZRtUaXyAYw@mail.gmail.com>
Message-ID: <CAN8CLgnQnD14k+0jKdTWcjmmpFhc+Rwaa-3ZGa1-fz4Wazx=PA@mail.gmail.com>

On 8 March 2015 at 13:31, Ben Hoyt <benhoyt at gmail.com> wrote:

> Thanks for committing this, Victor! And fixing the d_type issue on funky
> platforms.
>
> Others: if you want to benchmark this, the simplest way is to use my
> os.walk() benchmark.py test program here:
> https://github.com/benhoyt/scandir -- it compares the built-in os.walk()
> implemented with os.listdir() with a version of walk() implemented with
> os.scandir(). I see huge gains on Windows (12-50x) and modest gains on my
> Linux VM (3-5x).
>
> Note that the actual CPython version of os.walk() doesn't yet use
> os.scandir(). I intend to open a separate issue for that shortly (or Victor
> can). But that part should be fairly straight-forward, as I already have a
> version available in my GitHub project.
>

I think it would be a good idea to report the type of drive/mount along
with the results. I imagine that there might be significant differences
between solid state drives, hard drives and network mounts.

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150309/6861bbab/attachment-0001.html>

From phqnha at gmail.com  Mon Mar  9 01:17:16 2015
From: phqnha at gmail.com (nha pham)
Date: Sun, 8 Mar 2015 17:17:16 -0700
Subject: [Python-Dev] Tunning binary insertion sort algorithm in Timsort.
Message-ID: <CAMLcfj3usoq-new2gxh_ymP+1wv6XZj7+og6XMVAUj_sYeT+Pg@mail.gmail.com>

We can optimize the TimSort algorithm by optimizing its binary insertion
sort.



The current version of binary insertion sort use this idea:

Use binary search to find a final position in sorted list for a new element
X. Then insert X to that location.



I suggest another idea:

Use binary search to find a final postion in sorted list for a new element
X. Before insert X to that location, compare X with its next element.

For the next element, we already know if it is lower or bigger than X, so
we can reduce the search area to the left side or on the right side of X in
the sorted list.



I have applied my idea on java.util. ComparableTimSort.sort() and testing.
The execute time is reduced by 2%-6% with array of random integer.



Here is detail about algorithm and testing:
https://github.com/nhapq/Optimize_binary_insertion_sort



Sincerely.

phqnha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150308/9e3a5cd3/attachment.html>

From ethan at stoneleaf.us  Mon Mar  9 04:31:30 2015
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sun, 08 Mar 2015 20:31:30 -0700
Subject: [Python-Dev] boxing and unboxing data types
Message-ID: <54FD1412.3020002@stoneleaf.us>

When data is passed from Python to a native library (such as in an O/S call), how does the unboxing of data types occur?

For a specific instance, os.open allows an integer whose various bits express desired behavior as `flags` -- if flags is
1, for example, the file is open write-only.

If I pass an int-like object to os.open, __int__ is called; if I pass a thin wrapper over int to os.open, __int__ is not
called.

So the real question:  anywhere in Python where an int is expected (for lower-level API work), but not directly
received, should __int__ (or __index__) be called?  and failure to do so is a bug?

Here's my simple test code:

class Wrap:
  def __init__(self, value):
    self.value = value
  def __int__(self):
    print('__int__')
    return self.value
  def __index__(self):
    print('__index__')
    return self.value


class Thin(int):
    def __int__(self):
        print('__int__')
        return super().__int__()
    def __index__(self):
        print('__index__')
        return super().__index__()

two = Wrap(2)

[0, 1, 2][two]
# __index__
# 2

import struct
struct.pack('i', two)
# __index__
# b'\x02\x00\x00\x00'

t = Thin(1)
huh = os.open('blah.txt', t)
# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
# FileNotFoundError: [Errno 2] No such file or directory: 'blah.txt'

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150308/53685c73/attachment.sig>

From steve at pearwood.info  Mon Mar  9 05:12:44 2015
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 9 Mar 2015 15:12:44 +1100
Subject: [Python-Dev] boxing and unboxing data types
In-Reply-To: <54FD1412.3020002@stoneleaf.us>
References: <54FD1412.3020002@stoneleaf.us>
Message-ID: <20150309041243.GS7655@ando.pearwood.info>

On Sun, Mar 08, 2015 at 08:31:30PM -0700, Ethan Furman wrote:

> When data is passed from Python to a native library (such as in an O/S 
> call), how does the unboxing of data types occur?
[...]
> So the real question: anywhere in Python where an int is expected (for 
> lower-level API work), but not directly received, should __int__ (or 
> __index__) be called?  and failure to do so is a bug?

I think the answer is in the docs:

https://docs.python.org/3/reference/datamodel.html#object.__int__

Immediately below that __index__ is described, with this note:

    In order to have a coherent integer type class, when 
    __index__() is defined __int__() should also be defined, 
    and both should return the same value.


The PEP adding __index__ is also useful:

https://www.python.org/dev/peps/pep-0357/


My summary is as follows:

__int__ is used as the special method for int(), and it should coerce 
the object to an integer. This may be lossy e.g. int(2.999) --> 2 or may 
involve a conversion from a non-numeric type to integer e.g. int("2").

__index__ is used when the object in question actually represents an 
integer of some kind, e.g. a fixed-with integer. Conversion should be 
lossless and conceptually may be thought of a way of telling Python 
"this value actually is an int, even though it doesn't inherit from int" 
(for some definition of "is an int").

There's no built-in way of calling __index__ that I know of (no 
equivalent to int(obj)), but slicing at the very least will call it, 
e.g. seq[a:] will call type(a).__index__.

If you define __index__ for your class, you should also define __int__ 
and have the two return the same value. I would expect that an IntFlags 
object should inherit from int, and if that is not possible, practical 
or desirable for some reason, then it should define __index__ and 
__int__.

Failure to call __index__ is not necessarily a bug. I think it is 
allowed for functions to insist on an actual int, as slicing did for 
many years, but it is an obvious enhancement to allow such functions to 
accept arbitrary int-like objects.

Does that answer your questions?



-- 
Steve

From njs at pobox.com  Mon Mar  9 05:17:52 2015
From: njs at pobox.com (Nathaniel Smith)
Date: Sun, 8 Mar 2015 21:17:52 -0700
Subject: [Python-Dev] boxing and unboxing data types
In-Reply-To: <20150309041243.GS7655@ando.pearwood.info>
References: <54FD1412.3020002@stoneleaf.us>
 <20150309041243.GS7655@ando.pearwood.info>
Message-ID: <CAPJVwBnrWBtZo55wiYF7C5onRMZvuncACxX7ZYKxGq6O0x43jw@mail.gmail.com>

On Mar 8, 2015 9:13 PM, "Steven D'Aprano" <steve at pearwood.info> wrote:
>
> There's no built-in way of calling __index__ that I know of (no
> equivalent to int(obj)),

There's operator.index(obj), at least.

> but slicing at the very least will call it,
> e.g. seq[a:] will call type(a).__index__.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150308/5f07c12b/attachment.html>

From ethan at stoneleaf.us  Mon Mar  9 05:33:00 2015
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sun, 08 Mar 2015 21:33:00 -0700
Subject: [Python-Dev] boxing and unboxing data types
In-Reply-To: <20150309041243.GS7655@ando.pearwood.info>
References: <54FD1412.3020002@stoneleaf.us>
 <20150309041243.GS7655@ando.pearwood.info>
Message-ID: <54FD227C.5080807@stoneleaf.us>

On 03/08/2015 09:12 PM, Steven D'Aprano wrote:

> Does that answer your questions?

No, unfortunately.  You correctly guessed my question is motivated by the IntFlag discussion.

I guess it could boil down to:  if IntEnum was not based on 'int', but instead had the __int__ and __index__ methods
(plus all the other __xxx__ methods that int has), would it still be a drop-in replacement for actual ints?  Even when
being used to talk to non-Python libs?

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150308/ab6ea129/attachment.sig>

From storchaka at gmail.com  Mon Mar  9 07:07:28 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 09 Mar 2015 08:07:28 +0200
Subject: [Python-Dev] boxing and unboxing data types
In-Reply-To: <54FD227C.5080807@stoneleaf.us>
References: <54FD1412.3020002@stoneleaf.us>
 <20150309041243.GS7655@ando.pearwood.info> <54FD227C.5080807@stoneleaf.us>
Message-ID: <mdjdb1$q2q$1@ger.gmane.org>

On 09.03.15 06:33, Ethan Furman wrote:
> I guess it could boil down to:  if IntEnum was not based on 'int', but instead had the __int__ and __index__ methods
> (plus all the other __xxx__ methods that int has), would it still be a drop-in replacement for actual ints?  Even when
> being used to talk to non-Python libs?

If you don't call isinstance(x, int) (PyLong_Check* in C).

Most conversions from Python to C implicitly call __index__ or __int__, 
but unfortunately not all.

 >>> float(Thin(42))
42.0
 >>> float(Wrap(42))
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
TypeError: float() argument must be a string or a number, not 'Wrap'

 >>> '%*s' % (Thin(5), 'x')
'    x'
 >>> '%*s' % (Wrap(5), 'x')
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
TypeError: * wants int

 >>> OSError(Thin(2), 'No such file or directory')
FileNotFoundError(2, 'No such file or directory')
 >>> OSError(Wrap(2), 'No such file or directory')
OSError(<__main__.Wrap object at 0xb6fe81ac>, 'No such file or directory')

 >>> re.match('(x)', 'x').group(Thin(1))
'x'
 >>> re.match('(x)', 'x').group(Wrap(1))
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
IndexError: no such group

And to be ideal drop-in replacement IntEnum should override such methods 
as __eq__ and __hash__ (so it could be used as mapping key). If all 
methods should be overridden to quack as int, why not take an int?



From ethan at stoneleaf.us  Mon Mar  9 07:12:02 2015
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sun, 08 Mar 2015 23:12:02 -0700
Subject: [Python-Dev] boxing and unboxing data types
In-Reply-To: <mdjdb1$q2q$1@ger.gmane.org>
References: <54FD1412.3020002@stoneleaf.us>
 <20150309041243.GS7655@ando.pearwood.info> <54FD227C.5080807@stoneleaf.us>
 <mdjdb1$q2q$1@ger.gmane.org>
Message-ID: <54FD39B2.5000603@stoneleaf.us>

On 03/08/2015 11:07 PM, Serhiy Storchaka wrote:

> If you don't call isinstance(x, int) (PyLong_Check* in C).
> 
> Most conversions from Python to C implicitly call __index__ or __int__, but unfortunately not all.

[snip examples]

Thanks, Serhiy, that's what I was looking for.

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150308/a5c56fca/attachment.sig>

From tjreedy at udel.edu  Mon Mar  9 07:49:09 2015
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 09 Mar 2015 02:49:09 -0400
Subject: [Python-Dev] Tunning binary insertion sort algorithm in Timsort.
In-Reply-To: <CAMLcfj3usoq-new2gxh_ymP+1wv6XZj7+og6XMVAUj_sYeT+Pg@mail.gmail.com>
References: <CAMLcfj3usoq-new2gxh_ymP+1wv6XZj7+og6XMVAUj_sYeT+Pg@mail.gmail.com>
Message-ID: <mdjfp5$svr$1@ger.gmane.org>

On 3/8/2015 8:17 PM, nha pham wrote:
> We can optimize the TimSort algorithm by optimizing its binary insertion
> sort.
>
> The current version of binary insertion sort use this idea:
>
> Use binary search to find a final position in sorted list for a new
> element X. Then insert X to that location.
>
> I suggest another idea:
>
> Use binary search to find a final postion in sorted list for a new
> element X. Before insert X to that location, compare X with its next
> element.

> For the next element, we already know if it is lower or bigger than X,
> so we can reduce the search area to the left side or on the right side
> of X in the sorted list.
>
> I have applied my idea on java.util. ComparableTimSort.sort() and
> testing. The execute time is reduced by 2%-6% with array of random integer.
>
> Here is detail about algorithm and testing:
> https://github.com/nhapq/Optimize_binary_insertion_sort

Apply the idea to timsort in python, and try the different cases 
discussed in Tim's documentation.  If it still looks good, open an issue 
with the patch and put tim.peters as nosy.

-- 
Terry Jan Reedy


From storchaka at gmail.com  Mon Mar  9 08:13:57 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 09 Mar 2015 09:13:57 +0200
Subject: [Python-Dev] boxing and unboxing data types
In-Reply-To: <54FD39B2.5000603@stoneleaf.us>
References: <54FD1412.3020002@stoneleaf.us>
 <20150309041243.GS7655@ando.pearwood.info> <54FD227C.5080807@stoneleaf.us>
 <mdjdb1$q2q$1@ger.gmane.org> <54FD39B2.5000603@stoneleaf.us>
Message-ID: <mdjh7l$fjq$1@ger.gmane.org>

On 09.03.15 08:12, Ethan Furman wrote:
> On 03/08/2015 11:07 PM, Serhiy Storchaka wrote:
>
>> If you don't call isinstance(x, int) (PyLong_Check* in C).
>>
>> Most conversions from Python to C implicitly call __index__ or __int__, but unfortunately not all.
>
> [snip examples]
>
> Thanks, Serhiy, that's what I was looking for.

May be most if not all of these examples can be considered as bugs and 
slowly fixed, but we can't control third-party code.



From fijall at gmail.com  Mon Mar  9 09:19:50 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 9 Mar 2015 10:19:50 +0200
Subject: [Python-Dev] boxing and unboxing data types
In-Reply-To: <mdjdb1$q2q$1@ger.gmane.org>
References: <54FD1412.3020002@stoneleaf.us>
 <20150309041243.GS7655@ando.pearwood.info>
 <54FD227C.5080807@stoneleaf.us> <mdjdb1$q2q$1@ger.gmane.org>
Message-ID: <CAK5idxQFkxVH7Y8su1nnhG9u098zxLXgrVmna6DZr+Ab-cn+og@mail.gmail.com>

Not all your examples are good.

* float(x) calls __float__ (not __int__)

* re.group requires __eq__ (and __hash__)

* I'm unsure about OSError

* the % thing at the very least works on pypy

On Mon, Mar 9, 2015 at 8:07 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> On 09.03.15 06:33, Ethan Furman wrote:
>>
>> I guess it could boil down to:  if IntEnum was not based on 'int', but
>> instead had the __int__ and __index__ methods
>> (plus all the other __xxx__ methods that int has), would it still be a
>> drop-in replacement for actual ints?  Even when
>> being used to talk to non-Python libs?
>
>
> If you don't call isinstance(x, int) (PyLong_Check* in C).
>
> Most conversions from Python to C implicitly call __index__ or __int__, but
> unfortunately not all.
>
>>>> float(Thin(42))
> 42.0
>>>> float(Wrap(42))
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> TypeError: float() argument must be a string or a number, not 'Wrap'
>
>>>> '%*s' % (Thin(5), 'x')
> '    x'
>>>> '%*s' % (Wrap(5), 'x')
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> TypeError: * wants int
>
>>>> OSError(Thin(2), 'No such file or directory')
> FileNotFoundError(2, 'No such file or directory')
>>>> OSError(Wrap(2), 'No such file or directory')
> OSError(<__main__.Wrap object at 0xb6fe81ac>, 'No such file or directory')
>
>>>> re.match('(x)', 'x').group(Thin(1))
> 'x'
>>>> re.match('(x)', 'x').group(Wrap(1))
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> IndexError: no such group
>
> And to be ideal drop-in replacement IntEnum should override such methods as
> __eq__ and __hash__ (so it could be used as mapping key). If all methods
> should be overridden to quack as int, why not take an int?
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com

From solipsis at pitrou.net  Mon Mar  9 09:45:27 2015
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 9 Mar 2015 09:45:27 +0100
Subject: [Python-Dev] boxing and unboxing data types
References: <54FD1412.3020002@stoneleaf.us>
 <20150309041243.GS7655@ando.pearwood.info>
Message-ID: <20150309094527.135aac2d@fsol>

On Mon, 9 Mar 2015 15:12:44 +1100
Steven D'Aprano <steve at pearwood.info> wrote:
> 
> My summary is as follows:
> 
> __int__ is used as the special method for int(), and it should coerce 
> the object to an integer. This may be lossy e.g. int(2.999) --> 2 or may 
> involve a conversion from a non-numeric type to integer e.g. int("2").

Your example is misleading. Strings don't have an __int__:

>>> s = "3"
>>> s.__int__()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'str' object has no attribute '__int__'

Only int-compatible or int-coercible types (e.g. float, Decimal) should
have an __int__ method.

Regards

Antoine.



From storchaka at gmail.com  Mon Mar  9 10:10:09 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 09 Mar 2015 11:10:09 +0200
Subject: [Python-Dev] boxing and unboxing data types
In-Reply-To: <CAK5idxQFkxVH7Y8su1nnhG9u098zxLXgrVmna6DZr+Ab-cn+og@mail.gmail.com>
References: <54FD1412.3020002@stoneleaf.us>
 <20150309041243.GS7655@ando.pearwood.info> <54FD227C.5080807@stoneleaf.us>
 <mdjdb1$q2q$1@ger.gmane.org>
 <CAK5idxQFkxVH7Y8su1nnhG9u098zxLXgrVmna6DZr+Ab-cn+og@mail.gmail.com>
Message-ID: <mdjo1h$tba$1@ger.gmane.org>

On 09.03.15 10:19, Maciej Fijalkowski wrote:
> Not all your examples are good.
>
> * float(x) calls __float__ (not __int__)
>
> * re.group requires __eq__ (and __hash__)
>
> * I'm unsure about OSError
>
> * the % thing at the very least works on pypy

Yes, all these examples are implementation defined and can differ 
between CPython and PyPy. There is about a dozen of similar examples 
only in C part of CPython. Most of them have in common is that the 
behavior of the function depends on the argument type. For example in 
case of re.group an argument is either integer index or string group 
name. OSError constructor can produce OSError subtype if first argument 
is known integer errno. float either convert a number to float or parse 
a string (or bytes).

Python functions can be more lenient (if they allows ducktyping) or more 
strict (if they explicitly check the type). They rarely call __index__ 
or __int__.


From larry at hastings.org  Mon Mar  9 10:31:06 2015
From: larry at hastings.org (Larry Hastings)
Date: Mon, 09 Mar 2015 02:31:06 -0700
Subject: [Python-Dev] PEP 471 Final: os.scandir() merged into Python 3.5
In-Reply-To: <CAMpsgwYg=RZeyqTzLGn+Nr7xEiiff0EUDhEUgGS-So=B-DjJ1g@mail.gmail.com>
References: <CAMpsgwYg=RZeyqTzLGn+Nr7xEiiff0EUDhEUgGS-So=B-DjJ1g@mail.gmail.com>
Message-ID: <54FD685A.7030004@hastings.org>

On 03/07/2015 06:13 PM, Victor Stinner wrote:
> Hi,
>
> FYI I commited the implementation of os.scandir() written by Ben Hoyt.
> I hope that it will be part of Python 3.5 alpha 2

It is.


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150309/c234d251/attachment-0001.html>

From larry at hastings.org  Mon Mar  9 10:34:50 2015
From: larry at hastings.org (Larry Hastings)
Date: Mon, 09 Mar 2015 02:34:50 -0700
Subject: [Python-Dev] [RELEASED] Python 3.5.0a2 is now available
Message-ID: <54FD693A.5010704@hastings.org>



On behalf of the Python development community and the Python 3.5 release 
team, I'm thrilled to announce the availability of Python 3.5.0a2.   
Python 3.5.0a2 is the second alpha release of Python 3.5, which will be 
the next major release of Python.  Python 3.5 is still under heavy 
development, and is far from complete.

This is a preview release, and its use is not recommended for production 
settings.

Two important notes for Windows users about Python 3.5.0a2:

  * If you have previously installed Python 3.5.0a1, you must manually
    uninstall it before installing Python 3.5.0a2 (issue23612).
  * If installing Python 3.5.0a2 as a non-privileged user, you may need
    to escalate to administrator privileges to install an update to your
    C runtime libraries.


You can find Python 3.5.0a2 here:

    https://www.python.org/downloads/release/python-350a2/


Happy hacking,


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150309/b78f2f7e/attachment.html>

From p.f.moore at gmail.com  Mon Mar  9 11:20:54 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 9 Mar 2015 10:20:54 +0000
Subject: [Python-Dev] [RELEASED] Python 3.5.0a2 is now available
In-Reply-To: <54FD693A.5010704@hastings.org>
References: <54FD693A.5010704@hastings.org>
Message-ID: <CACac1F9E2Lj_Ls9YvufNhKuqL3_x=KAbUEEMokftFgRo+_cT6Q@mail.gmail.com>

On 9 March 2015 at 09:34, Larry Hastings <larry at hastings.org> wrote:
> On behalf of the Python development community and the Python 3.5 release
> team, I'm thrilled to announce the availability of Python 3.5.0a2.   Python
> 3.5.0a2 is the second alpha release of Python 3.5, which will be the next
> major release of Python.  Python 3.5 is still under heavy development, and
> is far from complete.

Hmm, I just tried to install the 64-bit "full installer" version, for
all users with the default options. This is on a PC that hasn't had
3.5 installed before, and doesn't have Visual Studio 2015 installed.
When it got to the step "precompiling standard library" I got an error
window pop up saying "python.exe - system error. The program can't
start because api-ms-win-crt-runtime-l1-1-0.dll is missing from your
computer. Try reinstalling the program to fix this problem." All there
was was an "OK" button. Pressing that told me "Setup was successful"
but then "py -3.5 -V" gives me nothing (no error, no version, just
returns me to the command prompt). Same result if I do "& 'C:\Program
Files\Python 3.5\python.exe' -V".

Python 3.5.0a2 (64-bit) is showing in my "Add/Remove Programs".

This is Windows 7, 64-bit.

Paul

From p.f.moore at gmail.com  Mon Mar  9 11:23:58 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 9 Mar 2015 10:23:58 +0000
Subject: [Python-Dev] [RELEASED] Python 3.5.0a2 is now available
In-Reply-To: <CACac1F9E2Lj_Ls9YvufNhKuqL3_x=KAbUEEMokftFgRo+_cT6Q@mail.gmail.com>
References: <54FD693A.5010704@hastings.org>
 <CACac1F9E2Lj_Ls9YvufNhKuqL3_x=KAbUEEMokftFgRo+_cT6Q@mail.gmail.com>
Message-ID: <CACac1F9+efYAh2FhbOVjfJ3qv+kemW-EvXyv_Nc6pa2_eyttYw@mail.gmail.com>

Submitted as http://bugs.python.org/issue23619

On 9 March 2015 at 10:20, Paul Moore <p.f.moore at gmail.com> wrote:
> On 9 March 2015 at 09:34, Larry Hastings <larry at hastings.org> wrote:
>> On behalf of the Python development community and the Python 3.5 release
>> team, I'm thrilled to announce the availability of Python 3.5.0a2.   Python
>> 3.5.0a2 is the second alpha release of Python 3.5, which will be the next
>> major release of Python.  Python 3.5 is still under heavy development, and
>> is far from complete.
>
> Hmm, I just tried to install the 64-bit "full installer" version, for
> all users with the default options. This is on a PC that hasn't had
> 3.5 installed before, and doesn't have Visual Studio 2015 installed.
> When it got to the step "precompiling standard library" I got an error
> window pop up saying "python.exe - system error. The program can't
> start because api-ms-win-crt-runtime-l1-1-0.dll is missing from your
> computer. Try reinstalling the program to fix this problem." All there
> was was an "OK" button. Pressing that told me "Setup was successful"
> but then "py -3.5 -V" gives me nothing (no error, no version, just
> returns me to the command prompt). Same result if I do "& 'C:\Program
> Files\Python 3.5\python.exe' -V".
>
> Python 3.5.0a2 (64-bit) is showing in my "Add/Remove Programs".
>
> This is Windows 7, 64-bit.
>
> Paul

From p.f.moore at gmail.com  Mon Mar  9 12:19:18 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 9 Mar 2015 11:19:18 +0000
Subject: [Python-Dev] Thoughts on running Python 3.5 on Windows (path,
	pip install --user, etc)
Message-ID: <CACac1F9WjM4Sih96A_2oK=jyMo7VQe-PBQQN_tH9m0NmC-Y6Rg@mail.gmail.com>

I just thought I'd give installing Python 3.5 a go on my PC, now that
3.5a2 has come out. I didn't get very far (see earlier message) but it
prompted me to think about how I'd use it, and what habits I'd need to
change.

I'd suggest that the "what's new in 3.5" document probably needs a
section on the new installer that explains this stuff...

First of all, I always use "all users" installs, so I have Python in
"Program Files" now. As a result, doing "pip install foo" requires
elevation. As that's a PITA, I probably need to switch to using "pip
install --user". All that's fine, and from there "py -3.5" works fine,
as does "py -3.5 -m foo". But even if it is, not every entry point has
a corresponding "-m" invocation (pygments' pygmentize command doesn't
seem to, for example)

But suppose I want to put Python 3.5 on my PATH. The installer has an
"add Python to PATH" checkbox, but (if I'm reading the WiX source
right, I didn't select that on install) that doesn't add the user
directory. So I need to add that to my PATH. Is that right? And of
course, that means I need to *know* the user site directory
($env:LOCALAPPDATA\Python\Python35\Scripts), correct?

It feels to me like this might be a frustrating step backwards for
Windows users who have recently (with the arrival of ensurepip) got to
the point where they can just run Python with "Add to path" and then
simply do

    pip install pygments
    pygmentize --help

Maybe the answer is that we simply start recommending that everyone on
Windows uses per-user installs. It makes little difference to me
(beyond the fact that when I want to look at the source of something
in the stdlib, the location of the file is a lot harder to remember
than C:\Apps\Python34\Lib\whatever.py) but I doubt it's what most
people will expect.

I'm completely OK with the idea that we move to "Program Files" as a
default location. And I have no real issue with Steve's position that
the "Add to Path" option has issues that can't really be solved
because of the way Windows constructs the PATH. But I know there have
been a lot of people frustrated by the complicated instructions needed
to get something like pygmentize working, for whom getting pip in 2.7
and 3.4 was a major improvement. So I think we need *some*
documentation helping them deal with what could well seem like a step
backwards in Python 3.5...

If I knew what the best (recommended) answer was, I'd be happy to
write it up. But I'm not sure I do (after all, the target audience is
people for whom "Add C:\PythonXY\Scripts to your PATH" was a problem
in the pre-3.4 days).

Should I raise this as a bug against the 3.5 documentation? If so,
should it be a release blocker for the final release?

Paul

From benhoyt at gmail.com  Mon Mar  9 12:58:26 2015
From: benhoyt at gmail.com (Ben Hoyt)
Date: Mon, 9 Mar 2015 07:58:26 -0400
Subject: [Python-Dev] PEP 471 Final: os.scandir() merged into Python 3.5
In-Reply-To: <CAM8kh5tL4QgJa4SaaxRsGzgV4RVpZxpdBC0VZ1f1ZKth4nF7HQ@mail.gmail.com>
References: <CAMpsgwYg=RZeyqTzLGn+Nr7xEiiff0EUDhEUgGS-So=B-DjJ1g@mail.gmail.com>
 <CAL9jXCEUOkR_EvGzA6kdodTk3S6+AfnWpRkTmJ27ZRtUaXyAYw@mail.gmail.com>
 <CAM8kh5tL4QgJa4SaaxRsGzgV4RVpZxpdBC0VZ1f1ZKth4nF7HQ@mail.gmail.com>
Message-ID: <CAL9jXCHrcvH48QhgCdZWTNUoLbSdJkH9QMOEHrnRPs==hTzxaw@mail.gmail.com>

Hi Ryan,

> ./configure --with-pydebug && make -j7
>
> I then ran ./python.exe ~/Workspace/python/scandir/benchmark.py and I got:
>
> Creating tree at /Users/rstuart/Workspace/python/scandir/benchtree: depth=4, num_dirs=5, num_files=50
> Using slower ctypes version of scandir
> Comparing against builtin version of os.walk()
> Priming the system's cache...
> Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree, repeat 1/3...
> Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree, repeat 2/3...
> Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree, repeat 3/3...
> os.walk took 0.184s, scandir.walk took 0.158s -- 1.2x as fast

Note that this benchmark is invalid for a couple of reasons. First,
you're compiling Python in debug mode (--with-pydebug), which produces
significantly slower code in my tests -- for example, on Windows
benchmark.py is about twice as slow when Python is compiled in debug
mode.

Second, as the output above shows, benchmark.py is "Using slower
ctypes version of scandir" and not a C version at all. If os.scandir()
is available, benchmark.py should use that, so there's something wrong
here -- maybe the patch didn't apply correctly or maybe you're testing
with a different version of Python than the one you built?

In any case, the easiest way to test it now is to download Python 3.5
alpha 2 which just came out:
https://www.python.org/downloads/release/python-350a2/

I just tried this on my Mac Mini (i5 2.3GHz, 2 GB RAM, HFS+ on
rotational drive) and got the following results:

Using Python 3.5's builtin os.scandir()
Comparing against builtin version of os.walk()
Priming the system's cache...
Benchmarking walks on benchtree, repeat 1/3...
Benchmarking walks on benchtree, repeat 2/3...
Benchmarking walks on benchtree, repeat 3/3...
os.walk took 0.074s, scandir.walk took 0.016s -- 4.7x as fast

> I then did ./python.exe ~/Workspace/python/scandir/benchmark.py -s and got:

Also note that "benchmark.py -s" tests the system os.walk() against a
get_tree_size() function using scandir's DirEntry.stat().st_size,
which provides huge gains on Windows (because stat().st_size doesn't
require on OS call) but only modest gains on POSIX systems, which
still require an OS stat call to get the size (though not the file
type, so at least it's only one stat call). I get "2.2x as fast" on my
Mac for "benchmark.py -s".

-Ben

From benhoyt at gmail.com  Mon Mar  9 13:29:24 2015
From: benhoyt at gmail.com (Ben Hoyt)
Date: Mon, 9 Mar 2015 08:29:24 -0400
Subject: [Python-Dev] [RELEASED] Python 3.5.0a2 is now available
In-Reply-To: <CACac1F9E2Lj_Ls9YvufNhKuqL3_x=KAbUEEMokftFgRo+_cT6Q@mail.gmail.com>
References: <54FD693A.5010704@hastings.org>
 <CACac1F9E2Lj_Ls9YvufNhKuqL3_x=KAbUEEMokftFgRo+_cT6Q@mail.gmail.com>
Message-ID: <CAL9jXCEzQPd9sF3hdAup5vNFs69xn0jSe+eAJtbL4m5JZrWkKg@mail.gmail.com>

I'm seeing the same issue (though I also get the missing-DLL error dialog
when I run python.exe). -Ben

On Mon, Mar 9, 2015 at 6:20 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 9 March 2015 at 09:34, Larry Hastings <larry at hastings.org> wrote:
> > On behalf of the Python development community and the Python 3.5 release
> > team, I'm thrilled to announce the availability of Python 3.5.0a2.
>  Python
> > 3.5.0a2 is the second alpha release of Python 3.5, which will be the next
> > major release of Python.  Python 3.5 is still under heavy development,
> and
> > is far from complete.
>
> Hmm, I just tried to install the 64-bit "full installer" version, for
> all users with the default options. This is on a PC that hasn't had
> 3.5 installed before, and doesn't have Visual Studio 2015 installed.
> When it got to the step "precompiling standard library" I got an error
> window pop up saying "python.exe - system error. The program can't
> start because api-ms-win-crt-runtime-l1-1-0.dll is missing from your
> computer. Try reinstalling the program to fix this problem." All there
> was was an "OK" button. Pressing that told me "Setup was successful"
> but then "py -3.5 -V" gives me nothing (no error, no version, just
> returns me to the command prompt). Same result if I do "& 'C:\Program
> Files\Python 3.5\python.exe' -V".
>
> Python 3.5.0a2 (64-bit) is showing in my "Add/Remove Programs".
>
> This is Windows 7, 64-bit.
>
> Paul
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/benhoyt%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150309/3f66944a/attachment.html>

From Steve.Dower at microsoft.com  Mon Mar  9 14:14:01 2015
From: Steve.Dower at microsoft.com (Steve Dower)
Date: Mon, 9 Mar 2015 13:14:01 +0000
Subject: [Python-Dev] [RELEASED] Python 3.5.0a2 is now available
In-Reply-To: <CAL9jXCEzQPd9sF3hdAup5vNFs69xn0jSe+eAJtbL4m5JZrWkKg@mail.gmail.com>
References: <54FD693A.5010704@hastings.org>
 <CACac1F9E2Lj_Ls9YvufNhKuqL3_x=KAbUEEMokftFgRo+_cT6Q@mail.gmail.com>,
 <CAL9jXCEzQPd9sF3hdAup5vNFs69xn0jSe+eAJtbL4m5JZrWkKg@mail.gmail.com>
Message-ID: <DM2PR0301MB0734369D059C276502111677F51B0@DM2PR0301MB0734.namprd03.prod.outlook.com>

Thanks for finding this. I'm following up on the issue of anyone else is having the same issue.

As an aside, it'd be great to hear if it's worked for anyone at all :)

Cheers,
Steve

Top-posted from my Windows Phone
________________________________
From: Ben Hoyt<mailto:benhoyt at gmail.com>
Sent: ?3/?9/?2015 5:29
To: Paul Moore<mailto:p.f.moore at gmail.com>
Cc: Python Dev<mailto:python-dev at python.org>
Subject: Re: [Python-Dev] [RELEASED] Python 3.5.0a2 is now available

I'm seeing the same issue (though I also get the missing-DLL error dialog when I run python.exe). -Ben

On Mon, Mar 9, 2015 at 6:20 AM, Paul Moore <p.f.moore at gmail.com<mailto:p.f.moore at gmail.com>> wrote:
On 9 March 2015 at 09:34, Larry Hastings <larry at hastings.org<mailto:larry at hastings.org>> wrote:
> On behalf of the Python development community and the Python 3.5 release
> team, I'm thrilled to announce the availability of Python 3.5.0a2.   Python
> 3.5.0a2 is the second alpha release of Python 3.5, which will be the next
> major release of Python.  Python 3.5 is still under heavy development, and
> is far from complete.

Hmm, I just tried to install the 64-bit "full installer" version, for
all users with the default options. This is on a PC that hasn't had
3.5 installed before, and doesn't have Visual Studio 2015 installed.
When it got to the step "precompiling standard library" I got an error
window pop up saying "python.exe - system error. The program can't
start because api-ms-win-crt-runtime-l1-1-0.dll is missing from your
computer. Try reinstalling the program to fix this problem." All there
was was an "OK" button. Pressing that told me "Setup was successful"
but then "py -3.5 -V" gives me nothing (no error, no version, just
returns me to the command prompt). Same result if I do "& 'C:\Program
Files\Python 3.5\python.exe' -V".

Python 3.5.0a2 (64-bit) is showing in my "Add/Remove Programs".

This is Windows 7, 64-bit.

Paul
_______________________________________________
Python-Dev mailing list
Python-Dev at python.org<mailto:Python-Dev at python.org>
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/benhoyt%40gmail.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150309/d21a035a/attachment.html>

From ncoghlan at gmail.com  Mon Mar  9 14:45:42 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 9 Mar 2015 23:45:42 +1000
Subject: [Python-Dev] Thoughts on running Python 3.5 on Windows (path,
 pip install --user, etc)
In-Reply-To: <CACac1F9WjM4Sih96A_2oK=jyMo7VQe-PBQQN_tH9m0NmC-Y6Rg@mail.gmail.com>
References: <CACac1F9WjM4Sih96A_2oK=jyMo7VQe-PBQQN_tH9m0NmC-Y6Rg@mail.gmail.com>
Message-ID: <CADiSq7eW5MX7w7un67aZ9MNe3f=_GzcyOd=VQEAQfNjA55F8hQ@mail.gmail.com>

On 9 March 2015 at 21:19, Paul Moore <p.f.moore at gmail.com> wrote:
> Should I raise this as a bug against the 3.5 documentation? If so,
> should it be a release blocker for the final release?

I'm happy to let the folks that use Windows for development regularly
decide on the best answer from a user experience perspective, but I
think a release blocker docs issue would be a good way to go for
ensuring it gets resolved be the release goes out.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From rmsr at lab.net  Mon Mar  9 06:57:30 2015
From: rmsr at lab.net (Ryan Smith-Roberts)
Date: Sun, 8 Mar 2015 22:57:30 -0700
Subject: [Python-Dev] Tunning binary insertion sort algorithm in Timsort.
In-Reply-To: <CAMLcfj3usoq-new2gxh_ymP+1wv6XZj7+og6XMVAUj_sYeT+Pg@mail.gmail.com>
References: <CAMLcfj3usoq-new2gxh_ymP+1wv6XZj7+og6XMVAUj_sYeT+Pg@mail.gmail.com>
Message-ID: <CABPWfCHou2N3pZ5Ltgv_ybAV485nFzYbCEKPtwbgj4qb5_f7gA@mail.gmail.com>

I suspect that you will find the Python community extremely conservative
about any changes to its sorting algorithm, given that it took thirteen
years and some really impressive automated verification software to find
this bug:

http://envisage-project.eu/proving-android-java-and-python-sorting-algorithm-is-broken-and-how-to-fix-it/

On Sun, Mar 8, 2015 at 5:17 PM, nha pham <phqnha at gmail.com> wrote:

> We can optimize the TimSort algorithm by optimizing its binary insertion
> sort.
>
>
>
> The current version of binary insertion sort use this idea:
>
> Use binary search to find a final position in sorted list for a new
> element X. Then insert X to that location.
>
>
>
> I suggest another idea:
>
> Use binary search to find a final postion in sorted list for a new element
> X. Before insert X to that location, compare X with its next element.
>
> For the next element, we already know if it is lower or bigger than X, so
> we can reduce the search area to the left side or on the right side of X in
> the sorted list.
>
>
>
> I have applied my idea on java.util. ComparableTimSort.sort() and testing.
> The execute time is reduced by 2%-6% with array of random integer.
>
>
>
> Here is detail about algorithm and testing:
> https://github.com/nhapq/Optimize_binary_insertion_sort
>
>
>
> Sincerely.
>
> phqnha
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/rmsr%40lab.net
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150308/335c9e79/attachment.html>

From mistersheik at gmail.com  Mon Mar  9 14:52:01 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Mon, 9 Mar 2015 09:52:01 -0400
Subject: [Python-Dev] boxing and unboxing data types
In-Reply-To: <mdjdb1$q2q$1@ger.gmane.org>
References: <54FD1412.3020002@stoneleaf.us>
 <20150309041243.GS7655@ando.pearwood.info>
 <54FD227C.5080807@stoneleaf.us> <mdjdb1$q2q$1@ger.gmane.org>
Message-ID: <CAA68w_nGXamqcYtxDbcXmCa2bRqAMtPDZO+pbmE=Fa0G1dbuDA@mail.gmail.com>

On Mon, Mar 9, 2015 at 2:07 AM, Serhiy Storchaka <storchaka at gmail.com>
wrote:

> On 09.03.15 06:33, Ethan Furman wrote:
>
>> I guess it could boil down to:  if IntEnum was not based on 'int', but
>> instead had the __int__ and __index__ methods
>> (plus all the other __xxx__ methods that int has), would it still be a
>> drop-in replacement for actual ints?  Even when
>> being used to talk to non-Python libs?
>>
>
> If you don't call isinstance(x, int) (PyLong_Check* in C).
>
> Most conversions from Python to C implicitly call __index__ or __int__,
> but unfortunately not all.
>
> >>> float(Thin(42))
> 42.0
> >>> float(Wrap(42))
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> TypeError: float() argument must be a string or a number, not 'Wrap'
>
> >>> '%*s' % (Thin(5), 'x')
> '    x'
> >>> '%*s' % (Wrap(5), 'x')
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> TypeError: * wants int
>
> >>> OSError(Thin(2), 'No such file or directory')
> FileNotFoundError(2, 'No such file or directory')
> >>> OSError(Wrap(2), 'No such file or directory')
> OSError(<__main__.Wrap object at 0xb6fe81ac>, 'No such file or directory')
>
> >>> re.match('(x)', 'x').group(Thin(1))
> 'x'
> >>> re.match('(x)', 'x').group(Wrap(1))
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> IndexError: no such group
>
> And to be ideal drop-in replacement IntEnum should override such methods
> as __eq__ and __hash__ (so it could be used as mapping key). If all methods
> should be overridden to quack as int, why not take an int?
>
>
You're absolutely right that if *all the methods should be overrriden to
quack as int, then you should subclass int (the Liskov substitution
principle).  But all methods should not be overridden ? mainly the methods
you overrode in your patch should be exposed.  Here is a list of methods on
int that should not be on IntFlags in my opinion (give or take a couple):

__abs__, __add__, __delattr__, __divmod__, __float__, __floor__,
__floordiv__, __index__, __lshift__, __mod__, __mul__, __pos__, __pow__,
__radd__, __rdivmod__, __rfloordiv__, __rlshift__, __rmod__, __rmul__,
__round__, __rpow__, __rrshift__, __rshift__, __rsub__, __rtruediv__,
__sub__, __truediv__, __trunc__, conjugate, denominator, imag, numerator,
real.

I don't think __index__ should be exposed either since are you really going
to slice a list using IntFlags?  Really?


>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> mistersheik%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150309/5010b6b6/attachment-0001.html>

From p.f.moore at gmail.com  Mon Mar  9 15:14:51 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 9 Mar 2015 14:14:51 +0000
Subject: [Python-Dev] Thoughts on running Python 3.5 on Windows (path,
 pip install --user, etc)
In-Reply-To: <CADiSq7eW5MX7w7un67aZ9MNe3f=_GzcyOd=VQEAQfNjA55F8hQ@mail.gmail.com>
References: <CACac1F9WjM4Sih96A_2oK=jyMo7VQe-PBQQN_tH9m0NmC-Y6Rg@mail.gmail.com>
 <CADiSq7eW5MX7w7un67aZ9MNe3f=_GzcyOd=VQEAQfNjA55F8hQ@mail.gmail.com>
Message-ID: <CACac1F_f_JKpyPgEw35aWJaTCX=knfmtxCaL8ovB=nuOYW90bw@mail.gmail.com>

On 9 March 2015 at 13:45, Nick Coghlan <ncoghlan at gmail.com> wrote:
> I'm happy to let the folks that use Windows for development regularly
> decide on the best answer from a user experience perspective, but I
> think a release blocker docs issue would be a good way to go for
> ensuring it gets resolved be the release goes out.

Done. http://bugs.python.org/issue23623

From storchaka at gmail.com  Mon Mar  9 16:11:50 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 09 Mar 2015 17:11:50 +0200
Subject: [Python-Dev] boxing and unboxing data types
In-Reply-To: <CAA68w_nVh8FWzwJdmRneix50fTPVsBmSKf5auNwMKBr6XgMLBw@mail.gmail.com>
References: <54FD1412.3020002@stoneleaf.us> <2847718.KSDy3zi7qt@raxxla>
 <CAA68w_nVh8FWzwJdmRneix50fTPVsBmSKf5auNwMKBr6XgMLBw@mail.gmail.com>
Message-ID: <3487932.K7gDH0h8FU@raxxla>

?????????, 09-???-2015 10:18:50 ?? ????????:
> On Mon, Mar 9, 2015 at 10:10 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> > ?????????, 09-???-2015 09:52:01 ?? ????????:
> > > On Mon, Mar 9, 2015 at 2:07 AM, Serhiy Storchaka <storchaka at gmail.com>
> > > > And to be ideal drop-in replacement IntEnum should override such methods
> > > > as __eq__ and __hash__ (so it could be used as mapping key). If all methods
> > > > should be overridden to quack as int, why not take an int?
> > > 
> > > You're absolutely right that if *all the methods should be overrriden to
> > > quack as int, then you should subclass int (the Liskov substitution
> > > principle).  But all methods should not be overridden ? mainly the methods
> > > you overrode in your patch should be exposed.  Here is a list of methods on
> > > int that should not be on IntFlags in my opinion (give or take a couple):
> > > 
> > > __abs__, __add__, __delattr__, __divmod__, __float__, __floor__,
> > > __floordiv__, __index__, __lshift__, __mod__, __mul__, __pos__, __pow__,
> > > __radd__, __rdivmod__, __rfloordiv__, __rlshift__, __rmod__, __rmul__,
> > > __round__, __rpow__, __rrshift__, __rshift__, __rsub__, __rtruediv__,
> > > __sub__, __truediv__, __trunc__, conjugate, denominator, imag, numerator,
> > > real.
> > > 
> > > I don't think __index__ should be exposed either since are you really going
> > > to slice a list using IntFlags?  Really?
> > 
> > Definitely __index__ should be exposed. __int__ is for lossy conversion to int
> > (as in float). __index__ is for lossless conversion.
> 
> Is it?  __index__ promises lossless conversion, but __index__ is *for*
> indexing.

I spite of its name it is for any lossless conversion.

> > __add__ should be exposed because some code can use + instead of | for
> > combining flags. But it shouldn't preserve the type, because this is not
> > recommended way.
> 
> I think it should be blocked because it can lead to all kinds of weird
> bugs.  If the flag is already set and you add it a copy, it silently spills
> over into other flags.  This is a mistake that a good interface prevents.

I think this is a case when backward compatibility has larger weight.

> > For the same reason I think __lshift__, __rshift__, __sub__,
> > __mul__, __divmod__, __floordiv__, __mod__, etc should be exposed too. So the
> > majority of the methods should be exposed, and there is a risk that we loss
> > something.
> 
> I totally disagree with all of those.
> 
> > For good compatibility with Python code IntFlags should expose also
> > __subclasscheck__ or __subclasshook__. And when we are at this point, why not
> > use int subclass?
> 
> Here's another reason.  What if someone wants to use an IntFlags object,
> but wants to use a fixed width type for storage, say numpy.int32?   Why
> shouldn't they be able to do that?  By using composition, you can easily
> provide such an option.

You can design abstract interface Flags that can be combined with int or other type. But why you want to use numpy.int32 as storage? This doesn't save much memory, because with composition the IntFlags class weighs more than int subclass.


From Steve.Dower at microsoft.com  Mon Mar  9 16:37:23 2015
From: Steve.Dower at microsoft.com (Steve Dower)
Date: Mon, 9 Mar 2015 15:37:23 +0000
Subject: [Python-Dev] Thoughts on running Python 3.5 on Windows (path,
 pip install --user, etc)
Message-ID: <DM2PR0301MB0734EC83B986CCC91A8A0043F51B0@DM2PR0301MB0734.namprd03.prod.outlook.com>

Paul Moore wrote:
> I just thought I'd give installing Python 3.5 a go on my PC, now that
> 3.5a2 has come out. I didn't get very far (see earlier message) but it prompted
> me to think about how I'd use it, and what habits I'd need to change.
> 
> I'd suggest that the "what's new in 3.5" document probably needs a section on
> the new installer that explains this stuff...

This is true. Right now I'm in experimentation mode, and being more aggressive about changing things than is probably a good idea (because it solicits feedback like this :) ). When things settle down I expect to end up closer to where we started, so there's not a huge amount of value in writing it all up right now. I'll get there.

> First of all, I always use "all users" installs, so I have Python in "Program
> Files" now. As a result, doing "pip install foo" requires elevation. As that's a
> PITA, I probably need to switch to using "pip install --user". All that's fine,
> and from there "py -3.5" works fine, as does "py -3.5 -m foo". But even if it
> is, not every entry point has a corresponding "-m" invocation (pygments'
> pygmentize command doesn't seem to, for example)

I know you're already involved in this Paul, but for everyone else there's a big discussion going on at https://github.com/pypa/pip/issues/1668 about changing pip's default behaviour to handle falling back to --user automatically.

> But suppose I want to put Python 3.5 on my PATH. The installer has an "add
> Python to PATH" checkbox, but (if I'm reading the WiX source right, I didn't
> select that on install) that doesn't add the user directory. So I need to add
> that to my PATH. Is that right? And of course, that means I need to *know* the
> user site directory ($env:LOCALAPPDATA\Python\Python35\Scripts), correct?

Correct. There's no way to add a per-user directory to PATH from an all-users installation (except for a few approaches that I expect/hope would trigger malware detectors...)

> Maybe the answer is that we simply start recommending that everyone on Windows
> uses per-user installs. It makes little difference to me (beyond the fact that
> when I want to look at the source of something in the stdlib, the location of
> the file is a lot harder to remember than C:\Apps\Python34\Lib\whatever.py) but
> I doubt it's what most people will expect.

I'm okay with this. Installing for all users is really something that could be considered an advanced option rather than the default, especially since the aim (AIUI) of the all-users install is to pretend that Python was shipped with the OS. (I'd kind of like to take that further by splitting things more sensibly between Program Files, Common Files and System32, but there's very little gain from that and much MUCH pain as long as people are still expecting C:\PythonXY installs...)

Cheers,
Steve

From steve at pearwood.info  Mon Mar  9 16:46:59 2015
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 10 Mar 2015 02:46:59 +1100
Subject: [Python-Dev] boxing and unboxing data types
In-Reply-To: <CAA68w_nGXamqcYtxDbcXmCa2bRqAMtPDZO+pbmE=Fa0G1dbuDA@mail.gmail.com>
References: <54FD1412.3020002@stoneleaf.us>
 <20150309041243.GS7655@ando.pearwood.info> <54FD227C.5080807@stoneleaf.us>
 <mdjdb1$q2q$1@ger.gmane.org>
 <CAA68w_nGXamqcYtxDbcXmCa2bRqAMtPDZO+pbmE=Fa0G1dbuDA@mail.gmail.com>
Message-ID: <20150309154659.GU7655@ando.pearwood.info>

On Mon, Mar 09, 2015 at 09:52:01AM -0400, Neil Girdhar wrote:

> Here is a list of methods on
> int that should not be on IntFlags in my opinion (give or take a couple):
> 
> __abs__, __add__, __delattr__, __divmod__, __float__, __floor__,
> __floordiv__, __index__, __lshift__, __mod__, __mul__, __pos__, __pow__,
> __radd__, __rdivmod__, __rfloordiv__, __rlshift__, __rmod__, __rmul__,
> __round__, __rpow__, __rrshift__, __rshift__, __rsub__, __rtruediv__,
> __sub__, __truediv__, __trunc__, conjugate, denominator, imag, numerator,
> real.
> 
> I don't think __index__ should be exposed either since are you really going
> to slice a list using IntFlags?  Really?

In what way is this an *Int*Flags object if it is nothing like an int? 
It sounds like what you want is a bunch of Enum inside a set with a custom 
__str__, not IntFlags.


-- 
Steve

From mistersheik at gmail.com  Mon Mar  9 16:50:14 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Mon, 9 Mar 2015 11:50:14 -0400
Subject: [Python-Dev] boxing and unboxing data types
In-Reply-To: <20150309154659.GU7655@ando.pearwood.info>
References: <54FD1412.3020002@stoneleaf.us>
 <20150309041243.GS7655@ando.pearwood.info>
 <54FD227C.5080807@stoneleaf.us> <mdjdb1$q2q$1@ger.gmane.org>
 <CAA68w_nGXamqcYtxDbcXmCa2bRqAMtPDZO+pbmE=Fa0G1dbuDA@mail.gmail.com>
 <20150309154659.GU7655@ando.pearwood.info>
Message-ID: <CAA68w_nMkFkK+SzFtPaRLuJt-1DX_nXir5dOy-3gkDDctJsO6A@mail.gmail.com>

On Mon, Mar 9, 2015 at 11:46 AM, Steven D'Aprano <steve at pearwood.info>
wrote:

> On Mon, Mar 09, 2015 at 09:52:01AM -0400, Neil Girdhar wrote:
>
> > Here is a list of methods on
> > int that should not be on IntFlags in my opinion (give or take a couple):
> >
> > __abs__, __add__, __delattr__, __divmod__, __float__, __floor__,
> > __floordiv__, __index__, __lshift__, __mod__, __mul__, __pos__, __pow__,
> > __radd__, __rdivmod__, __rfloordiv__, __rlshift__, __rmod__, __rmul__,
> > __round__, __rpow__, __rrshift__, __rshift__, __rsub__, __rtruediv__,
> > __sub__, __truediv__, __trunc__, conjugate, denominator, imag, numerator,
> > real.
> >
> > I don't think __index__ should be exposed either since are you really
> going
> > to slice a list using IntFlags?  Really?
>
> In what way is this an *Int*Flags object if it is nothing like an int?
> It sounds like what you want is a bunch of Enum inside a set with a custom
> __str__, not IntFlags.
>
>
It doesn't matter what you call it.  I believe the goal of this is to have
a flags object with flags operations and pretty-printing.  It makes more
sense to me to decide the interface and then the implementation.

>
> --
> Steve
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/mistersheik%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150309/68f463a6/attachment.html>

From steve at pearwood.info  Mon Mar  9 16:53:33 2015
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 10 Mar 2015 02:53:33 +1100
Subject: [Python-Dev] Tunning binary insertion sort algorithm in Timsort.
In-Reply-To: <CABPWfCHou2N3pZ5Ltgv_ybAV485nFzYbCEKPtwbgj4qb5_f7gA@mail.gmail.com>
References: <CAMLcfj3usoq-new2gxh_ymP+1wv6XZj7+og6XMVAUj_sYeT+Pg@mail.gmail.com>
 <CABPWfCHou2N3pZ5Ltgv_ybAV485nFzYbCEKPtwbgj4qb5_f7gA@mail.gmail.com>
Message-ID: <20150309155329.GV7655@ando.pearwood.info>

On Sun, Mar 08, 2015 at 10:57:30PM -0700, Ryan Smith-Roberts wrote:
> I suspect that you will find the Python community extremely conservative
> about any changes to its sorting algorithm, given that it took thirteen
> years and some really impressive automated verification software to find
> this bug:

On the other hand, the only person who really needs to be convinced is 
Tim Peters. It's really not up to the Python community.

The bug tracker is the right place for discussing this.

-- 
Steve

From mistersheik at gmail.com  Mon Mar  9 16:48:26 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Mon, 9 Mar 2015 11:48:26 -0400
Subject: [Python-Dev] boxing and unboxing data types
In-Reply-To: <3487932.K7gDH0h8FU@raxxla>
References: <54FD1412.3020002@stoneleaf.us> <2847718.KSDy3zi7qt@raxxla>
 <CAA68w_nVh8FWzwJdmRneix50fTPVsBmSKf5auNwMKBr6XgMLBw@mail.gmail.com>
 <3487932.K7gDH0h8FU@raxxla>
Message-ID: <CAA68w_md1CoyVn1yxP0mQof3P+PvONEgSv+Cwu9Lm2S3H1hSgw@mail.gmail.com>

On Mon, Mar 9, 2015 at 11:11 AM, Serhiy Storchaka <storchaka at gmail.com>
wrote:

> ?????????, 09-???-2015 10:18:50 ?? ????????:
> > On Mon, Mar 9, 2015 at 10:10 AM, Serhiy Storchaka <storchaka at gmail.com>
> wrote:
> > > ?????????, 09-???-2015 09:52:01 ?? ????????:
> > > > On Mon, Mar 9, 2015 at 2:07 AM, Serhiy Storchaka <
> storchaka at gmail.com>
> > > > > And to be ideal drop-in replacement IntEnum should override such
> methods
> > > > > as __eq__ and __hash__ (so it could be used as mapping key). If
> all methods
> > > > > should be overridden to quack as int, why not take an int?
> > > >
> > > > You're absolutely right that if *all the methods should be
> overrriden to
> > > > quack as int, then you should subclass int (the Liskov substitution
> > > > principle).  But all methods should not be overridden ? mainly the
> methods
> > > > you overrode in your patch should be exposed.  Here is a list of
> methods on
> > > > int that should not be on IntFlags in my opinion (give or take a
> couple):
> > > >
> > > > __abs__, __add__, __delattr__, __divmod__, __float__, __floor__,
> > > > __floordiv__, __index__, __lshift__, __mod__, __mul__, __pos__,
> __pow__,
> > > > __radd__, __rdivmod__, __rfloordiv__, __rlshift__, __rmod__,
> __rmul__,
> > > > __round__, __rpow__, __rrshift__, __rshift__, __rsub__, __rtruediv__,
> > > > __sub__, __truediv__, __trunc__, conjugate, denominator, imag,
> numerator,
> > > > real.
> > > >
> > > > I don't think __index__ should be exposed either since are you
> really going
> > > > to slice a list using IntFlags?  Really?
> > >
> > > Definitely __index__ should be exposed. __int__ is for lossy
> conversion to int
> > > (as in float). __index__ is for lossless conversion.
> >
> > Is it?  __index__ promises lossless conversion, but __index__ is *for*
> > indexing.
>
> I spite of its name it is for any lossless conversion.
>

You're right.

>
> > > __add__ should be exposed because some code can use + instead of | for
> > > combining flags. But it shouldn't preserve the type, because this is
> not
> > > recommended way.
> >
> > I think it should be blocked because it can lead to all kinds of weird
> > bugs.  If the flag is already set and you add it a copy, it silently
> spills
> > over into other flags.  This is a mistake that a good interface prevents.
>
> I think this is a case when backward compatibility has larger weight.
>
>
So you agree that the ideal solution is composition, but you prefer
inheritance in order to not break code?  Then,I think the big question is
how much code would actually break if you presented the ideal interface.  I
imagine that 99% of the code using flags only uses __or__ to compose and
__and__, __invert__ to erase flags.


> > > For the same reason I think __lshift__, __rshift__, __sub__,
> > > __mul__, __divmod__, __floordiv__, __mod__, etc should be exposed too.
> So the
> > > majority of the methods should be exposed, and there is a risk that we
> loss
> > > something.
> >
> > I totally disagree with all of those.
> >
> > > For good compatibility with Python code IntFlags should expose also
> > > __subclasscheck__ or __subclasshook__. And when we are at this point,
> why not
> > > use int subclass?
> >
> > Here's another reason.  What if someone wants to use an IntFlags object,
> > but wants to use a fixed width type for storage, say numpy.int32?   Why
> > shouldn't they be able to do that?  By using composition, you can easily
> > provide such an option.
>
> You can design abstract interface Flags that can be combined with int or
> other type. But why you want to use numpy.int32 as storage? This doesn't
> save much memory, because with composition the IntFlags class weighs more
> than int subclass.
>
>
Maybe you're storing a bunch of flags in a numpy array having dtype
np.int32?  It's contrived, I agree.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150309/9599b068/attachment.html>

From skip.montanaro at gmail.com  Mon Mar  9 17:28:55 2015
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Mon, 9 Mar 2015 11:28:55 -0500
Subject: [Python-Dev] Tunning binary insertion sort algorithm in Timsort.
In-Reply-To: <20150309155329.GV7655@ando.pearwood.info>
References: <CAMLcfj3usoq-new2gxh_ymP+1wv6XZj7+og6XMVAUj_sYeT+Pg@mail.gmail.com>
 <CABPWfCHou2N3pZ5Ltgv_ybAV485nFzYbCEKPtwbgj4qb5_f7gA@mail.gmail.com>
 <20150309155329.GV7655@ando.pearwood.info>
Message-ID: <CANc-5Uw4g7FvTaYrNF8EBT5ZQ0SRVe94z8LvDZ2ov3tJDNWbPQ@mail.gmail.com>

On Mon, Mar 9, 2015 at 10:53 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> On Sun, Mar 08, 2015 at 10:57:30PM -0700, Ryan Smith-Roberts wrote:
>> I suspect that you will find the Python community extremely conservative
>> about any changes to its sorting algorithm, given that it took thirteen
>> years and some really impressive automated verification software to find
>> this bug:
>
> On the other hand, the only person who really needs to be convinced is
> Tim Peters. It's really not up to the Python community.

Also, there's no sense discouraging people who have ideas to
contribute. So what if nha pham's contribution isn't accepted? You
never know when the next Tim Peters, Georg Brandl, Mark Dickinson, or
Brett Cannon will turn up. (That list is practically endless. Don't
feel slighted if I failed to mention you.) With seven billion people
out there, a few of them are bound to be pretty smart.

Skip

From donald at stufft.io  Mon Mar  9 17:35:44 2015
From: donald at stufft.io (Donald Stufft)
Date: Mon, 9 Mar 2015 12:35:44 -0400
Subject: [Python-Dev] Thoughts on running Python 3.5 on Windows (path,
	pip install --user, etc)
In-Reply-To: <DM2PR0301MB0734EC83B986CCC91A8A0043F51B0@DM2PR0301MB0734.namprd03.prod.outlook.com>
References: <DM2PR0301MB0734EC83B986CCC91A8A0043F51B0@DM2PR0301MB0734.namprd03.prod.outlook.com>
Message-ID: <4CD3C0B8-C9B2-4C7C-9257-C4F51276C8C2@stufft.io>


> On Mar 9, 2015, at 11:37 AM, Steve Dower <Steve.Dower at microsoft.com> wrote:
> 
> Paul Moore wrote:
>> I just thought I'd give installing Python 3.5 a go on my PC, now that
>> 3.5a2 has come out. I didn't get very far (see earlier message) but it prompted
>> me to think about how I'd use it, and what habits I'd need to change.
>> 
>> I'd suggest that the "what's new in 3.5" document probably needs a section on
>> the new installer that explains this stuff...
> 
> This is true. Right now I'm in experimentation mode, and being more aggressive about changing things than is probably a good idea (because it solicits feedback like this :) ). When things settle down I expect to end up closer to where we started, so there's not a huge amount of value in writing it all up right now. I'll get there.
> 
>> First of all, I always use "all users" installs, so I have Python in "Program
>> Files" now. As a result, doing "pip install foo" requires elevation. As that's a
>> PITA, I probably need to switch to using "pip install --user". All that's fine,
>> and from there "py -3.5" works fine, as does "py -3.5 -m foo". But even if it
>> is, not every entry point has a corresponding "-m" invocation (pygments'
>> pygmentize command doesn't seem to, for example)
> 
> I know you're already involved in this Paul, but for everyone else there's a big discussion going on at https://github.com/pypa/pip/issues/1668 about changing pip's default behaviour to handle falling back to --user automatically.
> 
>> But suppose I want to put Python 3.5 on my PATH. The installer has an "add
>> Python to PATH" checkbox, but (if I'm reading the WiX source right, I didn't
>> select that on install) that doesn't add the user directory. So I need to add
>> that to my PATH. Is that right? And of course, that means I need to *know* the
>> user site directory ($env:LOCALAPPDATA\Python\Python35\Scripts), correct?
> 
> Correct. There's no way to add a per-user directory to PATH from an all-users installation (except for a few approaches that I expect/hope would trigger malware detectors...)
> 
>> Maybe the answer is that we simply start recommending that everyone on Windows
>> uses per-user installs. It makes little difference to me (beyond the fact that
>> when I want to look at the source of something in the stdlib, the location of
>> the file is a lot harder to remember than C:\Apps\Python34\Lib\whatever.py) but
>> I doubt it's what most people will expect.
> 
> I'm okay with this. Installing for all users is really something that could be considered an advanced option rather than the default, especially since the aim (AIUI) of the all-users install is to pretend that Python was shipped with the OS. (I'd kind of like to take that further by splitting things more sensibly between Program Files, Common Files and System32, but there's very little gain from that and much MUCH pain as long as people are still expecting C:\PythonXY installs?)

Maybe the answer is to write up a PEP and standardize the idea of entry points, specifically the console_scripts and ui_scripts (or whatever it?s called) entrypoints and then give Python something like -m, but which executes a specific entry point name instead of a module name (or maybe -m can fall back to looking at entry points? I don?t know).

I?ve given this like? 30s worth of thought, but maybe:

    pip install pygmentize  # Implicit ?user
    py -e pygmetize

Is an OK UX for people to have without needing to add the user site bin directory to their PATH. Maybe it?s a horrible idea and we should all forget I mentioned it :)

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150309/032e7c72/attachment-0001.sig>

From p.f.moore at gmail.com  Mon Mar  9 17:50:41 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 9 Mar 2015 16:50:41 +0000
Subject: [Python-Dev] Thoughts on running Python 3.5 on Windows (path,
 pip install --user, etc)
In-Reply-To: <4CD3C0B8-C9B2-4C7C-9257-C4F51276C8C2@stufft.io>
References: <DM2PR0301MB0734EC83B986CCC91A8A0043F51B0@DM2PR0301MB0734.namprd03.prod.outlook.com>
 <4CD3C0B8-C9B2-4C7C-9257-C4F51276C8C2@stufft.io>
Message-ID: <CACac1F-GTv2gNWGy4O_RrPcQg4wxr5H98SNwzQtfgmDxYMUMJw@mail.gmail.com>

On 9 March 2015 at 16:35, Donald Stufft <donald at stufft.io> wrote:
>> I'm okay with this. Installing for all users is really something that could be considered an advanced option rather than the default, especially since the aim (AIUI) of the all-users install is to pretend that Python was shipped with the OS. (I'd kind of like to take that further by splitting things more sensibly between Program Files, Common Files and System32, but there's very little gain from that and much MUCH pain as long as people are still expecting C:\PythonXY installs?)
>
> Maybe the answer is to write up a PEP and standardize the idea of entry points, specifically the console_scripts and ui_scripts (or whatever it?s called) entrypoints and then give Python something like -m, but which executes a specific entry point name instead of a module name (or maybe -m can fall back to looking at entry points? I don?t know).
>
> I?ve given this like? 30s worth of thought, but maybe:
>
>     pip install pygmentize  # Implicit ?user
>     py -e pygmetize
>
> Is an OK UX for people to have without needing to add the user site bin directory to their PATH. Maybe it?s a horrible idea and we should all forget I mentioned it :)

That would be good. You can do it now using setuptools' entry point
API, so making it part of the core is certainly not impossible. But is
it practical on the 3.5 release timescales? It's worth taking into
account the fact that pygmentize is exposed by pygments, so there's no
way to deduce the package from the entry point name. Also, even if a
core feature appears, nobody will be using it for ages.

On 9 March 2015 at 15:37, Steve Dower <Steve.Dower at microsoft.com> wrote:
>> Maybe the answer is that we simply start recommending that everyone on Windows
>> uses per-user installs. It makes little difference to me (beyond the fact that
>> when I want to look at the source of something in the stdlib, the location of
>> the file is a lot harder to remember than C:\Apps\Python34\Lib\whatever.py) but
>> I doubt it's what most people will expect.
>
> I'm okay with this. Installing for all users is really something that could be considered an advanced option rather than the default, especially since the aim (AIUI) of the all-users install is to pretend that Python was shipped with the OS. (I'd kind of like to take that further by splitting things more sensibly between Program Files, Common Files and System32, but there's very little gain from that and much MUCH pain as long as people are still expecting C:\PythonXY installs...)

Just to be clear, I'm *not* okay with this in the form I proposed it.
I think we're a long way yet from a clean and understandable proposal
for having user installs be as usable as system installs. It's worth
noting that (as far as I know) users don't typically use user installs
even on Unix, where the issue of the system install being read-only
has always been the norm. To me, that implies that there are still
some pieces of the puzzle to be addressed.

And again, I don't know if this is going to be solved in Python 3.5
timescales. Beta 1 is May 24th - would a major change like a version
of pip defaulting to user installs be acceptable after that? More
specifically, would core changes to install processes and recommended
practices to support that be allowed after the beta?

Maybe I'm being pessimistic. A good solution would be fantastic. But
at the moment, I feel like 3.4 (and 2.7.9) with pip and the scripts
directory on PATH was a huge usability step forward for naive users on
Windows, and we're in danger here of reverting it. And getting "naive
user" feedback until after the release is notoriously hard... A
"practicality beats purity" solution of sticking to the old install
scheme for one more version seems like it should be left as an option,
at least as a fallback if we don't solve the issues in time.

Paul

From storchaka at gmail.com  Mon Mar  9 17:54:37 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 09 Mar 2015 18:54:37 +0200
Subject: [Python-Dev] boxing and unboxing data types
In-Reply-To: <CAA68w_md1CoyVn1yxP0mQof3P+PvONEgSv+Cwu9Lm2S3H1hSgw@mail.gmail.com>
References: <54FD1412.3020002@stoneleaf.us> <2847718.KSDy3zi7qt@raxxla>
 <CAA68w_nVh8FWzwJdmRneix50fTPVsBmSKf5auNwMKBr6XgMLBw@mail.gmail.com>
 <3487932.K7gDH0h8FU@raxxla>
 <CAA68w_md1CoyVn1yxP0mQof3P+PvONEgSv+Cwu9Lm2S3H1hSgw@mail.gmail.com>
Message-ID: <mdkj8d$mhi$1@ger.gmane.org>

On 09.03.15 17:48, Neil Girdhar wrote:
> So you agree that the ideal solution is composition, but you prefer
> inheritance in order to not break code?

Yes, I agree. There is two advantages in the inheritance: larger 
backward compatibility and simpler implementation.

> Then,I think the big question
> is how much code would actually break if you presented the ideal
> interface.  I imagine that 99% of the code using flags only uses __or__
> to compose and __and__, __invert__ to erase flags.

I don't know and don't want to guess. Let just follow the way of bool 
and IntEnum. When users will be encouraged to use IntEnum and IntFlags 
instead of plain ints we could consider the idea of dropping inheritance 
of bool, IntEnum and IntFlags from int. This is not near future.

>     > Here's another reason.  What if someone wants to use an IntFlags object,
>     > but wants to use a fixed width type for storage, say numpy.int32?   Why
>     > shouldn't they be able to do that?  By using composition, you can easily
>     > provide such an option.
>     You can design abstract interface Flags that can be combined with
>     int or other type. But why you want to use numpy.int32 as storage?
>     This doesn't save much memory, because with composition the IntFlags
>     class weighs more than int subclass.
> Maybe you're storing a bunch of flags in a numpy array having dtype
> np.int32?  It's contrived, I agree.

I afraid that composition will not help you with this. Can numpy array 
pack int-like objects into fixed-width integer array and then restore 
original type on unboxing?



From ischwabacher at wisc.edu  Mon Mar  9 18:39:18 2015
From: ischwabacher at wisc.edu (Isaac Schwabacher)
Date: Mon, 09 Mar 2015 12:39:18 -0500
Subject: [Python-Dev] Tunning binary insertion sort algorithm in Timsort.
In-Reply-To: <7790de4c2be2c.54fdd645@wiscmail.wisc.edu>
References: <CAMLcfj3usoq-new2gxh_ymP+1wv6XZj7+og6XMVAUj_sYeT+Pg@mail.gmail.com>
 <7750938a2fea2.54fdd09e@wiscmail.wisc.edu>
 <76e0c70929f48.54fdd0da@wiscmail.wisc.edu>
 <76f094a32c8ca.54fdd119@wiscmail.wisc.edu>
 <7740f7e62cf4b.54fdd155@wiscmail.wisc.edu>
 <778088e82a599.54fdd192@wiscmail.wisc.edu>
 <7740ebbf29f9e.54fdd2be@wiscmail.wisc.edu>
 <76f0f50b2c800.54fdd2fb@wiscmail.wisc.edu>
 <7710a1122c5ba.54fdd337@wiscmail.wisc.edu>
 <7780f4192fd2c.54fdd373@wiscmail.wisc.edu>
 <7790d04c299a7.54fdd590@wiscmail.wisc.edu>
 <7710a1822e0a9.54fdd5cc@wiscmail.wisc.edu>
 <7720dc73289e8.54fdd609@wiscmail.wisc.edu>
 <7790de4c2be2c.54fdd645@wiscmail.wisc.edu>
Message-ID: <76f0eced29ac4.54fd9476@wiscmail.wisc.edu>

On 15-03-08, nha pham 
 wrote:
> 
> We can optimize the TimSort algorithm by optimizing its binary insertion sort.
> 
> The current version of binary insertion sort use this idea:
> 
> Use binary search to find a final position in sorted list for a new element X. Then insert X to that location.
> 
> I suggest another idea:
> 
> Use binary search to find a final postion in sorted list for a new element X. Before insert X to that location, compare X with its next element.
> 
> For the next element, we already know if it is lower or bigger than X, so we can reduce the search area to the left side or on the right side of X in the sorted list.

I don't understand how this is an improvement, since with binary search the idea is that each comparison cuts the remaining list to search in half; i.e., each comparison yields one bit of information. Here, you're spending a comparison to cut the list to search at the element you just inserted, which is probably not right in the middle. If you miss the middle, you're getting on average less than a full bit of information from your comparison, so you're not reducing the remaining search space by as much as you would be if you just compared to the element in the middle of the list.

> I have applied my idea on java.util. ComparableTimSort.sort() and testing. The execute time is reduced by 2%-6% with array of random integer.

For all that, though, experiment trumps theory...

> Here is detail about algorithm and testing:?https://github.com/nhapq/Optimize_binary_insertion_sort
> 
> Sincerely.
> 
> phqnha

From phqnha at gmail.com  Mon Mar  9 19:49:46 2015
From: phqnha at gmail.com (nha pham)
Date: Mon, 9 Mar 2015 11:49:46 -0700
Subject: [Python-Dev] Tunning binary insertion sort algorithm in Timsort.
In-Reply-To: <CAMLcfj085vX827WZsA5R3ibtgySoodjprdCYFL4Ui5aNV9KKQg@mail.gmail.com>
References: <CAMLcfj3usoq-new2gxh_ymP+1wv6XZj7+og6XMVAUj_sYeT+Pg@mail.gmail.com>
 <7750938a2fea2.54fdd09e@wiscmail.wisc.edu>
 <76e0c70929f48.54fdd0da@wiscmail.wisc.edu>
 <76f094a32c8ca.54fdd119@wiscmail.wisc.edu>
 <7740f7e62cf4b.54fdd155@wiscmail.wisc.edu>
 <778088e82a599.54fdd192@wiscmail.wisc.edu>
 <7740ebbf29f9e.54fdd2be@wiscmail.wisc.edu>
 <76f0f50b2c800.54fdd2fb@wiscmail.wisc.edu>
 <7710a1122c5ba.54fdd337@wiscmail.wisc.edu>
 <7780f4192fd2c.54fdd373@wiscmail.wisc.edu>
 <7790d04c299a7.54fdd590@wiscmail.wisc.edu>
 <7710a1822e0a9.54fdd5cc@wiscmail.wisc.edu>
 <7720dc73289e8.54fdd609@wiscmail.wisc.edu>
 <7790de4c2be2c.54fdd645@wiscmail.wisc.edu>
 <76f0eced29ac4.54fd9476@wiscmail.wisc.edu>
 <CAMLcfj085vX827WZsA5R3ibtgySoodjprdCYFL4Ui5aNV9KKQg@mail.gmail.com>
Message-ID: <CAMLcfj0-wdoegAZExDm2nw3AW0DLpPrZceO_zWGLMoJMzfqhMQ@mail.gmail.com>

I do not know exactly, one thing I can imagine is: it turns the worst case
of binary insertion sort to best case.
With sorted array in range of 32 or 64 items, built from zero element. The
new element you put into the sorted list has a high chance of being the
smallest or the the highest of the sorted list (or nearly highest or nearly
smallest)

If that case happen, the old binary insertion sort will have the
investigate all the list, while with my idea, it just have to compare more
1-2 times.
I will try to run more test an more thinking to make sure though.

On Mon, Mar 9, 2015 at 11:48 AM, nha pham <phqnha at gmail.com> wrote:

> I do not know exactly, one thing I can imagine is: it turns the worst case
> of binary insertion sort to best case.
> With sorted array in range of 32 or 64 items, built from zero element. The
> new element you put into the sorted list has a high chance of being the
> smallest or the the highest of the sorted list (or nearly highest or nearly
> smallest)
>
> If that case happen, the old binary insertion sort will have the
> investigate all the list, while with my idea, it just have to compare more
> 1-2 times.
> I will try to run more test an more thinking to make sure though.
>
>
>
> On Mon, Mar 9, 2015 at 10:39 AM, Isaac Schwabacher <ischwabacher at wisc.edu>
> wrote:
>
>> On 15-03-08, nha pham
>>  wrote:
>> >
>> > We can optimize the TimSort algorithm by optimizing its binary
>> insertion sort.
>> >
>> > The current version of binary insertion sort use this idea:
>> >
>> > Use binary search to find a final position in sorted list for a new
>> element X. Then insert X to that location.
>> >
>> > I suggest another idea:
>> >
>> > Use binary search to find a final postion in sorted list for a new
>> element X. Before insert X to that location, compare X with its next
>> element.
>> >
>> > For the next element, we already know if it is lower or bigger than X,
>> so we can reduce the search area to the left side or on the right side of X
>> in the sorted list.
>>
>> I don't understand how this is an improvement, since with binary search
>> the idea is that each comparison cuts the remaining list to search in half;
>> i.e., each comparison yields one bit of information. Here, you're spending
>> a comparison to cut the list to search at the element you just inserted,
>> which is probably not right in the middle. If you miss the middle, you're
>> getting on average less than a full bit of information from your
>> comparison, so you're not reducing the remaining search space by as much as
>> you would be if you just compared to the element in the middle of the list.
>>
>> > I have applied my idea on java.util. ComparableTimSort.sort() and
>> testing. The execute time is reduced by 2%-6% with array of random integer.
>>
>> For all that, though, experiment trumps theory...
>>
>> > Here is detail about algorithm and testing:
>> https://github.com/nhapq/Optimize_binary_insertion_sort
>> >
>> > Sincerely.
>> >
>> > phqnha
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150309/6e7b1691/attachment.html>

From mistersheik at gmail.com  Mon Mar  9 21:42:48 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Mon, 9 Mar 2015 16:42:48 -0400
Subject: [Python-Dev] Tunning binary insertion sort algorithm in Timsort.
In-Reply-To: <CAMLcfj0-wdoegAZExDm2nw3AW0DLpPrZceO_zWGLMoJMzfqhMQ@mail.gmail.com>
References: <CAMLcfj3usoq-new2gxh_ymP+1wv6XZj7+og6XMVAUj_sYeT+Pg@mail.gmail.com>
 <7750938a2fea2.54fdd09e@wiscmail.wisc.edu>
 <76e0c70929f48.54fdd0da@wiscmail.wisc.edu>
 <76f094a32c8ca.54fdd119@wiscmail.wisc.edu>
 <7740f7e62cf4b.54fdd155@wiscmail.wisc.edu>
 <778088e82a599.54fdd192@wiscmail.wisc.edu>
 <7740ebbf29f9e.54fdd2be@wiscmail.wisc.edu>
 <76f0f50b2c800.54fdd2fb@wiscmail.wisc.edu>
 <7710a1122c5ba.54fdd337@wiscmail.wisc.edu>
 <7780f4192fd2c.54fdd373@wiscmail.wisc.edu>
 <7790d04c299a7.54fdd590@wiscmail.wisc.edu>
 <7710a1822e0a9.54fdd5cc@wiscmail.wisc.edu>
 <7720dc73289e8.54fdd609@wiscmail.wisc.edu>
 <7790de4c2be2c.54fdd645@wiscmail.wisc.edu>
 <76f0eced29ac4.54fd9476@wiscmail.wisc.edu>
 <CAMLcfj085vX827WZsA5R3ibtgySoodjprdCYFL4Ui5aNV9KKQg@mail.gmail.com>
 <CAMLcfj0-wdoegAZExDm2nw3AW0DLpPrZceO_zWGLMoJMzfqhMQ@mail.gmail.com>
Message-ID: <CAA68w_nbtiT1n6pka_PwkRTLryqpK6VpVVQq0ZGPozxfPk5FqA@mail.gmail.com>

It may be that the comparison that you do is between two elements that are
almost always in the same cache line whereas the binary search might often
incur a cache miss.

On Mon, Mar 9, 2015 at 2:49 PM, nha pham <phqnha at gmail.com> wrote:

> I do not know exactly, one thing I can imagine is: it turns the worst case
> of binary insertion sort to best case.
> With sorted array in range of 32 or 64 items, built from zero element. The
> new element you put into the sorted list has a high chance of being the
> smallest or the the highest of the sorted list (or nearly highest or nearly
> smallest)
>
> If that case happen, the old binary insertion sort will have the
> investigate all the list, while with my idea, it just have to compare more
> 1-2 times.
> I will try to run more test an more thinking to make sure though.
>
> On Mon, Mar 9, 2015 at 11:48 AM, nha pham <phqnha at gmail.com> wrote:
>
>> I do not know exactly, one thing I can imagine is: it turns the worst
>> case of binary insertion sort to best case.
>> With sorted array in range of 32 or 64 items, built from zero element.
>> The new element you put into the sorted list has a high chance of being the
>> smallest or the the highest of the sorted list (or nearly highest or nearly
>> smallest)
>>
>> If that case happen, the old binary insertion sort will have the
>> investigate all the list, while with my idea, it just have to compare more
>> 1-2 times.
>> I will try to run more test an more thinking to make sure though.
>>
>>
>>
>> On Mon, Mar 9, 2015 at 10:39 AM, Isaac Schwabacher <ischwabacher at wisc.edu
>> > wrote:
>>
>>> On 15-03-08, nha pham
>>>  wrote:
>>> >
>>> > We can optimize the TimSort algorithm by optimizing its binary
>>> insertion sort.
>>> >
>>> > The current version of binary insertion sort use this idea:
>>> >
>>> > Use binary search to find a final position in sorted list for a new
>>> element X. Then insert X to that location.
>>> >
>>> > I suggest another idea:
>>> >
>>> > Use binary search to find a final postion in sorted list for a new
>>> element X. Before insert X to that location, compare X with its next
>>> element.
>>> >
>>> > For the next element, we already know if it is lower or bigger than X,
>>> so we can reduce the search area to the left side or on the right side of X
>>> in the sorted list.
>>>
>>> I don't understand how this is an improvement, since with binary search
>>> the idea is that each comparison cuts the remaining list to search in half;
>>> i.e., each comparison yields one bit of information. Here, you're spending
>>> a comparison to cut the list to search at the element you just inserted,
>>> which is probably not right in the middle. If you miss the middle, you're
>>> getting on average less than a full bit of information from your
>>> comparison, so you're not reducing the remaining search space by as much as
>>> you would be if you just compared to the element in the middle of the list.
>>>
>>> > I have applied my idea on java.util. ComparableTimSort.sort() and
>>> testing. The execute time is reduced by 2%-6% with array of random integer.
>>>
>>> For all that, though, experiment trumps theory...
>>>
>>> > Here is detail about algorithm and testing:
>>> https://github.com/nhapq/Optimize_binary_insertion_sort
>>> >
>>> > Sincerely.
>>> >
>>> > phqnha
>>>
>>
>>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/mistersheik%40gmail.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150309/5d1f65c1/attachment.html>

From mistersheik at gmail.com  Mon Mar  9 21:49:27 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Mon, 9 Mar 2015 16:49:27 -0400
Subject: [Python-Dev] boxing and unboxing data types
In-Reply-To: <mdkj8d$mhi$1@ger.gmane.org>
References: <54FD1412.3020002@stoneleaf.us> <2847718.KSDy3zi7qt@raxxla>
 <CAA68w_nVh8FWzwJdmRneix50fTPVsBmSKf5auNwMKBr6XgMLBw@mail.gmail.com>
 <3487932.K7gDH0h8FU@raxxla>
 <CAA68w_md1CoyVn1yxP0mQof3P+PvONEgSv+Cwu9Lm2S3H1hSgw@mail.gmail.com>
 <mdkj8d$mhi$1@ger.gmane.org>
Message-ID: <CAA68w_nL8=8LQgj5m7wmjsNfUmH6Nt-XW3RtSUqpr=3RaooELg@mail.gmail.com>

On Mon, Mar 9, 2015 at 12:54 PM, Serhiy Storchaka <storchaka at gmail.com>
wrote:

> On 09.03.15 17:48, Neil Girdhar wrote:
>
>> So you agree that the ideal solution is composition, but you prefer
>> inheritance in order to not break code?
>>
>
> Yes, I agree. There is two advantages in the inheritance: larger backward
> compatibility and simpler implementation.
>
>
Inheritance might be more backwards compatible, but I believe that you
should check how much code is genuine not restricted to the idealized flags
interface.   It's not worth talking about "simpler implementation" since
the two solutions differ by only a couple dozen lines.

On the other hand, composition is better design.  It prevents you from
making mistakes like adding to flags and having carries, or using flags in
an unintended way.


>  Then,I think the big question
>> is how much code would actually break if you presented the ideal
>> interface.  I imagine that 99% of the code using flags only uses __or__
>> to compose and __and__, __invert__ to erase flags.
>>
>
> I don't know and don't want to guess. Let just follow the way of bool and
> IntEnum. When users will be encouraged to use IntEnum and IntFlags instead
> of plain ints we could consider the idea of dropping inheritance of bool,
> IntEnum and IntFlags from int. This is not near future.


I think it's the other way around.  You should typically start with the
modest interface and add methods as you need.  If you start with full blown
inheritance, you will find it only increasingly more difficult to remove
methods in changing your solution.  Using inheritance instead of
composition is one of the most common errors in objected oriented
programming, and I get the impression from your other paragraph that you're
seduced by the slightly shorter code.  I don't think it's worth giving in
to that without proof that composition will actually break a significant
amount of code.

Regarding IntEnum ? that should inherit from int since they are truly just
integer constants.  It's too late for bool; that ship has sailed
unfortunately.


>
>
>      > Here's another reason.  What if someone wants to use an IntFlags
>> object,
>>     > but wants to use a fixed width type for storage, say numpy.int32?
>>  Why
>>     > shouldn't they be able to do that?  By using composition, you can
>> easily
>>     > provide such an option.
>>     You can design abstract interface Flags that can be combined with
>>     int or other type. But why you want to use numpy.int32 as storage?
>>     This doesn't save much memory, because with composition the IntFlags
>>     class weighs more than int subclass.
>> Maybe you're storing a bunch of flags in a numpy array having dtype
>> np.int32?  It's contrived, I agree.
>>
>
> I afraid that composition will not help you with this. Can numpy array
> pack int-like objects into fixed-width integer array and then restore
> original type on unboxing?


You're right.

>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> mistersheik%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150309/d88d8da0/attachment-0001.html>

From ncoghlan at gmail.com  Tue Mar 10 00:11:53 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 10 Mar 2015 09:11:53 +1000
Subject: [Python-Dev] Thoughts on running Python 3.5 on Windows (path,
 pip install --user, etc)
In-Reply-To: <4CD3C0B8-C9B2-4C7C-9257-C4F51276C8C2@stufft.io>
References: <DM2PR0301MB0734EC83B986CCC91A8A0043F51B0@DM2PR0301MB0734.namprd03.prod.outlook.com>
 <4CD3C0B8-C9B2-4C7C-9257-C4F51276C8C2@stufft.io>
Message-ID: <CADiSq7cgMNJDp22uyFs1j4RKhEK2tkWWnMrrVx8Cs=dLvrP7+g@mail.gmail.com>

On 10 Mar 2015 02:37, "Donald Stufft" <donald at stufft.io> wrote:
> >
> > I'm okay with this. Installing for all users is really something that
could be considered an advanced option rather than the default, especially
since the aim (AIUI) of the all-users install is to pretend that Python was
shipped with the OS. (I'd kind of like to take that further by splitting
things more sensibly between Program Files, Common Files and System32, but
there's very little gain from that and much MUCH pain as long as people are
still expecting C:\PythonXY installs?)
>
> Maybe the answer is to write up a PEP and standardize the idea of entry
points, specifically the console_scripts and ui_scripts (or whatever it?s
called) entrypoints and then give Python something like -m, but which
executes a specific entry point name instead of a module name (or maybe -m
can fall back to looking at entry points? I don?t know).

While I like the idea of offering something more "built in" in this space,
my initial inclination is to prefer extending "-m" to accept the
"module.name:function.name" format to let you invoke entry points by the
name of the target function (Possible API name: runpy.run_cli_function),
and then add a "runpy.call" that can be used to call an arbitrary function
with positional and keyword string arguments based on sys.argv and
(optionally?) print the repr of the result.

It wouldn't be a universal panacea (and would need a PEP to work out the
exact UX details), but would likely make quite a few libraries more command
line accessible without needing to modify them.

Cheers,
Nick.

>
> I?ve given this like? 30s worth of thought, but maybe:
>
>     pip install pygmentize  # Implicit ?user
>     py -e pygmetize
>
> Is an OK UX for people to have without needing to add the user site bin
directory to their PATH. Maybe it?s a horrible idea and we should all
forget I mentioned it :)
>
> ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150310/5b80fd4e/attachment.html>

From donald at stufft.io  Tue Mar 10 00:16:35 2015
From: donald at stufft.io (Donald Stufft)
Date: Mon, 9 Mar 2015 19:16:35 -0400
Subject: [Python-Dev] Thoughts on running Python 3.5 on Windows (path,
	pip install --user, etc)
In-Reply-To: <CADiSq7cgMNJDp22uyFs1j4RKhEK2tkWWnMrrVx8Cs=dLvrP7+g@mail.gmail.com>
References: <DM2PR0301MB0734EC83B986CCC91A8A0043F51B0@DM2PR0301MB0734.namprd03.prod.outlook.com>
 <4CD3C0B8-C9B2-4C7C-9257-C4F51276C8C2@stufft.io>
 <CADiSq7cgMNJDp22uyFs1j4RKhEK2tkWWnMrrVx8Cs=dLvrP7+g@mail.gmail.com>
Message-ID: <284FD035-2925-4763-8685-CB5292104343@stufft.io>


> On Mar 9, 2015, at 7:11 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> 
> On 10 Mar 2015 02:37, "Donald Stufft" <donald at stufft.io <mailto:donald at stufft.io>> wrote:
> > >
> > > I'm okay with this. Installing for all users is really something that could be considered an advanced option rather than the default, especially since the aim (AIUI) of the all-users install is to pretend that Python was shipped with the OS. (I'd kind of like to take that further by splitting things more sensibly between Program Files, Common Files and System32, but there's very little gain from that and much MUCH pain as long as people are still expecting C:\PythonXY installs?)
> >
> > Maybe the answer is to write up a PEP and standardize the idea of entry points, specifically the console_scripts and ui_scripts (or whatever it?s called) entrypoints and then give Python something like -m, but which executes a specific entry point name instead of a module name (or maybe -m can fall back to looking at entry points? I don?t know).
> 
> While I like the idea of offering something more "built in" in this space, my initial inclination is to prefer extending "-m" to accept the "module.name:function.name <http://unction.name/>" format to let you invoke entry points by the name of the target function (Possible API name: runpy.run_cli_function), and then add a "runpy.call" that can be used to call an arbitrary function with positional and keyword string arguments based on sys.argv and (optionally?) print the repr of the result.
> 
> It wouldn't be a universal panacea (and would need a PEP to work out the exact UX details), but would likely make quite a few libraries more command line accessible without needing to modify them.
> 
> 

If I understand this correctly, you?re suggesting that to run ``pygmentize`` without using the script wrapper, you?d need to do ``py -m pygments.cmdline:main`` instead of ``pygmentize``? I don?t think that actually solves the problem (except by making it so that the script wrappers can maybe just be exactly #!/usr/bin/python -m pygments.cmdline:main but that?s a different thing..). I?m not against it in general though, I just don?t know that it solves the problem Paul was mentioning.


---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150309/512166ef/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150309/512166ef/attachment.sig>

From p.f.moore at gmail.com  Tue Mar 10 00:18:53 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 9 Mar 2015 23:18:53 +0000
Subject: [Python-Dev] Thoughts on running Python 3.5 on Windows (path,
 pip install --user, etc)
In-Reply-To: <CADiSq7cgMNJDp22uyFs1j4RKhEK2tkWWnMrrVx8Cs=dLvrP7+g@mail.gmail.com>
References: <DM2PR0301MB0734EC83B986CCC91A8A0043F51B0@DM2PR0301MB0734.namprd03.prod.outlook.com>
 <4CD3C0B8-C9B2-4C7C-9257-C4F51276C8C2@stufft.io>
 <CADiSq7cgMNJDp22uyFs1j4RKhEK2tkWWnMrrVx8Cs=dLvrP7+g@mail.gmail.com>
Message-ID: <CACac1F_6j82eEWfb_q-WJg58EdZPvuabXgrf7PxKujV+2sA_xw@mail.gmail.com>

On 9 March 2015 at 23:11, Nick Coghlan <ncoghlan at gmail.com> wrote:
> While I like the idea of offering something more "built in" in this space,
> my initial inclination is to prefer extending "-m" to accept the
> "module.name:function.name" format to let you invoke entry points by the
> name of the target function (Possible API name: runpy.run_cli_function), and
> then add a "runpy.call" that can be used to call an arbitrary function with
> positional and keyword string arguments based on sys.argv and (optionally?)
> print the repr of the result.
>
> It wouldn't be a universal panacea (and would need a PEP to work out the
> exact UX details), but would likely make quite a few libraries more command
> line accessible without needing to modify them.

Personally I doubt it would make much difference. If the docs say
"pygmentize" I'm unlikely to dig around to find that the incantation
"python -m pygments.somemodule:main" does the same thing using 3 times
as many characters. I'd just add Python to my PATH and say stuff it.

Paul

From ncoghlan at gmail.com  Tue Mar 10 00:22:06 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 10 Mar 2015 09:22:06 +1000
Subject: [Python-Dev] boxing and unboxing data types
In-Reply-To: <CAA68w_nL8=8LQgj5m7wmjsNfUmH6Nt-XW3RtSUqpr=3RaooELg@mail.gmail.com>
References: <54FD1412.3020002@stoneleaf.us> <2847718.KSDy3zi7qt@raxxla>
 <CAA68w_nVh8FWzwJdmRneix50fTPVsBmSKf5auNwMKBr6XgMLBw@mail.gmail.com>
 <3487932.K7gDH0h8FU@raxxla>
 <CAA68w_md1CoyVn1yxP0mQof3P+PvONEgSv+Cwu9Lm2S3H1hSgw@mail.gmail.com>
 <mdkj8d$mhi$1@ger.gmane.org>
 <CAA68w_nL8=8LQgj5m7wmjsNfUmH6Nt-XW3RtSUqpr=3RaooELg@mail.gmail.com>
Message-ID: <CADiSq7f0a=3BndeK+FsVRTarKVhMURjYzR7bZUOXX3Or_Rt2MQ@mail.gmail.com>

On 10 Mar 2015 06:51, "Neil Girdhar" <mistersheik at gmail.com> wrote:
>
>
>
> On Mon, Mar 9, 2015 at 12:54 PM, Serhiy Storchaka <storchaka at gmail.com>
wrote:
>>
>> On 09.03.15 17:48, Neil Girdhar wrote:
>>>
>>> So you agree that the ideal solution is composition, but you prefer
>>> inheritance in order to not break code?
>>
>>
>> Yes, I agree. There is two advantages in the inheritance: larger
backward compatibility and simpler implementation.
>>
>
> Inheritance might be more backwards compatible, but I believe that you
should check how much code is genuine not restricted to the idealized flags
interface.   It's not worth talking about "simpler implementation" since
the two solutions differ by only a couple dozen lines.

We literally can't do this, as the vast majority of Python code in the
world is locked up behind institutional firewalls or has otherwise never
been published. The open source stuff is merely the tip of a truly enormous
iceberg.

If we want to *use* IntFlags in the standard library (and that's the only
pay-off significant enough to justify having it in the standard library),
then it needs to inherit from int.

However, cloning the full enum module architecture to create
flags.FlagsMeta, flags.Flags and flags.IntFlags would make sense to me.

It would also make sense to try that idea out on PyPI for a while before
incorporating it into the stdlib.

Regards,
Nick.

>
> On the other hand, composition is better design.  It prevents you from
making mistakes like adding to flags and having carries, or using flags in
an unintended way.
>
>>>
>>> Then,I think the big question
>>> is how much code would actually break if you presented the ideal
>>> interface.  I imagine that 99% of the code using flags only uses __or__
>>> to compose and __and__, __invert__ to erase flags.
>>
>>
>> I don't know and don't want to guess. Let just follow the way of bool
and IntEnum. When users will be encouraged to use IntEnum and IntFlags
instead of plain ints we could consider the idea of dropping inheritance of
bool, IntEnum and IntFlags from int. This is not near future.
>
>
> I think it's the other way around.  You should typically start with the
modest interface and add methods as you need.  If you start with full blown
inheritance, you will find it only increasingly more difficult to remove
methods in changing your solution.  Using inheritance instead of
composition is one of the most common errors in objected oriented
programming, and I get the impression from your other paragraph that you're
seduced by the slightly shorter code.  I don't think it's worth giving in
to that without proof that composition will actually break a significant
amount of code.
>
> Regarding IntEnum ? that should inherit from int since they are truly
just integer constants.  It's too late for bool; that ship has sailed
unfortunately.
>
>>
>>
>>
>>>     > Here's another reason.  What if someone wants to use an IntFlags
object,
>>>     > but wants to use a fixed width type for storage, say
numpy.int32?   Why
>>>     > shouldn't they be able to do that?  By using composition, you can
easily
>>>     > provide such an option.
>>>     You can design abstract interface Flags that can be combined with
>>>     int or other type. But why you want to use numpy.int32 as storage?
>>>     This doesn't save much memory, because with composition the IntFlags
>>>     class weighs more than int subclass.
>>> Maybe you're storing a bunch of flags in a numpy array having dtype
>>> np.int32?  It's contrived, I agree.
>>
>>
>> I afraid that composition will not help you with this. Can numpy array
pack int-like objects into fixed-width integer array and then restore
original type on unboxing?
>
>
> You're right.
>>
>>
>>
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/mistersheik%40gmail.com
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150310/fc67e9b1/attachment.html>

From ryan.stuart.85 at gmail.com  Tue Mar 10 00:56:02 2015
From: ryan.stuart.85 at gmail.com (Ryan Stuart)
Date: Mon, 09 Mar 2015 23:56:02 +0000
Subject: [Python-Dev] PEP 471 Final: os.scandir() merged into Python 3.5
In-Reply-To: <CAL9jXCHrcvH48QhgCdZWTNUoLbSdJkH9QMOEHrnRPs==hTzxaw@mail.gmail.com>
References: <CAMpsgwYg=RZeyqTzLGn+Nr7xEiiff0EUDhEUgGS-So=B-DjJ1g@mail.gmail.com>
 <CAL9jXCEUOkR_EvGzA6kdodTk3S6+AfnWpRkTmJ27ZRtUaXyAYw@mail.gmail.com>
 <CAM8kh5tL4QgJa4SaaxRsGzgV4RVpZxpdBC0VZ1f1ZKth4nF7HQ@mail.gmail.com>
 <CAL9jXCHrcvH48QhgCdZWTNUoLbSdJkH9QMOEHrnRPs==hTzxaw@mail.gmail.com>
Message-ID: <CAM8kh5v8GHtxGEkHuMjUiuuTjkairvU8zddA3sZMgUizJFsB7g@mail.gmail.com>

Hi Ben,

On Mon, 9 Mar 2015 at 21:58 Ben Hoyt <benhoyt at gmail.com> wrote:

> Note that this benchmark is invalid for a couple of reasons. (...)
>

Thanks a lot for the guidance Ben, greatly appreciated. Just starting to
take an interest in the development of CPython and so something like
running a benchmark seemed like a good a place as any to start.

Since I want to get comfortable with compiling from source I tried this
again. Instead of applying the patch, since the issue is now closed, I just
compiled from the tip of the default branch which at the time
was 94920:0469af231d22. I also didn't configure with --with-pydebug. Here
are the new results:

*Ryans-MacBook-Pro:cpython rstuart$ ./python.exe
~/Workspace/python/scandir/benchmark.py*
Using Python 3.5's builtin os.scandir()
Comparing against builtin version of os.walk()
Priming the system's cache...
Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree,
repeat 1/3...
Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree,
repeat 2/3...
Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree,
repeat 3/3...
os.walk took 0.061s, scandir.walk took 0.012s -- 5.2x as fast

*Ryans-MacBook-Pro:cpython rstuart$ ./python.exe
~/Workspace/python/scandir/benchmark.py -s*
Using Python 3.5's builtin os.scandir()
Comparing against builtin version of os.walk()
Priming the system's cache...
Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree,
repeat 1/3...
Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree,
repeat 2/3...
Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree,
repeat 3/3...
os.walk size 23400, scandir.walk size 23400 -- equal
os.walk took 0.109s, scandir.walk took 0.049s -- 2.2x as fast

This is on a Retina Mid 2012 MacBook Pro with an SSD.

Cheers


> you're compiling Python in debug mode (--with-pydebug), which produces
> significantly slower code in my tests -- for example, on Windows
> benchmark.py is about twice as slow when Python is compiled in debug
> mode.
>
> Second, as the output above shows, benchmark.py is "Using slower
> ctypes version of scandir" and not a C version at all. If os.scandir()
> is available, benchmark.py should use that, so there's something wrong
> here -- maybe the patch didn't apply correctly or maybe you're testing
> with a different version of Python than the one you built?
>
> In any case, the easiest way to test it now is to download Python 3.5
> alpha 2 which just came out:
> https://www.python.org/downloads/release/python-350a2/
>
> I just tried this on my Mac Mini (i5 2.3GHz, 2 GB RAM, HFS+ on
> rotational drive) and got the following results:
>
> Using Python 3.5's builtin os.scandir()
> Comparing against builtin version of os.walk()
> Priming the system's cache...
> Benchmarking walks on benchtree, repeat 1/3...
> Benchmarking walks on benchtree, repeat 2/3...
> Benchmarking walks on benchtree, repeat 3/3...
> os.walk took 0.074s, scandir.walk took 0.016s -- 4.7x as fast
>
> > I then did ./python.exe ~/Workspace/python/scandir/benchmark.py -s and
> got:
>
> Also note that "benchmark.py -s" tests the system os.walk() against a
> get_tree_size() function using scandir's DirEntry.stat().st_size,
> which provides huge gains on Windows (because stat().st_size doesn't
> require on OS call) but only modest gains on POSIX systems, which
> still require an OS stat call to get the size (though not the file
> type, so at least it's only one stat call). I get "2.2x as fast" on my
> Mac for "benchmark.py -s".
>
> -Ben
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150309/2cc3ed8f/attachment.html>

From mistersheik at gmail.com  Tue Mar 10 01:23:18 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Mon, 9 Mar 2015 20:23:18 -0400
Subject: [Python-Dev] boxing and unboxing data types
In-Reply-To: <CADiSq7f0a=3BndeK+FsVRTarKVhMURjYzR7bZUOXX3Or_Rt2MQ@mail.gmail.com>
References: <54FD1412.3020002@stoneleaf.us> <2847718.KSDy3zi7qt@raxxla>
 <CAA68w_nVh8FWzwJdmRneix50fTPVsBmSKf5auNwMKBr6XgMLBw@mail.gmail.com>
 <3487932.K7gDH0h8FU@raxxla>
 <CAA68w_md1CoyVn1yxP0mQof3P+PvONEgSv+Cwu9Lm2S3H1hSgw@mail.gmail.com>
 <mdkj8d$mhi$1@ger.gmane.org>
 <CAA68w_nL8=8LQgj5m7wmjsNfUmH6Nt-XW3RtSUqpr=3RaooELg@mail.gmail.com>
 <CADiSq7f0a=3BndeK+FsVRTarKVhMURjYzR7bZUOXX3Or_Rt2MQ@mail.gmail.com>
Message-ID: <CAA68w_=KiA05yuo20cWyaSYB9YX+OF0V9jJt28KxQ6XnyhyM1Q@mail.gmail.com>

Totally agree
On 9 Mar 2015 19:22, "Nick Coghlan" <ncoghlan at gmail.com> wrote:

>
> On 10 Mar 2015 06:51, "Neil Girdhar" <mistersheik at gmail.com> wrote:
> >
> >
> >
> > On Mon, Mar 9, 2015 at 12:54 PM, Serhiy Storchaka <storchaka at gmail.com>
> wrote:
> >>
> >> On 09.03.15 17:48, Neil Girdhar wrote:
> >>>
> >>> So you agree that the ideal solution is composition, but you prefer
> >>> inheritance in order to not break code?
> >>
> >>
> >> Yes, I agree. There is two advantages in the inheritance: larger
> backward compatibility and simpler implementation.
> >>
> >
> > Inheritance might be more backwards compatible, but I believe that you
> should check how much code is genuine not restricted to the idealized flags
> interface.   It's not worth talking about "simpler implementation" since
> the two solutions differ by only a couple dozen lines.
>
> We literally can't do this, as the vast majority of Python code in the
> world is locked up behind institutional firewalls or has otherwise never
> been published. The open source stuff is merely the tip of a truly enormous
> iceberg.
>
> If we want to *use* IntFlags in the standard library (and that's the only
> pay-off significant enough to justify having it in the standard library),
> then it needs to inherit from int.
>
> However, cloning the full enum module architecture to create
> flags.FlagsMeta, flags.Flags and flags.IntFlags would make sense to me.
>
> It would also make sense to try that idea out on PyPI for a while before
> incorporating it into the stdlib.
>
> Regards,
> Nick.
>
> >
> > On the other hand, composition is better design.  It prevents you from
> making mistakes like adding to flags and having carries, or using flags in
> an unintended way.
> >
> >>>
> >>> Then,I think the big question
> >>> is how much code would actually break if you presented the ideal
> >>> interface.  I imagine that 99% of the code using flags only uses __or__
> >>> to compose and __and__, __invert__ to erase flags.
> >>
> >>
> >> I don't know and don't want to guess. Let just follow the way of bool
> and IntEnum. When users will be encouraged to use IntEnum and IntFlags
> instead of plain ints we could consider the idea of dropping inheritance of
> bool, IntEnum and IntFlags from int. This is not near future.
> >
> >
> > I think it's the other way around.  You should typically start with the
> modest interface and add methods as you need.  If you start with full blown
> inheritance, you will find it only increasingly more difficult to remove
> methods in changing your solution.  Using inheritance instead of
> composition is one of the most common errors in objected oriented
> programming, and I get the impression from your other paragraph that you're
> seduced by the slightly shorter code.  I don't think it's worth giving in
> to that without proof that composition will actually break a significant
> amount of code.
> >
> > Regarding IntEnum ? that should inherit from int since they are truly
> just integer constants.  It's too late for bool; that ship has sailed
> unfortunately.
> >
> >>
> >>
> >>
> >>>     > Here's another reason.  What if someone wants to use an IntFlags
> object,
> >>>     > but wants to use a fixed width type for storage, say
> numpy.int32?   Why
> >>>     > shouldn't they be able to do that?  By using composition, you
> can easily
> >>>     > provide such an option.
> >>>     You can design abstract interface Flags that can be combined with
> >>>     int or other type. But why you want to use numpy.int32 as storage?
> >>>     This doesn't save much memory, because with composition the
> IntFlags
> >>>     class weighs more than int subclass.
> >>> Maybe you're storing a bunch of flags in a numpy array having dtype
> >>> np.int32?  It's contrived, I agree.
> >>
> >>
> >> I afraid that composition will not help you with this. Can numpy array
> pack int-like objects into fixed-width integer array and then restore
> original type on unboxing?
> >
> >
> > You're right.
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> Python-Dev mailing list
> >> Python-Dev at python.org
> >> https://mail.python.org/mailman/listinfo/python-dev
> >> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/mistersheik%40gmail.com
> >
> >
> >
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150309/7b975664/attachment-0001.html>

From benhoyt at gmail.com  Tue Mar 10 01:40:50 2015
From: benhoyt at gmail.com (Ben Hoyt)
Date: Mon, 9 Mar 2015 20:40:50 -0400
Subject: [Python-Dev] PEP 471 Final: os.scandir() merged into Python 3.5
In-Reply-To: <CAM8kh5v8GHtxGEkHuMjUiuuTjkairvU8zddA3sZMgUizJFsB7g@mail.gmail.com>
References: <CAMpsgwYg=RZeyqTzLGn+Nr7xEiiff0EUDhEUgGS-So=B-DjJ1g@mail.gmail.com>
 <CAL9jXCEUOkR_EvGzA6kdodTk3S6+AfnWpRkTmJ27ZRtUaXyAYw@mail.gmail.com>
 <CAM8kh5tL4QgJa4SaaxRsGzgV4RVpZxpdBC0VZ1f1ZKth4nF7HQ@mail.gmail.com>
 <CAL9jXCHrcvH48QhgCdZWTNUoLbSdJkH9QMOEHrnRPs==hTzxaw@mail.gmail.com>
 <CAM8kh5v8GHtxGEkHuMjUiuuTjkairvU8zddA3sZMgUizJFsB7g@mail.gmail.com>
Message-ID: <CAL9jXCF5Nroqp8eOP296E0rsCf9zc=PSanMPD9HrB1q+j2Okmg@mail.gmail.com>

>
> os.walk took 0.061s, scandir.walk took 0.012s -- 5.2x as fast
>

Great, looks much better. :-) Even a bit better than what I'm seeing --
possibly due to your SSD.

-Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150309/c48935b6/attachment.html>

From phqnha at gmail.com  Tue Mar 10 04:46:12 2015
From: phqnha at gmail.com (nha pham)
Date: Mon, 9 Mar 2015 20:46:12 -0700
Subject: [Python-Dev] Tunning binary insertion sort algorithm in Timsort.
In-Reply-To: <76f0eced29ac4.54fd9476@wiscmail.wisc.edu>
References: <CAMLcfj3usoq-new2gxh_ymP+1wv6XZj7+og6XMVAUj_sYeT+Pg@mail.gmail.com>
 <7750938a2fea2.54fdd09e@wiscmail.wisc.edu>
 <76e0c70929f48.54fdd0da@wiscmail.wisc.edu>
 <76f094a32c8ca.54fdd119@wiscmail.wisc.edu>
 <7740f7e62cf4b.54fdd155@wiscmail.wisc.edu>
 <778088e82a599.54fdd192@wiscmail.wisc.edu>
 <7740ebbf29f9e.54fdd2be@wiscmail.wisc.edu>
 <76f0f50b2c800.54fdd2fb@wiscmail.wisc.edu>
 <7710a1122c5ba.54fdd337@wiscmail.wisc.edu>
 <7780f4192fd2c.54fdd373@wiscmail.wisc.edu>
 <7790d04c299a7.54fdd590@wiscmail.wisc.edu>
 <7710a1822e0a9.54fdd5cc@wiscmail.wisc.edu>
 <7720dc73289e8.54fdd609@wiscmail.wisc.edu>
 <7790de4c2be2c.54fdd645@wiscmail.wisc.edu>
 <76f0eced29ac4.54fd9476@wiscmail.wisc.edu>
Message-ID: <CAMLcfj0Fr_mGP+=xv0_Q1SVnR_GTKxK1cZs1DP-x+YaZBgHZrQ@mail.gmail.com>

Thank you for your comment. I admit that I did not thinking about the proof
before.
Here is my naive proof. I hope you can give me some comments:

=============
# This proof is an empirical thinking and is not completed, it just gives
us a closer look.
I hope someone can make it more mathematically.

In the proof, we assume we are dealing with unique values array (none of
them are equal together).
Because if they are equal, the "lucky search" can happen and it is
obviously not fair.

Statement_1: With an array of size N or less than N, we need at most
log2(N) comparisons to find a value
(or a position, incase the search miss), using the binary search algorithm.

proof: This statement is trivia, and I believe, someone outthere already
proved it. We can check again
by counting manually.

    let assume we have array of 32 items:
    32 => 16 => 8 => 4 => 2 => 1  (5 comparison)

    how about 24 items (24 < 32):
    24 => 12 => 6 => 3 => 2 => 1  (5 comparison)

ok, good enough. Let's just believe on it to move on.


Statement_2: If we divide an array into two parts, the more unbalanced
arrays we divide, the more
benefit we get from the binary search algorithm.

proof: Let's assume we have an array of 256 items.

    case1:
    If we divide in middle: 128 - 128
    Now, if we search on the left, it costs log2(128) = 7
    If we search on the right, it cost los2(128) = 7

    case2:
    If we divide unbalanced: 32 - 224
    Now, if we search on the left, it costs log2(32) = 5
    If we search on the right, it cost at max 8 comparisons (based on the
statement_1).
    You may not believe me, so let's count it by hand:
    224 => 112 => 56 => 28 => 14 => 7 => 4 => 2 => 1
    So, if we search on the left, we win 2 comparisons compare to case1.
    We search on the right, we lose 1 comparison compare to case1
    I call this is a "benefit".

    case3:
    What if we divide more unbalanced: 8 - 248
    Search on the left: log2(8) = 3 comparisons.
    Search on the right, it costs at max 8 comparisons.
    So if we search on the left, we win 4 comparisons.
    We search on the right, we lose 1 comparisons.
    It is "more benefit", isnt it?


Statement3: Because we are using random array. There is a 50-50 chance that
 next_X will be bigger or smaller than X.


Statement4: We call N is the size of the sorted list, "index" is the
position of X in the sorted list.
Because the array is random, index has an equal chance to exist in any
position in the sorted list.

Statement5: Now we build a model based on previous statements:

    My idea costs 1 comparison (between X and next_X) to devide the array
into two unbalanced parts.
    The old idea costs 1 comparison to divide the array into two balanced
parts.
    Now let's see which one can find position for next_X faster:

    If index belongs to [N/4 to 3N/4]: we may lose 1 comparison, or we may
not lose.
    If index belongs to [N/8 to N/4] or [3N/4 to 7N/8]: We may lose 1
comparison, or we win 1 comparison.
    If index belongs to [N/16 to N/8] or [7N/8 to 15N/16]: We may lose 1
comparison, or we win 2 comparison.
    If index belongs to [N/32 to N/16] or [15N/16 to 31N/32]: We may lose 1
comparison, or we win 3 comparison.
    If index belongs to [N/64 to N/32] or [31N/32 to 64N/64]: We may lose 1
comparison, or we win 4 comparison.
    ...
    and so on.

Statement6: Now we apply the model to a real example.

    Assume that we already has a sorted list with 16 items. And we already
know about "index" of X.
    We can think of it as a gamble game with 16 slots. In every slot, we
only can bid 1 dollar (statement4).

    From slot 5th to slot 12th, we may lose 1, or we may not lose, 50-50
chance.
    So after a huge play times, probability told us that we will lose (8 x
1)/2 = 4 dollars.

    For slot 3, slot 4, slot 13, slot 14, We may lose 1, or we win 1. So
after a huge play times,
    We wont lose or win anything.

    For slot 2, slot 15. We may lose 1, or we win 2. So after a huge play
times, we can win
    (2-1)x2 = 2 dollars.

    For slot 1, slot 16. We may lose 1, or we win 3. So after a huge play
times, we can win 4 dollars.

    In total, after a huge play times, we win 4 + 2 + 0 -4 = 2 dollars !!!!!

    You can test with sorted list 32 or 64 items or any number you want, I
believe the benefit is even more.

Conclusion:
    The unbalanced model give us more benefit than the balanced model. That
means with an array big enough,
    My idea give more benefit than the old idea.


    I think the lucky ticket companies is already know about this. It is a
shame that I do not know
    mathematic principle about this problem.


If I have something more, I will update my proof at:
https://github.com/nhapq/Optimize_binary_insertion_sort/blob/master/proof.txt

======
Thank you.
Nha Pham.

On Mon, Mar 9, 2015 at 10:39 AM, Isaac Schwabacher <ischwabacher at wisc.edu>
wrote:

> On 15-03-08, nha pham
>  wrote:
> >
> > We can optimize the TimSort algorithm by optimizing its binary insertion
> sort.
> >
> > The current version of binary insertion sort use this idea:
> >
> > Use binary search to find a final position in sorted list for a new
> element X. Then insert X to that location.
> >
> > I suggest another idea:
> >
> > Use binary search to find a final postion in sorted list for a new
> element X. Before insert X to that location, compare X with its next
> element.
> >
> > For the next element, we already know if it is lower or bigger than X,
> so we can reduce the search area to the left side or on the right side of X
> in the sorted list.
>
> I don't understand how this is an improvement, since with binary search
> the idea is that each comparison cuts the remaining list to search in half;
> i.e., each comparison yields one bit of information. Here, you're spending
> a comparison to cut the list to search at the element you just inserted,
> which is probably not right in the middle. If you miss the middle, you're
> getting on average less than a full bit of information from your
> comparison, so you're not reducing the remaining search space by as much as
> you would be if you just compared to the element in the middle of the list.
>
> > I have applied my idea on java.util. ComparableTimSort.sort() and
> testing. The execute time is reduced by 2%-6% with array of random integer.
>
> For all that, though, experiment trumps theory...
>
> > Here is detail about algorithm and testing:
> https://github.com/nhapq/Optimize_binary_insertion_sort
> >
> > Sincerely.
> >
> > phqnha
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150309/b8036606/attachment.html>

From tim.peters at gmail.com  Tue Mar 10 05:27:02 2015
From: tim.peters at gmail.com (Tim Peters)
Date: Mon, 9 Mar 2015 23:27:02 -0500
Subject: [Python-Dev] Tunning binary insertion sort algorithm in Timsort.
In-Reply-To: <CAMLcfj0Fr_mGP+=xv0_Q1SVnR_GTKxK1cZs1DP-x+YaZBgHZrQ@mail.gmail.com>
References: <CAMLcfj3usoq-new2gxh_ymP+1wv6XZj7+og6XMVAUj_sYeT+Pg@mail.gmail.com>
 <7750938a2fea2.54fdd09e@wiscmail.wisc.edu>
 <76e0c70929f48.54fdd0da@wiscmail.wisc.edu>
 <76f094a32c8ca.54fdd119@wiscmail.wisc.edu>
 <7740f7e62cf4b.54fdd155@wiscmail.wisc.edu>
 <778088e82a599.54fdd192@wiscmail.wisc.edu>
 <7740ebbf29f9e.54fdd2be@wiscmail.wisc.edu>
 <76f0f50b2c800.54fdd2fb@wiscmail.wisc.edu>
 <7710a1122c5ba.54fdd337@wiscmail.wisc.edu>
 <7780f4192fd2c.54fdd373@wiscmail.wisc.edu>
 <7790d04c299a7.54fdd590@wiscmail.wisc.edu>
 <7710a1822e0a9.54fdd5cc@wiscmail.wisc.edu>
 <7720dc73289e8.54fdd609@wiscmail.wisc.edu>
 <7790de4c2be2c.54fdd645@wiscmail.wisc.edu>
 <76f0eced29ac4.54fd9476@wiscmail.wisc.edu>
 <CAMLcfj0Fr_mGP+=xv0_Q1SVnR_GTKxK1cZs1DP-x+YaZBgHZrQ@mail.gmail.com>
Message-ID: <CAExdVNmi2RxncnJZrhL7thNaFLkf3aVPX=yQBQDD-HEV8zTzDw@mail.gmail.com>

[nha pham <phqnha at gmail.com>]
> Statement_1: With an array of size N or less than N, we need at most log2(N)
> comparisons to find a value
> (or a position, incase the search miss), using the binary search algorithm.
>
> proof: This statement is trivia, and I believe, someone outthere already
> proved it.

Sorry for the quick message here.  It's just a simple point where it
will pay not to get off on a wrong foot ;-)

Correct:  for an array of size N, binary search can require as many as
ceiling(log2(N+1)) comparisons.

That's because there are N+1 possible results for an array of size N.
For example, for an array of size 3, [A, B, C], "the answer" may be
"before A", "between A and B", "between B and C", or "after C".  3
elements, 3+1 = 4 possible results.  log2(3) comparisons are not
enough to distinguish among 4 results.

Make it trivial, an array of length 1.  Then 1 comparison is obviously
necessary and sufficient in all cases.  And, indeed,
ceiling(log2(1+1)) = 1.  log2(1) equals 0, too small.

For the rest, I haven't been able to understand your description or
your pseudo-code.  I'll try harder.  Some things clearly aren't doing
what you _intend_ them to do.  For example, in your Python code, each
time through the outer loop you're apparently trying to sort the next
CHUNK elements, but you end up appending CHUNK+1 values to data2 (or
data3).

Or in this part:

    for i in range(low,high):
        x = data[i]
        if x >= data[i-1]:

the first time that loop is executed low == 0, and so i == 0 on the
first iteration, and so the conditional is

       if x >= data[0-1]

That's referencing data[-1], which is the very last element in data -
which has nothing to do with the CHUNK you're trying to sort at the
time.

So there are a number of errors here, which makes it that much harder
to sort out (pun intended <wink>) what you're trying to do.  It would
help you to add some asserts to verify your code is doing what you
_hope_ it's doing.  For example, add

    assert data2[low: high] == sorted(data[low: high])
    assert len(data2) == high

to the end of your `sample` loop, and similarly for data3 in your
`new` loop.  Until those asserts all pass, you're not testing code
that's actually sorting correctly.  Repair the errors and you almost
certainly won't find `new` running over 10 times faster than `sample`
anymore.  I don't know what you _will_ discover, though.  If the code
doesn't have to sort correctly, there are much simpler ways to make it
run _very_ much faster ;-)

From phqnha at gmail.com  Tue Mar 10 06:05:39 2015
From: phqnha at gmail.com (nha pham)
Date: Mon, 9 Mar 2015 22:05:39 -0700
Subject: [Python-Dev] Tunning binary insertion sort algorithm in Timsort.
In-Reply-To: <CAExdVNmi2RxncnJZrhL7thNaFLkf3aVPX=yQBQDD-HEV8zTzDw@mail.gmail.com>
References: <CAMLcfj3usoq-new2gxh_ymP+1wv6XZj7+og6XMVAUj_sYeT+Pg@mail.gmail.com>
 <7750938a2fea2.54fdd09e@wiscmail.wisc.edu>
 <76e0c70929f48.54fdd0da@wiscmail.wisc.edu>
 <76f094a32c8ca.54fdd119@wiscmail.wisc.edu>
 <7740f7e62cf4b.54fdd155@wiscmail.wisc.edu>
 <778088e82a599.54fdd192@wiscmail.wisc.edu>
 <7740ebbf29f9e.54fdd2be@wiscmail.wisc.edu>
 <76f0f50b2c800.54fdd2fb@wiscmail.wisc.edu>
 <7710a1122c5ba.54fdd337@wiscmail.wisc.edu>
 <7780f4192fd2c.54fdd373@wiscmail.wisc.edu>
 <7790d04c299a7.54fdd590@wiscmail.wisc.edu>
 <7710a1822e0a9.54fdd5cc@wiscmail.wisc.edu>
 <7720dc73289e8.54fdd609@wiscmail.wisc.edu>
 <7790de4c2be2c.54fdd645@wiscmail.wisc.edu>
 <76f0eced29ac4.54fd9476@wiscmail.wisc.edu>
 <CAMLcfj0Fr_mGP+=xv0_Q1SVnR_GTKxK1cZs1DP-x+YaZBgHZrQ@mail.gmail.com>
 <CAExdVNmi2RxncnJZrhL7thNaFLkf3aVPX=yQBQDD-HEV8zTzDw@mail.gmail.com>
Message-ID: <CAMLcfj2Xq+dgO+cKmGbQ5NaZjnuKggCr3K-S4rt=GBSHXSGMVA@mail.gmail.com>

Thank you very much. I am very happy that I got a reply from Tim Peter.

You are correct, my mistake.

The python code should be:
    for i in range(low+1,high):              //because we already add
data[low]
        x = data[i]
        if x >= data[i-1]:

After I fix it, here is the result:

random array 10^6:
Old binsort:  1.3322
New binsort: 1.0015
ratio: 0.33

You are right, it is not ten times faster anymore. I will update other
results soon.

I do check the result of two sorting methods many times to make sure they
are the same. It is just because I do not know how to put assert into the
timeit.Timer class. I am pretty sure about this.

I will try to write the proof more clearly, sorry for inconvenience.

Thank you very much.
Nha Pham.

On Mon, Mar 9, 2015 at 9:27 PM, Tim Peters <tim.peters at gmail.com> wrote:

> [nha pham <phqnha at gmail.com>]
> > Statement_1: With an array of size N or less than N, we need at most
> log2(N)
> > comparisons to find a value
> > (or a position, incase the search miss), using the binary search
> algorithm.
> >
> > proof: This statement is trivia, and I believe, someone outthere already
> > proved it.
>
> Sorry for the quick message here.  It's just a simple point where it
> will pay not to get off on a wrong foot ;-)
>
> Correct:  for an array of size N, binary search can require as many as
> ceiling(log2(N+1)) comparisons.
>
> That's because there are N+1 possible results for an array of size N.
> For example, for an array of size 3, [A, B, C], "the answer" may be
> "before A", "between A and B", "between B and C", or "after C".  3
> elements, 3+1 = 4 possible results.  log2(3) comparisons are not
> enough to distinguish among 4 results.
>
> Make it trivial, an array of length 1.  Then 1 comparison is obviously
> necessary and sufficient in all cases.  And, indeed,
> ceiling(log2(1+1)) = 1.  log2(1) equals 0, too small.
>
> For the rest, I haven't been able to understand your description or
> your pseudo-code.  I'll try harder.  Some things clearly aren't doing
> what you _intend_ them to do.  For example, in your Python code, each
> time through the outer loop you're apparently trying to sort the next
> CHUNK elements, but you end up appending CHUNK+1 values to data2 (or
> data3).
>
> Or in this part:
>
>     for i in range(low,high):
>         x = data[i]
>         if x >= data[i-1]:
>
> the first time that loop is executed low == 0, and so i == 0 on the
> first iteration, and so the conditional is
>
>        if x >= data[0-1]
>
> That's referencing data[-1], which is the very last element in data -
> which has nothing to do with the CHUNK you're trying to sort at the
> time.
>
> So there are a number of errors here, which makes it that much harder
> to sort out (pun intended <wink>) what you're trying to do.  It would
> help you to add some asserts to verify your code is doing what you
> _hope_ it's doing.  For example, add
>
>     assert data2[low: high] == sorted(data[low: high])
>     assert len(data2) == high
>
> to the end of your `sample` loop, and similarly for data3 in your
> `new` loop.  Until those asserts all pass, you're not testing code
> that's actually sorting correctly.  Repair the errors and you almost
> certainly won't find `new` running over 10 times faster than `sample`
> anymore.  I don't know what you _will_ discover, though.  If the code
> doesn't have to sort correctly, there are much simpler ways to make it
> run _very_ much faster ;-)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150309/23d7bb73/attachment.html>

From tim.peters at gmail.com  Tue Mar 10 06:28:22 2015
From: tim.peters at gmail.com (Tim Peters)
Date: Tue, 10 Mar 2015 00:28:22 -0500
Subject: [Python-Dev] Tunning binary insertion sort algorithm in Timsort.
In-Reply-To: <CAMLcfj2Xq+dgO+cKmGbQ5NaZjnuKggCr3K-S4rt=GBSHXSGMVA@mail.gmail.com>
References: <CAMLcfj3usoq-new2gxh_ymP+1wv6XZj7+og6XMVAUj_sYeT+Pg@mail.gmail.com>
 <7750938a2fea2.54fdd09e@wiscmail.wisc.edu>
 <76e0c70929f48.54fdd0da@wiscmail.wisc.edu>
 <76f094a32c8ca.54fdd119@wiscmail.wisc.edu>
 <7740f7e62cf4b.54fdd155@wiscmail.wisc.edu>
 <778088e82a599.54fdd192@wiscmail.wisc.edu>
 <7740ebbf29f9e.54fdd2be@wiscmail.wisc.edu>
 <76f0f50b2c800.54fdd2fb@wiscmail.wisc.edu>
 <7710a1122c5ba.54fdd337@wiscmail.wisc.edu>
 <7780f4192fd2c.54fdd373@wiscmail.wisc.edu>
 <7790d04c299a7.54fdd590@wiscmail.wisc.edu>
 <7710a1822e0a9.54fdd5cc@wiscmail.wisc.edu>
 <7720dc73289e8.54fdd609@wiscmail.wisc.edu>
 <7790de4c2be2c.54fdd645@wiscmail.wisc.edu>
 <76f0eced29ac4.54fd9476@wiscmail.wisc.edu>
 <CAMLcfj0Fr_mGP+=xv0_Q1SVnR_GTKxK1cZs1DP-x+YaZBgHZrQ@mail.gmail.com>
 <CAExdVNmi2RxncnJZrhL7thNaFLkf3aVPX=yQBQDD-HEV8zTzDw@mail.gmail.com>
 <CAMLcfj2Xq+dgO+cKmGbQ5NaZjnuKggCr3K-S4rt=GBSHXSGMVA@mail.gmail.com>
Message-ID: <CAExdVNka3o3x-=DT3uk5wu1nDMyH9HLeyGDT2YnMiiQwQSA3VA@mail.gmail.com>

[nha pham <phqnha at gmail.com>]
> Thank you very much. I am very happy that I got a reply from Tim Peter.

My pleasure to speak with you too :-)


> You are correct, my mistake.
>
> The python code should be:
>     for i in range(low+1,high):              //because we already add
> data[low]
>         x = data[i]
>         if x >= data[i-1]:
>
> After I fix it, here is the result:
>
> random array 10^6:
> Old binsort:  1.3322
> New binsort: 1.0015
> ratio: 0.33
>
> You are right, it is not ten times faster anymore. I will update other
> results soon.
>
> I do check the result of two sorting methods many times to make sure they
> are the same. It is just because I do not know how to put assert into the
> timeit.Timer class.

`assert` is just another Python statement.  You simply add it to the
code - there's nothing tricky about this.  You could, e.g., simply
copy and paste the `assert`s I suggested last time.

Before you do, trying adding `print index` to your inner loops, and
make SIZE much smaller (say, 1000) so you're not overwhelmed with
output.  You'll be surprised by what you see on the second (and
following) CHUNKs.  For example, in both `sample` and `new` it will
print 900 ninety nine times in a row when doing the last CHUNK.  The
code still isn't doing what you intend.  Until it does, timing it
makes little sense :-)

> I am pretty sure about this.

Note that I'm talking about the Python code here, the code you run
through timeit.  You cannot have checked the results of running _that_
specific code, because it doesn't work at all.  You may have checked
_other_ code many times.  We may get to that later, but since I speak
Python, I'm not going to understand what you're doing until we have
Python code that works ;-)

From p.f.moore at gmail.com  Tue Mar 10 11:35:06 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 10 Mar 2015 10:35:06 +0000
Subject: [Python-Dev] Windows installer - File associations in "per user"
	installs
Message-ID: <CACac1F8-S12jkactoTr-Y0Ut_F2YXdEmj0cfF70N1BqR2VttrQ@mail.gmail.com>

On 9 March 2015 at 15:37, Steve Dower <Steve.Dower at microsoft.com> wrote:
>> Maybe the answer is that we simply start recommending that everyone on Windows
>> uses per-user installs. It makes little difference to me (beyond the fact that
>> when I want to look at the source of something in the stdlib, the location of
>> the file is a lot harder to remember than C:\Apps\Python34\Lib\whatever.py) but
>> I doubt it's what most people will expect.
>
> I'm okay with this. Installing for all users is really something that could be considered an advanced option rather than the default, especially since the aim (AIUI) of the all-users install is to pretend that Python was shipped with the OS. (I'd kind of like to take that further by splitting things more sensibly between Program Files, Common Files and System32, but there's very little gain from that and much MUCH pain as long as people are still expecting C:\PythonXY installs...)

I've just tried a per-user install of Python 3.5a2. The machine in
question previously had (and still has) a system install of 3.4, with
"Make this Python the default" selected (so the .py extension is
associated with that version and specifically the 3.4 launcher).

I didn't get the option to associate .py files with 3.5 (there's *no
way* I'd consider that to be advanced usage - if I'm installing
Python, why wouldn't I want to associate it with .py files [1]) and I
still seem to have .py associated with the 3.4 launcher, not the 3.5
one that's in my %APPDATA% folder.

    >cmd /c assoc .py
    .py=Python.File
    >cmd /c ftype python.file
    python.file="C:\WINDOWS\py.exe" "%1" %*

I'm happy if a per-user install of 3.5 makes a per-user filetype
association (assuming such a thing is possible, I've never tried it
before) but it's absolutely not OK if we're planning on recommending
an install type that doesn't create the right association.

Paul

[1] Given that I have 3.4 and am installing an experimental 3.5
version, it's not actually at all clear cut which version I want as my
default. In all honesty, I don't think this decision is actually
something that should be defaulted. Maybe the "don't make the user
make any choices in the default selection" approach has gone a little
too far here?

From Steve.Dower at microsoft.com  Tue Mar 10 14:09:55 2015
From: Steve.Dower at microsoft.com (Steve Dower)
Date: Tue, 10 Mar 2015 13:09:55 +0000
Subject: [Python-Dev] Windows installer - File associations in "per
	user" installs
In-Reply-To: <CACac1F8-S12jkactoTr-Y0Ut_F2YXdEmj0cfF70N1BqR2VttrQ@mail.gmail.com>
References: <CACac1F8-S12jkactoTr-Y0Ut_F2YXdEmj0cfF70N1BqR2VttrQ@mail.gmail.com>
Message-ID: <DM2PR0301MB07340AA373A62B9226C2CFDBF5180@DM2PR0301MB0734.namprd03.prod.outlook.com>

It's a bug. File and assign to me please.

Top-posted from my Windows Phone
________________________________
From: Paul Moore<mailto:p.f.moore at gmail.com>
Sent: ?3/?10/?2015 3:35
To: Steve Dower<mailto:Steve.Dower at microsoft.com>
Cc: Python Dev<mailto:python-dev at python.org>
Subject: Windows installer - File associations in "per user" installs

On 9 March 2015 at 15:37, Steve Dower <Steve.Dower at microsoft.com> wrote:
>> Maybe the answer is that we simply start recommending that everyone on Windows
>> uses per-user installs. It makes little difference to me (beyond the fact that
>> when I want to look at the source of something in the stdlib, the location of
>> the file is a lot harder to remember than C:\Apps\Python34\Lib\whatever.py) but
>> I doubt it's what most people will expect.
>
> I'm okay with this. Installing for all users is really something that could be considered an advanced option rather than the default, especially since the aim (AIUI) of the all-users install is to pretend that Python was shipped with the OS. (I'd kind of like to take that further by splitting things more sensibly between Program Files, Common Files and System32, but there's very little gain from that and much MUCH pain as long as people are still expecting C:\PythonXY installs...)

I've just tried a per-user install of Python 3.5a2. The machine in
question previously had (and still has) a system install of 3.4, with
"Make this Python the default" selected (so the .py extension is
associated with that version and specifically the 3.4 launcher).

I didn't get the option to associate .py files with 3.5 (there's *no
way* I'd consider that to be advanced usage - if I'm installing
Python, why wouldn't I want to associate it with .py files [1]) and I
still seem to have .py associated with the 3.4 launcher, not the 3.5
one that's in my %APPDATA% folder.

    >cmd /c assoc .py
    .py=Python.File
    >cmd /c ftype python.file
    python.file="C:\WINDOWS\py.exe" "%1" %*

I'm happy if a per-user install of 3.5 makes a per-user filetype
association (assuming such a thing is possible, I've never tried it
before) but it's absolutely not OK if we're planning on recommending
an install type that doesn't create the right association.

Paul

[1] Given that I have 3.4 and am installing an experimental 3.5
version, it's not actually at all clear cut which version I want as my
default. In all honesty, I don't think this decision is actually
something that should be defaulted. Maybe the "don't make the user
make any choices in the default selection" approach has gone a little
too far here?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150310/0ed4b10c/attachment.html>

From jimjjewett at gmail.com  Tue Mar 10 17:01:31 2015
From: jimjjewett at gmail.com (Jim J. Jewett)
Date: Tue, 10 Mar 2015 09:01:31 -0700 (PDT)
Subject: [Python-Dev] Thoughts on running Python 3.5 on Windows (path,
 pip install --user, etc)
In-Reply-To: <CACac1F_6j82eEWfb_q-WJg58EdZPvuabXgrf7PxKujV+2sA_xw@mail.gmail.com>
Message-ID: <54ff155b.c9298c0a.23d8.41a0@mx.google.com>



On 10 March 2015 at slightly after midnight. Paul Moore wrote:


> Personally I doubt it would make much difference. If the docs say
> "pygmentize" I'm unlikely to dig around to find that the incantation
> "python -m pygments.somemodule:main" does the same thing using 3 times
> as many characters. I'd just add Python to my PATH and say stuff it.

There is value in getting the incantation down to a single (preferably
short) line, because then it can be used as a shortcut.  That means it
can be created as a shortcut at installation time, and that someone
writing their own batch file can just cut and paste from the shortcut
properties' target.  Not as simple as just adding to the path, but
simpler than adding several directories to the path, or modifying other
environment variables, or fighting an existing but conflicting python
installation already on the path.

-jJ

--

If there are still threading problems with my replies, please
email me with details, so that I can try to resolve them.  -jJ

From tim.peters at gmail.com  Tue Mar 10 18:22:58 2015
From: tim.peters at gmail.com (Tim Peters)
Date: Tue, 10 Mar 2015 12:22:58 -0500
Subject: [Python-Dev] Tunning binary insertion sort algorithm in Timsort.
In-Reply-To: <CAExdVNka3o3x-=DT3uk5wu1nDMyH9HLeyGDT2YnMiiQwQSA3VA@mail.gmail.com>
References: <CAMLcfj3usoq-new2gxh_ymP+1wv6XZj7+og6XMVAUj_sYeT+Pg@mail.gmail.com>
 <7750938a2fea2.54fdd09e@wiscmail.wisc.edu>
 <76e0c70929f48.54fdd0da@wiscmail.wisc.edu>
 <76f094a32c8ca.54fdd119@wiscmail.wisc.edu>
 <7740f7e62cf4b.54fdd155@wiscmail.wisc.edu>
 <778088e82a599.54fdd192@wiscmail.wisc.edu>
 <7740ebbf29f9e.54fdd2be@wiscmail.wisc.edu>
 <76f0f50b2c800.54fdd2fb@wiscmail.wisc.edu>
 <7710a1122c5ba.54fdd337@wiscmail.wisc.edu>
 <7780f4192fd2c.54fdd373@wiscmail.wisc.edu>
 <7790d04c299a7.54fdd590@wiscmail.wisc.edu>
 <7710a1822e0a9.54fdd5cc@wiscmail.wisc.edu>
 <7720dc73289e8.54fdd609@wiscmail.wisc.edu>
 <7790de4c2be2c.54fdd645@wiscmail.wisc.edu>
 <76f0eced29ac4.54fd9476@wiscmail.wisc.edu>
 <CAMLcfj0Fr_mGP+=xv0_Q1SVnR_GTKxK1cZs1DP-x+YaZBgHZrQ@mail.gmail.com>
 <CAExdVNmi2RxncnJZrhL7thNaFLkf3aVPX=yQBQDD-HEV8zTzDw@mail.gmail.com>
 <CAMLcfj2Xq+dgO+cKmGbQ5NaZjnuKggCr3K-S4rt=GBSHXSGMVA@mail.gmail.com>
 <CAExdVNka3o3x-=DT3uk5wu1nDMyH9HLeyGDT2YnMiiQwQSA3VA@mail.gmail.com>
Message-ID: <CAExdVNkrB9JBhPGC5pBRD85HWsCHzoNADLCfBtSxvAL3WXQqqg@mail.gmail.com>

OK - here's what the current binsort does, ignoring that it skips an
already-sorted prefix (if any), and creating a new list instead of
modifying in-place:

    def oldsort(a):
        from bisect import bisect_right as br
        assert a
        result = [a[0]]
        for i in range(1, len(a)):
            x = a[i]
            index = br(result, x)
            result.insert(index, x)
        return result

And here's my best guess as to what you _intend_ the new version to
do.  Please confirm that, or, if I'm guessing wrong, please give a
Python function that _does_ implement your intent:

    def newsort(a):
        from bisect import bisect_right as br
        assert a
        oldx = a[0]
        result = [oldx]
        index = 0
        for i in range(1, len(a)):
            x = a[i]
            if x < oldx:
                index = br(result, x, 0, index)
            else:
                index = br(result, x, index + 1)
            result.insert(index, x)
            oldx = x
        return result

Now assuming that's right, I don't care about timing it ;-)  The only
basic question to me is whether it in fact reduces the number of
comparisons.  So here's an integer wrapper that bumps a global counter
whenever it's asked to compare:

    class IntWrap(object):
        def __init__(self, i):
            self.i = i

        def __cmp__(a, b):
            global gncmp
            gncmp += 1
            return cmp(a.i, b.i)

        def __repr__(self):
            return repr(self.i)

Now we can build lists containing that, and get exact comparison
counts.  To start, for a given length `n`, this counts the total
number of comparisons needed to sort all possible permutations of a
list of length n, under both the old and new ways:

    def drive(n):
        import itertools
        global gncmp
        base = [IntWrap(i) for i in range(n)]
        oldcount = newcount = 0
        numperms = 0
        for p in itertools.permutations(base):
            numperms += 1

            gncmp = 0
            oldresult = oldsort(p)
            oldcount += gncmp

            gncmp = 0
            newresult = newsort(p)
            newcount += gncmp

            assert oldresult == newresult == base
        print 'n', n, 'perms', numperms
        print 'old compares', oldcount
        print 'new compares', newcount
        print 'diff %', (newcount - oldcount) * 100.0 / max(oldcount, 1)

And, finally, a tiny loop to drive it:

    for n in range(1, 13):
        print
        drive(n)

It's still running as I type this, but the results aren't promising so
far - as soon as the list length gets non-trivial, the new way
requires more comparisons than the old way so far:

n 1 perms 1
old compares 0
new compares 0
diff % 0.0

n 2 perms 2
old compares 2
new compares 2
diff % 0.0

n 3 perms 6
old compares 16
new compares 16
diff % 0.0

n 4 perms 24
old compares 112
new compares 116
diff % 3.57142857143

n 5 perms 120
old compares 848
new compares 880
diff % 3.77358490566

n 6 perms 720
old compares 7008
new compares 7296
diff % 4.1095890411

n 7 perms 5040
old compares 63456
new compares 66432
diff % 4.68986384266

n 8 perms 40320
old compares 628608
new compares 662496
diff % 5.39095907147

n 9 perms 362880
old compares 6826752
new compares 7202304
diff % 5.50118123523

n 10 perms 3628800
old compares 80605440
new compares 85006080
diff % 5.45948263542

I believe it would be very difficult to analyze this rigorously - and
even if I saw an analysis it would be hard to trust it.  Raw counts
from simple code are hard to argue with ;-)

FYI, here are two ideas I had way back when, but didn't pursue:

1. Merge "2 at a time" instead of just 1.  That is, first "sort" the
next 2 elements to be merged (1 compare and a possible swap).  Then
binary search to find where the smaller belongs, and a shorter binary
search to find where the larger belongs.  Then shift both into place.
This can win on two counts:

A. Less data movement, since the already-sorted values after the
larger element get moved only once instead of twice.

B. A possible cut in number of compares.  Merging a sorted list of N
elements with a sorted list of 2 elements has been studied a lot
(e.g., search for "optimal merging of 2 elements" and find the paper
by Hwang and Lin).  The minimum average theoretical number of compares
needed is
ceiling(log2((N+2)*(N+1)/2)).

2. Instead of binary insertion sort, do an ordinary (but optimized)
bottom-up merge sort.  That may not cut the number of compares, but
would slash worst-case data movement cost from O(n**2) to O(n*log(n)).

As to why your code is sometimes faster, for the Python code in your
timing harness, well, that didn't actually sort anything, so wasn't
measuring anything interesting (or even explainable ;-) ).  For the
Java code, I have no guess - I don't know enough about Java internals.
Maybe "lucky" data, maybe cache effects, maybe a mistake - don't know,
and can't guess.  Or maybe my guess (above) at the intent of your code
is all wrong.

It _is_ an interesting idea, though!  Thanks for bringing it up :-)

From lox.xiao at gmail.com  Tue Mar 10 18:27:21 2015
From: lox.xiao at gmail.com (lou xiao)
Date: Wed, 11 Mar 2015 01:27:21 +0800
Subject: [Python-Dev] str.lstrip bug?
Message-ID: <CABHZ6sd7z085yLndxbjo5mbQ=NhOFzMgVR-5mnHco_45VRRX=Q@mail.gmail.com>

I find a bug in str.lstrip, when i call str.lstrip, i get this result.

tiny? ~ python
Python 2.7.5+ (default, Feb 27 2014, 19:37:08)
[GCC 4.8.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> a='device_info'
>>> a.lstrip('device')
'_info'
>>> a.lstrip('device_')
'nfo'
>>>
tiny? ~ uname -a
Linux tinymint 3.11.0-12-generic #19-Ubuntu SMP Wed Oct 9 16:20:46 UTC 2013
x86_64 x86_64 x86_64 GNU/Linux
tiny? ~
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150311/36d2fad5/attachment.html>

From rosuav at gmail.com  Tue Mar 10 18:33:44 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 11 Mar 2015 04:33:44 +1100
Subject: [Python-Dev] str.lstrip bug?
In-Reply-To: <CABHZ6sd7z085yLndxbjo5mbQ=NhOFzMgVR-5mnHco_45VRRX=Q@mail.gmail.com>
References: <CABHZ6sd7z085yLndxbjo5mbQ=NhOFzMgVR-5mnHco_45VRRX=Q@mail.gmail.com>
Message-ID: <CAPTjJmotpn4BW1FNpGncCoJTBMy9+3WZ5SB8Vb6uK8XANmn=5Q@mail.gmail.com>

On Wed, Mar 11, 2015 at 4:27 AM, lou xiao <lox.xiao at gmail.com> wrote:
> I find a bug in str.lstrip, when i call str.lstrip, i get this result.
>
> tiny? ~ python
> Python 2.7.5+ (default, Feb 27 2014, 19:37:08)
> [GCC 4.8.1] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> a='device_info'
>>>> a.lstrip('device')
> '_info'
>>>> a.lstrip('device_')
> 'nfo'
>>>>
> tiny? ~ uname -a
> Linux tinymint 3.11.0-12-generic #19-Ubuntu SMP Wed Oct 9 16:20:46 UTC 2013
> x86_64 x86_64 x86_64 GNU/Linux
> tiny? ~

It's not a bug, because it isn't doing what you think it is. It strips
a *set of characters*, not a prefix string.

https://docs.python.org/2/library/stdtypes.html#str.lstrip
https://docs.python.org/3/library/stdtypes.html#str.lstrip

ChrisA

From facundobatista at gmail.com  Tue Mar 10 18:34:26 2015
From: facundobatista at gmail.com (Facundo Batista)
Date: Tue, 10 Mar 2015 14:34:26 -0300
Subject: [Python-Dev] str.lstrip bug?
In-Reply-To: <CABHZ6sd7z085yLndxbjo5mbQ=NhOFzMgVR-5mnHco_45VRRX=Q@mail.gmail.com>
References: <CABHZ6sd7z085yLndxbjo5mbQ=NhOFzMgVR-5mnHco_45VRRX=Q@mail.gmail.com>
Message-ID: <CAM09pzSytbatyS6G9=z=Amo7Jqq=nRsRy6S5K=DWP1K2FDNjEw@mail.gmail.com>

On Tue, Mar 10, 2015 at 2:27 PM, lou xiao <lox.xiao at gmail.com> wrote:

> tiny? ~ python
> Python 2.7.5+ (default, Feb 27 2014, 19:37:08)
> [GCC 4.8.1] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> a='device_info'
>>>> a.lstrip('device')
> '_info'
>>>> a.lstrip('device_')
> 'nfo'

On one hand, this is the "development of python itself" list; this
issue was more aimed to the general python list, of you was sure that
this is a real bug, to the issue tracker.

On the other hand, this is not a bug! If you pass a parameter to
lstrip it will (quoted from its help) "remove characters in chars
instead.", so the moment you pass "device_", it removes all those
characers from the left... note that the 'i' is removed twice.

Regards,

-- 
.    Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/
Twitter: @facundobatista

From lox.xiao at gmail.com  Tue Mar 10 18:43:12 2015
From: lox.xiao at gmail.com (lou xiao)
Date: Wed, 11 Mar 2015 01:43:12 +0800
Subject: [Python-Dev] str.lstrip bug?
In-Reply-To: <CAPTjJmotpn4BW1FNpGncCoJTBMy9+3WZ5SB8Vb6uK8XANmn=5Q@mail.gmail.com>
References: <CABHZ6sd7z085yLndxbjo5mbQ=NhOFzMgVR-5mnHco_45VRRX=Q@mail.gmail.com>
 <CAPTjJmotpn4BW1FNpGncCoJTBMy9+3WZ5SB8Vb6uK8XANmn=5Q@mail.gmail.com>
Message-ID: <CABHZ6seCOzoCaDP35cQGs-e-6=3r5g=5KSM=7smeQSWyAuz+nQ@mail.gmail.com>

ths, i misunderstood the method

2015-03-11 1:33 GMT+08:00 Chris Angelico <rosuav at gmail.com>:

> On Wed, Mar 11, 2015 at 4:27 AM, lou xiao <lox.xiao at gmail.com> wrote:
> > I find a bug in str.lstrip, when i call str.lstrip, i get this result.
> >
> > tiny? ~ python
> > Python 2.7.5+ (default, Feb 27 2014, 19:37:08)
> > [GCC 4.8.1] on linux2
> > Type "help", "copyright", "credits" or "license" for more information.
> >>>> a='device_info'
> >>>> a.lstrip('device')
> > '_info'
> >>>> a.lstrip('device_')
> > 'nfo'
> >>>>
> > tiny? ~ uname -a
> > Linux tinymint 3.11.0-12-generic #19-Ubuntu SMP Wed Oct 9 16:20:46 UTC
> 2013
> > x86_64 x86_64 x86_64 GNU/Linux
> > tiny? ~
>
> It's not a bug, because it isn't doing what you think it is. It strips
> a *set of characters*, not a prefix string.
>
> https://docs.python.org/2/library/stdtypes.html#str.lstrip
> https://docs.python.org/3/library/stdtypes.html#str.lstrip
>
> ChrisA
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150311/359d7950/attachment.html>

From me at the-compiler.org  Tue Mar 10 18:34:34 2015
From: me at the-compiler.org (Florian Bruhin)
Date: Tue, 10 Mar 2015 18:34:34 +0100
Subject: [Python-Dev] str.lstrip bug?
In-Reply-To: <CABHZ6sd7z085yLndxbjo5mbQ=NhOFzMgVR-5mnHco_45VRRX=Q@mail.gmail.com>
References: <CABHZ6sd7z085yLndxbjo5mbQ=NhOFzMgVR-5mnHco_45VRRX=Q@mail.gmail.com>
Message-ID: <20150310173434.GO11094@tonks>

* lou xiao <lox.xiao at gmail.com> [2015-03-11 01:27:21 +0800]:
> I find a bug in str.lstrip, when i call str.lstrip, i get this result.

You're misunderstanding how str.strip works. It takes a set of
characters and removes them all from the beginning:

    >>> "abababcd".lstrip('ab')
    >>> 'cd'

Florian

-- 
http://www.the-compiler.org | me at the-compiler.org (Mail/XMPP)
   GPG: 916E B0C8 FD55 A072 | http://the-compiler.org/pubkey.asc
         I love long mails! | http://email.is-not-s.ms/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150310/e6de258b/attachment.sig>

From stephen at xemacs.org  Tue Mar 10 19:42:20 2015
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 11 Mar 2015 03:42:20 +0900
Subject: [Python-Dev]  str.lstrip bug?
In-Reply-To: <CABHZ6sd7z085yLndxbjo5mbQ=NhOFzMgVR-5mnHco_45VRRX=Q@mail.gmail.com>
References: <CABHZ6sd7z085yLndxbjo5mbQ=NhOFzMgVR-5mnHco_45VRRX=Q@mail.gmail.com>
Message-ID: <87ioe87kz7.fsf@uwakimon.sk.tsukuba.ac.jp>

lou xiao writes:

 > I find a bug in str.lstrip, when i call str.lstrip, i get this result.

 > >>> a.lstrip('device_')
 > 'nfo'
 > >>>

Try

    a.lstrip('_ecived')

You'll see that you get the same result.  I suspect that you
misunderstand the meaning of the argument, which is not a sequence of
characters, but a *set* of characters.


From ischwabacher at wisc.edu  Tue Mar 10 18:57:41 2015
From: ischwabacher at wisc.edu (Isaac Schwabacher)
Date: Tue, 10 Mar 2015 12:57:41 -0500
Subject: [Python-Dev] str.lstrip bug?
In-Reply-To: <776081621755c1.54ff3070@wiscmail.wisc.edu>
References: <CABHZ6sd7z085yLndxbjo5mbQ=NhOFzMgVR-5mnHco_45VRRX=Q@mail.gmail.com>
 <CAM09pzSytbatyS6G9=z=Amo7Jqq=nRsRy6S5K=DWP1K2FDNjEw@mail.gmail.com>
 <747088be171788.54ff2d5e@wiscmail.wisc.edu>
 <7760d6ad177272.54ff2d9b@wiscmail.wisc.edu>
 <7570adf21732e2.54ff2e4f@wiscmail.wisc.edu>
 <7640b7c6171335.54ff2e8c@wiscmail.wisc.edu>
 <7640fb6f174491.54ff2ec8@wiscmail.wisc.edu>
 <748099b41731c5.54ff2f04@wiscmail.wisc.edu>
 <7570b96e1706b0.54ff2f41@wiscmail.wisc.edu>
 <7480fc3f17687a.54ff2f7d@wiscmail.wisc.edu>
 <7750a881177e82.54ff2fba@wiscmail.wisc.edu>
 <75809c6c17326a.54ff2ff6@wiscmail.wisc.edu>
 <7760d124172a35.54ff3033@wiscmail.wisc.edu>
 <776081621755c1.54ff3070@wiscmail.wisc.edu>
Message-ID: <7490a88c171d64.54feea45@wiscmail.wisc.edu>

On 15-03-10, Facundo Batista  wrote:
> On Tue, Mar 10, 2015 at 2:27 PM, lou xiao <lox.xiao at gmail.com> wrote:
> 
> > tiny? ~ python
> > Python 2.7.5+ (default, Feb 27 2014, 19:37:08)
> > [GCC 4.8.1] on linux2
> > Type "help", "copyright", "credits" or "license" for more information.
> >>>> a='device_info'
> >>>> a.lstrip('device')
> > '_info'
> >>>> a.lstrip('device_')
> > 'nfo'
> 
> On one hand, this is the "development of python itself" list; this
> issue was more aimed to the general python list, of you was sure that
> this is a real bug, to the issue tracker.
> 
> On the other hand, this is not a bug! If you pass a parameter to
> lstrip it will (quoted from its help) "remove characters in chars
> instead.", so the moment you pass "device_", it removes all those
> characers from the left... note that the 'i' is removed twice.

That said, I bet this is the most common string-munging operations that *isn't* available as a single function in the stdlib. I know my bash code is full of ${filename%.suffix} and such, and the fact that in python I have to either import re or resort to some combination of (starts|ends)with, r?partition, slicing and an if-clause makes that sort of code much more verbose and harder to read. Pathlib's Path.with_suffix helps in some but not all of these cases. Maybe the stdlib should have a simple way to do this? It could even be added as a kwarg (exact=False) to str.[lr]strip to minimize the effect on the API; alternatively it could be str.strip(prefix|suffix).

ijs

> Regards,
> 
> -- 
> . Facundo
> 
> Blog: http://www.taniquetil.com.ar/plog/
> PyAr: http://www.python.org/ar/
> Twitter: @facundobatista
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/ischwabacher%40wisc.edu

From ethan at stoneleaf.us  Tue Mar 10 20:12:40 2015
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 10 Mar 2015 12:12:40 -0700
Subject: [Python-Dev] str.lstrip bug?
In-Reply-To: <7490a88c171d64.54feea45@wiscmail.wisc.edu>
References: <CABHZ6sd7z085yLndxbjo5mbQ=NhOFzMgVR-5mnHco_45VRRX=Q@mail.gmail.com>
 <CAM09pzSytbatyS6G9=z=Amo7Jqq=nRsRy6S5K=DWP1K2FDNjEw@mail.gmail.com>
 <747088be171788.54ff2d5e@wiscmail.wisc.edu>
 <7760d6ad177272.54ff2d9b@wiscmail.wisc.edu>
 <7570adf21732e2.54ff2e4f@wiscmail.wisc.edu>
 <7640b7c6171335.54ff2e8c@wiscmail.wisc.edu>
 <7640fb6f174491.54ff2ec8@wiscmail.wisc.edu>
 <748099b41731c5.54ff2f04@wiscmail.wisc.edu>
 <7570b96e1706b0.54ff2f41@wiscmail.wisc.edu>
 <7480fc3f17687a.54ff2f7d@wiscmail.wisc.edu>
 <7750a881177e82.54ff2fba@wiscmail.wisc.edu>
 <75809c6c17326a.54ff2ff6@wiscmail.wisc.edu>
 <7760d124172a35.54ff3033@wiscmail.wisc.edu>
 <776081621755c1.54ff3070@wiscmail.wisc.edu>
 <7490a88c171d64.54feea45@wiscmail.wisc.edu>
Message-ID: <54FF4228.8070802@stoneleaf.us>

On 03/10/2015 10:57 AM, Isaac Schwabacher wrote:
> On 15-03-10, Facundo Batista  wrote:
>> On Tue, Mar 10, 2015 at 2:27 PM, lou xiao wrote:
>>
>>> tiny? ~ python
>>> Python 2.7.5+ (default, Feb 27 2014, 19:37:08)
>>> [GCC 4.8.1] on linux2
>>> Type "help", "copyright", "credits" or "license" for more information.
>>> >>> a='device_info'
>>> >>> a.lstrip('device')
>>> '_info'
>>> >>> a.lstrip('device_')
>>> 'nfo'
>>
>> On one hand, this is the "development of python itself" list; this
>> issue was more aimed to the general python list, of you was sure that
>> this is a real bug, to the issue tracker.
>>
>> On the other hand, this is not a bug! If you pass a parameter to
>> lstrip it will (quoted from its help) "remove characters in chars
>> instead.", so the moment you pass "device_", it removes all those
>> characers from the left... note that the 'i' is removed twice.
> 
> That said, I bet this is the most common string-munging operations that *isn't* available as a single function in the stdlib. I know my bash code is full of ${filename%.suffix} and such, and the fact that in python I have to either import re or resort to some combination of (starts|ends)with, r?partition, slicing and an if-clause makes that sort of code much more verbose and harder to read. Pathlib's Path.with_suffix helps in some but not all of these cases. Maybe the stdlib should have a simple way to do this? It could even be added as a kwarg (exact=False) to str.[lr]strip to minimize the effect on the API; alternatively it could be str.strip(prefix|suffix).

Nice way to bring it on-topic!

And yes, that would be nice.

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150310/7676701e/attachment.sig>

From ischwabacher at wisc.edu  Tue Mar 10 21:32:03 2015
From: ischwabacher at wisc.edu (Isaac Schwabacher)
Date: Tue, 10 Mar 2015 15:32:03 -0500
Subject: [Python-Dev] str.lstrip bug?
In-Reply-To: <75f0bdb5173bcb.54ff544b@wiscmail.wisc.edu>
References: <CABHZ6sd7z085yLndxbjo5mbQ=NhOFzMgVR-5mnHco_45VRRX=Q@mail.gmail.com>
 <CAM09pzSytbatyS6G9=z=Amo7Jqq=nRsRy6S5K=DWP1K2FDNjEw@mail.gmail.com>
 <747088be171788.54ff2d5e@wiscmail.wisc.edu>
 <7760d6ad177272.54ff2d9b@wiscmail.wisc.edu>
 <7570adf21732e2.54ff2e4f@wiscmail.wisc.edu>
 <7640b7c6171335.54ff2e8c@wiscmail.wisc.edu>
 <7640fb6f174491.54ff2ec8@wiscmail.wisc.edu>
 <748099b41731c5.54ff2f04@wiscmail.wisc.edu>
 <7570b96e1706b0.54ff2f41@wiscmail.wisc.edu>
 <7480fc3f17687a.54ff2f7d@wiscmail.wisc.edu>
 <7750a881177e82.54ff2fba@wiscmail.wisc.edu>
 <75809c6c17326a.54ff2ff6@wiscmail.wisc.edu>
 <7760d124172a35.54ff3033@wiscmail.wisc.edu>
 <776081621755c1.54ff3070@wiscmail.wisc.edu>
 <7490a88c171d64.54feea45@wiscmail.wisc.edu> <54FF4228.8070802@stoneleaf.us>
 <75f0bdb5173bcb.54ff544b@wiscmail.wisc.edu>
Message-ID: <772082aa1711d1.54ff0e73@wiscmail.wisc.edu>

On 15-03-10, Ethan Furman  wrote:
> On 03/10/2015 10:57 AM, Isaac Schwabacher wrote:
> > On 15-03-10, Facundo Batista wrote:
> >> On Tue, Mar 10, 2015 at 2:27 PM, lou xiao wrote:
> >>
> >>> tiny? ~ python
> >>> Python 2.7.5+ (default, Feb 27 2014, 19:37:08)
> >>> [GCC 4.8.1] on linux2
> >>> Type "help", "copyright", "credits" or "license" for more information.
> >>> >>> a='device_info'
> >>> >>> a.lstrip('device')
> >>> '_info'
> >>> >>> a.lstrip('device_')
> >>> 'nfo'
> >>
> >> On one hand, this is the "development of python itself" list; this
> >> issue was more aimed to the general python list, of you was sure that
> >> this is a real bug, to the issue tracker.
> >>
> >> On the other hand, this is not a bug! If you pass a parameter to
> >> lstrip it will (quoted from its help) "remove characters in chars
> >> instead.", so the moment you pass "device_", it removes all those
> >> characers from the left... note that the 'i' is removed twice.
> > 
> > That said, I bet this is the most common string-munging operations that *isn't* available as a single function in the stdlib. I know my bash code is full of ${filename%.suffix} and such, and the fact that in python I have to either import re or resort to some combination of (starts|ends)with, r?partition, slicing and an if-clause makes that sort of code much more verbose and harder to read. Pathlib's Path.with_suffix helps in some but not all of these cases. Maybe the stdlib should have a simple way to do this? It could even be added as a kwarg (exact=False) to str.[lr]strip to minimize the effect on the API; alternatively it could be str.strip(prefix|suffix).
> 
> Nice way to bring it on-topic!
> 
> And yes, that would be nice.

Yes. When you are converting a code base from bash to python, you *really* don't want python to push back.

> --
> ~Ethan~

From arigo at tunes.org  Wed Mar 11 09:26:12 2015
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 11 Mar 2015 09:26:12 +0100
Subject: [Python-Dev] Tunning binary insertion sort algorithm in Timsort.
In-Reply-To: <CAExdVNkrB9JBhPGC5pBRD85HWsCHzoNADLCfBtSxvAL3WXQqqg@mail.gmail.com>
References: <CAMLcfj3usoq-new2gxh_ymP+1wv6XZj7+og6XMVAUj_sYeT+Pg@mail.gmail.com>
 <7750938a2fea2.54fdd09e@wiscmail.wisc.edu>
 <76e0c70929f48.54fdd0da@wiscmail.wisc.edu>
 <76f094a32c8ca.54fdd119@wiscmail.wisc.edu>
 <7740f7e62cf4b.54fdd155@wiscmail.wisc.edu>
 <778088e82a599.54fdd192@wiscmail.wisc.edu>
 <7740ebbf29f9e.54fdd2be@wiscmail.wisc.edu>
 <76f0f50b2c800.54fdd2fb@wiscmail.wisc.edu>
 <7710a1122c5ba.54fdd337@wiscmail.wisc.edu>
 <7780f4192fd2c.54fdd373@wiscmail.wisc.edu>
 <7790d04c299a7.54fdd590@wiscmail.wisc.edu>
 <7710a1822e0a9.54fdd5cc@wiscmail.wisc.edu>
 <7720dc73289e8.54fdd609@wiscmail.wisc.edu>
 <7790de4c2be2c.54fdd645@wiscmail.wisc.edu>
 <76f0eced29ac4.54fd9476@wiscmail.wisc.edu>
 <CAMLcfj0Fr_mGP+=xv0_Q1SVnR_GTKxK1cZs1DP-x+YaZBgHZrQ@mail.gmail.com>
 <CAExdVNmi2RxncnJZrhL7thNaFLkf3aVPX=yQBQDD-HEV8zTzDw@mail.gmail.com>
 <CAMLcfj2Xq+dgO+cKmGbQ5NaZjnuKggCr3K-S4rt=GBSHXSGMVA@mail.gmail.com>
 <CAExdVNka3o3x-=DT3uk5wu1nDMyH9HLeyGDT2YnMiiQwQSA3VA@mail.gmail.com>
 <CAExdVNkrB9JBhPGC5pBRD85HWsCHzoNADLCfBtSxvAL3WXQqqg@mail.gmail.com>
Message-ID: <CAMSv6X2phsGvcZjHU0Z=bjxv9dy3032JQUHE0b_OehcdwEdfbQ@mail.gmail.com>

Hi Tim,

On 10 March 2015 at 18:22, Tim Peters <tim.peters at gmail.com> wrote:
> 1. Merge "2 at a time" instead of just 1.  That is, first "sort" the
> next 2 elements to be merged (1 compare and a possible swap).  Then
> binary search to find where the smaller belongs, and a shorter binary
> search to find where the larger belongs.  Then shift both into place.

Good idea, but when I tried that it seemed to increase the total
number of comparisons (on random inputs with only up to 136 items).
The increase is on the order of 5%.  I'm not sure reduced data
movement can compensate for that in Python.

Test and code available here:
https://bitbucket.org/arigo/arigo/src/default/hack/pypy-hack/list_sort/

The change to insert two items at a time is here:
https://bitbucket.org/arigo/arigo/commits/68e04d143dc242cfd9e3934451321f685a68a8e2

(This is taken straight from PyPy's code.)


A bient?t,

Armin.

From storchaka at gmail.com  Wed Mar 11 12:25:39 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Wed, 11 Mar 2015 13:25:39 +0200
Subject: [Python-Dev] Not documented special methods
Message-ID: <mdp8nk$8ob$1@ger.gmane.org>

There are many special names used in Python core and the stdlib, but 
absent in the documentation index [1]. If you see names that are defined 
or used in the module or area maintained by you, please add references 
to these names to the index and document them if it is needed.

Repeat the lists here.

Module level names used in pydoc:
     __author__
     __credits__
     __date__
     __version__

Module level name used in doctest:
     __test__

Other module level names:
     __about__               (heapq only)
     __copyright__           (many modules)
     __cvsid__               (tarfile only)
     __docformat__           (doctest only)
     __email__               (test_with and test_keywordonlyarg only)
     __libmpdec_version__    (decimal only)
     __status__              (logging only)


type attributes (mostly used in tests):
     __abstractmethods__     (used in abc, functools)
     __base__
     __basicsize__
     __dictoffset__
     __flags__               (used in inspect, copyreg)
     __itemsize__
     __weakrefoffset__

super() attributes:
     __self_class__
     __thisclass__

Used in sqlite:
     __adapt__
     __conform__

Used in ctypes:
     __ctype_be__
     __ctype_le__
     __ctypes_from_outparam__

Used in unittest:
     __unittest_expecting_failure__
     __unittest_skip__
     __unittest_skip_why__

float methods, for testing:
     __getformat__
     __setformat__

Used in IDLE RPC:
     __attributes__
     __methods__

Others:
     __alloc__               (bytearray method)
     __args__                (used in bdb)
     __build_class__         (builtins function, used in eval loop)
     __builtins__            (module attribute)
     __decimal_context__     (used in decimal)
     __exception__           (used in pdb)
     __getinitargs__         (used in pickle, datetime)
     __initializing__        (used in importlib)
     __isabstractmethod__    (function/method/descriptor attribute,
                              used in abc, functools, types)
     __ltrace__              (used in eval loop, never set)
     __members__             (Enum attribute, used in many modules)
     __mp_main__             (used in multiprocessing)
     __new_member__          (Enum attribute, used in enum internally)
     __newobj__              (copyreg function,
                              used in pickle, object.__reduce_ex__)
     __newobj_ex__           (copyreg function,
                              used in pickle, object.__reduce_ex__)
     __objclass__            (descriptor/enum attribute, used in
                              inspect, pydoc, doctest, multiprocessing)
     __prepare__             (metaclass method,
                              used in builtins.__build_class__, types)
     __pycache__             (cache directory name)
     __return__              (used in pdb)
     __signature__           (used in inspect, never set)
     __sizeof__              (standard method, used in sys.getsizeof)
     __slotnames__           (used in object.__getstate__ for caching)
     __text_signature__      (function/method/descriptor attribute,
                              used in inspect)
     __trunc__               (used in math.trunc, int, etc)
     __warningregistry__     (used in warnings)
     __weakref__             (used in weakref)
     __wrapped__             (used in inspect, functools, contextlib,
                              asyncio)


[1] http://bugs.python.org/issue23639


From brett at python.org  Wed Mar 11 18:27:58 2015
From: brett at python.org (Brett Cannon)
Date: Wed, 11 Mar 2015 17:27:58 +0000
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <549891df.c332e00a.54c7.34f8@mx.google.com>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
Message-ID: <CAP1=2W7=DRv-rfy5jv=pwamXhC4xU8qnPm3fDAjb2WTFKOYC3A@mail.gmail.com>

On Mon, Dec 22, 2014 at 4:49 PM Jim J. Jewett <jimjjewett at gmail.com> wrote:

>
>
> On Thu, Dec 18, 2014, at 14:13, Maciej Fijalkowski wrote:
> > ... http://bugs.python.org/issue23085 ...
> > is there any reason any more for libffi being included in CPython?
>
>
> Paul Moore wrote:
> > Probably the easiest way of moving this forward would be for someone
> > to identify the CPython-specific patches in the current version ...
>
> Christian Heimes wrote:
> > That's easy. All patches are tracked in the diff file
> > https://hg.python.org/cpython/file/3de678cd184d/Modules/_
> ctypes/libffi.diff
>
> That (200+ lines) doesn't seem to have all the C changes, such as the
> win64 sizeof changes from issue 11835.
>
> Besides http://bugs.python.org/issue23085, there is at least
>     http://bugs.python.org/issue22733
>     http://bugs.python.org/issue20160
>     http://bugs.python.org/issue11835
>
> which sort of drives home the point that making sure we have a
> good merge isn't trivial, and this isn't an area where we should
> just assume that tests will catch everything.  I don't think it
> is just a quicky waiting on permission.
>
> I've no doubt that upstream libffi is better in many ways, but
> those are ways people have already learned to live with.
>
> That said, I haven't seen any objections in principle, except
> perhaps from Steve Dower in the issues.  (I *think* he was
> just saying "not worth the time to me", but it was ambiguous.)
>
> I do believe that Christian or Maciej *could* sort things out well
> enough; I have no insight into whether they have (or someone else
> has) the time to actually do so.
>

Did anyone ever step forward to do this? I'm a bit worried about the
long-term viability of ctypes if we don't have a maintainer or at least
someone making sure we are staying up-to-date with upstream libffi. The
ctypes module is a dangerous thing, so having a chunk of C code that isn't
being properly maintained seems to me to make it even more dangerous.

I'm going to propose a somewhat controversial idea: let's deprecate the
ctypes module. We now have things like cffi and Cython for people who need
to interface with C code. Both of those projects are maintained. And they
are not overly difficult to work with. All of this seems to match the
reasons we added ctypes in the first place while also being safer to use
than ctypes.

And I'm not saying we need to remove it in Python 3.6 or something. But I
think it would be wise to deprecate the module to promote people to use
third-party solutions to interfacing with C code and to eventually remove
the maintenance burden at some point when we clear out all of our
deprecated cruft (I call that Python 4, you can call it "some day way in
the future" if you prefer).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150311/6abb413f/attachment.html>

From brett at python.org  Wed Mar 11 18:34:10 2015
From: brett at python.org (Brett Cannon)
Date: Wed, 11 Mar 2015 17:34:10 +0000
Subject: [Python-Dev] PEP 488: elimination of PYO files
In-Reply-To: <1425685751.3874542.236704353.11A94396@webmail.messagingengine.com>
References: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
 <54F9EBAF.1080400@hotpy.org>
 <CAP1=2W6JUnDKpQFcr1Ud_b9rCBayk3RCp8XQQhd3zSQDYwmxtQ@mail.gmail.com>
 <CAA68w_nJW9=_=U=fwfTWN+h5vdaWT-Si9YpqxddULJLduPA-ew@mail.gmail.com>
 <CAP1=2W4g-sUwh74Zdu4rORqjOF8O24eD=HUatVinQ_HnVdKbVQ@mail.gmail.com>
 <1425668955.3364036.236612829.494092A7@webmail.messagingengine.com>
 <CAP1=2W4FbUQjaK5a-FgsfS1c3kP5qbqi=iZHE5h1+CxDYrJqtA@mail.gmail.com>
 <1425685751.3874542.236704353.11A94396@webmail.messagingengine.com>
Message-ID: <CAP1=2W5jmY88Z5AfT0MD9yfmaKq3q5Zo8BAQ3Tu96HZseeTcFQ@mail.gmail.com>

On Fri, Mar 6, 2015 at 6:49 PM Benjamin Peterson <benjamin at python.org>
wrote:

>
>
> On Fri, Mar 6, 2015, at 15:11, Brett Cannon wrote:
> >
> > OK, but that doesn't influence the PEP's goal of dropping .pyo files.
>
> Correct.
>
> >
> > Are you suggesting that the tag be changed to be less specific to
> > optimizations and more free-form? Like
> > `importlib.cpython-35.__no-asserts_no-docstrings__.pyc`? Otherwise stuff
> > like this gets baked into the .pyc file itself instead of the file name,
> > but I don't think we should just drop the ability to switch off asserts
> > and
> > docstrings like Mark seemed to be suggesting.
>
> Basically, though the filename strings could perhaps be more compact.
>

I have a poll going on G+ to see what people think of the various proposed
file name formats at
https://plus.google.com/u/0/+BrettCannon/posts/fZynLNwHWGm . Feel free to
vote if you have an opinion.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150311/fc12cda5/attachment.html>

From solipsis at pitrou.net  Wed Mar 11 18:50:28 2015
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 11 Mar 2015 18:50:28 +0100
Subject: [Python-Dev] libffi embedded in CPython
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAP1=2W7=DRv-rfy5jv=pwamXhC4xU8qnPm3fDAjb2WTFKOYC3A@mail.gmail.com>
Message-ID: <20150311185028.6f05973c@fsol>

On Wed, 11 Mar 2015 17:27:58 +0000
Brett Cannon <brett at python.org> wrote:
> 
> Did anyone ever step forward to do this? I'm a bit worried about the
> long-term viability of ctypes if we don't have a maintainer or at least
> someone making sure we are staying up-to-date with upstream libffi. The
> ctypes module is a dangerous thing, so having a chunk of C code that isn't
> being properly maintained seems to me to make it even more dangerous.

Depends what you call "dangerous". C code doesn't rot quicker than pure
Python code :-) Also, libffi really offers a wrapper around platform
ABIs, which rarely change.

> I'm going to propose a somewhat controversial idea: let's deprecate the
> ctypes module.

This is gratuitous.

> We now have things like cffi and Cython for people who need
> to interface with C code. Both of those projects are maintained. And they
> are not overly difficult to work with.

But they are not ctypes. For example, cffi wouldn't be obvious to use
for interfacing with non-C code, since it requires you to write C-like
declarations.

> All of this seems to match the
> reasons we added ctypes in the first place while also being safer to use
> than ctypes.

I don't understand why cffi would be safer than ctypes. At least not in
the operation mode where it doesn't need to invoke a C compiler.
Cython is a completely different beast, it requires a separate
compilation pass which makes it useless in some situations.

Regards

Antoine.



From fijall at gmail.com  Wed Mar 11 18:54:58 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 11 Mar 2015 19:54:58 +0200
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <20150311185028.6f05973c@fsol>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAP1=2W7=DRv-rfy5jv=pwamXhC4xU8qnPm3fDAjb2WTFKOYC3A@mail.gmail.com>
 <20150311185028.6f05973c@fsol>
Message-ID: <CAK5idxQYyWbWq_frrvWV0ZYdfeWgCm8pW0dgotf0uE-op4WNBg@mail.gmail.com>

On Wed, Mar 11, 2015 at 7:50 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Wed, 11 Mar 2015 17:27:58 +0000
> Brett Cannon <brett at python.org> wrote:
>>
>> Did anyone ever step forward to do this? I'm a bit worried about the
>> long-term viability of ctypes if we don't have a maintainer or at least
>> someone making sure we are staying up-to-date with upstream libffi. The
>> ctypes module is a dangerous thing, so having a chunk of C code that isn't
>> being properly maintained seems to me to make it even more dangerous.
>
> Depends what you call "dangerous". C code doesn't rot quicker than pure
> Python code :-) Also, libffi really offers a wrapper around platform
> ABIs, which rarely change.

And yet, lesser known ABIs in libffi contain bugs (as we discovered
trying to work there with anything else than x86 really). Also there
*are* ABI differencies that change slowly over time (e.g. requiring
stack to be 16 byte aligned)

>
>> I'm going to propose a somewhat controversial idea: let's deprecate the
>> ctypes module.
>
> This is gratuitous.

I'm +1 on deprecating ctypes

>
>> We now have things like cffi and Cython for people who need
>> to interface with C code. Both of those projects are maintained. And they
>> are not overly difficult to work with.
>
> But they are not ctypes. For example, cffi wouldn't be obvious to use
> for interfacing with non-C code, since it requires you to write C-like
> declarations.

You mean like Fortran? Or what precisely?

> I don't understand why cffi would be safer than ctypes. At least not in
> the operation mode where it doesn't need to invoke a C compiler.
> Cython is a completely different beast, it requires a separate
> compilation pass which makes it useless in some situations.
>

Our main motivation for "safer" is "comes with less magic and less
gotchas", which also means "does less stuff". It's also smaller.

From p.f.moore at gmail.com  Wed Mar 11 18:56:14 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 11 Mar 2015 17:56:14 +0000
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <CAP1=2W7=DRv-rfy5jv=pwamXhC4xU8qnPm3fDAjb2WTFKOYC3A@mail.gmail.com>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAP1=2W7=DRv-rfy5jv=pwamXhC4xU8qnPm3fDAjb2WTFKOYC3A@mail.gmail.com>
Message-ID: <CACac1F9uRauWtOvagp23RpJdks+j+yyvtkp+1t+oCU_5f6HjEg@mail.gmail.com>

On 11 March 2015 at 17:27, Brett Cannon <brett at python.org> wrote:
> I'm going to propose a somewhat controversial idea: let's deprecate the
> ctypes module. We now have things like cffi and Cython for people who need
> to interface with C code. Both of those projects are maintained. And they
> are not overly difficult to work with. All of this seems to match the
> reasons we added ctypes in the first place while also being safer to use
> than ctypes.

-1. On Windows a huge amount of code uses ctypes to interface with the
Windows API. Many of those projects transitioned from pywin32 because
that is a big dependency, and a binary build. Not having in-core
access to the Windows API would be a huge problem for many projects.
Who would rewrite projects like pyreadline or colorama, for example?
How would pip (which has to vendor everything and can't include binary
dependencies because of its nature) locate the correct windows config
file folders without ctypes in core?

Paul

From solipsis at pitrou.net  Wed Mar 11 19:05:57 2015
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 11 Mar 2015 19:05:57 +0100
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <CAK5idxQYyWbWq_frrvWV0ZYdfeWgCm8pW0dgotf0uE-op4WNBg@mail.gmail.com>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAP1=2W7=DRv-rfy5jv=pwamXhC4xU8qnPm3fDAjb2WTFKOYC3A@mail.gmail.com>
 <20150311185028.6f05973c@fsol>
 <CAK5idxQYyWbWq_frrvWV0ZYdfeWgCm8pW0dgotf0uE-op4WNBg@mail.gmail.com>
Message-ID: <20150311190557.25eebc16@fsol>

On Wed, 11 Mar 2015 19:54:58 +0200
Maciej Fijalkowski <fijall at gmail.com> wrote:
> >
> > Depends what you call "dangerous". C code doesn't rot quicker than pure
> > Python code :-) Also, libffi really offers a wrapper around platform
> > ABIs, which rarely change.
> 
> And yet, lesser known ABIs in libffi contain bugs (as we discovered
> trying to work there with anything else than x86 really). Also there
> *are* ABI differencies that change slowly over time (e.g. requiring
> stack to be 16 byte aligned)

Well, sure. The point is, such bugs are unlikely to appear at a fast
rate... Also, I don't understand why libffi issues would affect cffi
any less than it affects ctypes, at least in the compiler-less mode of
operation.

> >> We now have things like cffi and Cython for people who need
> >> to interface with C code. Both of those projects are maintained. And they
> >> are not overly difficult to work with.
> >
> > But they are not ctypes. For example, cffi wouldn't be obvious to use
> > for interfacing with non-C code, since it requires you to write C-like
> > declarations.
> 
> You mean like Fortran? Or what precisely?

Any toolchain that can generate native code. It can be Fortran, but it
can also be code generated at runtime without there being any external
declaration. Having to generate "C declarations" for such code would be
a distraction.

Of course, if cffi gains the same ability as ctypes (namely to lookup
a function and declare its signature without going through the
FFI.cdef() interface), that issue disappears.

As a side note, ctypes has a large number of users, so even if it were
deprecated that wouldn't be a good reason to stop maintaining it.

And calling cffi "simple" while it relies on a parser of the C language
(which would then have to be bundled with Python) is a bit misleading
IMO.

Regards

Antoine.

From solipsis at pitrou.net  Wed Mar 11 19:17:50 2015
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 11 Mar 2015 19:17:50 +0100
Subject: [Python-Dev] (ctypes example) libffi embedded in CPython
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAP1=2W7=DRv-rfy5jv=pwamXhC4xU8qnPm3fDAjb2WTFKOYC3A@mail.gmail.com>
 <20150311185028.6f05973c@fsol>
 <CAK5idxQYyWbWq_frrvWV0ZYdfeWgCm8pW0dgotf0uE-op4WNBg@mail.gmail.com>
 <20150311190557.25eebc16@fsol>
Message-ID: <20150311191750.7cf23a40@fsol>

On Wed, 11 Mar 2015 19:05:57 +0100
Antoine Pitrou <solipsis at pitrou.net> wrote:
> > >
> > > But they are not ctypes. For example, cffi wouldn't be obvious to use
> > > for interfacing with non-C code, since it requires you to write C-like
> > > declarations.
> > 
> > You mean like Fortran? Or what precisely?
> 
> Any toolchain that can generate native code. It can be Fortran, but it
> can also be code generated at runtime without there being any external
> declaration. Having to generate "C declarations" for such code would be
> a distraction.

For instance, you can look at the compiler example that Eli wrote using
llvmlite. It implements a JIT compiler for a toy language. The
JIT-compiled function is then declared and called using a simple ctypes
declaration:

https://github.com/eliben/pykaleidoscope/blob/master/chapter7.py#L937

Regards

Antoine.



From wes.turner at gmail.com  Wed Mar 11 19:31:23 2015
From: wes.turner at gmail.com (Wes Turner)
Date: Wed, 11 Mar 2015 13:31:23 -0500
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <CAK5idxQYyWbWq_frrvWV0ZYdfeWgCm8pW0dgotf0uE-op4WNBg@mail.gmail.com>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAP1=2W7=DRv-rfy5jv=pwamXhC4xU8qnPm3fDAjb2WTFKOYC3A@mail.gmail.com>
 <20150311185028.6f05973c@fsol>
 <CAK5idxQYyWbWq_frrvWV0ZYdfeWgCm8pW0dgotf0uE-op4WNBg@mail.gmail.com>
Message-ID: <CACfEFw_WuB1DeFROdjyQ7y4njsb6Hw+ojTHAHaQ0wXNbfMCSUw@mail.gmail.com>

On Mar 11, 2015 12:55 PM, "Maciej Fijalkowski" <fijall at gmail.com> wrote:
>
> On Wed, Mar 11, 2015 at 7:50 PM, Antoine Pitrou <solipsis at pitrou.net>
wrote:
> > On Wed, 11 Mar 2015 17:27:58 +0000
> > Brett Cannon <brett at python.org> wrote:
> >>
> >> Did anyone ever step forward to do this? I'm a bit worried about the
> >> long-term viability of ctypes if we don't have a maintainer or at least
> >> someone making sure we are staying up-to-date with upstream libffi. The
> >> ctypes module is a dangerous thing, so having a chunk of C code that
isn't
> >> being properly maintained seems to me to make it even more dangerous.
> >
> > Depends what you call "dangerous". C code doesn't rot quicker than pure
> > Python code :-) Also, libffi really offers a wrapper around platform
> > ABIs, which rarely change.
>
> And yet, lesser known ABIs in libffi contain bugs (as we discovered
> trying to work there with anything else than x86 really). Also there
> *are* ABI differencies that change slowly over time (e.g. requiring
> stack to be 16 byte aligned)

Are there tests for this?

>
> >
> >> I'm going to propose a somewhat controversial idea: let's deprecate the
> >> ctypes module.
> >
> > This is gratuitous.
>
> I'm +1 on deprecating ctypes

-1. These docs are helpful for understanding the pros and cons of different
CPython <-> C interfaces.

https://scipy-lectures.github.io/advanced/interfacing_with_c/interfacing_with_c.html

(https://github.com/scipy-lectures/scipy-lecture-notes/issues/131 discusses
documenting CFFI here as well)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150311/41567522/attachment-0001.html>

From tim.peters at gmail.com  Wed Mar 11 20:19:20 2015
From: tim.peters at gmail.com (Tim Peters)
Date: Wed, 11 Mar 2015 14:19:20 -0500
Subject: [Python-Dev] Tunning binary insertion sort algorithm in Timsort.
In-Reply-To: <CAMSv6X2phsGvcZjHU0Z=bjxv9dy3032JQUHE0b_OehcdwEdfbQ@mail.gmail.com>
References: <CAMLcfj3usoq-new2gxh_ymP+1wv6XZj7+og6XMVAUj_sYeT+Pg@mail.gmail.com>
 <7750938a2fea2.54fdd09e@wiscmail.wisc.edu>
 <76e0c70929f48.54fdd0da@wiscmail.wisc.edu>
 <76f094a32c8ca.54fdd119@wiscmail.wisc.edu>
 <7740f7e62cf4b.54fdd155@wiscmail.wisc.edu>
 <778088e82a599.54fdd192@wiscmail.wisc.edu>
 <7740ebbf29f9e.54fdd2be@wiscmail.wisc.edu>
 <76f0f50b2c800.54fdd2fb@wiscmail.wisc.edu>
 <7710a1122c5ba.54fdd337@wiscmail.wisc.edu>
 <7780f4192fd2c.54fdd373@wiscmail.wisc.edu>
 <7790d04c299a7.54fdd590@wiscmail.wisc.edu>
 <7710a1822e0a9.54fdd5cc@wiscmail.wisc.edu>
 <7720dc73289e8.54fdd609@wiscmail.wisc.edu>
 <7790de4c2be2c.54fdd645@wiscmail.wisc.edu>
 <76f0eced29ac4.54fd9476@wiscmail.wisc.edu>
 <CAMLcfj0Fr_mGP+=xv0_Q1SVnR_GTKxK1cZs1DP-x+YaZBgHZrQ@mail.gmail.com>
 <CAExdVNmi2RxncnJZrhL7thNaFLkf3aVPX=yQBQDD-HEV8zTzDw@mail.gmail.com>
 <CAMLcfj2Xq+dgO+cKmGbQ5NaZjnuKggCr3K-S4rt=GBSHXSGMVA@mail.gmail.com>
 <CAExdVNka3o3x-=DT3uk5wu1nDMyH9HLeyGDT2YnMiiQwQSA3VA@mail.gmail.com>
 <CAExdVNkrB9JBhPGC5pBRD85HWsCHzoNADLCfBtSxvAL3WXQqqg@mail.gmail.com>
 <CAMSv6X2phsGvcZjHU0Z=bjxv9dy3032JQUHE0b_OehcdwEdfbQ@mail.gmail.com>
Message-ID: <CAExdVNm+hk-vf63-iZHmY_YWVOZvZoneR8g5+6WuPZdwfbjFPg@mail.gmail.com>

[Tim]
>> 1. Merge "2 at a time" instead of just 1.  That is, first "sort" the
>> next 2 elements to be merged (1 compare and a possible swap).  Then
>> binary search to find where the smaller belongs, and a shorter binary
>> search to find where the larger belongs.  Then shift both into place.

[Armin]
> Good idea, but when I tried that it seemed to increase the total
> number of comparisons (on random inputs with only up to 136 items).
> The increase is on the order of 5%.  I'm not sure reduced data
> movement can compensate for that in Python.

Which is another way of saying "bad idea" - that must be why I didn't
pursue it to begin with ;-)

Thanks for trying!  I plugged a similar routine into the code I showed
before to count the # of comparisons in Nha Pham's idea, and this
"merge 2 at a time" thing has a higher average # of compares (over all
permutations) than Nha's (which in turn has a higher average than the
status quo).

That makes some sense, thinking about what they do.  Nha's algorithm
has some supernaturally good cases (input array already ascending or
already descending), but "merge 2 at a time" doesn't appear to have
any.

In any case, the information-theoretic minimum average number of
comparisons for merging N sorted elements with 2 sorted elements is
("where do the 2 belong in the final list of N+2 elements?" =
comb(N+2, 2)):

    log2((N+2)*(N+1)/2) = log2(N+2) + log2(N+1) - 1

Add a comparison to get the 2 elements in order to begin with, and we're up to

    log2(N+2) + log2(N+1)

Two independent binary inserts (first to a list of size N, and then to
a list of size N+1) comes out to the same.  So even being
supernaturally clever can't reduce the average number of compares this
way.  And since, in context, we're only looking at short arrays, a
marginal saving in data movement costs (which are still O(N**2) worst
case) are unlikely to be significant.

Still, if anyone wants to go nuts ... ;-)

From cournape at gmail.com  Wed Mar 11 21:36:21 2015
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 11 Mar 2015 20:36:21 +0000
Subject: [Python-Dev] Why does python use relative instead of absolute path
	when calling LoadLibrary*
Message-ID: <CAGY4rcUvP5HqfUc3ZDhOr6U=iVVmbrW_v_9ofzOU=JaiinKc6w@mail.gmail.com>

Hi,

While looking at the import code of python for C extensions, I was
wondering why we pass a relative path instead of an absolute path to
LoadLibraryEx (see bottom for some context).

In python 2.7, the full path existence was even checked before calling into
LoadLibraryEx (
https://github.com/python/cpython/blob/2.7/Python/dynload_win.c#L189), but
it looks like this check was removed in python 3.x branch.

Is there any defined behaviour that depends on this path to be relative ?

Context
-----------

The reason why I am interested in this is the potential use of
SetDllDirectory to share dlls between multiple python extensions.
Currently, the only solutions I am aware of are:

1. putting the dlls in the PATH
2. bundling the dlls side by side the .pyd
3. patching packages to use preloading (using e.g. ctypes)

I am investigating a solution 4, where the dlls would be put in a separate
"private" directory only known of python itself, without the need to modify
PATH.

Patching python to use SetDllDirectory("some private paths specific to a
python interpreter") works perfectly, except that it slightly changes the
semantics of LoadLibraryEx not to look for dlls in the current directory.
This breaks importing extensions built in place, unless I modify the call
in ;https://github.com/python/cpython/blob/2.7/Python/dynload_win.c#L195
from:

    hDLL = LoadLibraryEx(pathname, NULL LOAD_WITH_ALTERED_SEARCH_PATH)

to

    hDLL = LoadLibraryEx(pathbuf, NULL LOAD_WITH_ALTERED_SEARCH_PATH)

That seems to work, but I am quite worried about changing any import
semantics by accident.

David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150311/56ceb6dd/attachment.html>

From fijall at gmail.com  Wed Mar 11 22:10:14 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 11 Mar 2015 23:10:14 +0200
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <20150311190557.25eebc16@fsol>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAP1=2W7=DRv-rfy5jv=pwamXhC4xU8qnPm3fDAjb2WTFKOYC3A@mail.gmail.com>
 <20150311185028.6f05973c@fsol>
 <CAK5idxQYyWbWq_frrvWV0ZYdfeWgCm8pW0dgotf0uE-op4WNBg@mail.gmail.com>
 <20150311190557.25eebc16@fsol>
Message-ID: <CAK5idxS2P45pVK+XAXEfOTsZhWebB+yL953KL-+4zzMTcz3L_g@mail.gmail.com>

On Wed, Mar 11, 2015 at 8:05 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Wed, 11 Mar 2015 19:54:58 +0200
> Maciej Fijalkowski <fijall at gmail.com> wrote:
>> >
>> > Depends what you call "dangerous". C code doesn't rot quicker than pure
>> > Python code :-) Also, libffi really offers a wrapper around platform
>> > ABIs, which rarely change.
>>
>> And yet, lesser known ABIs in libffi contain bugs (as we discovered
>> trying to work there with anything else than x86 really). Also there
>> *are* ABI differencies that change slowly over time (e.g. requiring
>> stack to be 16 byte aligned)
>
> Well, sure. The point is, such bugs are unlikely to appear at a fast
> rate... Also, I don't understand why libffi issues would affect cffi
> any less than it affects ctypes, at least in the compiler-less mode of
> operation.

My point here was only about shipping own libffi vs using the system
one (and it does affect cffi equally with or without compiler)

>
>> >> We now have things like cffi and Cython for people who need
>> >> to interface with C code. Both of those projects are maintained. And they
>> >> are not overly difficult to work with.
>> >
>> > But they are not ctypes. For example, cffi wouldn't be obvious to use
>> > for interfacing with non-C code, since it requires you to write C-like
>> > declarations.
>>
>> You mean like Fortran? Or what precisely?
>
> Any toolchain that can generate native code. It can be Fortran, but it
> can also be code generated at runtime without there being any external
> declaration. Having to generate "C declarations" for such code would be
> a distraction.
>
> Of course, if cffi gains the same ability as ctypes (namely to lookup
> a function and declare its signature without going through the
> FFI.cdef() interface), that issue disappears.
>
> As a side note, ctypes has a large number of users, so even if it were
> deprecated that wouldn't be a good reason to stop maintaining it.
>
> And calling cffi "simple" while it relies on a parser of the C language
> (which would then have to be bundled with Python) is a bit misleading
> IMO.
>
> Regards
>
> Antoine.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com

From fijall at gmail.com  Wed Mar 11 22:11:41 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 11 Mar 2015 23:11:41 +0200
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <CACfEFw_WuB1DeFROdjyQ7y4njsb6Hw+ojTHAHaQ0wXNbfMCSUw@mail.gmail.com>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAP1=2W7=DRv-rfy5jv=pwamXhC4xU8qnPm3fDAjb2WTFKOYC3A@mail.gmail.com>
 <20150311185028.6f05973c@fsol>
 <CAK5idxQYyWbWq_frrvWV0ZYdfeWgCm8pW0dgotf0uE-op4WNBg@mail.gmail.com>
 <CACfEFw_WuB1DeFROdjyQ7y4njsb6Hw+ojTHAHaQ0wXNbfMCSUw@mail.gmail.com>
Message-ID: <CAK5idxQPGEdBB2nRyKHR3bEgC76miPiJH5RU=J4fMjMbqb+oJA@mail.gmail.com>

On Wed, Mar 11, 2015 at 8:31 PM, Wes Turner <wes.turner at gmail.com> wrote:
>
> On Mar 11, 2015 12:55 PM, "Maciej Fijalkowski" <fijall at gmail.com> wrote:
>>
>> On Wed, Mar 11, 2015 at 7:50 PM, Antoine Pitrou <solipsis at pitrou.net>
>> wrote:
>> > On Wed, 11 Mar 2015 17:27:58 +0000
>> > Brett Cannon <brett at python.org> wrote:
>> >>
>> >> Did anyone ever step forward to do this? I'm a bit worried about the
>> >> long-term viability of ctypes if we don't have a maintainer or at least
>> >> someone making sure we are staying up-to-date with upstream libffi. The
>> >> ctypes module is a dangerous thing, so having a chunk of C code that
>> >> isn't
>> >> being properly maintained seems to me to make it even more dangerous.
>> >
>> > Depends what you call "dangerous". C code doesn't rot quicker than pure
>> > Python code :-) Also, libffi really offers a wrapper around platform
>> > ABIs, which rarely change.
>>
>> And yet, lesser known ABIs in libffi contain bugs (as we discovered
>> trying to work there with anything else than x86 really). Also there
>> *are* ABI differencies that change slowly over time (e.g. requiring
>> stack to be 16 byte aligned)
>
> Are there tests for this?
>

What do you mean? The usual failure mode is "will segfault every now
and again if the moon is in the right position" (e.g. the stack
alignment thing only happens if the underlaying function uses certain
SSE instructions that compilers emit these days in certain
circumstances)

From fijall at gmail.com  Wed Mar 11 22:13:29 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 11 Mar 2015 23:13:29 +0200
Subject: [Python-Dev] (ctypes example) libffi embedded in CPython
In-Reply-To: <20150311191750.7cf23a40@fsol>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAP1=2W7=DRv-rfy5jv=pwamXhC4xU8qnPm3fDAjb2WTFKOYC3A@mail.gmail.com>
 <20150311185028.6f05973c@fsol>
 <CAK5idxQYyWbWq_frrvWV0ZYdfeWgCm8pW0dgotf0uE-op4WNBg@mail.gmail.com>
 <20150311190557.25eebc16@fsol> <20150311191750.7cf23a40@fsol>
Message-ID: <CAK5idxQD+2uJ8vjJQcx7WKitZyc6biP3AL-LUVZELsMVow_EQw@mail.gmail.com>

On Wed, Mar 11, 2015 at 8:17 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Wed, 11 Mar 2015 19:05:57 +0100
> Antoine Pitrou <solipsis at pitrou.net> wrote:
>> > >
>> > > But they are not ctypes. For example, cffi wouldn't be obvious to use
>> > > for interfacing with non-C code, since it requires you to write C-like
>> > > declarations.
>> >
>> > You mean like Fortran? Or what precisely?
>>
>> Any toolchain that can generate native code. It can be Fortran, but it
>> can also be code generated at runtime without there being any external
>> declaration. Having to generate "C declarations" for such code would be
>> a distraction.
>
> For instance, you can look at the compiler example that Eli wrote using
> llvmlite. It implements a JIT compiler for a toy language. The
> JIT-compiled function is then declared and called using a simple ctypes
> declaration:
>
> https://github.com/eliben/pykaleidoscope/blob/master/chapter7.py#L937
>
> Regards
>
> Antoine.

It might be a matter of taste, but I don't find declaring C functions
any more awkward than using strange interface that ctypes comes with.
the equivalent in cffi would be ffi.cast("double (*)()", x)

From arigo at tunes.org  Wed Mar 11 22:28:53 2015
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 11 Mar 2015 22:28:53 +0100
Subject: [Python-Dev] PEP 488: elimination of PYO files
In-Reply-To: <CAP1=2W6JUnDKpQFcr1Ud_b9rCBayk3RCp8XQQhd3zSQDYwmxtQ@mail.gmail.com>
References: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
 <54F9EBAF.1080400@hotpy.org>
 <CAP1=2W6JUnDKpQFcr1Ud_b9rCBayk3RCp8XQQhd3zSQDYwmxtQ@mail.gmail.com>
Message-ID: <CAMSv6X0ePfq+H69fa+f6-X3UViQbezD6i9yXpJTBSvZVftRFeg@mail.gmail.com>

Hi Brett,

On 6 March 2015 at 19:11, Brett Cannon <brett at python.org> wrote:
> I disagree with your premise that .pyo files don't have a noticeable effect
> on performance. If you don't use asserts a lot then there is no effect, but
> if you use them heavily or have them perform expensive calculations then
> there is an impact.

Maybe you'd be interested to learn that PyPy (at least the 2.x branch)
uses a new bytecode, JUMP_IF_NOT_DEBUG, to conditionally jump over the
"assert" line.  In "optimized" mode PyPy follows the jumps; in
"non-optimized" mode it doesn't.  This mode is initialized with the -O
flag but can be changed dynamically, as the bytecode is the same.  We
introduced it as a simple solution to the mess of .pyc versus .pyo.
(We didn't consider the case of -OO very closely because PyPy is much
bigger than CPython as a binary to start with, so the demand for that
is lower.)


A bient?t,

Armin.

From victor.stinner at gmail.com  Wed Mar 11 22:34:47 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 11 Mar 2015 22:34:47 +0100
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <CAP1=2W7=DRv-rfy5jv=pwamXhC4xU8qnPm3fDAjb2WTFKOYC3A@mail.gmail.com>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAP1=2W7=DRv-rfy5jv=pwamXhC4xU8qnPm3fDAjb2WTFKOYC3A@mail.gmail.com>
Message-ID: <CAMpsgwaAaxiMquZHhHCL5RofAB_neci8TvDuLA4-W_XMThwpCA@mail.gmail.com>

Le 11 mars 2015 18:29, "Brett Cannon" <brett at python.org> a ?crit :
> I'm going to propose a somewhat controversial idea: let's deprecate the
ctypes module.

In the past I tried to deprecate many functions or modules because they are
rarely or never used. Many developers prefered to keep them. By the way, I
still want to remove plat-xxx modules like IN or CDROM :-)

Getopt was deprecated when optparse was added to the stdlib. Then optparse
was deprecated when argparse was added to the stdlib.

Cython and cffi are not part of the stdlib and can be hard to install on
some platforms. Ctypes is cool because it doesn't require C headers nor a C
compiler.

Is it possible to use cffi without a C compiler/headers as easily than
ctypes?

If you want to move forward, you should help to integrate cffi into the
stdlib. What's the status?

Victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150311/de67c248/attachment.html>

From fijall at gmail.com  Wed Mar 11 22:45:15 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 11 Mar 2015 23:45:15 +0200
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <CAMpsgwaAaxiMquZHhHCL5RofAB_neci8TvDuLA4-W_XMThwpCA@mail.gmail.com>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAP1=2W7=DRv-rfy5jv=pwamXhC4xU8qnPm3fDAjb2WTFKOYC3A@mail.gmail.com>
 <CAMpsgwaAaxiMquZHhHCL5RofAB_neci8TvDuLA4-W_XMThwpCA@mail.gmail.com>
Message-ID: <CAK5idxTJ=hfq8_xqSZbq=3h69_joBcLvGk0m1a002+XcHgddhQ@mail.gmail.com>

On Wed, Mar 11, 2015 at 11:34 PM, Victor Stinner
<victor.stinner at gmail.com> wrote:
>
> Le 11 mars 2015 18:29, "Brett Cannon" <brett at python.org> a ?crit :
>> I'm going to propose a somewhat controversial idea: let's deprecate the
>> ctypes module.
>
> In the past I tried to deprecate many functions or modules because they are
> rarely or never used. Many developers prefered to keep them. By the way, I
> still want to remove plat-xxx modules like IN or CDROM :-)
>
> Getopt was deprecated when optparse was added to the stdlib. Then optparse
> was deprecated when argparse was added to the stdlib.
>
> Cython and cffi are not part of the stdlib and can be hard to install on
> some platforms. Ctypes is cool because it doesn't require C headers nor a C
> compiler.
>
> Is it possible to use cffi without a C compiler/headers as easily than
> ctypes?

yes, it has two modes, one that does that and the other that does
extra safety at the cost of a C compiler

From p.f.moore at gmail.com  Wed Mar 11 23:03:33 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 11 Mar 2015 22:03:33 +0000
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <CAK5idxTJ=hfq8_xqSZbq=3h69_joBcLvGk0m1a002+XcHgddhQ@mail.gmail.com>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAP1=2W7=DRv-rfy5jv=pwamXhC4xU8qnPm3fDAjb2WTFKOYC3A@mail.gmail.com>
 <CAMpsgwaAaxiMquZHhHCL5RofAB_neci8TvDuLA4-W_XMThwpCA@mail.gmail.com>
 <CAK5idxTJ=hfq8_xqSZbq=3h69_joBcLvGk0m1a002+XcHgddhQ@mail.gmail.com>
Message-ID: <CACac1F9mK4+W3=Zpau4CdJ3yZ1YU42CMJ2A=YUEw-PD+DXK7Ag@mail.gmail.com>

On 11 March 2015 at 21:45, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> Is it possible to use cffi without a C compiler/headers as easily than
>> ctypes?
>
> yes, it has two modes, one that does that and the other that does
> extra safety at the cost of a C compiler

So if someone were to propose a practical approach to including cffi
into the stdlib, *and* assisting the many Windows projects using
ctypes for access to the Windows API [1], then there may be a
reasonable argument for deprecating ctypes. But nobody seems to be
doing that, rather the suggestion appears to be just to deprecate a
widely used part of the stdlib offering no migration path :-(

Paul

[1] There may be Unix users of ctypes, but AFAIK, the vast majority of
use cases are on Windows.

From brett at python.org  Wed Mar 11 23:06:23 2015
From: brett at python.org (Brett Cannon)
Date: Wed, 11 Mar 2015 22:06:23 +0000
Subject: [Python-Dev] Why does python use relative instead of absolute
 path when calling LoadLibrary*
In-Reply-To: <CAGY4rcUvP5HqfUc3ZDhOr6U=iVVmbrW_v_9ofzOU=JaiinKc6w@mail.gmail.com>
References: <CAGY4rcUvP5HqfUc3ZDhOr6U=iVVmbrW_v_9ofzOU=JaiinKc6w@mail.gmail.com>
Message-ID: <CAP1=2W5D1h3rmuCnxdET6bTewQRStuCAJD3BTJkZmV5OF2dnBA@mail.gmail.com>

On Wed, Mar 11, 2015 at 4:37 PM David Cournapeau <cournape at gmail.com> wrote:

> Hi,
>
> While looking at the import code of python for C extensions, I was
> wondering why we pass a relative path instead of an absolute path to
> LoadLibraryEx (see bottom for some context).
>
> In python 2.7, the full path existence was even checked before calling
> into LoadLibraryEx (
> https://github.com/python/cpython/blob/2.7/Python/dynload_win.c#L189),
> but it looks like this check was removed in python 3.x branch.
>

I personally don't know of any specific reason.

-Brett


>
> Is there any defined behaviour that depends on this path to be relative ?
>
> Context
> -----------
>
> The reason why I am interested in this is the potential use of
> SetDllDirectory to share dlls between multiple python extensions.
> Currently, the only solutions I am aware of are:
>
> 1. putting the dlls in the PATH
> 2. bundling the dlls side by side the .pyd
> 3. patching packages to use preloading (using e.g. ctypes)
>
> I am investigating a solution 4, where the dlls would be put in a separate
> "private" directory only known of python itself, without the need to modify
> PATH.
>
> Patching python to use SetDllDirectory("some private paths specific to a
> python interpreter") works perfectly, except that it slightly changes the
> semantics of LoadLibraryEx not to look for dlls in the current directory.
> This breaks importing extensions built in place, unless I modify the call
> in ;https://github.com/python/cpython/blob/2.7/Python/dynload_win.c#L195
> from:
>
>     hDLL = LoadLibraryEx(pathname, NULL LOAD_WITH_ALTERED_SEARCH_PATH)
>
> to
>
>     hDLL = LoadLibraryEx(pathbuf, NULL LOAD_WITH_ALTERED_SEARCH_PATH)
>
> That seems to work, but I am quite worried about changing any import
> semantics by accident.
>
> David
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150311/1ecd5e8f/attachment-0001.html>

From brett at python.org  Wed Mar 11 23:08:21 2015
From: brett at python.org (Brett Cannon)
Date: Wed, 11 Mar 2015 22:08:21 +0000
Subject: [Python-Dev] PEP 488: elimination of PYO files
In-Reply-To: <CAMSv6X0ePfq+H69fa+f6-X3UViQbezD6i9yXpJTBSvZVftRFeg@mail.gmail.com>
References: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
 <54F9EBAF.1080400@hotpy.org>
 <CAP1=2W6JUnDKpQFcr1Ud_b9rCBayk3RCp8XQQhd3zSQDYwmxtQ@mail.gmail.com>
 <CAMSv6X0ePfq+H69fa+f6-X3UViQbezD6i9yXpJTBSvZVftRFeg@mail.gmail.com>
Message-ID: <CAP1=2W4ovp6ccJE-vfR-TLA5NNm8Gmc-eC2F=fv=UC0grpF0hw@mail.gmail.com>

On Wed, Mar 11, 2015 at 5:29 PM Armin Rigo <arigo at tunes.org> wrote:

> Hi Brett,
>
> On 6 March 2015 at 19:11, Brett Cannon <brett at python.org> wrote:
> > I disagree with your premise that .pyo files don't have a noticeable
> effect
> > on performance. If you don't use asserts a lot then there is no effect,
> but
> > if you use them heavily or have them perform expensive calculations then
> > there is an impact.
>
> Maybe you'd be interested to learn that PyPy (at least the 2.x branch)
> uses a new bytecode, JUMP_IF_NOT_DEBUG, to conditionally jump over the
> "assert" line.  In "optimized" mode PyPy follows the jumps; in
> "non-optimized" mode it doesn't.  This mode is initialized with the -O
> flag but can be changed dynamically, as the bytecode is the same.  We
> introduced it as a simple solution to the mess of .pyc versus .pyo.
> (We didn't consider the case of -OO very closely because PyPy is much
> bigger than CPython as a binary to start with, so the demand for that
> is lower.)
>

Interesting, so you simply merged the optimization levels 0 and 1 in the
bytecode and basically drop .pyo files thanks to it. That might be some
motivation to support the default file name not having any specified
optimization level at all.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150311/62be8e81/attachment.html>

From brett at python.org  Wed Mar 11 23:20:58 2015
From: brett at python.org (Brett Cannon)
Date: Wed, 11 Mar 2015 22:20:58 +0000
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <CACac1F9mK4+W3=Zpau4CdJ3yZ1YU42CMJ2A=YUEw-PD+DXK7Ag@mail.gmail.com>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAP1=2W7=DRv-rfy5jv=pwamXhC4xU8qnPm3fDAjb2WTFKOYC3A@mail.gmail.com>
 <CAMpsgwaAaxiMquZHhHCL5RofAB_neci8TvDuLA4-W_XMThwpCA@mail.gmail.com>
 <CAK5idxTJ=hfq8_xqSZbq=3h69_joBcLvGk0m1a002+XcHgddhQ@mail.gmail.com>
 <CACac1F9mK4+W3=Zpau4CdJ3yZ1YU42CMJ2A=YUEw-PD+DXK7Ag@mail.gmail.com>
Message-ID: <CAP1=2W7tcKtF3mxWiqhLhrFG-5=N_8cOyVQ8L3sNCK5-AJxV6w@mail.gmail.com>

On Wed, Mar 11, 2015 at 6:03 PM Paul Moore <p.f.moore at gmail.com> wrote:

> On 11 March 2015 at 21:45, Maciej Fijalkowski <fijall at gmail.com> wrote:
> >> Is it possible to use cffi without a C compiler/headers as easily than
> >> ctypes?
> >
> > yes, it has two modes, one that does that and the other that does
> > extra safety at the cost of a C compiler
>
> So if someone were to propose a practical approach to including cffi
> into the stdlib, *and* assisting the many Windows projects using
> ctypes for access to the Windows API [1], then there may be a
> reasonable argument for deprecating ctypes. But nobody seems to be
> doing that, rather the suggestion appears to be just to deprecate a
> widely used part of the stdlib offering no migration path :-(
>

You're ignoring that it's not maintained, which is the entire reason I
brought this up. No one seems to want to touch the code. Who knows what
improvements, bugfixes, etc. exist upstream in libffi that we lack because
no one wants to go through and figure it out. If someone would come forward
and help maintain it then I have no issue with it sticking around.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150311/bdc55d0a/attachment.html>

From solipsis at pitrou.net  Wed Mar 11 23:28:18 2015
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 11 Mar 2015 23:28:18 +0100
Subject: [Python-Dev] libffi embedded in CPython
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAP1=2W7=DRv-rfy5jv=pwamXhC4xU8qnPm3fDAjb2WTFKOYC3A@mail.gmail.com>
 <CAMpsgwaAaxiMquZHhHCL5RofAB_neci8TvDuLA4-W_XMThwpCA@mail.gmail.com>
 <CAK5idxTJ=hfq8_xqSZbq=3h69_joBcLvGk0m1a002+XcHgddhQ@mail.gmail.com>
 <CACac1F9mK4+W3=Zpau4CdJ3yZ1YU42CMJ2A=YUEw-PD+DXK7Ag@mail.gmail.com>
 <CAP1=2W7tcKtF3mxWiqhLhrFG-5=N_8cOyVQ8L3sNCK5-AJxV6w@mail.gmail.com>
Message-ID: <20150311232818.5997d7cc@fsol>

On Wed, 11 Mar 2015 22:20:58 +0000
Brett Cannon <brett at python.org> wrote:
> 
> You're ignoring that it's not maintained,

According to which metric/criterion?

changeset:   94932:86c9ef950288
user:        Steve Dower <steve.dower at microsoft.com>
date:        Tue Mar 10 09:56:38 2015 -0700
summary:     Issue #23606: Disable ctypes.util.find_library("c") on
Windows so tests are skipped while we figure out how best to approach
the CRT change

changeset:   94756:7c6e3358221a
user:        Serhiy Storchaka <storchaka at gmail.com>
date:        Thu Feb 26 15:27:57 2015 +0200
summary:     Silenced minor GCC warnings.

changeset:   94653:a84ae2ccd220
user:        Serhiy Storchaka <storchaka at gmail.com>
date:        Mon Feb 16 20:52:17 2015 +0200
summary:     Issue #23450: Fixed possible integer overflows.

[etc.]

> Who knows what
> improvements, bugfixes, etc. exist upstream in libffi that we lack because
> no one wants to go through and figure it out.

Well, who knows? How about you enlighten us about that?
And why do you think cffi, which *also* requires libffi, would be
better served in that regard?

Regards

Antoine.



From solipsis at pitrou.net  Wed Mar 11 23:31:45 2015
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 11 Mar 2015 23:31:45 +0100
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <CAK5idxS2P45pVK+XAXEfOTsZhWebB+yL953KL-+4zzMTcz3L_g@mail.gmail.com>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAP1=2W7=DRv-rfy5jv=pwamXhC4xU8qnPm3fDAjb2WTFKOYC3A@mail.gmail.com>
 <20150311185028.6f05973c@fsol>
 <CAK5idxQYyWbWq_frrvWV0ZYdfeWgCm8pW0dgotf0uE-op4WNBg@mail.gmail.com>
 <20150311190557.25eebc16@fsol>
 <CAK5idxS2P45pVK+XAXEfOTsZhWebB+yL953KL-+4zzMTcz3L_g@mail.gmail.com>
Message-ID: <20150311233145.5a6cd376@fsol>

On Wed, 11 Mar 2015 23:10:14 +0200
Maciej Fijalkowski <fijall at gmail.com> wrote:
> >
> > Well, sure. The point is, such bugs are unlikely to appear at a fast
> > rate... Also, I don't understand why libffi issues would affect cffi
> > any less than it affects ctypes, at least in the compiler-less mode of
> > operation.
> 
> My point here was only about shipping own libffi vs using the system
> one (and it does affect cffi equally with or without compiler)

So what? If ctypes used the system libffi as cffi does, it would by
construction be at least portable as cffi is.  The only reason the
bundled libffi was patched at some point was to be *more* portable than
vanilla libffi is.

So, really, I don't see how switching from ctypes to cffi solves any of
this.

Regards

Antoine.

From fijall at gmail.com  Wed Mar 11 23:31:55 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 12 Mar 2015 00:31:55 +0200
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <CAP1=2W7tcKtF3mxWiqhLhrFG-5=N_8cOyVQ8L3sNCK5-AJxV6w@mail.gmail.com>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAP1=2W7=DRv-rfy5jv=pwamXhC4xU8qnPm3fDAjb2WTFKOYC3A@mail.gmail.com>
 <CAMpsgwaAaxiMquZHhHCL5RofAB_neci8TvDuLA4-W_XMThwpCA@mail.gmail.com>
 <CAK5idxTJ=hfq8_xqSZbq=3h69_joBcLvGk0m1a002+XcHgddhQ@mail.gmail.com>
 <CACac1F9mK4+W3=Zpau4CdJ3yZ1YU42CMJ2A=YUEw-PD+DXK7Ag@mail.gmail.com>
 <CAP1=2W7tcKtF3mxWiqhLhrFG-5=N_8cOyVQ8L3sNCK5-AJxV6w@mail.gmail.com>
Message-ID: <CAK5idxTWy6UDjo9XiefnT8UwP0AFjLkNDFbidfo-1MwNPcVmKg@mail.gmail.com>

On Thu, Mar 12, 2015 at 12:20 AM, Brett Cannon <brett at python.org> wrote:
>
>
> On Wed, Mar 11, 2015 at 6:03 PM Paul Moore <p.f.moore at gmail.com> wrote:
>>
>> On 11 March 2015 at 21:45, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> >> Is it possible to use cffi without a C compiler/headers as easily than
>> >> ctypes?
>> >
>> > yes, it has two modes, one that does that and the other that does
>> > extra safety at the cost of a C compiler
>>
>> So if someone were to propose a practical approach to including cffi
>> into the stdlib, *and* assisting the many Windows projects using
>> ctypes for access to the Windows API [1], then there may be a
>> reasonable argument for deprecating ctypes. But nobody seems to be
>> doing that, rather the suggestion appears to be just to deprecate a
>> widely used part of the stdlib offering no migration path :-(
>
>
> You're ignoring that it's not maintained, which is the entire reason I
> brought this up. No one seems to want to touch the code. Who knows what
> improvements, bugfixes, etc. exist upstream in libffi that we lack because
> no one wants to go through and figure it out. If someone would come forward
> and help maintain it then I have no issue with it sticking around.

It's a bit worse than that. Each time someone wants to touch the code
(e.g. push back the upstream libffi fixes), there is "we need to
review it, but there is noone to do it", "noone knows how it works,
don't touch it" kind of feedback, which leads to disincentives to
potential maintainers. I would be likely willing to rip off the libffi
from CPython as it is for example (and just use the upstream one)

From fijall at gmail.com  Wed Mar 11 23:33:23 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 12 Mar 2015 00:33:23 +0200
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <20150311233145.5a6cd376@fsol>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAP1=2W7=DRv-rfy5jv=pwamXhC4xU8qnPm3fDAjb2WTFKOYC3A@mail.gmail.com>
 <20150311185028.6f05973c@fsol>
 <CAK5idxQYyWbWq_frrvWV0ZYdfeWgCm8pW0dgotf0uE-op4WNBg@mail.gmail.com>
 <20150311190557.25eebc16@fsol>
 <CAK5idxS2P45pVK+XAXEfOTsZhWebB+yL953KL-+4zzMTcz3L_g@mail.gmail.com>
 <20150311233145.5a6cd376@fsol>
Message-ID: <CAK5idxS=D4nMBN9zNaJtiGA=5gdTPX9g+4xywhiN1nxT_WQmOw@mail.gmail.com>

On Thu, Mar 12, 2015 at 12:31 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Wed, 11 Mar 2015 23:10:14 +0200
> Maciej Fijalkowski <fijall at gmail.com> wrote:
>> >
>> > Well, sure. The point is, such bugs are unlikely to appear at a fast
>> > rate... Also, I don't understand why libffi issues would affect cffi
>> > any less than it affects ctypes, at least in the compiler-less mode of
>> > operation.
>>
>> My point here was only about shipping own libffi vs using the system
>> one (and it does affect cffi equally with or without compiler)
>
> So what? If ctypes used the system libffi as cffi does, it would by
> construction be at least portable as cffi is.  The only reason the
> bundled libffi was patched at some point was to be *more* portable than
> vanilla libffi is.
>
> So, really, I don't see how switching from ctypes to cffi solves any of
> this.

You're missing my point. Ripping off the libffi from CPython is a good
idea to start with. Maybe deprecating ctypes is *also* a good idea,
but it's a separate discussion point. It certainly does not solve the
libffi problem.

From p.f.moore at gmail.com  Wed Mar 11 23:55:33 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 11 Mar 2015 22:55:33 +0000
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <CAK5idxS=D4nMBN9zNaJtiGA=5gdTPX9g+4xywhiN1nxT_WQmOw@mail.gmail.com>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAP1=2W7=DRv-rfy5jv=pwamXhC4xU8qnPm3fDAjb2WTFKOYC3A@mail.gmail.com>
 <20150311185028.6f05973c@fsol>
 <CAK5idxQYyWbWq_frrvWV0ZYdfeWgCm8pW0dgotf0uE-op4WNBg@mail.gmail.com>
 <20150311190557.25eebc16@fsol>
 <CAK5idxS2P45pVK+XAXEfOTsZhWebB+yL953KL-+4zzMTcz3L_g@mail.gmail.com>
 <20150311233145.5a6cd376@fsol>
 <CAK5idxS=D4nMBN9zNaJtiGA=5gdTPX9g+4xywhiN1nxT_WQmOw@mail.gmail.com>
Message-ID: <CACac1F979sg5O1MALXHs781dVNR0g0+zFtjMny--9pmzAa=XGg@mail.gmail.com>

On 11 March 2015 at 22:33, Maciej Fijalkowski <fijall at gmail.com> wrote:
> You're missing my point. Ripping off the libffi from CPython is a good
> idea to start with. Maybe deprecating ctypes is *also* a good idea,
> but it's a separate discussion point. It certainly does not solve the
> libffi problem.

OK, so let's focus on the libffi side of things and ignore deprecating
or replacing ctypes.

I guess I don't see a problem with a proof-of-concept patch to upgrade
the libffi (obviously it's not possible to rely on a "system" libffi
on Windows, but treating it as an external might work). If it passes
all the tests on all platforms, maybe it could be considered.

I don't see how anyone can say "yes, we'll do it" without seeing
evidence that it'll work. But equally, I don't think there's any
reason to say it absolutely wouldn't be accepted regardless of
evidence. So why not prepare a patch for 3.6 (I assume it's too late
to make such a big change for 3.5) and we'll see what happens. Or even
better, prepare a test build of 3.5 or even 3.4 that switches libffi -
people can replace an existing install (I'd be willing to) and test it
in live situations.

But unless someone provides a patch, the status quo will win. At least
until an actual bug that affects live code forces the issue.

Paul

From steve at pearwood.info  Thu Mar 12 00:03:53 2015
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 12 Mar 2015 10:03:53 +1100
Subject: [Python-Dev] PEP 488: elimination of PYO files
In-Reply-To: <CAP1=2W5jmY88Z5AfT0MD9yfmaKq3q5Zo8BAQ3Tu96HZseeTcFQ@mail.gmail.com>
References: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
 <54F9EBAF.1080400@hotpy.org>
 <CAP1=2W6JUnDKpQFcr1Ud_b9rCBayk3RCp8XQQhd3zSQDYwmxtQ@mail.gmail.com>
 <CAA68w_nJW9=_=U=fwfTWN+h5vdaWT-Si9YpqxddULJLduPA-ew@mail.gmail.com>
 <CAP1=2W4g-sUwh74Zdu4rORqjOF8O24eD=HUatVinQ_HnVdKbVQ@mail.gmail.com>
 <1425668955.3364036.236612829.494092A7@webmail.messagingengine.com>
 <CAP1=2W4FbUQjaK5a-FgsfS1c3kP5qbqi=iZHE5h1+CxDYrJqtA@mail.gmail.com>
 <1425685751.3874542.236704353.11A94396@webmail.messagingengine.com>
 <CAP1=2W5jmY88Z5AfT0MD9yfmaKq3q5Zo8BAQ3Tu96HZseeTcFQ@mail.gmail.com>
Message-ID: <20150311230352.GB7655@ando.pearwood.info>

On Wed, Mar 11, 2015 at 05:34:10PM +0000, Brett Cannon wrote:

> I have a poll going on G+ to see what people think of the various proposed
> file name formats at
> https://plus.google.com/u/0/+BrettCannon/posts/fZynLNwHWGm . Feel free to
> vote if you have an opinion.

G+ hates my browser and won't let me vote. I click on the button and 
nothing happens. I have Javascript enabled and I'm not using any ad 
blockers.

For the record, I think only the first two options 

importlib.cpython-35.opt-0.pyc
importlib.cpython-35.opt0.pyc


are sane, and I prefer the first. I'm mildly inclined to leave out the 
opt* part for default, unoptimized code. In other words, the file name 
holds two or three '.' delimited fields, plus the extension:

<module>.<interpreter>-<version>.[opt-<optimization code>].pyc

where [...] is optional and the optimization codes for CPython will be 1 
for -O and 2 for -OO. And 0 for unoptimized, if you decide that it 
should be mandatory.

Thank you for moving forward on this, I think it is a good plan.



-- 
Steve

From solipsis at pitrou.net  Thu Mar 12 00:11:40 2015
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 12 Mar 2015 00:11:40 +0100
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <CACac1F979sg5O1MALXHs781dVNR0g0+zFtjMny--9pmzAa=XGg@mail.gmail.com>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAP1=2W7=DRv-rfy5jv=pwamXhC4xU8qnPm3fDAjb2WTFKOYC3A@mail.gmail.com>
 <20150311185028.6f05973c@fsol>
 <CAK5idxQYyWbWq_frrvWV0ZYdfeWgCm8pW0dgotf0uE-op4WNBg@mail.gmail.com>
 <20150311190557.25eebc16@fsol>
 <CAK5idxS2P45pVK+XAXEfOTsZhWebB+yL953KL-+4zzMTcz3L_g@mail.gmail.com>
 <20150311233145.5a6cd376@fsol>
 <CAK5idxS=D4nMBN9zNaJtiGA=5gdTPX9g+4xywhiN1nxT_WQmOw@mail.gmail.com>
 <CACac1F979sg5O1MALXHs781dVNR0g0+zFtjMny--9pmzAa=XGg@mail.gmail.com>
Message-ID: <20150312001140.64419595@fsol>

On Wed, 11 Mar 2015 22:55:33 +0000
Paul Moore <p.f.moore at gmail.com> wrote:

> On 11 March 2015 at 22:33, Maciej Fijalkowski <fijall at gmail.com> wrote:
> > You're missing my point. Ripping off the libffi from CPython is a good
> > idea to start with. Maybe deprecating ctypes is *also* a good idea,
> > but it's a separate discussion point. It certainly does not solve the
> > libffi problem.
> 
> OK, so let's focus on the libffi side of things and ignore deprecating
> or replacing ctypes.
> 
> I guess I don't see a problem with a proof-of-concept patch to upgrade
> the libffi (obviously it's not possible to rely on a "system" libffi
> on Windows, but treating it as an external might work). If it passes
> all the tests on all platforms, maybe it could be considered.

Agreed. We have enough tests and enough buildbots that it can be
attempted.
(besides, the only more-or-less officially supported platforms are
Linux, Windows, OS X; we also try to not to be too broken on the BSDs,
but that's it)

Regards

Antoine.

From guido at python.org  Thu Mar 12 01:20:33 2015
From: guido at python.org (Guido van Rossum)
Date: Wed, 11 Mar 2015 17:20:33 -0700
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <CAP1=2W7tcKtF3mxWiqhLhrFG-5=N_8cOyVQ8L3sNCK5-AJxV6w@mail.gmail.com>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAP1=2W7=DRv-rfy5jv=pwamXhC4xU8qnPm3fDAjb2WTFKOYC3A@mail.gmail.com>
 <CAMpsgwaAaxiMquZHhHCL5RofAB_neci8TvDuLA4-W_XMThwpCA@mail.gmail.com>
 <CAK5idxTJ=hfq8_xqSZbq=3h69_joBcLvGk0m1a002+XcHgddhQ@mail.gmail.com>
 <CACac1F9mK4+W3=Zpau4CdJ3yZ1YU42CMJ2A=YUEw-PD+DXK7Ag@mail.gmail.com>
 <CAP1=2W7tcKtF3mxWiqhLhrFG-5=N_8cOyVQ8L3sNCK5-AJxV6w@mail.gmail.com>
Message-ID: <CAP7+vJKUEsD-erzzzNKU9BaraQSW44dCGbNkhJZV90+cTNuy7w@mail.gmail.com>

Can we please decouple the ctypes deprecation discussion from efforts to
upgrade cffi? They can coexist just fine, and they don't even really solve
the same problem.

On Wed, Mar 11, 2015 at 3:20 PM, Brett Cannon <brett at python.org> wrote:

>
>
> On Wed, Mar 11, 2015 at 6:03 PM Paul Moore <p.f.moore at gmail.com> wrote:
>
>> On 11 March 2015 at 21:45, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> >> Is it possible to use cffi without a C compiler/headers as easily than
>> >> ctypes?
>> >
>> > yes, it has two modes, one that does that and the other that does
>> > extra safety at the cost of a C compiler
>>
>> So if someone were to propose a practical approach to including cffi
>> into the stdlib, *and* assisting the many Windows projects using
>> ctypes for access to the Windows API [1], then there may be a
>> reasonable argument for deprecating ctypes. But nobody seems to be
>> doing that, rather the suggestion appears to be just to deprecate a
>> widely used part of the stdlib offering no migration path :-(
>>
>
> You're ignoring that it's not maintained, which is the entire reason I
> brought this up. No one seems to want to touch the code. Who knows what
> improvements, bugfixes, etc. exist upstream in libffi that we lack because
> no one wants to go through and figure it out. If someone would come forward
> and help maintain it then I have no issue with it sticking around.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150311/9e9acf2d/attachment-0001.html>

From wes.turner at gmail.com  Thu Mar 12 03:41:37 2015
From: wes.turner at gmail.com (Wes Turner)
Date: Wed, 11 Mar 2015 21:41:37 -0500
Subject: [Python-Dev] Why does python use relative instead of absolute
 path when calling LoadLibrary*
In-Reply-To: <CAGY4rcUvP5HqfUc3ZDhOr6U=iVVmbrW_v_9ofzOU=JaiinKc6w@mail.gmail.com>
References: <CAGY4rcUvP5HqfUc3ZDhOr6U=iVVmbrW_v_9ofzOU=JaiinKc6w@mail.gmail.com>
Message-ID: <CACfEFw-U4kok9VjcpVGifABBY==7cEMS9f0Jvi_Q_hJLJcyLug@mail.gmail.com>

On Mar 11, 2015 3:36 PM, "David Cournapeau" <cournape at gmail.com> wrote:
>
> Hi,
>
> While looking at the import code of python for C extensions, I was
wondering why we pass a relative path instead of an absolute path to
LoadLibraryEx (see bottom for some context).
>
> In python 2.7, the full path existence was even checked before calling
into LoadLibraryEx (
https://github.com/python/cpython/blob/2.7/Python/dynload_win.c#L189), but
it looks like this check was removed in python 3.x branch.
>
> Is there any defined behaviour that depends on this path to be relative ?

Just a guess: does it have to do with resolving symlinks (w/ POSIX
filesystems)?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150311/96493110/attachment.html>

From soeren.j.loevborg at siemens.com  Thu Mar 12 13:37:16 2015
From: soeren.j.loevborg at siemens.com (Loevborg, Soeren Jakob)
Date: Thu, 12 Mar 2015 12:37:16 +0000
Subject: [Python-Dev] Request for comments: [issue22941] IPv4Interface
 arithmetic changes subnet mask
Message-ID: <927E0DF62C47C940A006AFF297D7F512120C11@DEFTHW99EK0MSX.ww902.siemens.net>

Hi,

I'm looking for feedback on issue 22941, "IPv4Interface arithmetic changes subnet mask".

As stated in the bug, I'd be happy to write a patch, if anyone would comment on my proposal.

http://bugs.python.org/issue22941

Thank you,
S?ren L?vborg

---

Addition and subtraction of integers are documented for ipaddress.IPv4Address and ipaddress.IPv6Address, but also work for IPv4Interface and IPv6Interface (though the only documentation of this is a brief mention that the Interface classes inherit from the respective Address classes). That's good.

The problem is that adding/subtracting an integer to an Interface object changes the subnet mask (to max_prefixlen), something which is 1) not documented and 2) not the least surprising result.

>>> import ipaddress
>>> ipaddress.IPv4Interface('10.0.0.1/8') + 1
IPv4Interface('10.0.0.2/32')
>>> ipaddress.IPv6Interface('fd00::1/64') + 1
IPv6Interface('fd00::2/128')

Proposed behavior:

>>> import ipaddress
>>> ipaddress.IPv4Interface('10.0.0.1/8') + 1
IPv4Interface('10.0.0.2/8')
>>> ipaddress.IPv6Interface('fd00::1/64') + 1
IPv6Interface('fd00::2/64')

It all comes down to a judgment call: is this a bug, or expected (but undocumented) behavior?

In PEP 387 lingo: Is this a "reasonable bug fix"? Or is it a "design mistake", thus calling for a FutureWarning first, and the actual change later?

    >>> ipaddress.IPv4Interface('1.2.3.4/16') + 1
    __main__:1: FutureWarning: Arithmetic on IPv4Interface objects will
    change in Python 3.6. If you rely on the current behavior, change
    'interface + n' to 'IPv4Interface(interface.ip + n, 32)'


From solipsis at pitrou.net  Thu Mar 12 15:00:23 2015
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 12 Mar 2015 15:00:23 +0100
Subject: [Python-Dev] Request for comments: [issue22941] IPv4Interface
 arithmetic changes subnet mask
References: <927E0DF62C47C940A006AFF297D7F512120C11@DEFTHW99EK0MSX.ww902.siemens.net>
Message-ID: <20150312150023.04051b02@fsol>

On Thu, 12 Mar 2015 12:37:16 +0000
"Loevborg, Soeren Jakob" <soeren.j.loevborg at siemens.com> wrote:
> 
> >>> import ipaddress
> >>> ipaddress.IPv4Interface('10.0.0.1/8') + 1
> IPv4Interface('10.0.0.2/32')
> >>> ipaddress.IPv6Interface('fd00::1/64') + 1
> IPv6Interface('fd00::2/128')

Given that the behaviour is unlikely to match any use case, I'd say
it's a bug.

Related issue:

>>> ipaddress.IPv4Interface('10.0.0.255/8') + 1
IPv4Interface('10.0.1.0/32')

Should the result be IPv4Interface('10.0.0.0/8')
(i.e. wrap around)? Should OverflowError be raised?

Regards

Antoine.



From brett at python.org  Thu Mar 12 15:05:15 2015
From: brett at python.org (Brett Cannon)
Date: Thu, 12 Mar 2015 14:05:15 +0000
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <CAP7+vJKUEsD-erzzzNKU9BaraQSW44dCGbNkhJZV90+cTNuy7w@mail.gmail.com>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAP1=2W7=DRv-rfy5jv=pwamXhC4xU8qnPm3fDAjb2WTFKOYC3A@mail.gmail.com>
 <CAMpsgwaAaxiMquZHhHCL5RofAB_neci8TvDuLA4-W_XMThwpCA@mail.gmail.com>
 <CAK5idxTJ=hfq8_xqSZbq=3h69_joBcLvGk0m1a002+XcHgddhQ@mail.gmail.com>
 <CACac1F9mK4+W3=Zpau4CdJ3yZ1YU42CMJ2A=YUEw-PD+DXK7Ag@mail.gmail.com>
 <CAP1=2W7tcKtF3mxWiqhLhrFG-5=N_8cOyVQ8L3sNCK5-AJxV6w@mail.gmail.com>
 <CAP7+vJKUEsD-erzzzNKU9BaraQSW44dCGbNkhJZV90+cTNuy7w@mail.gmail.com>
Message-ID: <CAP1=2W7S4p1i-UZLsHrHjpYK+P9=43kTZT=CRbcwxV6tbzk0Wg@mail.gmail.com>

On Wed, Mar 11, 2015 at 8:20 PM Guido van Rossum <guido at python.org> wrote:

> Can we please decouple the ctypes deprecation discussion from efforts to
> upgrade cffi? They can coexist just fine, and they don't even really solve
> the same problem.
>

I mostly proposed deprecating ctypes because we were not keeping up with
libffi upstream. If we solve the latter I'm not bothered enough to
personally pursue the former.

-Brett


>
> On Wed, Mar 11, 2015 at 3:20 PM, Brett Cannon <brett at python.org> wrote:
>
>>
>>
>> On Wed, Mar 11, 2015 at 6:03 PM Paul Moore <p.f.moore at gmail.com> wrote:
>>
>>> On 11 March 2015 at 21:45, Maciej Fijalkowski <fijall at gmail.com> wrote:
>>> >> Is it possible to use cffi without a C compiler/headers as easily than
>>> >> ctypes?
>>> >
>>> > yes, it has two modes, one that does that and the other that does
>>> > extra safety at the cost of a C compiler
>>>
>>> So if someone were to propose a practical approach to including cffi
>>> into the stdlib, *and* assisting the many Windows projects using
>>> ctypes for access to the Windows API [1], then there may be a
>>> reasonable argument for deprecating ctypes. But nobody seems to be
>>> doing that, rather the suggestion appears to be just to deprecate a
>>> widely used part of the stdlib offering no migration path :-(
>>>
>>
>> You're ignoring that it's not maintained, which is the entire reason I
>> brought this up. No one seems to want to touch the code. Who knows what
>> improvements, bugfixes, etc. exist upstream in libffi that we lack because
>> no one wants to go through and figure it out. If someone would come forward
>> and help maintain it then I have no issue with it sticking around.
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>>
> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150312/9a756084/attachment.html>

From thomas at python.org  Thu Mar 12 15:25:18 2015
From: thomas at python.org (Thomas Wouters)
Date: Thu, 12 Mar 2015 15:25:18 +0100
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <CAP7+vJKUEsD-erzzzNKU9BaraQSW44dCGbNkhJZV90+cTNuy7w@mail.gmail.com>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAP1=2W7=DRv-rfy5jv=pwamXhC4xU8qnPm3fDAjb2WTFKOYC3A@mail.gmail.com>
 <CAMpsgwaAaxiMquZHhHCL5RofAB_neci8TvDuLA4-W_XMThwpCA@mail.gmail.com>
 <CAK5idxTJ=hfq8_xqSZbq=3h69_joBcLvGk0m1a002+XcHgddhQ@mail.gmail.com>
 <CACac1F9mK4+W3=Zpau4CdJ3yZ1YU42CMJ2A=YUEw-PD+DXK7Ag@mail.gmail.com>
 <CAP1=2W7tcKtF3mxWiqhLhrFG-5=N_8cOyVQ8L3sNCK5-AJxV6w@mail.gmail.com>
 <CAP7+vJKUEsD-erzzzNKU9BaraQSW44dCGbNkhJZV90+cTNuy7w@mail.gmail.com>
Message-ID: <CAPdQG2qohCDxa4WQY-nQZG00yOkUOgC38rE1wPbirtvWVOPeqg@mail.gmail.com>

On Thu, Mar 12, 2015 at 1:20 AM, Guido van Rossum <guido at python.org> wrote:

> Can we please decouple the ctypes deprecation discussion from efforts to
> upgrade cffi? They can coexist just fine, and they don't even really solve
> the same problem.
>

ctypes and cffi do actually solve the same problem, just not in the same
way.



>
> On Wed, Mar 11, 2015 at 3:20 PM, Brett Cannon <brett at python.org> wrote:
>
>>
>>
>> On Wed, Mar 11, 2015 at 6:03 PM Paul Moore <p.f.moore at gmail.com> wrote:
>>
>>> On 11 March 2015 at 21:45, Maciej Fijalkowski <fijall at gmail.com> wrote:
>>> >> Is it possible to use cffi without a C compiler/headers as easily than
>>> >> ctypes?
>>> >
>>> > yes, it has two modes, one that does that and the other that does
>>> > extra safety at the cost of a C compiler
>>>
>>> So if someone were to propose a practical approach to including cffi
>>> into the stdlib, *and* assisting the many Windows projects using
>>> ctypes for access to the Windows API [1], then there may be a
>>> reasonable argument for deprecating ctypes. But nobody seems to be
>>> doing that, rather the suggestion appears to be just to deprecate a
>>> widely used part of the stdlib offering no migration path :-(
>>>
>>
>> You're ignoring that it's not maintained, which is the entire reason I
>> brought this up. No one seems to want to touch the code. Who knows what
>> improvements, bugfixes, etc. exist upstream in libffi that we lack because
>> no one wants to go through and figure it out. If someone would come forward
>> and help maintain it then I have no issue with it sticking around.
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/thomas%40python.org
>
>


-- 
Thomas Wouters <thomas at python.org>

Hi! I'm an email virus! Think twice before sending your email to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150312/fc77c4be/attachment.html>

From zachary.ware+pydev at gmail.com  Thu Mar 12 15:39:14 2015
From: zachary.ware+pydev at gmail.com (Zachary Ware)
Date: Thu, 12 Mar 2015 09:39:14 -0500
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <549891df.c332e00a.54c7.34f8@mx.google.com>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
Message-ID: <CAKJDb-N1ewaSsWyEycVHWwDASx8OjXG9LAWw3k3ntCZtw6oJSQ@mail.gmail.com>

I started this message about 3 months ago; at this point I'm just
getting it posted so it stops rotting in my Drafts folder.

On Mon, Dec 22, 2014 at 3:49 PM, Jim J. Jewett <jimjjewett at gmail.com> wrote:
> On Thu, Dec 18, 2014, at 14:13, Maciej Fijalkowski wrote:
>> ... http://bugs.python.org/issue23085 ...
>> is there any reason any more for libffi being included in CPython?
>
>
> Paul Moore wrote:
>> Probably the easiest way of moving this forward would be for someone
>> to identify the CPython-specific patches in the current version ...
>
> Christian Heimes wrote:
>> That's easy. All patches are tracked in the diff file
>> https://hg.python.org/cpython/file/3de678cd184d/Modules/_ctypes/libffi.diff
>
> That (200+ lines) doesn't seem to have all the C changes, such as the
> win64 sizeof changes from issue 11835.

I took a little bit of time to look at this a while ago.  Something
that quickly became clear that I'm not sure everybody is on the same
page about (I, for one, had no idea previously) is that libffi for
Windows and libffi for everything else are entirely separate beasts:
the Windows build pulls all of its libffi source from
Modules/_ctypes/libffi_msvc and doesn't even touch
Modules/_ctypes/libffi (which is libffi for nearly everything else).
Most of the files in libffi_msvc haven't been touched in about 4
years, except for a couple fixes by Steve Dower recently.
README.ctypes in that folder states that the files were taken from the
libffi CVS tree in 2004 and there have been some custom modifications
since then, so there is really very little resemblance between
libffi_msvc and current libffi.

There's also 'libffi_arm_wince' and 'libffi_osx'.  I would be fairly
amazed if Python itself built on any version of Windows CE; personally
I would be for ripping out any supposed support that we have for it
(though if somebody is actually using it and it works, it's not
hurting anything and should stay).  That's a completely different
issue, though.

I'm all for ditching our 'libffi_msvc' in favor of adding libffi as
another 'external' for the Windows build.  I have managed to get
_ctypes to build on Windows using vanilla libffi sources, prepared
using their configure script from within Git Bash and built with our
usual Windows build system (properly patched).  Unfortunately, making
things usable will take some work on ctypes itself, which I'm not
qualified to do. I'm happy to pass on my procedure and patches for
getting to the point of successful compilation to anyone who feels up
to fixing the things that are broken.

As for the lightly-patched version of libffi that we keep around for
Linux et. al., looking at the diff file I don't see that there's a
whole lot of reason to keep it around.

-- 
Zach

From eric at trueblade.com  Thu Mar 12 15:46:34 2015
From: eric at trueblade.com (Eric V. Smith)
Date: Thu, 12 Mar 2015 10:46:34 -0400
Subject: [Python-Dev] Request for comments: [issue22941] IPv4Interface
 arithmetic changes subnet mask
In-Reply-To: <20150312150023.04051b02@fsol>
References: <927E0DF62C47C940A006AFF297D7F512120C11@DEFTHW99EK0MSX.ww902.siemens.net>
 <20150312150023.04051b02@fsol>
Message-ID: <5501A6CA.2080301@trueblade.com>

On 3/12/2015 10:00 AM, Antoine Pitrou wrote:
> On Thu, 12 Mar 2015 12:37:16 +0000
> "Loevborg, Soeren Jakob" <soeren.j.loevborg at siemens.com> wrote:
>>
>>>>> import ipaddress
>>>>> ipaddress.IPv4Interface('10.0.0.1/8') + 1
>> IPv4Interface('10.0.0.2/32')
>>>>> ipaddress.IPv6Interface('fd00::1/64') + 1
>> IPv6Interface('fd00::2/128')
> 
> Given that the behaviour is unlikely to match any use case, I'd say
> it's a bug.

Agreed.

> Related issue:
> 
>>>> ipaddress.IPv4Interface('10.0.0.255/8') + 1
> IPv4Interface('10.0.1.0/32')
> 
> Should the result be IPv4Interface('10.0.0.0/8')
> (i.e. wrap around)? Should OverflowError be raised?

I think it should be an OverflowError. 10.0.1.0/8 makes no sense, and
10.0.0.0/8 is unlikely to be desirable.

-- 
Eric.

From brett at python.org  Thu Mar 12 18:26:02 2015
From: brett at python.org (Brett Cannon)
Date: Thu, 12 Mar 2015 17:26:02 +0000
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <CAKJDb-N1ewaSsWyEycVHWwDASx8OjXG9LAWw3k3ntCZtw6oJSQ@mail.gmail.com>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAKJDb-N1ewaSsWyEycVHWwDASx8OjXG9LAWw3k3ntCZtw6oJSQ@mail.gmail.com>
Message-ID: <CAP1=2W7Cx5JpQV_pr61RQs1uBUSJF5f6KG0cD-QCwR2+9ijegg@mail.gmail.com>

On Thu, Mar 12, 2015 at 10:39 AM Zachary Ware <zachary.ware+pydev at gmail.com>
wrote:

> I started this message about 3 months ago; at this point I'm just
> getting it posted so it stops rotting in my Drafts folder.
>

Thanks for looking into this!


>
> On Mon, Dec 22, 2014 at 3:49 PM, Jim J. Jewett <jimjjewett at gmail.com>
> wrote:
> > On Thu, Dec 18, 2014, at 14:13, Maciej Fijalkowski wrote:
> >> ... http://bugs.python.org/issue23085 ...
> >> is there any reason any more for libffi being included in CPython?
> >
> >
> > Paul Moore wrote:
> >> Probably the easiest way of moving this forward would be for someone
> >> to identify the CPython-specific patches in the current version ...
> >
> > Christian Heimes wrote:
> >> That's easy. All patches are tracked in the diff file
> >> https://hg.python.org/cpython/file/3de678cd184d/Modules/_
> ctypes/libffi.diff
> >
> > That (200+ lines) doesn't seem to have all the C changes, such as the
> > win64 sizeof changes from issue 11835.
>
> I took a little bit of time to look at this a while ago.  Something
> that quickly became clear that I'm not sure everybody is on the same
> page about (I, for one, had no idea previously) is that libffi for
> Windows and libffi for everything else are entirely separate beasts:
> the Windows build pulls all of its libffi source from
> Modules/_ctypes/libffi_msvc and doesn't even touch
> Modules/_ctypes/libffi (which is libffi for nearly everything else).
> Most of the files in libffi_msvc haven't been touched in about 4
> years, except for a couple fixes by Steve Dower recently.
> README.ctypes in that folder states that the files were taken from the
> libffi CVS tree in 2004 and there have been some custom modifications
> since then, so there is really very little resemblance between
> libffi_msvc and current libffi.
>

That's what I was afraid of.


>
> There's also 'libffi_arm_wince' and 'libffi_osx'.  I would be fairly
> amazed if Python itself built on any version of Windows CE; personally
> I would be for ripping out any supposed support that we have for it
> (though if somebody is actually using it and it works, it's not
> hurting anything and should stay).  That's a completely different
> issue, though.
>
> I'm all for ditching our 'libffi_msvc' in favor of adding libffi as
> another 'external' for the Windows build.  I have managed to get
> _ctypes to build on Windows using vanilla libffi sources, prepared
> using their configure script from within Git Bash and built with our
> usual Windows build system (properly patched).  Unfortunately, making
> things usable will take some work on ctypes itself, which I'm not
> qualified to do. I'm happy to pass on my procedure and patches for
> getting to the point of successful compilation to anyone who feels up
> to fixing the things that are broken.
>

So it seems possible to use upstream libffi but will require some work.


>
> As for the lightly-patched version of libffi that we keep around for
> Linux et. al., looking at the diff file I don't see that there's a
> whole lot of reason to keep it around.
>

For UNIX OSs we could probably rely on the system libffi then. What's the
situation on OS X? Anyone know if it has libffi, or would be need to be
pulled in to be used like on Windows?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150312/3a150556/attachment.html>

From p.f.moore at gmail.com  Thu Mar 12 18:44:27 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 12 Mar 2015 17:44:27 +0000
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <CAP1=2W7Cx5JpQV_pr61RQs1uBUSJF5f6KG0cD-QCwR2+9ijegg@mail.gmail.com>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAKJDb-N1ewaSsWyEycVHWwDASx8OjXG9LAWw3k3ntCZtw6oJSQ@mail.gmail.com>
 <CAP1=2W7Cx5JpQV_pr61RQs1uBUSJF5f6KG0cD-QCwR2+9ijegg@mail.gmail.com>
Message-ID: <CACac1F86h7qQ=ggKS5Z8FCR4w1L1k-smoqP0362TpqK23autJw@mail.gmail.com>

On 12 March 2015 at 17:26, Brett Cannon <brett at python.org> wrote:
>> I'm all for ditching our 'libffi_msvc' in favor of adding libffi as
>> another 'external' for the Windows build.  I have managed to get
>> _ctypes to build on Windows using vanilla libffi sources, prepared
>> using their configure script from within Git Bash and built with our
>> usual Windows build system (properly patched).  Unfortunately, making
>> things usable will take some work on ctypes itself, which I'm not
>> qualified to do. I'm happy to pass on my procedure and patches for
>> getting to the point of successful compilation to anyone who feels up
>> to fixing the things that are broken.
>
>
> So it seems possible to use upstream libffi but will require some work.

I'd be willing to contemplate helping out on the Windows side of
things, if nobody else steps up (with the proviso that I have little
free time, and I'm saying this without much idea of what's involved
:-)) If Zachary can give a bit more detail on what the work on ctypes
is, and/or put what he has somewhere that I could have a look at, that
might help.

Paul.

From scott+python-dev at scottdial.com  Thu Mar 12 19:08:01 2015
From: scott+python-dev at scottdial.com (Scott Dial)
Date: Thu, 12 Mar 2015 14:08:01 -0400
Subject: [Python-Dev] Request for comments: [issue22941] IPv4Interface
 arithmetic changes subnet mask
In-Reply-To: <5501A6CA.2080301@trueblade.com>
References: <927E0DF62C47C940A006AFF297D7F512120C11@DEFTHW99EK0MSX.ww902.siemens.net>
 <20150312150023.04051b02@fsol> <5501A6CA.2080301@trueblade.com>
Message-ID: <5501D601.9090105@scottdial.com>

On 2015-03-12 10:46 AM, Eric V. Smith wrote:
> On 3/12/2015 10:00 AM, Antoine Pitrou wrote:
>> Related issue:
>>
>>>>> ipaddress.IPv4Interface('10.0.0.255/8') + 1
>> IPv4Interface('10.0.1.0/32')
>>
>> Should the result be IPv4Interface('10.0.0.0/8')
>> (i.e. wrap around)? Should OverflowError be raised?
> 
> I think it should be an OverflowError. 10.0.1.0/8 makes no sense, and
> 10.0.0.0/8 is unlikely to be desirable.
> 

No, you are both imbuing meaning to the dot-notation that is not there.
A.B.C.D is just an ugly way to represent a single 32-bit unsigned
integer. The only contentious part would be when your addition would
overlfow the host-portion of the address:

>>> ipaddress.IPv4Interface('10.255.255.255/8') + 1
IPv4Interface('11.0.0.0/8')

However, if you take the perspective that the address is just a 32-bit
unsigned integer, then it makes perfect sense. I would argue it's likely
to be a source of bugs, but I can't say either way because I never
adopted using this library due to issues that are cataloged in the
mailing list archives. (In the end, I wrote my own and have never found
the need to support such operands.)

-- 
Scott Dial
scott at scottdial.com

From solipsis at pitrou.net  Thu Mar 12 19:17:17 2015
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 12 Mar 2015 19:17:17 +0100
Subject: [Python-Dev] Request for comments: [issue22941] IPv4Interface
 arithmetic changes subnet mask
References: <927E0DF62C47C940A006AFF297D7F512120C11@DEFTHW99EK0MSX.ww902.siemens.net>
 <20150312150023.04051b02@fsol> <5501A6CA.2080301@trueblade.com>
 <5501D601.9090105@scottdial.com>
Message-ID: <20150312191717.30acbd7b@fsol>

On Thu, 12 Mar 2015 14:08:01 -0400
Scott Dial <scott+python-dev at scottdial.com> wrote:

> On 2015-03-12 10:46 AM, Eric V. Smith wrote:
> > On 3/12/2015 10:00 AM, Antoine Pitrou wrote:
> >> Related issue:
> >>
> >>>>> ipaddress.IPv4Interface('10.0.0.255/8') + 1
> >> IPv4Interface('10.0.1.0/32')
> >>
> >> Should the result be IPv4Interface('10.0.0.0/8')
> >> (i.e. wrap around)? Should OverflowError be raised?
> > 
> > I think it should be an OverflowError. 10.0.1.0/8 makes no sense, and
> > 10.0.0.0/8 is unlikely to be desirable.
> > 
> 
> No, you are both imbuing meaning to the dot-notation that is not there.
> A.B.C.D is just an ugly way to represent a single 32-bit unsigned
> integer. The only contentious part would be when your addition would
> overlfow the host-portion of the address:
> 
> >>> ipaddress.IPv4Interface('10.255.255.255/8') + 1
> IPv4Interface('11.0.0.0/8')

Oh, well, you're right, I borked my subnet. So, for the record, I meant
"10.0.0.255/24" where the issue actually appears.

Regards

Antoine.



From nad at acm.org  Thu Mar 12 19:35:55 2015
From: nad at acm.org (Ned Deily)
Date: Thu, 12 Mar 2015 11:35:55 -0700
Subject: [Python-Dev] libffi embedded in CPython
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAKJDb-N1ewaSsWyEycVHWwDASx8OjXG9LAWw3k3ntCZtw6oJSQ@mail.gmail.com>
 <CAP1=2W7Cx5JpQV_pr61RQs1uBUSJF5f6KG0cD-QCwR2+9ijegg@mail.gmail.com>
Message-ID: <nad-BDEEA5.11355512032015@news.gmane.org>

In article 
<CAP1=2W7Cx5JpQV_pr61RQs1uBUSJF5f6KG0cD-QCwR2+9ijegg at mail.gmail.com>,
> For UNIX OSs we could probably rely on the system libffi then. What's the
> situation on OS X? Anyone know if it has libffi, or would be need to be
> pulled in to be used like on Windows?

Ronald (in http://bugs.python.org/issue23534):
"On OSX the internal copy of libffi that's used is based on the one in 
PyObjC, which in turn is based on the version of libffi on 
opensource.apple.com (IIRC with some small patches that fix minor issues 
found by the PyObjC testsuite)."

-- 
 Ned Deily,
 nad at acm.org


From zachary.ware+pydev at gmail.com  Thu Mar 12 19:54:16 2015
From: zachary.ware+pydev at gmail.com (Zachary Ware)
Date: Thu, 12 Mar 2015 13:54:16 -0500
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <CACac1F86h7qQ=ggKS5Z8FCR4w1L1k-smoqP0362TpqK23autJw@mail.gmail.com>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAKJDb-N1ewaSsWyEycVHWwDASx8OjXG9LAWw3k3ntCZtw6oJSQ@mail.gmail.com>
 <CAP1=2W7Cx5JpQV_pr61RQs1uBUSJF5f6KG0cD-QCwR2+9ijegg@mail.gmail.com>
 <CACac1F86h7qQ=ggKS5Z8FCR4w1L1k-smoqP0362TpqK23autJw@mail.gmail.com>
Message-ID: <CAKJDb-PPYjb3RYgnTxmebCASJOQ2Qney5kHjXRNeaECiTZBNiw@mail.gmail.com>

On Thu, Mar 12, 2015 at 12:44 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> I'd be willing to contemplate helping out on the Windows side of
> things, if nobody else steps up (with the proviso that I have little
> free time, and I'm saying this without much idea of what's involved
> :-)) If Zachary can give a bit more detail on what the work on ctypes
> is, and/or put what he has somewhere that I could have a look at, that
> might help.

I'll send you everything I've got next chance I have, but I don't know
when that will be.

-- 
Zach

From eric at trueblade.com  Thu Mar 12 19:55:05 2015
From: eric at trueblade.com (Eric V. Smith)
Date: Thu, 12 Mar 2015 14:55:05 -0400
Subject: [Python-Dev] Request for comments: [issue22941] IPv4Interface
 arithmetic changes subnet mask
In-Reply-To: <20150312191717.30acbd7b@fsol>
References: <927E0DF62C47C940A006AFF297D7F512120C11@DEFTHW99EK0MSX.ww902.siemens.net>
 <20150312150023.04051b02@fsol> <5501A6CA.2080301@trueblade.com>
 <5501D601.9090105@scottdial.com> <20150312191717.30acbd7b@fsol>
Message-ID: <5501E109.7090408@trueblade.com>

On 3/12/2015 2:17 PM, Antoine Pitrou wrote:
> On Thu, 12 Mar 2015 14:08:01 -0400
> Scott Dial <scott+python-dev at scottdial.com> wrote:
> 
>> On 2015-03-12 10:46 AM, Eric V. Smith wrote:
>>> On 3/12/2015 10:00 AM, Antoine Pitrou wrote:
>>>> Related issue:
>>>>
>>>>>>> ipaddress.IPv4Interface('10.0.0.255/8') + 1
>>>> IPv4Interface('10.0.1.0/32')
>>>>
>>>> Should the result be IPv4Interface('10.0.0.0/8')
>>>> (i.e. wrap around)? Should OverflowError be raised?
>>>
>>> I think it should be an OverflowError. 10.0.1.0/8 makes no sense, and
>>> 10.0.0.0/8 is unlikely to be desirable.
>>>
>>
>> No, you are both imbuing meaning to the dot-notation that is not there.
>> A.B.C.D is just an ugly way to represent a single 32-bit unsigned
>> integer. The only contentious part would be when your addition would
>> overlfow the host-portion of the address:
>>
>>>>> ipaddress.IPv4Interface('10.255.255.255/8') + 1
>> IPv4Interface('11.0.0.0/8')
> 
> Oh, well, you're right, I borked my subnet. So, for the record, I meant
> "10.0.0.255/24" where the issue actually appears.

I, too, was thinking /24. I think that overflowing the host portion
should raise OverflowError.

-- 
Eric.

From p.f.moore at gmail.com  Thu Mar 12 20:13:08 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 12 Mar 2015 19:13:08 +0000
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <CAKJDb-PPYjb3RYgnTxmebCASJOQ2Qney5kHjXRNeaECiTZBNiw@mail.gmail.com>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAKJDb-N1ewaSsWyEycVHWwDASx8OjXG9LAWw3k3ntCZtw6oJSQ@mail.gmail.com>
 <CAP1=2W7Cx5JpQV_pr61RQs1uBUSJF5f6KG0cD-QCwR2+9ijegg@mail.gmail.com>
 <CACac1F86h7qQ=ggKS5Z8FCR4w1L1k-smoqP0362TpqK23autJw@mail.gmail.com>
 <CAKJDb-PPYjb3RYgnTxmebCASJOQ2Qney5kHjXRNeaECiTZBNiw@mail.gmail.com>
Message-ID: <CACac1F_A7krghQp7KbzracTbKq2JpxPR=572nB6Ux-hww6k4Qg@mail.gmail.com>

On 12 March 2015 at 18:54, Zachary Ware <zachary.ware+pydev at gmail.com> wrote:
> On Thu, Mar 12, 2015 at 12:44 PM, Paul Moore <p.f.moore at gmail.com> wrote:
>> I'd be willing to contemplate helping out on the Windows side of
>> things, if nobody else steps up (with the proviso that I have little
>> free time, and I'm saying this without much idea of what's involved
>> :-)) If Zachary can give a bit more detail on what the work on ctypes
>> is, and/or put what he has somewhere that I could have a look at, that
>> might help.
>
> I'll send you everything I've got next chance I have, but I don't know
> when that will be.

Thanks. No rush :-)
Paul

From fijall at gmail.com  Thu Mar 12 20:36:45 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 12 Mar 2015 21:36:45 +0200
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <nad-BDEEA5.11355512032015@news.gmane.org>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAKJDb-N1ewaSsWyEycVHWwDASx8OjXG9LAWw3k3ntCZtw6oJSQ@mail.gmail.com>
 <CAP1=2W7Cx5JpQV_pr61RQs1uBUSJF5f6KG0cD-QCwR2+9ijegg@mail.gmail.com>
 <nad-BDEEA5.11355512032015@news.gmane.org>
Message-ID: <CAK5idxQ_hcs7qCk8a1LBO35Bg2Ni=jXOieXdQJwXyfnbhL+L-w@mail.gmail.com>

On Thu, Mar 12, 2015 at 8:35 PM, Ned Deily <nad at acm.org> wrote:
> In article
> <CAP1=2W7Cx5JpQV_pr61RQs1uBUSJF5f6KG0cD-QCwR2+9ijegg at mail.gmail.com>,
>> For UNIX OSs we could probably rely on the system libffi then. What's the
>> situation on OS X? Anyone know if it has libffi, or would be need to be
>> pulled in to be used like on Windows?
>
> Ronald (in http://bugs.python.org/issue23534):
> "On OSX the internal copy of libffi that's used is based on the one in
> PyObjC, which in turn is based on the version of libffi on
> opensource.apple.com (IIRC with some small patches that fix minor issues
> found by the PyObjC testsuite)."
>
> --
>  Ned Deily,
>  nad at acm.org

>From pypy experience, libffi installed on OS X tends to just work (we
never had any issues with those)

From thomas at python.org  Thu Mar 12 20:58:32 2015
From: thomas at python.org (Thomas Wouters)
Date: Thu, 12 Mar 2015 20:58:32 +0100
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <CAP1=2W7Cx5JpQV_pr61RQs1uBUSJF5f6KG0cD-QCwR2+9ijegg@mail.gmail.com>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAKJDb-N1ewaSsWyEycVHWwDASx8OjXG9LAWw3k3ntCZtw6oJSQ@mail.gmail.com>
 <CAP1=2W7Cx5JpQV_pr61RQs1uBUSJF5f6KG0cD-QCwR2+9ijegg@mail.gmail.com>
Message-ID: <CAPdQG2rgkRJCiEkQKCjM9R=z1+BBM_8nw5gq-TEiwWAVtxY+3Q@mail.gmail.com>

On Thu, Mar 12, 2015 at 6:26 PM, Brett Cannon <brett at python.org> wrote:
>
>
> For UNIX OSs we could probably rely on the system libffi then.
>

Yes, it's possible to use the system libffi -- that's what most linux
distros already do -- but only if you use dynamic linking. If you want to
statically link libffi (useful when embedding, making installations that
are a little more portable across systems, or when you're a company like
Google that wants to statically link all the things) it's a little more
confusing. (The reason Brett started this thread is that I mentioned these
issues in an internal Google meeting that he happened to be in ;-P)

_ctypes currently contains a bunch of duplicate definitions of the
ffi_type_* structs (in Modules/_ctypes/cfield.c.) It also contains a
separate implementation of ffi_closure_alloc and ffi_closure_free (in
Modules/_ctypes/malloc_closure.c) which is only used on OSX. I guess those
redefinitions are to cope with (much) older libffis, but that's just a
guess. The redefinitions aren't a problem when using dynamic linking, as
long as the new definitions aren't incompatible, but if libffi were to
change the ABI of the ffi_type structs, yuck. It can be a problem when
statically linking, depending on how you link -- and the only reason it
works with the bundled libffi is that ctypes *does not build libffi the
normal way*. Instead it runs libffi's configure and then selectively builds
a few source files from it, using distutils.

It would be a really, really good idea to clean up that mess if we're not
going to bundle libffi anymore :-P

Screaming-and-stabbing-at-setup.py'ly y'rs,
-- 
Thomas Wouters <thomas at python.org>

Hi! I'm an email virus! Think twice before sending your email to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150312/bff0d257/attachment.html>

From cournape at gmail.com  Fri Mar 13 09:00:41 2015
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 13 Mar 2015 08:00:41 +0000
Subject: [Python-Dev] Why does python use relative instead of absolute
 path when calling LoadLibrary*
In-Reply-To: <CACfEFw-U4kok9VjcpVGifABBY==7cEMS9f0Jvi_Q_hJLJcyLug@mail.gmail.com>
References: <CAGY4rcUvP5HqfUc3ZDhOr6U=iVVmbrW_v_9ofzOU=JaiinKc6w@mail.gmail.com>
 <CACfEFw-U4kok9VjcpVGifABBY==7cEMS9f0Jvi_Q_hJLJcyLug@mail.gmail.com>
Message-ID: <CAGY4rcU=7XMyg+ALdCmqNddTeA1vdr0oc3RycB1wG9z42C8mWQ@mail.gmail.com>

Thank you both for your answers.

I will go away with this modification, and see how it goes.

David

On Thu, Mar 12, 2015 at 2:41 AM, Wes Turner <wes.turner at gmail.com> wrote:

>
> On Mar 11, 2015 3:36 PM, "David Cournapeau" <cournape at gmail.com> wrote:
> >
> > Hi,
> >
> > While looking at the import code of python for C extensions, I was
> wondering why we pass a relative path instead of an absolute path to
> LoadLibraryEx (see bottom for some context).
> >
> > In python 2.7, the full path existence was even checked before calling
> into LoadLibraryEx (
> https://github.com/python/cpython/blob/2.7/Python/dynload_win.c#L189),
> but it looks like this check was removed in python 3.x branch.
> >
> > Is there any defined behaviour that depends on this path to be relative ?
>
> Just a guess: does it have to do with resolving symlinks (w/ POSIX
> filesystems)?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150313/d8eb13e9/attachment.html>

From bcannon at gmail.com  Fri Mar 13 14:39:59 2015
From: bcannon at gmail.com (Brett Cannon)
Date: Fri, 13 Mar 2015 13:39:59 +0000
Subject: [Python-Dev] PEP 488: elimination of PYO files
In-Reply-To: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
References: <CAP1=2W4=ygF-kvaqxhXaba-PVr03V_phMTQ0TJzfC9pFuu1VqA@mail.gmail.com>
Message-ID: <CAP1=2W48vM-be2Ts-k7SXBAPjFV2uvOqWJO2fH32B9thK7goPw@mail.gmail.com>

Here is the latest draft of the PEP. I have closed the issue of file name
formatting thanks to the informal poll results being very clear on the
preferred format and also closed the idea of embedding the optimization
level in the bytecode file metadata (that can be another PEP if someone
cares to write that one). The only remaining open issue is whether to
special-case non-optimized bytecode file names and completely leave out the
optimization level in that instance.

I think this PEP is close enough to be finished to ask whether Guido wants
to wield his BDFL stick for this PEP or if he wants to delegate to a BDFAP.

================================================
PEP: 488
Title: Elimination of PYO files
Version: $Revision$
Last-Modified: $Date$
Author: Brett Cannon <brett at python.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 20-Feb-2015
Post-History:
    2015-03-06
    2015-03-13

Abstract
========

This PEP proposes eliminating the concept of PYO files from Python.
To continue the support of the separation of bytecode files based on
their optimization level, this PEP proposes extending the PYC file
name to include the optimization level in bytecode repository
directory (i.e., the ``__pycache__`` directory).


Rationale
=========

As of today, bytecode files come in two flavours: PYC and PYO. A PYC
file is the bytecode file generated and read from when no
optimization level is specified at interpreter startup (i.e., ``-O``
is not specified). A PYO file represents the bytecode file that is
read/written when **any** optimization level is specified (i.e., when
``-O`` is specified, including ``-OO``). This means that while PYC
files clearly delineate the optimization level used when they were
generated -- namely no optimizations beyond the peepholer -- the same
is not true for PYO files. Put in terms of optimization levels and
the file extension:

  - 0: ``.pyc``
  - 1 (``-O``): ``.pyo``
  - 2 (``-OO``): ``.pyo``

The reuse of the ``.pyo`` file extension for both level 1 and 2
optimizations means that there is no clear way to tell what
optimization level was used to generate the bytecode file. In terms
of reading PYO files, this can lead to an interpreter using a mixture
of optimization levels with its code if the user was not careful to
make sure all PYO files were generated using the same optimization
level (typically done by blindly deleting all PYO files and then
using the `compileall` module to compile all-new PYO files [1]_).
This issue is only compounded when people optimize Python code beyond
what the interpreter natively supports, e.g., using the astoptimizer
project [2]_.

In terms of writing PYO files, the need to delete all PYO files
every time one either changes the optimization level they want to use
or are unsure of what optimization was used the last time PYO files
were generated leads to unnecessary file churn. The change proposed
by this PEP also allows for **all** optimization levels to be
pre-compiled for bytecode files ahead of time, something that is
currently impossible thanks to the reuse of the ``.pyo`` file
extension for multiple optimization levels.

As for distributing bytecode-only modules, having to distribute both
``.pyc`` and ``.pyo`` files is unnecessary for the common use-case
of code obfuscation and smaller file deployments.


Proposal
========

To eliminate the ambiguity that PYO files present, this PEP proposes
eliminating the concept of PYO files and their accompanying ``.pyo``
file extension. To allow for the optimization level to be unambiguous
as well as to avoid having to regenerate optimized bytecode files
needlessly in the `__pycache__` directory, the optimization level
used to generate a PYC file will be incorporated into the bytecode
file name. Currently bytecode file names are created by
``importlib.util.cache_from_source()``, approximately using the
following expression defined by PEP 3147 [3]_, [4]_, [5]_::

    '{name}.{cache_tag}.pyc'.format(name=module_name,
                                    cache_tag=sys.implementation.cache_tag)

This PEP proposes to change the expression to::

    '{name}.{cache_tag}.opt-{optimization}.pyc'.format(
            name=module_name,
            cache_tag=sys.implementation.cache_tag,
            optimization=str(sys.flags.optimize))

The "opt-" prefix was chosen so as to provide a visual separator
from the cache tag. The placement of the optimization level after
the cache tag was chosen to preserve lexicographic sort order of
bytecode file names based on module name and cache tag which will
not vary for a single interpreter. The "opt-" prefix was chosen over
"o" so as to be somewhat self-documenting. The "opt-" prefix was
chosen over "O" so as to not have any confusion with "0" while being
so close to the interpreter version number.

A period was chosen over a hyphen as a separator so as to distinguish
clearly that the optimization level is not part of the interpreter
version as specified by the cache tag. It also lends to the use of
the period in the file name to delineate semantically different
concepts.

For example, the bytecode file name of ``importlib.cpython-35.pyc``
would become ``importlib.cpython-35.opt-0.pyc``. If ``-OO`` had been
passed to the interpreter then instead of
``importlib.cpython-35.pyo`` the file name would be
``importlib.cpython-35.opt-2.pyc``.

It should be noted that this change in no way affects the performance
of import. Since the import system looks for a single bytecode file
based on the optimization level of the interpreter already and
generates a new bytecode file if it doesn't exist, the introduction
of potentially more bytecode files in the ``__pycache__`` directory
has no effect. The interpreter will continue to look for only a
single bytecode file based on the optimization level and thus no
increase in stat calls will occur.


Implementation
==============

importlib
---------

As ``importlib.util.cache_from_source()`` is the API that exposes
bytecode file paths as well as being directly used by importlib, it
requires the most critical change. As of Python 3.4, the function's
signature is::

  importlib.util.cache_from_source(path, debug_override=None)

This PEP proposes changing the signature in Python 3.5 to::

  importlib.util.cache_from_source(path, debug_override=None, *,
optimization=None)

The introduced ``optimization`` keyword-only parameter will control
what optimization level is specified in the file name. If the
argument is ``None`` then the current optimization level of the
interpreter will be assumed. Any argument given for ``optimization``
will be passed to ``str()`` and must have ``str.isalnum()`` be true,
else ``ValueError`` will be raised (this prevents invalid characters
being used in the file name). If the empty string is passed in for
``optimization`` then the addition of the optimization will be
suppressed, reverting to the file name format which predates this
PEP.

It is expected that beyond Python's own
0-2 optimization levels, third-party code will use a hash of
optimization names to specify the optimization level, e.g.
``hashlib.sha256(','.join(['dead code elimination', 'constant
folding'])).hexdigest()``.
While this might lead to long file names, it is assumed that most
users never look at the contents of the __pycache__ directory and so
this won't be an issue.

The ``debug_override`` parameter will be deprecated. As the parameter
expects a boolean, the integer value of the boolean will be used as
if it had been provided as the argument to ``optimization`` (a
``None`` argument will mean the same as for ``optimization``). A
deprecation warning will be raised when ``debug_override`` is given a
value other than ``None``, but there are no plans for the complete
removal of the parameter at this time (but removal will be no later
than Python 4).

The various module attributes for importlib.machinery which relate to
bytecode file suffixes will be updated [7]_. The
``DEBUG_BYTECODE_SUFFIXES`` and ``OPTIMIZED_BYTECODE_SUFFIXES`` will
both be documented as deprecated and set to the same value as
``BYTECODE_SUFFIXES`` (removal of ``DEBUG_BYTECODE_SUFFIXES`` and
``OPTIMIZED_BYTECODE_SUFFIXES`` is not currently planned, but will be
not later than Python 4).

All various finders and loaders will also be updated as necessary,
but updating the previous mentioned parts of importlib should be all
that is required.


Rest of the standard library
----------------------------

The various functions exposed by the ``py_compile`` and
``compileall`` functions will be updated as necessary to make sure
they follow the new bytecode file name semantics [6]_, [1]_. The CLI
for the ``compileall`` module will not be directly affected (the
``-b`` flag will be implicit as it will no longer generate ``.pyo``
files when ``-O`` is specified).


Compatibility Considerations
============================

Any code directly manipulating bytecode files from Python 3.2 on
will need to consider the impact of this change on their code (prior
to Python 3.2 -- including all of Python 2 -- there was no
__pycache__ which already necessitates bifurcating bytecode file
handling support). If code was setting the ``debug_override``
argument to ``importlib.util.cache_from_source()`` then care will be
needed if they want the path to a bytecode file with an optimization
level of 2. Otherwise only code **not** using
``importlib.util.cache_from_source()`` will need updating.

As for people who distribute bytecode-only modules (i.e., use a
bytecode file instead of a source file), they will have to choose
which optimization level they want their bytecode files to be since
distributing a ``.pyo`` file with a ``.pyc`` file will no longer be
of any use. Since people typically only distribute bytecode files for
code obfuscation purposes or smaller distribution size then only
having to distribute a single ``.pyc`` should actually be beneficial
to these use-cases. And since the magic number for bytecode files
changed in Python 3.5 to support PEP 465 there is no need to support
pre-existing ``.pyo`` files [8]_.


Rejected Ideas
==============

Completely dropping optimization levels from CPython
----------------------------------------------------

Some have suggested that instead of accommodating the various
optimization levels in CPython, we should instead drop them
entirely. The argument is that significant performance gains would
occur from runtime optimizations through something like a JIT and not
through pre-execution bytecode optimizations.

This idea is rejected for this PEP as that ignores the fact that
there are people who do find the pre-existing optimization levels for
CPython useful. It also assumes that no other Python interpreter
would find what this PEP proposes useful.


Alternative formatting of the optimization level in the file name
-----------------------------------------------------------------

Using the "opt-" prefix and placing the optimization level between
the cache tag and file extension is not critical. All options which
have been considered are:

* ``importlib.cpython-35.opt-0.pyc``
* ``importlib.cpython-35.opt0.pyc``
* ``importlib.cpython-35.o0.pyc``
* ``importlib.cpython-35.O0.pyc``
* ``importlib.cpython-35.0.pyc``
* ``importlib.cpython-35-O0.pyc``
* ``importlib.O0.cpython-35.pyc``
* ``importlib.o0.cpython-35.pyc``
* ``importlib.0.cpython-35.pyc``

These were initially rejected either because they would change the
sort order of bytecode files, possible ambiguity with the cache tag,
or were not self-documenting enough. An informal poll was taken and
people clearly preferred the formatting proposed by the PEP [9]_.
Since this topic is non-technical and of personal choice, the issue
is considered solved.


Embedding the optimization level in the bytecode metadata
---------------------------------------------------------

Some have suggested that rather than embedding the optimization level
of bytecode in the file name that it be included in the file's
metadata instead. This would mean every interpreter had a single copy
of bytecode at any time. Changing the optimization level would thus
require rewriting the bytecode, but there would also only be a single
file to care about.

This has been rejected due to the fact that Python is often installed
as a root-level application and thus modifying the bytecode file for
modules in the standard library are always possible. In this
situation integrators would need to guess at what a reasonable
optimization level was for users for any/all situations. By
allowing multiple optimization levels to co-exist simultaneously it
frees integrators from having to guess what users want and allows
users to utilize the optimization level they want.


Open Issues
===========

Not specifying the optimization level when it is at 0
-----------------------------------------------------

It has been suggested that for the common case of when the
optimizations are at level 0 that the entire part of the file name
relating to the optimization level be left out. This would allow for
file names of ``.pyc`` files to go unchanged, potentially leading to
less backwards-compatibility issues (although Python 3.5 introduces a
new magic number for bytecode so all bytecode files will have to be
regenerated regardless of the outcome of this PEP).

It would also allow a potentially redundant bit of information to be
left out of the file name if an implementation of Python did not
allow for optimizing bytecode. This would only occur, though, if the
interpreter didn't support ``-O`` **and** didn't implement the ast
module, else users could implement their own optimizations.

Arguments against allowing this special case is "explicit is better
than implicit" and "special cases aren't special enough to break the
rules".

At this people have weakly supporting this idea while no one has
explicitly come out against it.


References
==========

.. [1] The compileall module
   (https://docs.python.org/3/library/compileall.html#module-compileall)

.. [2] The astoptimizer project
   (https://pypi.python.org/pypi/astoptimizer)

.. [3] ``importlib.util.cache_from_source()``
   (
https://docs.python.org/3.5/library/importlib.html#importlib.util.cache_from_source
)

.. [4] Implementation of ``importlib.util.cache_from_source()`` from
CPython 3.4.3rc1
   (
https://hg.python.org/cpython/file/038297948389/Lib/importlib/_bootstrap.py#l437
)

.. [5] PEP 3147, PYC Repository Directories, Warsaw
   (http://www.python.org/dev/peps/pep-3147)

.. [6] The py_compile module
   (https://docs.python.org/3/library/compileall.html#module-compileall)

.. [7] The importlib.machinery module
   (
https://docs.python.org/3/library/importlib.html#module-importlib.machinery)

.. [8] ``importlib.util.MAGIC_NUMBER``
   (
https://docs.python.org/3/library/importlib.html#importlib.util.MAGIC_NUMBER
)

.. [9] Informal poll of file name format options on Google+
   (https://plus.google.com/u/0/+BrettCannon/posts/fZynLNwHWGm)


Copyright
=========

This document has been placed in the public domain.


..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End:
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150313/cb341236/attachment.html>

From brett at python.org  Fri Mar 13 15:50:52 2015
From: brett at python.org (Brett Cannon)
Date: Fri, 13 Mar 2015 14:50:52 +0000
Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving
 Python ZIP Application Support
In-Reply-To: <CAP1=2W7ma-O8Z9Qj37PRuF73aiv-bOUbpPbXCpH5psJ5B_ouRg@mail.gmail.com>
References: <CACac1F_E_a9hB818EMG4nbDTqqoOEc9B0_ZPZUzop6JKjk-Y3A@mail.gmail.com>
 <CAP1=2W4APXunO=wSYWpEyuYmYmkYjuSgEumvT_9=s6QdKobUgQ@mail.gmail.com>
 <CAG8k2+4JbRarMZmVg0dT18u1CnfWnS9f44=JvF++TVmMUJEgnw@mail.gmail.com>
 <CAP1=2W6GkDNvH9y2+gbh-WBpWvfM0tTX1BE92mniu74En1r7fA@mail.gmail.com>
 <CACac1F_t3uj8r5gTV1Y0ynGKWUF3oH3OHbX6Tf-H3inkxQOwvA@mail.gmail.com>
 <CAG8k2+4Mn9f1oujLr3mVY2DnNjfn52L5+Mc2vyJNJTvOtk1O5w@mail.gmail.com>
 <54EB7DDE.9030405@stoneleaf.us>
 <CAPdQG2pvRgeTiA6T-Lb6E7Th0eM1DZAzwrS9f85mPPWkPqAH4w@mail.gmail.com>
 <CAP7+vJ+=AzrnGdQfjCkq7Tc3tqHz2qq=jOYjQkyiuuDht5Hn-Q@mail.gmail.com>
 <CACac1F9W9vQ2TckokfGqEscUZMk-CLbt+ODEgxxkGz_oOnAg3g@mail.gmail.com>
 <CADiSq7fdPPQ=wRDhwZEv8Rh7iUm_Vt+k9MBgFhvo54svV0L2xA@mail.gmail.com>
 <CAP7+vJJ45esj8bbfHpBsitUhoxHHLN91yZyX72nLTHBkFe+6Cw@mail.gmail.com>
 <CACac1F_LsMGqBzF=ONe2D=AO44xZBZYkPXz8e=FHZhkUkj51QQ@mail.gmail.com>
 <CAP7+vJKXX=Ybz0rnfkQgH1+kgY-uVmvjJcOOhVR17Nhafp7N=w@mail.gmail.com>
 <CAP7+vJL_1danWZefJivqpDbNYM3DzNAeN_+q5PeqgC-ZMae-xw@mail.gmail.com>
 <CACac1F_tYTecaK0NursqPp4y4HwFDuhi4RQQ4Ggp9JKhC5ZUnA@mail.gmail.com>
 <CAP7+vJK-V=ipy4ALW_MYSENSVH=Mvm4jeTxey_B+iNuFppieCg@mail.gmail.com>
 <CACac1F96rbrc4gZbV9fkxKiAEMv7uVxPF0pPD60WwZHWZfyBEw@mail.gmail.com>
 <CACac1F9yQdOA5zKN31hzkX+Efsn5W-k=xngB0vi8WmnnkQioeg@mail.gmail.com>
 <CAP1=2W7ma-O8Z9Qj37PRuF73aiv-bOUbpPbXCpH5psJ5B_ouRg@mail.gmail.com>
Message-ID: <CAP1=2W4ztzmYY7kCqhyE0jgr3BSuw_BGcMcsrcryox8+Hq-t8A@mail.gmail.com>

On Sun, Mar 8, 2015 at 10:35 AM Brett Cannon <brett at python.org> wrote:

>
>  On Sun, Mar 8, 2015, 08:40 Paul Moore <p.f.moore at gmail.com> wrote:
>
> On 26 February 2015 at 21:48, Paul Moore <p.f.moore at gmail.com> wrote:
> > On 26 February 2015 at 21:34, Guido van Rossum <guido at python.org> wrote:
> >> Accepted!
> >>
> >> Thanks for your patience, Paul, and thanks everyone for their feedback.
> >>
> >> I know there are still a few small edits to the PEP, but those don't
> affect
> >> my acceptance. Congrats!
> >
> > Excellent, thanks to everyone for the helpful comments throughout.
>
> I'm still looking for someone who has the time to commit the final
> patch on http://bugs.python.org/issue23491. Python 3.5a2 is due out
> today (at least according to the PEP) so I've probably missed that,
> but it would be nice if it could be in for alpha 3 at least.
>
>  If no one gets to it I can submit on Friday.
>

Submitted in https://hg.python.org/cpython/rev/d1e9f337fea1 .

Thanks for the hard work, Paul!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150313/a5a99f98/attachment.html>

From status at bugs.python.org  Fri Mar 13 18:08:21 2015
From: status at bugs.python.org (Python tracker)
Date: Fri, 13 Mar 2015 18:08:21 +0100 (CET)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20150313170821.CF063560D1@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2015-03-06 - 2015-03-13)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    4817 (+13)
  closed 30604 (+49)
  total  35421 (+62)

Open issues with patches: 2266 


Issues opened (49)
==================

#11726: clarify that linecache only works on files that can be decoded
http://bugs.python.org/issue11726  reopened by r.david.murray

#23566: RFE: faulthandler.register() should support file descriptors
http://bugs.python.org/issue23566  reopened by haypo

#23599: single and double quotes stripped upon paste with MacPorts lib
http://bugs.python.org/issue23599  opened by Jeff Doak

#23600: tizinfo.fromutc changed for tzinfo wih StdOffset=0, DstOffset=
http://bugs.python.org/issue23600  opened by peterjclaw

#23601: use small object allocator for dict key storage
http://bugs.python.org/issue23601  opened by jtaylor

#23602: Implement __format__ for Fraction
http://bugs.python.org/issue23602  opened by tuomas.suutari

#23603: Embedding Python3.4 - PyUnicode_Check fails (MinGW-W64)
http://bugs.python.org/issue23603  opened by Ashish Sadanandan

#23605: Use the new os.scandir() function in os.walk()
http://bugs.python.org/issue23605  opened by haypo

#23606: ctypes.util.find_library("c") no longer makes sense
http://bugs.python.org/issue23606  opened by steve.dower

#23607: Inconsistency in datetime.utcfromtimestamp(Decimal)
http://bugs.python.org/issue23607  opened by serhiy.storchaka

#23609: Export PyModuleObject in moduleobject.h
http://bugs.python.org/issue23609  opened by h.venev

#23611: Pickle nested names with protocols < 4
http://bugs.python.org/issue23611  opened by serhiy.storchaka

#23612: 3.5.0a2 Windows installer does not remove 3.5.0a1
http://bugs.python.org/issue23612  opened by steve.dower

#23614: Opaque error message on UTF-8 decoding to surrogates
http://bugs.python.org/issue23614  opened by Rosuav

#23616: Idle: conflict between loop execution and undo shortcut.
http://bugs.python.org/issue23616  opened by Davide Okami

#23618: PEP 475: handle EINTR in the socket module
http://bugs.python.org/issue23618  opened by haypo

#23619: Python 3.5.0a2 installer fails on Windows
http://bugs.python.org/issue23619  opened by pmoore

#23621: Uninstalling Python 3.5 removes a "py.exe" that was installed 
http://bugs.python.org/issue23621  opened by pmoore

#23622: Deprecate unrecognized backslash+letter escapes
http://bugs.python.org/issue23622  opened by serhiy.storchaka

#23623: Python 3.5 docs need to clarify how to set PATH, etc
http://bugs.python.org/issue23623  opened by pmoore

#23624: str.center inconsistent with format "^"
http://bugs.python.org/issue23624  opened by Vedran.??a??i??

#23626: Windows per-user install of 3.5a2 doesn't associate .py files 
http://bugs.python.org/issue23626  opened by pmoore

#23628: See if os.scandir() could help speed up importlib
http://bugs.python.org/issue23628  opened by brett.cannon

#23630: support multiple hosts in create_server/start_server
http://bugs.python.org/issue23630  opened by sebastien.bourdeauducq

#23631: 3.5 (a2) traceback regression snarls Idle
http://bugs.python.org/issue23631  opened by terry.reedy

#23632: memoryview doesn't allow tuple-indexing
http://bugs.python.org/issue23632  opened by pitrou

#23633: Improve py launcher help, index, and doc
http://bugs.python.org/issue23633  opened by terry.reedy

#23634: os.fdopen reopening a read-only fd on windows
http://bugs.python.org/issue23634  opened by mattip

#23636: Add scgi to urllib.parse.uses_netloc
http://bugs.python.org/issue23636  opened by anthonyryan1

#23637: Warnings error with non-ascii chars.
http://bugs.python.org/issue23637  opened by Luk????.N??mec

#23639: Not documented special names
http://bugs.python.org/issue23639  opened by serhiy.storchaka

#23640: Enum.from_bytes() is broken
http://bugs.python.org/issue23640  opened by serhiy.storchaka

#23641: Got rid of bad dunder names
http://bugs.python.org/issue23641  opened by serhiy.storchaka

#23642: Interaction of ModuleSpec and C Extension Modules
http://bugs.python.org/issue23642  opened by dabeaz

#23644: g++ module compile fails with ???_Atomic??? does not name a ty
http://bugs.python.org/issue23644  opened by Joshua.J.Cogliati

#23645: Incorrect doc for __getslice__
http://bugs.python.org/issue23645  opened by gdementen

#23646: PEP 475: handle EINTR in the time module, retry sleep()
http://bugs.python.org/issue23646  opened by haypo

#23647: imaplib.py MAXLINE value is too low for gmail
http://bugs.python.org/issue23647  opened by arnt

#23648: PEP 475 meta issue
http://bugs.python.org/issue23648  opened by haypo

#23649: tarfile not re-entrant for multi-threading
http://bugs.python.org/issue23649  opened by sgnn7

#23652: ifdef uses of EPOLLxxx macros so we can compile on systems tha
http://bugs.python.org/issue23652  opened by WanderingLogic

#23653: Make inspect._empty test to False
http://bugs.python.org/issue23653  opened by Lucretiel

#23654: infinite loop in faulthandler._stack_overflow
http://bugs.python.org/issue23654  opened by WanderingLogic

#23655: Memory corruption using pickle over pipe to subprocess
http://bugs.python.org/issue23655  opened by nagle

#23656: shutil.copyfile (or built-in open) releases sqlite3's transact
http://bugs.python.org/issue23656  opened by h-miyajima

#23657: Don't do isinstance checks in zipapp
http://bugs.python.org/issue23657  opened by brett.cannon

#23658: multiprocessing: string arg to SystemExit
http://bugs.python.org/issue23658  opened by Dan Nawrocki

#23659: csv.register_dialect  doc string
http://bugs.python.org/issue23659  opened by Vladimir Ulupov

#23660: Turtle left/right inverted when using different coordinates or
http://bugs.python.org/issue23660  opened by aroberge



Most recent 15 issues with no replies (15)
==========================================

#23660: Turtle left/right inverted when using different coordinates or
http://bugs.python.org/issue23660

#23658: multiprocessing: string arg to SystemExit
http://bugs.python.org/issue23658

#23656: shutil.copyfile (or built-in open) releases sqlite3's transact
http://bugs.python.org/issue23656

#23653: Make inspect._empty test to False
http://bugs.python.org/issue23653

#23652: ifdef uses of EPOLLxxx macros so we can compile on systems tha
http://bugs.python.org/issue23652

#23648: PEP 475 meta issue
http://bugs.python.org/issue23648

#23628: See if os.scandir() could help speed up importlib
http://bugs.python.org/issue23628

#23626: Windows per-user install of 3.5a2 doesn't associate .py files 
http://bugs.python.org/issue23626

#23622: Deprecate unrecognized backslash+letter escapes
http://bugs.python.org/issue23622

#23621: Uninstalling Python 3.5 removes a "py.exe" that was installed 
http://bugs.python.org/issue23621

#23609: Export PyModuleObject in moduleobject.h
http://bugs.python.org/issue23609

#23585: patchcheck doesn't depend on all
http://bugs.python.org/issue23585

#23584: test_doctest lineendings fails in verbose mode
http://bugs.python.org/issue23584

#23578: struct.pack error messages do not indicate which argument was 
http://bugs.python.org/issue23578

#23577: Add tests for wsgiref.validate
http://bugs.python.org/issue23577



Most recent 15 issues waiting for review (15)
=============================================

#23657: Don't do isinstance checks in zipapp
http://bugs.python.org/issue23657

#23654: infinite loop in faulthandler._stack_overflow
http://bugs.python.org/issue23654

#23652: ifdef uses of EPOLLxxx macros so we can compile on systems tha
http://bugs.python.org/issue23652

#23649: tarfile not re-entrant for multi-threading
http://bugs.python.org/issue23649

#23647: imaplib.py MAXLINE value is too low for gmail
http://bugs.python.org/issue23647

#23646: PEP 475: handle EINTR in the time module, retry sleep()
http://bugs.python.org/issue23646

#23645: Incorrect doc for __getslice__
http://bugs.python.org/issue23645

#23641: Got rid of bad dunder names
http://bugs.python.org/issue23641

#23640: Enum.from_bytes() is broken
http://bugs.python.org/issue23640

#23632: memoryview doesn't allow tuple-indexing
http://bugs.python.org/issue23632

#23630: support multiple hosts in create_server/start_server
http://bugs.python.org/issue23630

#23622: Deprecate unrecognized backslash+letter escapes
http://bugs.python.org/issue23622

#23618: PEP 475: handle EINTR in the socket module
http://bugs.python.org/issue23618

#23611: Pickle nested names with protocols < 4
http://bugs.python.org/issue23611

#23607: Inconsistency in datetime.utcfromtimestamp(Decimal)
http://bugs.python.org/issue23607



Top 10 most discussed issues (10)
=================================

#23605: Use the new os.scandir() function in os.walk()
http://bugs.python.org/issue23605  24 msgs

#23619: Python 3.5.0a2 installer fails on Windows
http://bugs.python.org/issue23619  23 msgs

#23496: Steps for Android Native Build of Python 3.4.2
http://bugs.python.org/issue23496  14 msgs

#23246: distutils fails to locate vcvarsall with Visual C++ Compiler f
http://bugs.python.org/issue23246  11 msgs

#23566: RFE: faulthandler.register() should support file descriptors
http://bugs.python.org/issue23566  11 msgs

#23602: Implement __format__ for Fraction
http://bugs.python.org/issue23602  10 msgs

#23632: memoryview doesn't allow tuple-indexing
http://bugs.python.org/issue23632  10 msgs

#23524: Use _set_thread_local_invalid_parameter_handler in posixmodule
http://bugs.python.org/issue23524   9 msgs

#23633: Improve py launcher help, index, and doc
http://bugs.python.org/issue23633   9 msgs

#12916: Add inspect.splitdoc
http://bugs.python.org/issue12916   8 msgs



Issues closed (46)
==================

#5527: multiprocessing won't work with Tkinter (under Linux)
http://bugs.python.org/issue5527  closed by davin

#7503: multiprocessing AuthenticationError "digest sent was rejected"
http://bugs.python.org/issue7503  closed by davin

#9191: winreg.c:Reg2Py() may leak memory (in unusual circumstances)
http://bugs.python.org/issue9191  closed by Claudiu.Popa

#10805: traceback.print_exception throws AttributeError when exception
http://bugs.python.org/issue10805  closed by Claudiu.Popa

#11705: sys.excepthook doesn't work in imported modules
http://bugs.python.org/issue11705  closed by Claudiu.Popa

#17403: Robotparser fails to parse some robots.txt
http://bugs.python.org/issue17403  closed by berker.peksag

#19909: Best practice docs for new SSL features
http://bugs.python.org/issue19909  closed by christian.heimes

#20617: test_ssl does not use mock
http://bugs.python.org/issue20617  closed by berker.peksag

#20659: Want to make a class method a property by combining decorators
http://bugs.python.org/issue20659  closed by ncoghlan

#20876: python -m test test_pathlib fails
http://bugs.python.org/issue20876  closed by pitrou

#21610: load_module not closing opened files
http://bugs.python.org/issue21610  closed by python-dev

#22522: sys.excepthook doesn't receive the traceback when called from 
http://bugs.python.org/issue22522  closed by Claudiu.Popa

#22524: PEP 471 implementation: os.scandir() directory scanning functi
http://bugs.python.org/issue22524  closed by haypo

#22853: Multiprocessing.Queue._feed deadlocks on import
http://bugs.python.org/issue22853  closed by serhiy.storchaka

#22928: HTTP header injection in urrlib2/urllib/httplib/http.client
http://bugs.python.org/issue22928  closed by serhiy.storchaka

#23003: traceback.{print_exc,print_exception,format_exc,format_excepti
http://bugs.python.org/issue23003  closed by berker.peksag

#23051: multiprocessing.pool methods imap()[_unordered()] deadlock
http://bugs.python.org/issue23051  closed by serhiy.storchaka

#23081: Document PySequence_List(o) as equivalent to list(o)
http://bugs.python.org/issue23081  closed by berker.peksag

#23138: cookiejar parses cookie value as int with empty name-value pai
http://bugs.python.org/issue23138  closed by serhiy.storchaka

#23192: Generator return value ignored in lambda function
http://bugs.python.org/issue23192  closed by serhiy.storchaka

#23279: test_site/test_startup_imports fails when mpl_toolkit or logil
http://bugs.python.org/issue23279  closed by brett.cannon

#23432: Duplicate content in SystemExit documentation
http://bugs.python.org/issue23432  closed by berker.peksag

#23467: Improve byte formatting compatibility between Py2 and Py3
http://bugs.python.org/issue23467  closed by python-dev

#23486: Enum member lookup is 20x slower than normal class attribute l
http://bugs.python.org/issue23486  closed by python-dev

#23491: PEP 441 - Improving Python Zip Application Support
http://bugs.python.org/issue23491  closed by brett.cannon

#23526: Silence resource warnings in test_httplib
http://bugs.python.org/issue23526  closed by vadmium

#23571: Raise SystemError if a function returns a result with an excep
http://bugs.python.org/issue23571  closed by haypo

#23579: Amazon.com links
http://bugs.python.org/issue23579  closed by python-dev

#23581: unittest.mock.MagicMock doesn't support matmul (@) operator
http://bugs.python.org/issue23581  closed by berker.peksag

#23595: Split the math module into _math (C) + math (py)
http://bugs.python.org/issue23595  closed by haypo

#23598: No backspace on curses on iTerm (mac)
http://bugs.python.org/issue23598  closed by ned.deily

#23604: Python 3.4 and 2.7 installation no Script folder and no pip in
http://bugs.python.org/issue23604  closed by Daiyue Weng

#23608: ssl.VERIFY_X509_TRUSTED_FIRST: Incorrect value for versionadde
http://bugs.python.org/issue23608  closed by python-dev

#23610: -m arg behavior
http://bugs.python.org/issue23610  closed by zach.ware

#23613: searchindex.js is annoying
http://bugs.python.org/issue23613  closed by georg.brandl

#23615: Reloading tokenize breaks tokenize.open()
http://bugs.python.org/issue23615  closed by serhiy.storchaka

#23617: Incorrect plural "includes" on Python Standard Library front p
http://bugs.python.org/issue23617  closed by ned.deily

#23620: cross-type comparison is different in python minor versions
http://bugs.python.org/issue23620  closed by r.david.murray

#23625: load_module() docs: zipped eggs are not loaded.
http://bugs.python.org/issue23625  closed by brett.cannon

#23627: Incorrect string formatting of float values in Python 2.7
http://bugs.python.org/issue23627  closed by eric.smith

#23629: Default __sizeof__ is wrong for var-sized objects
http://bugs.python.org/issue23629  closed by pitrou

#23635: Python won't install correctly.
http://bugs.python.org/issue23635  closed by Player72

#23638: shutil.copytree makedirs exception
http://bugs.python.org/issue23638  closed by r.david.murray

#23643: Segmentation Fault with Large Simple Function
http://bugs.python.org/issue23643  closed by Michael Klein

#23650: xml.etree.ElementTreee.write can't parse its own output
http://bugs.python.org/issue23650  closed by ned.deily

#23651: Typo in argparse allow_abrev
http://bugs.python.org/issue23651  closed by berker.peksag

From stefano.borini at ferrara.linux.it  Fri Mar 13 19:25:36 2015
From: stefano.borini at ferrara.linux.it (Stefano Borini)
Date: Fri, 13 Mar 2015 19:25:36 +0100
Subject: [Python-Dev] Why does python use relative instead of absolute
 path when calling LoadLibrary*
In-Reply-To: <CAGY4rcUvP5HqfUc3ZDhOr6U=iVVmbrW_v_9ofzOU=JaiinKc6w@mail.gmail.com>
References: <CAGY4rcUvP5HqfUc3ZDhOr6U=iVVmbrW_v_9ofzOU=JaiinKc6w@mail.gmail.com>
Message-ID: <55032BA0.2030706@ferrara.linux.it>



On 3/11/15 9:36 PM, David Cournapeau wrote:
> Hi,
>
> While looking at the import code of python for C extensions, I was
> wondering why we pass a relative path instead of an absolute path to
> LoadLibraryEx (see bottom for some context).
>
> In python 2.7, the full path existence was even checked before calling
> into LoadLibraryEx
> (https://github.com/python/cpython/blob/2.7/Python/dynload_win.c#L189),
> but it looks like this check was removed in python 3.x branch.
>
> Is there any defined behaviour that depends on this path to be relative ?

My two cents. According to the MSDN documentation, passing a relative 
path is actually undefined

https://msdn.microsoft.com/en-us/library/windows/desktop/ms684179%28v=vs.85%29.aspx

----------------------------
LOAD_WITH_ALTERED_SEARCH_PATH
0x00000008
If this value is used and lpFileName specifies an absolute path, the 
system uses the alternate file search strategy discussed in the Remarks 
section to find associated executable modules that the specified module 
causes to be loaded. If this value is used and lpFileName specifies a 
relative path, the behavior is undefined.
---------------------------

meaning that wpathname _must_ be absolute.

https://github.com/python/cpython/blob/master/Python/dynload_win.c#L222

From francismb at email.de  Sat Mar 14 12:04:57 2015
From: francismb at email.de (francis)
Date: Sat, 14 Mar 2015 12:04:57 +0100
Subject: [Python-Dev] Request for comments: [issue22941] IPv4Interface
 arithmetic changes subnet mask
In-Reply-To: <5501E109.7090408@trueblade.com>
References: <927E0DF62C47C940A006AFF297D7F512120C11@DEFTHW99EK0MSX.ww902.siemens.net>
 <20150312150023.04051b02@fsol> <5501A6CA.2080301@trueblade.com>
 <5501D601.9090105@scottdial.com> <20150312191717.30acbd7b@fsol>
 <5501E109.7090408@trueblade.com>
Message-ID: <550415D9.8000100@email.de>

Hi,
> I, too, was thinking /24. I think that overflowing the host portion
> should raise OverflowError.
> 
Just curiosity, why not a modulo calculation on the subnet instead
of raising the error?

Thanks in advance!
francis


From francismb at email.de  Sat Mar 14 12:34:48 2015
From: francismb at email.de (francis)
Date: Sat, 14 Mar 2015 12:34:48 +0100
Subject: [Python-Dev] (ctypes example) libffi embedded in CPython
In-Reply-To: <CAK5idxQD+2uJ8vjJQcx7WKitZyc6biP3AL-LUVZELsMVow_EQw@mail.gmail.com>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAP1=2W7=DRv-rfy5jv=pwamXhC4xU8qnPm3fDAjb2WTFKOYC3A@mail.gmail.com>
 <20150311185028.6f05973c@fsol>
 <CAK5idxQYyWbWq_frrvWV0ZYdfeWgCm8pW0dgotf0uE-op4WNBg@mail.gmail.com>
 <20150311190557.25eebc16@fsol> <20150311191750.7cf23a40@fsol>
 <CAK5idxQD+2uJ8vjJQcx7WKitZyc6biP3AL-LUVZELsMVow_EQw@mail.gmail.com>
Message-ID: <55041CD8.4010608@email.de>

Hi
> 
> It might be a matter of taste, but I don't find declaring C functions
> any more awkward than using strange interface that ctypes comes with.
> the equivalent in cffi would be ffi.cast("double (*)()", x)

- Could you please elaborate on "using strange interface"?
- Is there an easy way to convert the current code using ctypes to
cffi?

Thanks in advance!
francis


From doko at ubuntu.com  Sat Mar 14 15:42:45 2015
From: doko at ubuntu.com (Matthias Klose)
Date: Sat, 14 Mar 2015 15:42:45 +0100
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <CAP1=2W7=DRv-rfy5jv=pwamXhC4xU8qnPm3fDAjb2WTFKOYC3A@mail.gmail.com>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAP1=2W7=DRv-rfy5jv=pwamXhC4xU8qnPm3fDAjb2WTFKOYC3A@mail.gmail.com>
Message-ID: <550448E5.8030405@ubuntu.com>

On 03/11/2015 06:27 PM, Brett Cannon wrote:
> On Mon, Dec 22, 2014 at 4:49 PM Jim J. Jewett <jimjjewett at gmail.com> wrote:
> 
>>
>>
>> On Thu, Dec 18, 2014, at 14:13, Maciej Fijalkowski wrote:
>>> ... http://bugs.python.org/issue23085 ...
>>> is there any reason any more for libffi being included in CPython?
>>
>>
>> Paul Moore wrote:
>>> Probably the easiest way of moving this forward would be for someone
>>> to identify the CPython-specific patches in the current version ...
>>
>> Christian Heimes wrote:
>>> That's easy. All patches are tracked in the diff file
>>> https://hg.python.org/cpython/file/3de678cd184d/Modules/_
>> ctypes/libffi.diff
>>
>> That (200+ lines) doesn't seem to have all the C changes, such as the
>> win64 sizeof changes from issue 11835.
>>
>> Besides http://bugs.python.org/issue23085, there is at least
>>     http://bugs.python.org/issue22733
>>     http://bugs.python.org/issue20160
>>     http://bugs.python.org/issue11835
>>
>> which sort of drives home the point that making sure we have a
>> good merge isn't trivial, and this isn't an area where we should
>> just assume that tests will catch everything.  I don't think it
>> is just a quicky waiting on permission.
>>
>> I've no doubt that upstream libffi is better in many ways, but
>> those are ways people have already learned to live with.
>>
>> That said, I haven't seen any objections in principle, except
>> perhaps from Steve Dower in the issues.  (I *think* he was
>> just saying "not worth the time to me", but it was ambiguous.)
>>
>> I do believe that Christian or Maciej *could* sort things out well
>> enough; I have no insight into whether they have (or someone else
>> has) the time to actually do so.
>>
> 
> Did anyone ever step forward to do this? I'm a bit worried about the
> long-term viability of ctypes if we don't have a maintainer or at least
> someone making sure we are staying up-to-date with upstream libffi.

my plan was to merge the next libffi release, something along the lines which is
released with GCC 5. But this release isn't yet done.


From eric at trueblade.com  Sat Mar 14 21:52:27 2015
From: eric at trueblade.com (Eric V. Smith)
Date: Sat, 14 Mar 2015 16:52:27 -0400
Subject: [Python-Dev] Request for comments: [issue22941] IPv4Interface
 arithmetic changes subnet mask
In-Reply-To: <550415D9.8000100@email.de>
References: <927E0DF62C47C940A006AFF297D7F512120C11@DEFTHW99EK0MSX.ww902.siemens.net>
 <20150312150023.04051b02@fsol> <5501A6CA.2080301@trueblade.com>
 <5501D601.9090105@scottdial.com> <20150312191717.30acbd7b@fsol>
 <5501E109.7090408@trueblade.com> <550415D9.8000100@email.de>
Message-ID: <55049F8B.9000504@trueblade.com>

On 3/14/2015 7:04 AM, francis wrote:
> Hi,
>> I, too, was thinking /24. I think that overflowing the host portion
>> should raise OverflowError.
>>
> Just curiosity, why not a modulo calculation on the subnet instead
> of raising the error?

Personally, I can't imaging wanting that behavior. I can't say I've ever
needed additional at all with an IP address, but if I did, it would only
be to stay within the host portion. To wrap around seems odd. If you had
a /24 address and added 1000, should that really be the same as adding
232 (= 1000 % 256)? It seems more likely it's an error.

I'm +0 on adding addition with an OverflowError for the host portion,
and -1 on addition with modulo arithmetic.

-- 
Eric.

From eric at trueblade.com  Sat Mar 14 22:15:22 2015
From: eric at trueblade.com (Eric V. Smith)
Date: Sat, 14 Mar 2015 17:15:22 -0400
Subject: [Python-Dev] Request for comments: [issue22941] IPv4Interface
 arithmetic changes subnet mask
In-Reply-To: <55049F8B.9000504@trueblade.com>
References: <927E0DF62C47C940A006AFF297D7F512120C11@DEFTHW99EK0MSX.ww902.siemens.net>
 <20150312150023.04051b02@fsol> <5501A6CA.2080301@trueblade.com>
 <5501D601.9090105@scottdial.com> <20150312191717.30acbd7b@fsol>
 <5501E109.7090408@trueblade.com> <550415D9.8000100@email.de>
 <55049F8B.9000504@trueblade.com>
Message-ID: <5504A4EA.7080509@trueblade.com>

On 3/14/2015 4:52 PM, Eric V. Smith wrote:
> On 3/14/2015 7:04 AM, francis wrote:
>> Hi,
>>> I, too, was thinking /24. I think that overflowing the host portion
>>> should raise OverflowError.
>>>
>> Just curiosity, why not a modulo calculation on the subnet instead
>> of raising the error?
> 
> Personally, I can't imaging wanting that behavior. I can't say I've ever
> needed additional at all with an IP address, but if I did, it would only

That should be "I can't say I've ever needed addition at all...".

> be to stay within the host portion. To wrap around seems odd. If you had
> a /24 address and added 1000, should that really be the same as adding
> 232 (= 1000 % 256)? It seems more likely it's an error.
> 
> I'm +0 on adding addition with an OverflowError for the host portion,
> and -1 on addition with modulo arithmetic.

-- 
Eric.

From larry at hastings.org  Sun Mar 15 03:48:05 2015
From: larry at hastings.org (Larry Hastings)
Date: Sat, 14 Mar 2015 19:48:05 -0700
Subject: [Python-Dev] Minor update to Python 3.5 release schedule
Message-ID: <5504F2E5.3030707@hastings.org>



I always intended all my releases to be on Sundays--that all the release 
engineering work is done on weekends, which is generally easier for 
everybody.  But I goofed up the 3.5 release schedule and had proposed 
3.5.0a3 to be released Saturday March 28th.  With the assent of the team 
I bumped it forward a day to Sunday March 29th. I apologize for my goof.

Cheers,


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150314/b11e9605/attachment.html>

From stephen at xemacs.org  Sun Mar 15 06:22:06 2015
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sun, 15 Mar 2015 14:22:06 +0900
Subject: [Python-Dev] Request for comments: [issue22941] IPv4Interface
 arithmetic changes subnet mask
In-Reply-To: <55049F8B.9000504@trueblade.com>
References: <927E0DF62C47C940A006AFF297D7F512120C11@DEFTHW99EK0MSX.ww902.siemens.net>
 <20150312150023.04051b02@fsol> <5501A6CA.2080301@trueblade.com>
 <5501D601.9090105@scottdial.com> <20150312191717.30acbd7b@fsol>
 <5501E109.7090408@trueblade.com> <550415D9.8000100@email.de>
 <55049F8B.9000504@trueblade.com>
Message-ID: <87oanuesxt.fsf@uwakimon.sk.tsukuba.ac.jp>

Eric V. Smith writes:

 > Personally, I can't imaging wanting that behavior. I can't say I've ever
 > needed additional at all with an IP address, but if I did, it would only
 > be to stay within the host portion. To wrap around seems odd.

It's worse than odd.  I've occasionally had use for iterating
something like a "ping" over a network, and if I make an off-by-one
error "boom" I hit the network address or (probably worse) the
broadcast address.

The ipaddress module appears to have an __iter__ method on
IPv4Network that dtrts.  I don't see a real need for an addition
operator given that.

From ethan at stoneleaf.us  Sun Mar 15 06:52:17 2015
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 14 Mar 2015 22:52:17 -0700
Subject: [Python-Dev] backwards and forwards compatibility,
 the exact contents of pickle files, and IntEnum
Message-ID: <55051E11.8030603@stoneleaf.us>

I'm not sure exactly how to phrase this inquiry, so please bear with me.

What exactly does backwards compatibility mean as far as pickle goes?  We have the various protocols, we have the
contents of pickle files created at those protocols, and we have different versions of Python.

Should a pickle file created in Python 3.4 with protocol 3 and the contents of socket.AF_INET be unpicklable with a
Python 3.3 system?  Because currently it cannot be as socket.AF_INET became an IntEnum in 3.4.

I'm thinking the answer is yes, it should be.

So how do we fix it?  There are a couple different options:

  - modify IntEnum pickle methods to return the name only

  - modify IntEnum pickle methods to pickle just like the int they represent

The first option has the advantage that in 3.4 and above, you'll get back the IntEnum version.

The second option has the advantage that the actual pickle contents are the same [1] as in previous versions.

So, the final question:  Do the contents of a pickle file at a certain protocol have to be the some between versions, or
is it enough if unpickling them returns the correct object?

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150314/f8359cc5/attachment.sig>

From ethan at stoneleaf.us  Sun Mar 15 07:05:20 2015
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 14 Mar 2015 23:05:20 -0700
Subject: [Python-Dev] backwards and forwards compatibility,
 the exact contents of pickle files, and IntEnum
In-Reply-To: <55051E11.8030603@stoneleaf.us>
References: <55051E11.8030603@stoneleaf.us>
Message-ID: <55052120.2030705@stoneleaf.us>

[1] The second option hasn't been coded yet, but the first one has.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150314/a89eb89d/attachment.sig>

From v+python at g.nevcal.com  Sun Mar 15 07:46:04 2015
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Sat, 14 Mar 2015 23:46:04 -0700
Subject: [Python-Dev] backwards and forwards compatibility,
 the exact contents of pickle files, and IntEnum
In-Reply-To: <55051E11.8030603@stoneleaf.us>
References: <55051E11.8030603@stoneleaf.us>
Message-ID: <55052AAC.6090907@g.nevcal.com>

On 3/14/2015 10:52 PM, Ethan Furman wrote:
> I'm not sure exactly how to phrase this inquiry, so please bear with me.
>
> What exactly does backwards compatibility mean as far as pickle goes?  We have the various protocols, we have the
> contents of pickle files created at those protocols, and we have different versions of Python.
>
> Should a pickle file created in Python 3.4 with protocol 3 and the contents of socket.AF_INET be unpicklable with a
> Python 3.3 system?  Because currently it cannot be as socket.AF_INET became an IntEnum in 3.4.
>
> I'm thinking the answer is yes, it should be.
>
> So how do we fix it?  There are a couple different options:
>
>    - modify IntEnum pickle methods to return the name only
>
>    - modify IntEnum pickle methods to pickle just like the int they represent
>
> The first option has the advantage that in 3.4 and above, you'll get back the IntEnum version.
>
> The second option has the advantage that the actual pickle contents are the same [1] as in previous versions.
>
> So, the final question:  Do the contents of a pickle file at a certain protocol have to be the some between versions, or
> is it enough if unpickling them returns the correct object?
In general, it would seem to me that each version of Python could add 
object types that are foreign to prior versions, and so as long as 
version N can unpickle anything produced by versions <=N, all would be well.

While I am _not_ an expert in pickle, this would just be my expectation 
based on years of experience with compatibility issues in non-Python 
non-pickle systems.  Pickle may have stronger compatibility goals than I 
would expect, and so must you, if you are thinking the answer to the 
above is "yes, it should be". :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150314/e6f3d62a/attachment.html>

From ethan at stoneleaf.us  Sun Mar 15 07:58:23 2015
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 14 Mar 2015 23:58:23 -0700
Subject: [Python-Dev] backwards and forwards compatibility,
 the exact contents of pickle files, and IntEnum
In-Reply-To: <55052AAC.6090907@g.nevcal.com>
References: <55051E11.8030603@stoneleaf.us> <55052AAC.6090907@g.nevcal.com>
Message-ID: <55052D8F.8080601@stoneleaf.us>

On 03/14/2015 11:46 PM, Glenn Linderman wrote:

> In general, it would seem to me that each version of Python could add object types that are foreign to prior versions,
> and so as long as version N can unpickle anything produced by versions <=N, all would be well.

The question to me is because we not so much adding something brand-new (socket.AF_INET has been there for ages), as
changing the type of something already existing -- I'm just not sure if we need "end-result" compatibility (achieved by
the first option) or if we need "bug for bug" compatibility (achieved by the second option).

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150314/b5187bec/attachment.sig>

From storchaka at gmail.com  Sun Mar 15 08:13:55 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sun, 15 Mar 2015 09:13:55 +0200
Subject: [Python-Dev] backwards and forwards compatibility,
 the exact contents of pickle files, and IntEnum
In-Reply-To: <55051E11.8030603@stoneleaf.us>
References: <55051E11.8030603@stoneleaf.us>
Message-ID: <me3bfk$l7o$1@ger.gmane.org>

On 15.03.15 07:52, Ethan Furman wrote:
> So how do we fix it?  There are a couple different options:
>
>    - modify IntEnum pickle methods to return the name only
>
>    - modify IntEnum pickle methods to pickle just like the int they represent
>
> The first option has the advantage that in 3.4 and above, you'll get back the IntEnum version.
>
> The second option has the advantage that the actual pickle contents are the same [1] as in previous versions.
>
> So, the final question:  Do the contents of a pickle file at a certain protocol have to be the some between versions, or
> is it enough if unpickling them returns the correct object?

With the second option you lost the type even for 3.5+. This is a step back.



From ethan at stoneleaf.us  Sun Mar 15 15:20:00 2015
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sun, 15 Mar 2015 07:20:00 -0700
Subject: [Python-Dev] backwards and forwards compatibility,
 the exact contents of pickle files, and IntEnum
In-Reply-To: <5505324E.3090205@g.nevcal.com>
References: <55051E11.8030603@stoneleaf.us> <55052AAC.6090907@g.nevcal.com>
 <55052D8F.8080601@stoneleaf.us> <5505324E.3090205@g.nevcal.com>
Message-ID: <55059510.2020806@stoneleaf.us>

Bringing back on list.  :)

On 03/15/2015 12:18 AM, Glenn Linderman wrote:
> On 3/14/2015 11:58 PM, Ethan Furman wrote:
>> On 03/14/2015 11:46 PM, Glenn Linderman wrote:
>>
>>> In general, it would seem to me that each version of Python could
>>> add object types that are foreign to prior versions, and so as long
>>> as version N can unpickle anything produced by versions <=N, all
>>> would be well.
>>
>> The question to me is because we are not so much adding something
>> brand-new (socket.AF_INET has been there for ages), as changing the
>> type of something already existing -- I'm just not sure if we need
>> "end-result" compatibility (achieved by the first option) or if we
>> need "bug for bug" compatibility (achieved by the second option).
> 
> Yes, the fact that it is not brand-new does add a mind-twist to the compatibility expectations.  But while it is not
> brand-new, it is changed.

Exactly.  IntEnum was advertised as being a "drop-in replacement" with the only change being better reprs and strs;  so
being unable to un-pickle in prior versions invalidates that claim.

Part 2 of my question (Part 1 of which is "which method do we use to fix the IntEnum being used in the stdlib") is: do
we just change the IntEnum subclasses (such as socket.AddressFamily) to have the fixed behavior, or do we change IntEnum
itself?

Having thought about it for a few minutes, we cannot make that change in IntEnum itself as it only works if the names
are exported to the global namespace -- so this fix will work for the stdlib (since we export the names), but won't work
otherwise.

Unless someone steps forward and says that the actual pickle contents must be the same, I'm going to go with pickling
the name: this is both unpicklable in prior versions, and doesn't lose the type in present versions.

Just to be clear, there are no issue unpickling in the same version, only in prior versions.

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150315/d5448bcb/attachment.sig>

From solipsis at pitrou.net  Sun Mar 15 16:10:51 2015
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 15 Mar 2015 16:10:51 +0100
Subject: [Python-Dev] backwards and forwards compatibility,
 the exact contents of pickle files, and IntEnum
References: <55051E11.8030603@stoneleaf.us>
Message-ID: <20150315161051.77d5e7ca@fsol>

On Sat, 14 Mar 2015 22:52:17 -0700
Ethan Furman <ethan at stoneleaf.us> wrote:

> I'm not sure exactly how to phrase this inquiry, so please bear with me.
> 
> What exactly does backwards compatibility mean as far as pickle goes?  We have the various protocols, we have the
> contents of pickle files created at those protocols, and we have different versions of Python.
> 
> Should a pickle file created in Python 3.4 with protocol 3 and the contents of socket.AF_INET be unpicklable with a
> Python 3.3 system?  Because currently it cannot be as socket.AF_INET became an IntEnum in 3.4.
> 
> I'm thinking the answer is yes, it should be.

I agree with you that this should not break.

> So, the final question:  Do the contents of a pickle file at a certain protocol have to be the some between versions, or
> is it enough if unpickling them returns the correct object?

It's certainly enough if unpickling returns the correct object. Exact
serialization is an implementation detail.

Regards

Antoine.



From ncoghlan at gmail.com  Mon Mar 16 00:01:57 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 16 Mar 2015 09:01:57 +1000
Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving
 Python ZIP Application Support
In-Reply-To: <CAP1=2W4ztzmYY7kCqhyE0jgr3BSuw_BGcMcsrcryox8+Hq-t8A@mail.gmail.com>
References: <CACac1F_E_a9hB818EMG4nbDTqqoOEc9B0_ZPZUzop6JKjk-Y3A@mail.gmail.com>
 <CAP1=2W4APXunO=wSYWpEyuYmYmkYjuSgEumvT_9=s6QdKobUgQ@mail.gmail.com>
 <CAG8k2+4JbRarMZmVg0dT18u1CnfWnS9f44=JvF++TVmMUJEgnw@mail.gmail.com>
 <CAP1=2W6GkDNvH9y2+gbh-WBpWvfM0tTX1BE92mniu74En1r7fA@mail.gmail.com>
 <CACac1F_t3uj8r5gTV1Y0ynGKWUF3oH3OHbX6Tf-H3inkxQOwvA@mail.gmail.com>
 <CAG8k2+4Mn9f1oujLr3mVY2DnNjfn52L5+Mc2vyJNJTvOtk1O5w@mail.gmail.com>
 <54EB7DDE.9030405@stoneleaf.us>
 <CAPdQG2pvRgeTiA6T-Lb6E7Th0eM1DZAzwrS9f85mPPWkPqAH4w@mail.gmail.com>
 <CAP7+vJ+=AzrnGdQfjCkq7Tc3tqHz2qq=jOYjQkyiuuDht5Hn-Q@mail.gmail.com>
 <CACac1F9W9vQ2TckokfGqEscUZMk-CLbt+ODEgxxkGz_oOnAg3g@mail.gmail.com>
 <CADiSq7fdPPQ=wRDhwZEv8Rh7iUm_Vt+k9MBgFhvo54svV0L2xA@mail.gmail.com>
 <CAP7+vJJ45esj8bbfHpBsitUhoxHHLN91yZyX72nLTHBkFe+6Cw@mail.gmail.com>
 <CACac1F_LsMGqBzF=ONe2D=AO44xZBZYkPXz8e=FHZhkUkj51QQ@mail.gmail.com>
 <CAP7+vJKXX=Ybz0rnfkQgH1+kgY-uVmvjJcOOhVR17Nhafp7N=w@mail.gmail.com>
 <CAP7+vJL_1danWZefJivqpDbNYM3DzNAeN_+q5PeqgC-ZMae-xw@mail.gmail.com>
 <CACac1F_tYTecaK0NursqPp4y4HwFDuhi4RQQ4Ggp9JKhC5ZUnA@mail.gmail.com>
 <CAP7+vJK-V=ipy4ALW_MYSENSVH=Mvm4jeTxey_B+iNuFppieCg@mail.gmail.com>
 <CACac1F96rbrc4gZbV9fkxKiAEMv7uVxPF0pPD60WwZHWZfyBEw@mail.gmail.com>
 <CACac1F9yQdOA5zKN31hzkX+Efsn5W-k=xngB0vi8WmnnkQioeg@mail.gmail.com>
 <CAP1=2W7ma-O8Z9Qj37PRuF73aiv-bOUbpPbXCpH5psJ5B_ouRg@mail.gmail.com>
 <CAP1=2W4ztzmYY7kCqhyE0jgr3BSuw_BGcMcsrcryox8+Hq-t8A@mail.gmail.com>
Message-ID: <CADiSq7e1PoDimbSnDEzeceXQ9QMhDOQoNRUWk4s4auy1euR-Vw@mail.gmail.com>

Huzzah! Thanks for working on this one folks, it should make the zipfile
execution support much easier for current and future Python users to
discover :)

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150316/8ab57fff/attachment.html>

From encukou at gmail.com  Mon Mar 16 13:38:13 2015
From: encukou at gmail.com (Petr Viktorin)
Date: Mon, 16 Mar 2015 13:38:13 +0100
Subject: [Python-Dev] PEP 489: Redesigning extension module loading
Message-ID: <5506CEB5.7050105@gmail.com>

Hello,
On import-sig, I've agreed to continue Nick Coghlan's work on making 
extension modules act more like Python ones, work well with PEP 451 
(ModuleSpec), and encourage proper subinterpreter and reloading support. 
Here is the resulting PEP.

I don't have a patch yet, but I'm working on it.

There's a remaining open issue: providing a tool that can be run in test 
suites to check if a module behaves well with subinterpreters/reloading. 
I believe it's out of scope for this PEP but speak out if you disagree.

Please discuss on import-sig.

=======================

PEP: 489
Title: Redesigning extension module loading
Version: $Revision$
Last-Modified: $Date$
Author: Petr Viktorin <encukou at gmail.com>,
         Stefan Behnel <stefan_ml at behnel.de>,
         Nick Coghlan <ncoghlan at gmail.com>
Discussions-To: import-sig at python.org
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 11-Aug-2013
Python-Version: 3.5
Post-History: 23-Aug-2013, 20-Feb-2015
Resolution:


Abstract
========

This PEP proposes a redesign of the way in which extension modules interact
with the import machinery. This was last revised for Python 3.0 in PEP
3121, but did not solve all problems at the time. The goal is to solve them
by bringing extension modules closer to the way Python modules behave;
specifically to hook into the ModuleSpec-based loading mechanism
introduced in PEP 451.

Extensions that do not require custom memory layout for their module objects
may be executed in arbitrary pre-defined namespaces, paving the way for
extension modules being runnable with Python's ``-m`` switch.
Other extensions can use custom types for their module implementation.
Module types are no longer restricted to types.ModuleType.

This proposal makes it easy to support properties at the module
level and to safely store arbitrary global state in the module that is
covered by normal garbage collection and supports reloading and
sub-interpreters.
Extension authors are encouraged to take these issues into account
when using the new API.



Motivation
==========

Python modules and extension modules are not being set up in the same way.
For Python modules, the module is created and set up first, then the module
code is being executed (PEP 302).
A ModuleSpec object (PEP 451) is used to hold information about the module,
and passed to the relevant hooks.
For extensions, i.e. shared libraries, the module
init function is executed straight away and does both the creation and
initialisation. The initialisation function is not passed ModuleSpec
information about the loaded module, such as the __file__ or fully-qualified
name. This hinders relative imports and resource loading.

This is specifically a problem for Cython generated modules, for which it's
not uncommon that the module init code has the same level of complexity as
that of any 'regular' Python module. Also, the lack of __file__ and __name__
information hinders the compilation of __init__.py modules, i.e. packages,
especially when relative imports are being used at module init time.

The other disadvantage of the discrepancy is that existing Python 
programmers
learning C cannot effectively map concepts between the two domains.
As long as extension modules are fundamentally different from pure 
Python ones
in the way they're initialised, they are harder for people to pick up 
without
relying on something like cffi, SWIG or Cython to handle the actual 
extension
module creation.

Currently, extension modules are also not added to sys.modules until 
they are
fully initialized, which means that a (potentially transitive)
re-import of the module will really try to reimport it and thus run into an
infinite loop when it executes the module init function again.
Without the fully qualified module name, it is not trivial to correctly add
the module to sys.modules either.

Furthermore, the majority of currently existing extension modules has
problems with sub-interpreter support and/or reloading, and, while it is
possible with the current infrastructure to support these
features, it is neither easy nor efficient.
Addressing these issues was the goal of PEP 3121, but many extensions,
including some in the standard library, took the least-effort approach
to porting to Python 3, leaving these issues unresolved.
This PEP keeps the backwards-compatible behavior, which should reduce 
pressure
and give extension authors adequate time to consider these issues when 
porting.


The current process
===================

Currently, extension modules export an initialisation function named
"PyInit_modulename", named after the file name of the shared library. This
function is executed by the import machinery and must return either NULL in
the case of an exception, or a fully initialised module object. The
function receives no arguments, so it has no way of knowing about its
import context.

During its execution, the module init function creates a module object
based on a PyModuleDef struct. It then continues to initialise it by adding
attributes to the module dict, creating types, etc.

In the back, the shared library loader keeps a note of the fully qualified
module name of the last module that it loaded, and when a module gets
created that has a matching name, this global variable is used to determine
the fully qualified name of the module object. This is not entirely safe 
as it
relies on the module init function creating its own module object first,
but this assumption usually holds in practice.


The proposal
============

The current extension module initialisation will be deprecated in favour of
a new initialisation scheme. Since the current scheme will continue to be
available, existing code will continue to work unchanged, including binary
compatibility.

Extension modules that support the new initialisation scheme must export
the public symbol "PyModuleExec_modulename", and optionally
"PyModuleCreate_modulename", where "modulename" is the
name of the module. This mimics the previous naming convention for
the "PyInit_modulename" function.

If defined, these symbols must resolve to C functions with the following
signatures, respectively::

     int (*PyModuleExecFunction)(PyObject* module)
     PyObject* (*PyModuleCreateFunction)(PyObject* module_spec)


The PyModuleExec function
-------------------------

The PyModuleExec function is used to implement "loader.exec_module"
defined in PEP 451.

It function will be called to initialize a module. (Usually, this amounts to
setting the module's initial attributes.)
This happens in two situations: when the module is first initialized for
a given (sub-)interpreter, and possibly later when the module is reloaded.

When PyModuleExec is called, the module has already been added to
sys.modules, and import-related attributes specified in
PEP 451 [#pep-0451-attributes]_) have been set on the module.

The "module" argument receives the module object to initialize.

If PyModuleCreate is defined, "module" will generally be the the object
returned by it.
It is possible for a custom loader to pass any object to
PyModuleExec, so this function should check and fail with TypeError
if the module's type is unsupported.
Any other assumptions should also be checked.

If PyModuleCreate is not defined, PyModuleExec is expected to operate
on any Python object for which attributes can be added by PyObject_GetAttr*
and retrieved by PyObject_SetAttr*.
This allows loading an extension into a pre-created module, making it 
possible
to run it as __main__ in the future, participate in certain lazy-loading
schemes [#lazy_import_concerns]_, or enable other creative uses.

If PyModuleExec replaces the module's entry in sys.modules,
the new object will be used and returned by importlib machinery.
(This mirrors the behavior of Python modules. Note that for extensions,
implementing PyModuleCreate is usually a better solution for the use cases
this serves.)

The function must return ``0`` on success, or, on error, set an 
exception and
return ``-1``.


The PyModuleCreate function
---------------------------

The optional PyModuleCreate function is used to implement
"loader.create_module" defined in PEP 451.
By exporting it, an extension module indicates that it uses a custom
module object.
This prevents loading the extension in a pre-created module,
but gives greater flexibility in allowing a custom C-level layout
of the module object.
Most extensions will not need to implement this function.

The "module_spec" argument receives a "ModuleSpec" instance, as defined in
PEP 451.

When called, this function must create and return a module object,
or set an exception and return NULL.
There is no requirement for the returned object to be an instance of
types.ModuleType. Any type can be used, as long as it supports setting and
getting attributes, including at least the import-related attributes
specified in PEP 451 [#pep-0451-attributes]_.
This follows the current support for allowing arbitrary objects in 
sys.modules
and makes it easier for extension modules to define a type that exactly 
matches
their needs for holding module state.

Note that when this function is called, the module's entry in sys.modules
is not populated yet. Attempting to import the same module again
(possibly transitively), may lead to an infinite loop.
Extension authors are advised to keep PyModuleCreate minimal, an in 
particular
to not call user code from it.

If PyModuleCreate is not defined, the default loader will construct
a module object as if with PyModule_New.


Initialization helper functions
-------------------------------

For two initialization tasks previously done by PyModule_Create,
two functions are introduced::

     int PyModule_SetDocString(PyObject *m, const char *doc)
     int PyModule_AddFunctions(PyObject *m, PyMethodDef *functions)

These set the module docstring, and add the module functions, respectively.
Both will work on any Python object that supports setting attributes.
They return ``0`` on success, and on failure, they set the exception
and return ``-1``.


PyCapsule convenience functions
-------------------------------

Instead of custom module objects, PyCapsule will become the preferred
mechanism for storing per-module C data.
Two new convenience functions will be added to help with this.

*
     ::

         PyObject *PyModule_AddCapsule(
             PyObject *module,
             const char *module_name,
             const char *attribute_name,
             void *pointer,
             PyCapsule_Destructor destructor)

     Add a new PyCapsule to *module* as *attribute_name*.
     The capsule name is formed by joining *module_name* and 
*attribute_name*
     by a dot.

     This convenience function can be used from a module initialization 
function
     instead of separate calls to PyCapsule_New and PyModule_AddObject.

     Returns a borrowed reference to the new capsule,
     or NULL (with exception set) on failure.

*
     ::

         void *PyModule_GetCapsulePointer(
             PyObject *module,
             const char *module_name,
             const char *attribute_name)

     Returns the pointer stored in *module* as *attribute_name*, or NULL
     (with an exception set) on failure. The capsule name is formed by 
joining
     *module_name* and *attribute_name* by a dot.

     This convenience function can be used instead of separate calls to
     PyObject_GetAttr and PyCapsule_GetPointer.

Extension authors are encouraged to define a macro to
call PyModule_GetCapsulePointer and cast the result to an appropriate type.


Generalizing PyModule_* functions
---------------------------------

The following functions and macros will be modified to work on any object
that supports attribute access:

     * PyModule_GetNameObject
     * PyModule_GetName
     * PyModule_GetFilenameObject
     * PyModule_GetFilename
     * PyModule_AddIntConstant
     * PyModule_AddStringConstant
     * PyModule_AddIntMacro
     * PyModule_AddStringMacro
     * PyModule_AddObject

The PyModule_GetDict function will continue to only work on true module
objects. This means that it should not be used on extension modules that 
only
define PyModuleExec.


Legacy Init
-----------

If PyModuleExec is not defined, the import machinery will try to initialize
the module using the PyModuleInit hook, as described in PEP 3121.

If PyModuleExec is defined, PyModuleInit will be ignored.
Modules requiring compatibility with previous versions of CPython may 
implement
PyModuleInit in addition to the new hook.


Subinterpreters and Interpreter Reloading
-----------------------------------------

Extensions using the new initialization scheme are expected to support
subinterpreters and multiple Py_Initialize/Py_Finalize cycles correctly.
The mechanism is designed to make this easy, but care is still required
on the part of the extension author.
No user-defined functions, methods, or instances may leak to different
interpreters.
To achieve this, all module-level state should be kept in either the module
dict, or in the module object.
A simple rule of thumb is: Do not define any static data, except 
built-in types
with no mutable or user-settable class attributes.


Module Reloading
----------------

Reloading an extension module will re-execute its PyModuleInit function.
Similar caveats apply to reloading an extension module as to reloading
a Python module. Notably, attributes or any other state of the module
are not reset before reloading.

Additionally, due to limitations in shared library loading (both dlopen on
POSIX and LoadModuleEx on Windows), it is not generally possible to load
a modified library after it has changed on disk.
Therefore, reloading extension modules is of limited use.


Multiple modules in one library
-------------------------------

To support multiple Python modules in one shared library, the library
must export appropriate PyModuleExec_<name> or PyModuleCreate_<name> hooks
for each exported module.
The modules are loaded using a ModuleSpec with origin set to the name of the
library file, and name set to the module name.

Note that this mechanism can currently only be used to *load* such modules,
not to *find* them.

XXX: This is an existing issue; either fix it/wait for a fix or provide
an example of how to load such modules.


Implementation
==============

XXX - not started


Open issues
===========

We should expose some kind of API in importlib.util (or a better place?) 
that
can be used to check that a module works with reloading and subinterpreters.


Related issues
==============

The runpy module will need to be modified to take advantage of PEP 451
and this PEP. This is out of scope for this PEP.


Previous Approaches
===================

Stefan Behnel's initial proto-PEP [#stefans_protopep]_
had a "PyInit_modulename" hook that would create a module class,
whose ``__init__`` would be then called to create the module.
This proposal did not correspond to the (then nonexistent) PEP 451,
where module creation and initialization is broken into distinct steps.
It also did not support loading an extension into pre-existing module 
objects.

Nick Coghlan proposed the Create annd Exec hooks, and wrote a prototype
implementation [#nicks-prototype]_.
At this time PEP 451 was still not implemented, so the prototype
does not use ModuleSpec.


References
==========

.. [#lazy_import_concerns]
    https://mail.python.org/pipermail/python-dev/2013-August/128129.html

.. [#pep-0451-attributes]
    https://www.python.org/dev/peps/pep-0451/#attributes

.. [#stefans_protopep]
    https://mail.python.org/pipermail/python-dev/2013-August/128087.html

.. [#nicks-prototype]
    https://mail.python.org/pipermail/python-dev/2013-August/128101.html


Copyright
=========

This document has been placed in the public domain.

From jimjjewett at gmail.com  Mon Mar 16 16:42:26 2015
From: jimjjewett at gmail.com (Jim J. Jewett)
Date: Mon, 16 Mar 2015 08:42:26 -0700 (PDT)
Subject: [Python-Dev] PEP 489: Redesigning extension module loading
In-Reply-To: <5506CEB5.7050105@gmail.com>
Message-ID: <5506f9e2.15148c0a.2361.ffff8a31@mx.google.com>


On 16 March 2015 Petr Viktorin wrote:

> If PyModuleCreate is not defined, PyModuleExec is expected to operate
> on any Python object for which attributes can be added by PyObject_GetAttr*
> and retrieved by PyObject_SetAttr*.

I assume it is the other way around (add with Set and retrieve with Get),
rather than a description of the required form of magic.


>         PyObject *PyModule_AddCapsule(
>             PyObject *module,
>             const char *module_name,
>             const char *attribute_name,
>             void *pointer,
>             PyCapsule_Destructor destructor)

What happens if module_name doesn't match the module's __name__?
Does it become a hidden attribute?  A dotted attribute?  Is the
result undefined?

Later, there is

>         void *PyModule_GetCapsulePointer(
>             PyObject *module,
>             const char *module_name,
>             const char *attribute_name)

with the same apparently redundant arguments, but not a
PyModule_SetCapsulePointer.  Are capsule pointers read-only, or can
they be replaced with another call to PyModule_AddCapsule, or by a
simple PyObject_SetAttr?

> Subinterpreters and Interpreter Reloading
...
> No user-defined functions, methods, or instances may leak to different
> interpreters.

By "user-defined" do you mean "defined in python, as opposed to in
the extension itself"?

If so, what is the recommendation for modules that do want to support,
say, callbacks?  A dual-layer mapping that uses the interpreter as the
first key?  Naming it _module and only using it indirectly through
module.py, which is not shared across interpreters?  Not using this
API at all?

> To achieve this, all module-level state should be kept in either the module
> dict, or in the module object.

I don't see how that is related to leakage.

> A simple rule of thumb is: Do not define any static data, except 
> built-in types
> with no mutable or user-settable class attributes.

What about singleton instances?  Should they be per-interpreter?
What about constants, such as PI?
Where should configuration variables (e.g., MAX_SEARCH_DEPTH) be
kept?


What happens if this no-leakage rule is violated?  Does the module
not load, or does it just maybe lead to a crash down the road?

-jJ

--

If there are still threading problems with my replies, please
email me with details, so that I can try to resolve them.  -jJ

From encukou at gmail.com  Mon Mar 16 18:42:44 2015
From: encukou at gmail.com (Petr Viktorin)
Date: Mon, 16 Mar 2015 18:42:44 +0100
Subject: [Python-Dev] PEP 489: Redesigning extension module loading
In-Reply-To: <5506f9e2.15148c0a.2361.ffff8a31@mx.google.com>
References: <5506CEB5.7050105@gmail.com>
 <5506f9e2.15148c0a.2361.ffff8a31@mx.google.com>
Message-ID: <CA+=+wqB2i2FOJ6+VQ5WMJ7tdPTOwfy2AaDw9YHaiUxuU0fGUKA@mail.gmail.com>

On Mon, Mar 16, 2015 at 4:42 PM, Jim J. Jewett <jimjjewett at gmail.com> wrote:
>
> On 16 March 2015 Petr Viktorin wrote:
>
>> If PyModuleCreate is not defined, PyModuleExec is expected to operate
>> on any Python object for which attributes can be added by PyObject_GetAttr*
>> and retrieved by PyObject_SetAttr*.
>
> I assume it is the other way around (add with Set and retrieve with Get),
> rather than a description of the required form of magic.

Right you are, I mixed that up.

>>         PyObject *PyModule_AddCapsule(
>>             PyObject *module,
>>             const char *module_name,
>>             const char *attribute_name,
>>             void *pointer,
>>             PyCapsule_Destructor destructor)
>
> What happens if module_name doesn't match the module's __name__?
> Does it become a hidden attribute?  A dotted attribute?  Is the
> result undefined?

The module_name is used to name the capsule, following the convention
from PyCapsule_Import. The "module.__name__" is not used or checked.
The function would do this:
    capsule_name = module_name + '.' + attribute_name
    capsule = PyCapsule_New(pointer, capsule_name, destructor)
    PyModule_AddObject(module, attribute_name, capsule)
just with error handling, and suitable C code for the "+".
I will add the pseudocode to the PEP.

> Later, there is
>
>>         void *PyModule_GetCapsulePointer(
>>             PyObject *module,
>>             const char *module_name,
>>             const char *attribute_name)
>
> with the same apparently redundant arguments,

Here the behavior would be:
    capsule_name = module_name + '.' + attribute_name
    capsule = PyObject_GetAttr(module, attribute_name)
    return PyCapsule_GetPointer(capsule, capsule_name)

> but not a
> PyModule_SetCapsulePointer.  Are capsule pointers read-only, or can
> they be replaced with another call to PyModule_AddCapsule, or by a
> simple PyObject_SetAttr?

You can replace the capsule using any of those two, or set the pointer
using PyCapsule_SetPointer, or (most likely) change the data the
pointer points to.
The added functions are just simple helpers for common operations,
meant to encourage keeping per-module state.

>> Subinterpreters and Interpreter Reloading
> ...
>> No user-defined functions, methods, or instances may leak to different
>> interpreters.
>
> By "user-defined" do you mean "defined in python, as opposed to in
> the extension itself"?

Yes.

> If so, what is the recommendation for modules that do want to support,
> say, callbacks?  A dual-layer mapping that uses the interpreter as the
> first key?  Naming it _module and only using it indirectly through
> module.py, which is not shared across interpreters?  Not using this
> API at all?

There is a separate module object, with its own dict, for each
subinterpreter (as when creating the module with "PyModuleDef.m_size
== 0" today).
Callbacks should be stored on the appropriate module instance.
Does that answer your question? I'm not sure how you meant "callbacks".

>> To achieve this, all module-level state should be kept in either the module
>> dict, or in the module object.
>
> I don't see how that is related to leakage.
>
>> A simple rule of thumb is: Do not define any static data, except
>> built-in types
>> with no mutable or user-settable class attributes.
>
> What about singleton instances?  Should they be per-interpreter?

Yes, definitely.

> What about constants, such as PI?

In PyModuleExec, create the constant using PyFloat_FromDouble, and add
it using PyModule_FromObject. That will do the right thing.
(Float constants can be shared, since they cannot refer to
user-defined code. But this PEP shields you from needing to know this
for every type.)

> Where should configuration variables (e.g., MAX_SEARCH_DEPTH) be
> kept?

On the module object.

> What happens if this no-leakage rule is violated?  Does the module
> not load, or does it just maybe lead to a crash down the road?

It may, as today, lead to unexpected behavior down the road. This is
explained here:
https://docs.python.org/3/c-api/init.html#sub-interpreter-support
Unfortunately, there's no good way to detect such leakage. This PEP
adds the tools, documentation, and guidelines to make it easy to do
the right thing, but won't prevent you from shooting yourself in the
foot in C code.


Thank you for sharing your concerns! I will keep them in mind when
writing the docs for this.

From mistersheik at gmail.com  Tue Mar 17 00:10:50 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Mon, 16 Mar 2015 19:10:50 -0400
Subject: [Python-Dev] PEP 448 review
In-Reply-To: <CAA68w_ki+sAti_WX1a+ACUCbRexh38a=aQpSG7BgEgbhWmsiig@mail.gmail.com>
References: <CAP7+vJK58SYgBExknnSO2xwiQm5+AO5_=f0yAarNJUkbmczcPg@mail.gmail.com>
 <CAA68w_m1cB4MmMeNX=zA5MjfaxytgU9+=8fVqEK173DGy3K5Zw@mail.gmail.com>
 <CAMpsgwa8JY3rCobEPOe2CN2ovcQJE7AkHfRPtw5SZE83eF1+fQ@mail.gmail.com>
 <CAA68w_kFVmMQJsk+hdgnKtOui2rqYdecQmaNEjk_UPk7N7Q_xg@mail.gmail.com>
 <CAA68w_ngnkX0KPnDni8Vvi9iG=RgiN4B_0dp9NuEMvaNO6=5OA@mail.gmail.com>
 <CAA68w_ki+sAti_WX1a+ACUCbRexh38a=aQpSG7BgEgbhWmsiig@mail.gmail.com>
Message-ID: <CAA68w_mpd-7Ai5as3V=6F-7Lw4er0s=H10jdznB4CJ=zJyFcpA@mail.gmail.com>

Hi everyone,

I was wondering what is left with the PEP 448 (
http://bugs.python.org/issue2292) code review?  Big thanks to Benjamin,
Ethan, and Serhiy for reviewing some (all?) of the code.  What is the next
step of this process?

Thanks,

Neil


On Sun, Mar 8, 2015 at 4:38 PM, Neil Girdhar <mistersheik at gmail.com> wrote:

> Anyone have time to do a code review?
>
> http://bugs.python.org/issue2292
>
>
> On Mon, Mar 2, 2015 at 4:54 PM, Neil Girdhar <mistersheik at gmail.com>
> wrote:
>
>> It's from five days ago.  I asked Joshua to take a look at something, but
>> I guess he is busy.
>>
>> Best,
>>
>> Neil
>>
>> ?
>>
>> The latest file there is from Feb 26, while your message that the patch
>> was ready for review is from today -- so is the
>> patch from five days ago the most recent?
>>
>> --
>> ~Ethan~
>>
>> On Mon, Mar 2, 2015 at 3:18 PM, Neil Girdhar <mistersheik at gmail.com>
>> wrote:
>>
>>> http://bugs.python.org/issue2292
>>>
>>> On Mon, Mar 2, 2015 at 3:17 PM, Victor Stinner <victor.stinner at gmail.com
>>> > wrote:
>>>
>>>> Where is the patch?
>>>>
>>>> Victor
>>>>
>>>> Le lundi 2 mars 2015, Neil Girdhar <mistersheik at gmail.com> a ?crit :
>>>>
>>>> Hi everyone,
>>>>>
>>>>> The patch is ready for review now, and I should have time this week to
>>>>> make changes and respond to comments.
>>>>>
>>>>> Best,
>>>>>
>>>>> Neil
>>>>>
>>>>> On Wed, Feb 25, 2015 at 2:42 PM, Guido van Rossum <guido at python.org>
>>>>> wrote:
>>>>>
>>>>>> I'm back, I've re-read the PEP, and I've re-read the long thread with
>>>>>> "(no subject)".
>>>>>>
>>>>>> I think Georg Brandl nailed it:
>>>>>>
>>>>>> """
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *I like the "sequence and dict flattening" part of the PEP, mostly
>>>>>> because itis consistent and should be easy to understand, but the
>>>>>> comprehension syntaxenhancements seem to be bad for readability and
>>>>>> "comprehending" what the codedoes.The call syntax part is a mixed bag on
>>>>>> the one hand it is nice to be consistent with the extended possibilities in
>>>>>> literals (flattening), but on the other hand there would be small but
>>>>>> annoying inconsistencies anyways (e.g. the duplicate kwarg case above).*
>>>>>> """
>>>>>>
>>>>>> Greg Ewing followed up explaining that the inconsistency between dict
>>>>>> flattening and call syntax is inherent in the pre-existing different rules
>>>>>> for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while f(a=1,
>>>>>> a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1,
>>>>>> **{'a': 1}) is a TypeError.)
>>>>>>
>>>>>> For me, allowing f(*a, *b) and f(**d, **e) and all the other
>>>>>> combinations for function calls proposed by the PEP is an easy +1 -- it's a
>>>>>> straightforward extension of the existing pattern, and anybody who knows
>>>>>> what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d,
>>>>>> **e) means shouldn't be hard either. Understanding the edge case for
>>>>>> duplicate keys with f(**d, **e) is a little harder, but the error messages
>>>>>> are pretty clear, and it is not a new edge case.
>>>>>>
>>>>>> The sequence and dict flattening syntax proposals are also clean and
>>>>>> logical -- we already have *-unpacking on the receiving side, so allowing
>>>>>> *x in tuple expressions reads pretty naturally (and the similarity with *a
>>>>>> in argument lists certainly helps). From here, having [a, *x, b, *y] is
>>>>>> also natural, and then the extension to other displays is natural: {a, *x,
>>>>>> b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me.
>>>>>>
>>>>>> So that leaves comprehensions. IIRC, during the development of the
>>>>>> patch we realized that f(*x for x in xs) is sufficiently ambiguous that we
>>>>>> decided to disallow it -- note that f(x for x in xs) is already somewhat of
>>>>>> a special case because an argument can only be a "bare" generator
>>>>>> expression if it is the only argument. The same reasoning doesn't apply (in
>>>>>> that form) to list, set and dict comprehensions -- while f(x for x in xs)
>>>>>> is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the
>>>>>> same as [(x for x in xs)] (that's a list of one element, and the element is
>>>>>> a generator expression).
>>>>>>
>>>>>> The basic premise of this part of the proposal is that if you have a
>>>>>> few iterables, the new proposal (without comprehensions) lets you create a
>>>>>> list or generator expression that iterates over all of them, essentially
>>>>>> flattening them:
>>>>>>
>>>>>>     >>> xs = [1, 2, 3]
>>>>>>     >>> ys = ['abc', 'def']
>>>>>>     >>> zs = [99]
>>>>>>     >>> [*xs, *ys, *zs]
>>>>>>     [1, 2, 3, 'abc', 'def', 99]
>>>>>>     >>>
>>>>>>
>>>>>> But now suppose you have a list of iterables:
>>>>>>
>>>>>>     >>> xss = [[1, 2, 3], ['abc', 'def'], [99]]
>>>>>>     >>> [*xss[0], *xss[1], *xss[2]]
>>>>>>     [1, 2, 3, 'abc', 'def', 99]
>>>>>>     >>>
>>>>>>
>>>>>> Wouldn't it be nice if you could write the latter using a
>>>>>> comprehension?
>>>>>>
>>>>>>     >>> xss = [[1, 2, 3], ['abc', 'def'], [99]]
>>>>>>     >>> [*xs for xs in xss]
>>>>>>     [1, 2, 3, 'abc', 'def', 99]
>>>>>>     >>>
>>>>>>
>>>>>> This is somewhat seductive, and the following is even nicer: the *xs
>>>>>> position may be an expression, e.g.:
>>>>>>
>>>>>>     >>> xss = [[1, 2, 3], ['abc', 'def'], [99]]
>>>>>>     >>> [*xs[:2] for xs in xss]
>>>>>>     [1, 2, 'abc', 'def', 99]
>>>>>>     >>>
>>>>>>
>>>>>> On the other hand, I had to explore the possibilities here by
>>>>>> experimenting in the interpreter, and I discovered some odd edge cases
>>>>>> (e.g. you can parenthesize the starred expression, but that seems a
>>>>>> syntactic accident).
>>>>>>
>>>>>> All in all I am personally +0 on the comprehension part of the PEP,
>>>>>> and I like that it provides a way to "flatten" a sequence of sequences, but
>>>>>> I think very few people in the thread have supported this part. Therefore I
>>>>>> would like to ask Neil to update the PEP and the patch to take out the
>>>>>> comprehension part, so that the two "easy wins" can make it into Python 3.5
>>>>>> (basically, I am accepting two-thirds of the PEP :-). There is some time
>>>>>> yet until alpha 2.
>>>>>>
>>>>>> I would also like code reviewers (Benjamin?) to start reviewing the
>>>>>> patch <http://bugs.python.org/issue2292>, taking into account that
>>>>>> the comprehension part needs to be removed.
>>>>>>
>>>>>> --
>>>>>> --Guido van Rossum (python.org/~guido)
>>>>>>
>>>>>>
>>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150316/31dccfff/attachment-0001.html>

From brett at python.org  Tue Mar 17 14:49:58 2015
From: brett at python.org (Brett Cannon)
Date: Tue, 17 Mar 2015 13:49:58 +0000
Subject: [Python-Dev] PEP 448 review
In-Reply-To: <CAA68w_mpd-7Ai5as3V=6F-7Lw4er0s=H10jdznB4CJ=zJyFcpA@mail.gmail.com>
References: <CAP7+vJK58SYgBExknnSO2xwiQm5+AO5_=f0yAarNJUkbmczcPg@mail.gmail.com>
 <CAA68w_m1cB4MmMeNX=zA5MjfaxytgU9+=8fVqEK173DGy3K5Zw@mail.gmail.com>
 <CAMpsgwa8JY3rCobEPOe2CN2ovcQJE7AkHfRPtw5SZE83eF1+fQ@mail.gmail.com>
 <CAA68w_kFVmMQJsk+hdgnKtOui2rqYdecQmaNEjk_UPk7N7Q_xg@mail.gmail.com>
 <CAA68w_ngnkX0KPnDni8Vvi9iG=RgiN4B_0dp9NuEMvaNO6=5OA@mail.gmail.com>
 <CAA68w_ki+sAti_WX1a+ACUCbRexh38a=aQpSG7BgEgbhWmsiig@mail.gmail.com>
 <CAA68w_mpd-7Ai5as3V=6F-7Lw4er0s=H10jdznB4CJ=zJyFcpA@mail.gmail.com>
Message-ID: <CAP1=2W5yq+GESZXyr9g5BEe64Kq_-ZHBgUb4sZ5QykDoRvp4Ow@mail.gmail.com>

On Mon, Mar 16, 2015 at 7:11 PM Neil Girdhar <mistersheik at gmail.com> wrote:

> Hi everyone,
>
> I was wondering what is left with the PEP 448 (
> http://bugs.python.org/issue2292) code review?  Big thanks to Benjamin,
> Ethan, and Serhiy for reviewing some (all?) of the code.  What is the next
> step of this process?
>

My suspicion is that if no one steps up between now and PyCon to do a
complete code review of the final patch, we as a group will try to get it
done at the PyCon sprints. I have made the issue a release blocker to help
make sure it gets reviewed and doesn't accidentally get left behind.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150317/377ecc00/attachment.html>

From szport at gmail.com  Tue Mar 17 20:05:16 2015
From: szport at gmail.com (Zaur Shibzukhov)
Date: Tue, 17 Mar 2015 22:05:16 +0300
Subject: [Python-Dev] Possible wrong behavior of the dict?
Message-ID: <CAPOE21TuvFU-=fdBWw1jCS+X-EXLOS5pBy8=o5nVr4+A6Hky+Q@mail.gmail.com>

Hello!

In order to explain, let define subclass of dict:

class Pair:
    def __init__(self, key, val):
        self.key = key
        self.val = val

class MyDict(dict):
    #
    def __init__(self, *args, **kwds):
        if len(args) > 1:
            raise TypeError('Expected at most 1 arguments, but got %d' %
len(args))

        for key, val in args[0]:
            self[key] = val

        for key, val in kwds.items():
            self[key] = val

    def __getitem__(self, key):
        pair = dict.__getitem__(key)
        return pair.value

    def __setitem__(self, key, val):
        if key in self:
            pair = dict.__getitem__(key)
            pair.value = value
        else:
            pair = Pair(key, val)
            dict.__setitem__(self, key, pair)

    def values(self):
        for key in self:
            p = dict.__getitem__(self, key)
            yield p.value

    def items(self):
        for key, p in dict.__iter__(self):
            yield p.key, p.value


The simple test give me strange result:

>>> d = MyDict([('a', 1), ('b', 2), ('c', 3)])
>>> dict(d)
{'a': <__main__.Pair at 0x104ca9e48>,
 'b': <__main__.Pair at 0x104ca9e80>,
 'c': <__main__.Pair at 0x104ca9eb8>}

instead of {'a':1, 'b':2, 'c':3}.


Is this right behavior of the dict?

---
Zaur Shibzukhov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150317/82827f46/attachment.html>

From brett at python.org  Tue Mar 17 20:12:36 2015
From: brett at python.org (Brett Cannon)
Date: Tue, 17 Mar 2015 19:12:36 +0000
Subject: [Python-Dev] Possible wrong behavior of the dict?
In-Reply-To: <CAPOE21TuvFU-=fdBWw1jCS+X-EXLOS5pBy8=o5nVr4+A6Hky+Q@mail.gmail.com>
References: <CAPOE21TuvFU-=fdBWw1jCS+X-EXLOS5pBy8=o5nVr4+A6Hky+Q@mail.gmail.com>
Message-ID: <CAP1=2W5wiYAEAMqFa4WGvOnMY1dH_p5Sy8piydcwDvt+u2+QDQ@mail.gmail.com>

On Tue, Mar 17, 2015 at 3:05 PM Zaur Shibzukhov <szport at gmail.com> wrote:

> Hello!
>
> In order to explain, let define subclass of dict:
>
> class Pair:
>     def __init__(self, key, val):
>         self.key = key
>         self.val = val
>
> class MyDict(dict):
>     #
>     def __init__(self, *args, **kwds):
>         if len(args) > 1:
>             raise TypeError('Expected at most 1 arguments, but got %d' %
> len(args))
>
>         for key, val in args[0]:
>             self[key] = val
>
>         for key, val in kwds.items():
>             self[key] = val
>
>     def __getitem__(self, key):
>         pair = dict.__getitem__(key)
>         return pair.value
>
>     def __setitem__(self, key, val):
>         if key in self:
>             pair = dict.__getitem__(key)
>             pair.value = value
>         else:
>             pair = Pair(key, val)
>             dict.__setitem__(self, key, pair)
>
>     def values(self):
>         for key in self:
>             p = dict.__getitem__(self, key)
>             yield p.value
>
>     def items(self):
>         for key, p in dict.__iter__(self):
>             yield p.key, p.value
>
>
> The simple test give me strange result:
>
> >>> d = MyDict([('a', 1), ('b', 2), ('c', 3)])
> >>> dict(d)
> {'a': <__main__.Pair at 0x104ca9e48>,
>  'b': <__main__.Pair at 0x104ca9e80>,
>  'c': <__main__.Pair at 0x104ca9eb8>}
>
> instead of {'a':1, 'b':2, 'c':3}.
>
>
> Is this right behavior of the dict?
>

Yes because in your __setitem__ call you are storing the value as the Pair.
So when dict prints its repr it prints the key and value, and in this case
the value is a Pair.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150317/0f90b3ce/attachment.html>

From szport at gmail.com  Tue Mar 17 20:29:07 2015
From: szport at gmail.com (Zaur Shibzukhov)
Date: Tue, 17 Mar 2015 22:29:07 +0300
Subject: [Python-Dev] Possible wrong behavior of the dict?
In-Reply-To: <CAP1=2W5wiYAEAMqFa4WGvOnMY1dH_p5Sy8piydcwDvt+u2+QDQ@mail.gmail.com>
References: <CAPOE21TuvFU-=fdBWw1jCS+X-EXLOS5pBy8=o5nVr4+A6Hky+Q@mail.gmail.com>
 <CAP1=2W5wiYAEAMqFa4WGvOnMY1dH_p5Sy8piydcwDvt+u2+QDQ@mail.gmail.com>
Message-ID: <CAPOE21TcOXv61N9KJYqc7yxzkBnZnFDws6ZPHhqw1FfWQYeZfw@mail.gmail.com>

Yes... But I expected that dict constructor will use `__getitem__`  or
`items` method of MyDict instance  in order to retrieve items of the MyDict
instance during construction of the dict instance... Instead it interpreted
MyDict instance as the dict instance during construction of new dict.This
exactly caused my confusion.

---
*Zaur Shibzukhov*


2015-03-17 22:12 GMT+03:00 Brett Cannon <brett at python.org>:

>
>
> On Tue, Mar 17, 2015 at 3:05 PM Zaur Shibzukhov <szport at gmail.com> wrote:
>
>> Hello!
>>
>> In order to explain, let define subclass of dict:
>>
>> class Pair:
>>     def __init__(self, key, val):
>>         self.key = key
>>         self.val = val
>>
>> class MyDict(dict):
>>     #
>>     def __init__(self, *args, **kwds):
>>         if len(args) > 1:
>>             raise TypeError('Expected at most 1 arguments, but got %d' %
>> len(args))
>>
>>         for key, val in args[0]:
>>             self[key] = val
>>
>>         for key, val in kwds.items():
>>             self[key] = val
>>
>>     def __getitem__(self, key):
>>         pair = dict.__getitem__(key)
>>         return pair.value
>>
>>     def __setitem__(self, key, val):
>>         if key in self:
>>             pair = dict.__getitem__(key)
>>             pair.value = value
>>         else:
>>             pair = Pair(key, val)
>>             dict.__setitem__(self, key, pair)
>>
>>     def values(self):
>>         for key in self:
>>             p = dict.__getitem__(self, key)
>>             yield p.value
>>
>>     def items(self):
>>         for key, p in dict.__iter__(self):
>>             yield p.key, p.value
>>
>>
>> The simple test give me strange result:
>>
>> >>> d = MyDict([('a', 1), ('b', 2), ('c', 3)])
>> >>> dict(d)
>> {'a': <__main__.Pair at 0x104ca9e48>,
>>  'b': <__main__.Pair at 0x104ca9e80>,
>>  'c': <__main__.Pair at 0x104ca9eb8>}
>>
>> instead of {'a':1, 'b':2, 'c':3}.
>>
>>
>> Is this right behavior of the dict?
>>
>
> Yes because in your __setitem__ call you are storing the value as the
> Pair. So when dict prints its repr it prints the key and value, and in this
> case the value is a Pair.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150317/2ca598e9/attachment.html>

From brett at python.org  Tue Mar 17 20:38:59 2015
From: brett at python.org (Brett Cannon)
Date: Tue, 17 Mar 2015 19:38:59 +0000
Subject: [Python-Dev] Possible wrong behavior of the dict?
In-Reply-To: <CAPOE21TcOXv61N9KJYqc7yxzkBnZnFDws6ZPHhqw1FfWQYeZfw@mail.gmail.com>
References: <CAPOE21TuvFU-=fdBWw1jCS+X-EXLOS5pBy8=o5nVr4+A6Hky+Q@mail.gmail.com>
 <CAP1=2W5wiYAEAMqFa4WGvOnMY1dH_p5Sy8piydcwDvt+u2+QDQ@mail.gmail.com>
 <CAPOE21TcOXv61N9KJYqc7yxzkBnZnFDws6ZPHhqw1FfWQYeZfw@mail.gmail.com>
Message-ID: <CAP1=2W4wKZ8Aqn_Jnk_+oFng8yXO671Datjp=SO=S-+a-Q30hQ@mail.gmail.com>

On Tue, Mar 17, 2015 at 3:29 PM Zaur Shibzukhov <szport at gmail.com> wrote:

> Yes... But I expected that dict constructor will use `__getitem__`  or
> `items` method of MyDict instance  in order to retrieve items of the MyDict
> instance during construction of the dict instance... Instead it interpreted
> MyDict instance as the dict instance during construction of new dict.This
> exactly caused my confusion.
>

It's because you subclassed dict. Copying is optimized to skip over using
the methods you listed when the object is a dict and so we know the
structure of the object at the C level. You can look at
https://hg.python.org/cpython/file/22a0c925a7c2/Objects/dictobject.c#l1997
to see the actual code.

-Brett


>
> ---
> *Zaur Shibzukhov*
>
>
> 2015-03-17 22:12 GMT+03:00 Brett Cannon <brett at python.org>:
>
>>
>>
>> On Tue, Mar 17, 2015 at 3:05 PM Zaur Shibzukhov <szport at gmail.com> wrote:
>>
>>> Hello!
>>>
>>> In order to explain, let define subclass of dict:
>>>
>>> class Pair:
>>>     def __init__(self, key, val):
>>>         self.key = key
>>>         self.val = val
>>>
>>> class MyDict(dict):
>>>     #
>>>     def __init__(self, *args, **kwds):
>>>         if len(args) > 1:
>>>             raise TypeError('Expected at most 1 arguments, but got %d' %
>>> len(args))
>>>
>>>         for key, val in args[0]:
>>>             self[key] = val
>>>
>>>         for key, val in kwds.items():
>>>             self[key] = val
>>>
>>>     def __getitem__(self, key):
>>>         pair = dict.__getitem__(key)
>>>         return pair.value
>>>
>>>     def __setitem__(self, key, val):
>>>         if key in self:
>>>             pair = dict.__getitem__(key)
>>>             pair.value = value
>>>         else:
>>>             pair = Pair(key, val)
>>>             dict.__setitem__(self, key, pair)
>>>
>>>     def values(self):
>>>         for key in self:
>>>             p = dict.__getitem__(self, key)
>>>             yield p.value
>>>
>>>     def items(self):
>>>         for key, p in dict.__iter__(self):
>>>             yield p.key, p.value
>>>
>>>
>>> The simple test give me strange result:
>>>
>>> >>> d = MyDict([('a', 1), ('b', 2), ('c', 3)])
>>> >>> dict(d)
>>> {'a': <__main__.Pair at 0x104ca9e48>,
>>>  'b': <__main__.Pair at 0x104ca9e80>,
>>>  'c': <__main__.Pair at 0x104ca9eb8>}
>>>
>>> instead of {'a':1, 'b':2, 'c':3}.
>>>
>>>
>>> Is this right behavior of the dict?
>>>
>>
>> Yes because in your __setitem__ call you are storing the value as the
>> Pair. So when dict prints its repr it prints the key and value, and in this
>> case the value is a Pair.
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150317/a0e65e0f/attachment.html>

From szport at gmail.com  Tue Mar 17 20:45:50 2015
From: szport at gmail.com (Zaur Shibzukhov)
Date: Tue, 17 Mar 2015 22:45:50 +0300
Subject: [Python-Dev] Possible wrong behavior of the dict?
In-Reply-To: <CAP1=2W4wKZ8Aqn_Jnk_+oFng8yXO671Datjp=SO=S-+a-Q30hQ@mail.gmail.com>
References: <CAPOE21TuvFU-=fdBWw1jCS+X-EXLOS5pBy8=o5nVr4+A6Hky+Q@mail.gmail.com>
 <CAP1=2W5wiYAEAMqFa4WGvOnMY1dH_p5Sy8piydcwDvt+u2+QDQ@mail.gmail.com>
 <CAPOE21TcOXv61N9KJYqc7yxzkBnZnFDws6ZPHhqw1FfWQYeZfw@mail.gmail.com>
 <CAP1=2W4wKZ8Aqn_Jnk_+oFng8yXO671Datjp=SO=S-+a-Q30hQ@mail.gmail.com>
Message-ID: <CAPOE21QjfvJzTsd-rOcC7sZBkNE+aSL-NTAe6vK92Xh0DUWXKg@mail.gmail.com>

So in such cases it should not subclassed `dict`, but
`collections.MutableMapping`, for example?

---
*Zaur Shibzukhov*


2015-03-17 22:38 GMT+03:00 Brett Cannon <brett at python.org>:

>
>
> On Tue, Mar 17, 2015 at 3:29 PM Zaur Shibzukhov <szport at gmail.com> wrote:
>
>> Yes... But I expected that dict constructor will use `__getitem__`  or
>> `items` method of MyDict instance  in order to retrieve items of the MyDict
>> instance during construction of the dict instance... Instead it interpreted
>> MyDict instance as the dict instance during construction of new dict.This
>> exactly caused my confusion.
>>
>
> It's because you subclassed dict. Copying is optimized to skip over using
> the methods you listed when the object is a dict and so we know the
> structure of the object at the C level. You can look at
> https://hg.python.org/cpython/file/22a0c925a7c2/Objects/dictobject.c#l1997
> to see the actual code.
>
> -Brett
>
>
>>
>> ---
>> *Zaur Shibzukhov*
>>
>>
>> 2015-03-17 22:12 GMT+03:00 Brett Cannon <brett at python.org>:
>>
>>>
>>>
>>> On Tue, Mar 17, 2015 at 3:05 PM Zaur Shibzukhov <szport at gmail.com>
>>> wrote:
>>>
>>>> Hello!
>>>>
>>>> In order to explain, let define subclass of dict:
>>>>
>>>> class Pair:
>>>>     def __init__(self, key, val):
>>>>         self.key = key
>>>>         self.val = val
>>>>
>>>> class MyDict(dict):
>>>>     #
>>>>     def __init__(self, *args, **kwds):
>>>>         if len(args) > 1:
>>>>             raise TypeError('Expected at most 1 arguments, but got %d'
>>>> % len(args))
>>>>
>>>>         for key, val in args[0]:
>>>>             self[key] = val
>>>>
>>>>         for key, val in kwds.items():
>>>>             self[key] = val
>>>>
>>>>     def __getitem__(self, key):
>>>>         pair = dict.__getitem__(key)
>>>>         return pair.value
>>>>
>>>>     def __setitem__(self, key, val):
>>>>         if key in self:
>>>>             pair = dict.__getitem__(key)
>>>>             pair.value = value
>>>>         else:
>>>>             pair = Pair(key, val)
>>>>             dict.__setitem__(self, key, pair)
>>>>
>>>>     def values(self):
>>>>         for key in self:
>>>>             p = dict.__getitem__(self, key)
>>>>             yield p.value
>>>>
>>>>     def items(self):
>>>>         for key, p in dict.__iter__(self):
>>>>             yield p.key, p.value
>>>>
>>>>
>>>> The simple test give me strange result:
>>>>
>>>> >>> d = MyDict([('a', 1), ('b', 2), ('c', 3)])
>>>> >>> dict(d)
>>>> {'a': <__main__.Pair at 0x104ca9e48>,
>>>>  'b': <__main__.Pair at 0x104ca9e80>,
>>>>  'c': <__main__.Pair at 0x104ca9eb8>}
>>>>
>>>> instead of {'a':1, 'b':2, 'c':3}.
>>>>
>>>>
>>>> Is this right behavior of the dict?
>>>>
>>>
>>> Yes because in your __setitem__ call you are storing the value as the
>>> Pair. So when dict prints its repr it prints the key and value, and in this
>>> case the value is a Pair.
>>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150317/25bbd4da/attachment.html>

From brett at python.org  Tue Mar 17 20:48:14 2015
From: brett at python.org (Brett Cannon)
Date: Tue, 17 Mar 2015 19:48:14 +0000
Subject: [Python-Dev] Possible wrong behavior of the dict?
In-Reply-To: <CAPOE21QjfvJzTsd-rOcC7sZBkNE+aSL-NTAe6vK92Xh0DUWXKg@mail.gmail.com>
References: <CAPOE21TuvFU-=fdBWw1jCS+X-EXLOS5pBy8=o5nVr4+A6Hky+Q@mail.gmail.com>
 <CAP1=2W5wiYAEAMqFa4WGvOnMY1dH_p5Sy8piydcwDvt+u2+QDQ@mail.gmail.com>
 <CAPOE21TcOXv61N9KJYqc7yxzkBnZnFDws6ZPHhqw1FfWQYeZfw@mail.gmail.com>
 <CAP1=2W4wKZ8Aqn_Jnk_+oFng8yXO671Datjp=SO=S-+a-Q30hQ@mail.gmail.com>
 <CAPOE21QjfvJzTsd-rOcC7sZBkNE+aSL-NTAe6vK92Xh0DUWXKg@mail.gmail.com>
Message-ID: <CAP1=2W774a_ypnpMUnLE0yAkzxM+ghoT0_xwZoFPYgGRUfj_dg@mail.gmail.com>

On Tue, Mar 17, 2015 at 3:46 PM Zaur Shibzukhov <szport at gmail.com> wrote:

> So in such cases it should not subclassed `dict`, but
> `collections.MutableMapping`, for example?
>

Yes (see the comment at
https://hg.python.org/cpython/file/22a0c925a7c2/Objects/dictobject.c#l2003
).

-Brett


>
> ---
> *Zaur Shibzukhov*
>
>
> 2015-03-17 22:38 GMT+03:00 Brett Cannon <brett at python.org>:
>
>>
>>
>> On Tue, Mar 17, 2015 at 3:29 PM Zaur Shibzukhov <szport at gmail.com> wrote:
>>
>>> Yes... But I expected that dict constructor will use `__getitem__`  or
>>> `items` method of MyDict instance  in order to retrieve items of the MyDict
>>> instance during construction of the dict instance... Instead it interpreted
>>> MyDict instance as the dict instance during construction of new dict.This
>>> exactly caused my confusion.
>>>
>>
>> It's because you subclassed dict. Copying is optimized to skip over using
>> the methods you listed when the object is a dict and so we know the
>> structure of the object at the C level. You can look at
>> https://hg.python.org/cpython/file/22a0c925a7c2/Objects/dictobject.c#l1997
>> to see the actual code.
>>
>> -Brett
>>
>>
>>>
>>> ---
>>> *Zaur Shibzukhov*
>>>
>>>
>>> 2015-03-17 22:12 GMT+03:00 Brett Cannon <brett at python.org>:
>>>
>>>>
>>>>
>>>> On Tue, Mar 17, 2015 at 3:05 PM Zaur Shibzukhov <szport at gmail.com>
>>>> wrote:
>>>>
>>>>> Hello!
>>>>>
>>>>> In order to explain, let define subclass of dict:
>>>>>
>>>>> class Pair:
>>>>>     def __init__(self, key, val):
>>>>>         self.key = key
>>>>>         self.val = val
>>>>>
>>>>> class MyDict(dict):
>>>>>     #
>>>>>     def __init__(self, *args, **kwds):
>>>>>         if len(args) > 1:
>>>>>             raise TypeError('Expected at most 1 arguments, but got %d'
>>>>> % len(args))
>>>>>
>>>>>         for key, val in args[0]:
>>>>>             self[key] = val
>>>>>
>>>>>         for key, val in kwds.items():
>>>>>             self[key] = val
>>>>>
>>>>>     def __getitem__(self, key):
>>>>>         pair = dict.__getitem__(key)
>>>>>         return pair.value
>>>>>
>>>>>     def __setitem__(self, key, val):
>>>>>         if key in self:
>>>>>             pair = dict.__getitem__(key)
>>>>>             pair.value = value
>>>>>         else:
>>>>>             pair = Pair(key, val)
>>>>>             dict.__setitem__(self, key, pair)
>>>>>
>>>>>     def values(self):
>>>>>         for key in self:
>>>>>             p = dict.__getitem__(self, key)
>>>>>             yield p.value
>>>>>
>>>>>     def items(self):
>>>>>         for key, p in dict.__iter__(self):
>>>>>             yield p.key, p.value
>>>>>
>>>>>
>>>>> The simple test give me strange result:
>>>>>
>>>>> >>> d = MyDict([('a', 1), ('b', 2), ('c', 3)])
>>>>> >>> dict(d)
>>>>> {'a': <__main__.Pair at 0x104ca9e48>,
>>>>>  'b': <__main__.Pair at 0x104ca9e80>,
>>>>>  'c': <__main__.Pair at 0x104ca9eb8>}
>>>>>
>>>>> instead of {'a':1, 'b':2, 'c':3}.
>>>>>
>>>>>
>>>>> Is this right behavior of the dict?
>>>>>
>>>>
>>>> Yes because in your __setitem__ call you are storing the value as the
>>>> Pair. So when dict prints its repr it prints the key and value, and in this
>>>> case the value is a Pair.
>>>>
>>>
>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150317/23888aa7/attachment-0001.html>

From szport at gmail.com  Tue Mar 17 20:50:47 2015
From: szport at gmail.com (Zaur Shibzukhov)
Date: Tue, 17 Mar 2015 22:50:47 +0300
Subject: [Python-Dev] Possible wrong behavior of the dict?
In-Reply-To: <CAP1=2W774a_ypnpMUnLE0yAkzxM+ghoT0_xwZoFPYgGRUfj_dg@mail.gmail.com>
References: <CAPOE21TuvFU-=fdBWw1jCS+X-EXLOS5pBy8=o5nVr4+A6Hky+Q@mail.gmail.com>
 <CAP1=2W5wiYAEAMqFa4WGvOnMY1dH_p5Sy8piydcwDvt+u2+QDQ@mail.gmail.com>
 <CAPOE21TcOXv61N9KJYqc7yxzkBnZnFDws6ZPHhqw1FfWQYeZfw@mail.gmail.com>
 <CAP1=2W4wKZ8Aqn_Jnk_+oFng8yXO671Datjp=SO=S-+a-Q30hQ@mail.gmail.com>
 <CAPOE21QjfvJzTsd-rOcC7sZBkNE+aSL-NTAe6vK92Xh0DUWXKg@mail.gmail.com>
 <CAP1=2W774a_ypnpMUnLE0yAkzxM+ghoT0_xwZoFPYgGRUfj_dg@mail.gmail.com>
Message-ID: <CAPOE21Ti3-xeen+HL1-pJ4Ba0z4a9cF_CVo8+Hv7p=aPBjxmnQ@mail.gmail.com>

Thanks.

---
*Zaur Shibzukhov*


2015-03-17 22:48 GMT+03:00 Brett Cannon <brett at python.org>:

>
>
> On Tue, Mar 17, 2015 at 3:46 PM Zaur Shibzukhov <szport at gmail.com> wrote:
>
>> So in such cases it should not subclassed `dict`, but
>> `collections.MutableMapping`, for example?
>>
>
> Yes (see the comment at
> https://hg.python.org/cpython/file/22a0c925a7c2/Objects/dictobject.c#l2003
> ).
>
> -Brett
>
>
>>
>> ---
>> *Zaur Shibzukhov*
>>
>>
>> 2015-03-17 22:38 GMT+03:00 Brett Cannon <brett at python.org>:
>>
>>>
>>>
>>> On Tue, Mar 17, 2015 at 3:29 PM Zaur Shibzukhov <szport at gmail.com>
>>> wrote:
>>>
>>>> Yes... But I expected that dict constructor will use `__getitem__`  or
>>>> `items` method of MyDict instance  in order to retrieve items of the MyDict
>>>> instance during construction of the dict instance... Instead it interpreted
>>>> MyDict instance as the dict instance during construction of new dict.This
>>>> exactly caused my confusion.
>>>>
>>>
>>> It's because you subclassed dict. Copying is optimized to skip over
>>> using the methods you listed when the object is a dict and so we know the
>>> structure of the object at the C level. You can look at
>>> https://hg.python.org/cpython/file/22a0c925a7c2/Objects/dictobject.c#l1997
>>> to see the actual code.
>>>
>>> -Brett
>>>
>>>
>>>>
>>>> ---
>>>> *Zaur Shibzukhov*
>>>>
>>>>
>>>> 2015-03-17 22:12 GMT+03:00 Brett Cannon <brett at python.org>:
>>>>
>>>>>
>>>>>
>>>>> On Tue, Mar 17, 2015 at 3:05 PM Zaur Shibzukhov <szport at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hello!
>>>>>>
>>>>>> In order to explain, let define subclass of dict:
>>>>>>
>>>>>> class Pair:
>>>>>>     def __init__(self, key, val):
>>>>>>         self.key = key
>>>>>>         self.val = val
>>>>>>
>>>>>> class MyDict(dict):
>>>>>>     #
>>>>>>     def __init__(self, *args, **kwds):
>>>>>>         if len(args) > 1:
>>>>>>             raise TypeError('Expected at most 1 arguments, but got
>>>>>> %d' % len(args))
>>>>>>
>>>>>>         for key, val in args[0]:
>>>>>>             self[key] = val
>>>>>>
>>>>>>         for key, val in kwds.items():
>>>>>>             self[key] = val
>>>>>>
>>>>>>     def __getitem__(self, key):
>>>>>>         pair = dict.__getitem__(key)
>>>>>>         return pair.value
>>>>>>
>>>>>>     def __setitem__(self, key, val):
>>>>>>         if key in self:
>>>>>>             pair = dict.__getitem__(key)
>>>>>>             pair.value = value
>>>>>>         else:
>>>>>>             pair = Pair(key, val)
>>>>>>             dict.__setitem__(self, key, pair)
>>>>>>
>>>>>>     def values(self):
>>>>>>         for key in self:
>>>>>>             p = dict.__getitem__(self, key)
>>>>>>             yield p.value
>>>>>>
>>>>>>     def items(self):
>>>>>>         for key, p in dict.__iter__(self):
>>>>>>             yield p.key, p.value
>>>>>>
>>>>>>
>>>>>> The simple test give me strange result:
>>>>>>
>>>>>> >>> d = MyDict([('a', 1), ('b', 2), ('c', 3)])
>>>>>> >>> dict(d)
>>>>>> {'a': <__main__.Pair at 0x104ca9e48>,
>>>>>>  'b': <__main__.Pair at 0x104ca9e80>,
>>>>>>  'c': <__main__.Pair at 0x104ca9eb8>}
>>>>>>
>>>>>> instead of {'a':1, 'b':2, 'c':3}.
>>>>>>
>>>>>>
>>>>>> Is this right behavior of the dict?
>>>>>>
>>>>>
>>>>> Yes because in your __setitem__ call you are storing the value as the
>>>>> Pair. So when dict prints its repr it prints the key and value, and in this
>>>>> case the value is a Pair.
>>>>>
>>>>
>>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150317/9cc18f85/attachment.html>

From me at the-compiler.org  Tue Mar 17 20:33:46 2015
From: me at the-compiler.org (Florian Bruhin)
Date: Tue, 17 Mar 2015 20:33:46 +0100
Subject: [Python-Dev] Possible wrong behavior of the dict?
In-Reply-To: <CAPOE21TcOXv61N9KJYqc7yxzkBnZnFDws6ZPHhqw1FfWQYeZfw@mail.gmail.com>
References: <CAPOE21TuvFU-=fdBWw1jCS+X-EXLOS5pBy8=o5nVr4+A6Hky+Q@mail.gmail.com>
 <CAP1=2W5wiYAEAMqFa4WGvOnMY1dH_p5Sy8piydcwDvt+u2+QDQ@mail.gmail.com>
 <CAPOE21TcOXv61N9KJYqc7yxzkBnZnFDws6ZPHhqw1FfWQYeZfw@mail.gmail.com>
Message-ID: <20150317193345.GI10871@tonks>

* Zaur Shibzukhov <szport at gmail.com> [2015-03-17 22:29:07 +0300]:
> Yes... But I expected that dict constructor will use `__getitem__`  or
> `items` method of MyDict instance  in order to retrieve items of the MyDict
> instance during construction of the dict instance... Instead it interpreted
> MyDict instance as the dict instance during construction of new dict.This
> exactly caused my confusion.

Subclassing builtins is always a recipe for trouble, because the C
implementation doesn't necessarily call your Python methods.

You should probably use collections.UserDict or
collections.abc.(Mutable)Mapping instead:

https://docs.python.org/3/library/collections.html#collections.UserDict
https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping

Florian

-- 
http://www.the-compiler.org | me at the-compiler.org (Mail/XMPP)
   GPG: 916E B0C8 FD55 A072 | http://the-compiler.org/pubkey.asc
         I love long mails! | http://email.is-not-s.ms/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150317/aa7fc5a0/attachment.sig>

From breamoreboy at yahoo.co.uk  Tue Mar 17 21:41:28 2015
From: breamoreboy at yahoo.co.uk (Mark Lawrence)
Date: Tue, 17 Mar 2015 20:41:28 +0000
Subject: [Python-Dev] PEP 489: Redesigning extension module loading
In-Reply-To: <5506CEB5.7050105@gmail.com>
References: <5506CEB5.7050105@gmail.com>
Message-ID: <mea3ho$2i2$1@ger.gmane.org>

On 16/03/2015 12:38, Petr Viktorin wrote:
> Hello,

Can you use anything from the meta issue 
http://bugs.python.org/issue15787 for PEP 3121 and PEP 384 or will the 
work that you are doing render everything done previously redundant?

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence


From ncoghlan at gmail.com  Wed Mar 18 15:52:05 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 19 Mar 2015 00:52:05 +1000
Subject: [Python-Dev] PEP 448 review
In-Reply-To: <CAP1=2W5yq+GESZXyr9g5BEe64Kq_-ZHBgUb4sZ5QykDoRvp4Ow@mail.gmail.com>
References: <CAP7+vJK58SYgBExknnSO2xwiQm5+AO5_=f0yAarNJUkbmczcPg@mail.gmail.com>
 <CAA68w_m1cB4MmMeNX=zA5MjfaxytgU9+=8fVqEK173DGy3K5Zw@mail.gmail.com>
 <CAMpsgwa8JY3rCobEPOe2CN2ovcQJE7AkHfRPtw5SZE83eF1+fQ@mail.gmail.com>
 <CAA68w_kFVmMQJsk+hdgnKtOui2rqYdecQmaNEjk_UPk7N7Q_xg@mail.gmail.com>
 <CAA68w_ngnkX0KPnDni8Vvi9iG=RgiN4B_0dp9NuEMvaNO6=5OA@mail.gmail.com>
 <CAA68w_ki+sAti_WX1a+ACUCbRexh38a=aQpSG7BgEgbhWmsiig@mail.gmail.com>
 <CAA68w_mpd-7Ai5as3V=6F-7Lw4er0s=H10jdznB4CJ=zJyFcpA@mail.gmail.com>
 <CAP1=2W5yq+GESZXyr9g5BEe64Kq_-ZHBgUb4sZ5QykDoRvp4Ow@mail.gmail.com>
Message-ID: <CADiSq7eYMxOaXxtVa8ObaW-ROmOvYvyZcDPOKAEdfAhsesOb9Q@mail.gmail.com>

On 17 March 2015 at 23:49, Brett Cannon <brett at python.org> wrote:
>
>
> On Mon, Mar 16, 2015 at 7:11 PM Neil Girdhar <mistersheik at gmail.com> wrote:
>>
>> Hi everyone,
>>
>> I was wondering what is left with the PEP 448
>> (http://bugs.python.org/issue2292) code review?  Big thanks to Benjamin,
>> Ethan, and Serhiy for reviewing some (all?) of the code.  What is the next
>> step of this process?
>
>
> My suspicion is that if no one steps up between now and PyCon to do a
> complete code review of the final patch, we as a group will try to get it
> done at the PyCon sprints. I have made the issue a release blocker to help
> make sure it gets reviewed and doesn't accidentally get left behind.

Good idea - I just bumped the PEP 479 issue
(http://bugs.python.org/issue22906) similarly, as well as giving it an
initial review (Neil had already noted it needed tests for the new
behaviour, and the language reference doc updates look surprisingly
minimal to me).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Wed Mar 18 16:01:14 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 19 Mar 2015 01:01:14 +1000
Subject: [Python-Dev] PEP 489: Redesigning extension module loading
In-Reply-To: <mea3ho$2i2$1@ger.gmane.org>
References: <5506CEB5.7050105@gmail.com>
	<mea3ho$2i2$1@ger.gmane.org>
Message-ID: <CADiSq7d9NrH8R6+g+Hocfgfy2oMED6Cmrt5mWdqeOyonc7M23A@mail.gmail.com>

On 18 March 2015 at 06:41, Mark Lawrence <breamoreboy at yahoo.co.uk> wrote:
> On 16/03/2015 12:38, Petr Viktorin wrote:
>>
>> Hello,
>
>
> Can you use anything from the meta issue http://bugs.python.org/issue15787
> for PEP 3121 and PEP 384 or will the work that you are doing render
> everything done previously redundant?

Nothing should break in relation to PEP 3121 or 384, so I think that
determination would still need to be made on a case by case basis.
Alternatively, it may be possible to update the abitype.py converter
to also switch to the new module initialisation hooks (if we can
figure out a good way of automating that).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From orion at cora.nwra.com  Wed Mar 18 16:46:12 2015
From: orion at cora.nwra.com (Orion Poplawski)
Date: Wed, 18 Mar 2015 15:46:12 +0000 (UTC)
Subject: [Python-Dev] Use ptyhon -s as default shbang for system python
	executables/daemons
Message-ID: <loom.20150318T164041-536@post.gmane.org>

We're starting a discussion in Fedora about setting the default shbang for
system python executables and/or daemons to python -s or python -Es (or ?).
 See also:

https://bugzilla.redhat.com/show_bug.cgi?id=1202151
https://fedorahosted.org/fpc/ticket/513

Basically we're wanting to avoid locally installed items causing security
issues or other bad behavior, without too adversely affecting users'
abilities to work around issues or intentionally alter behavior.

It would be good to get some feedback from the broader python community
before implementing anything, so I'm asking for feedback here.

Thank you, 

 Orion


From barry at python.org  Wed Mar 18 17:22:03 2015
From: barry at python.org (Barry Warsaw)
Date: Wed, 18 Mar 2015 12:22:03 -0400
Subject: [Python-Dev] Use ptyhon -s as default shbang for system python
 executables/daemons
In-Reply-To: <loom.20150318T164041-536@post.gmane.org>
References: <loom.20150318T164041-536@post.gmane.org>
Message-ID: <20150318122203.12ff1ee3@limelight.wooz.org>

On Mar 18, 2015, at 03:46 PM, Orion Poplawski wrote:

>We're starting a discussion in Fedora about setting the default shbang for
>system python executables and/or daemons to python -s or python -Es (or ?).

We've talked about this in Debian/Ubuntu land and the general consensus is
that for Python 2, use -Es and for Python 3 use -I (which implies -Es).  I'm
not sure we're consistent yet in making sure our build tools install these
switches in our shebangs, but I'm hoping after Jessie we can make some
progress on that.

Cheers,
-Barry

From victor.stinner at gmail.com  Wed Mar 18 17:31:09 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 18 Mar 2015 17:31:09 +0100
Subject: [Python-Dev] Use ptyhon -s as default shbang for system python
	executables/daemons
In-Reply-To: <loom.20150318T164041-536@post.gmane.org>
References: <loom.20150318T164041-536@post.gmane.org>
Message-ID: <CAMpsgwa3F2Abnptu8uFtxzCD_T-x9BCK9KtuOsVDW3kF1N88-g@mail.gmail.com>

2015-03-18 16:46 GMT+01:00 Orion Poplawski <orion at cora.nwra.com>:
> We're starting a discussion in Fedora about setting the default shbang for
> system python executables and/or daemons to python -s or python -Es (or ?).

Python 3.4 has -I which is more strict than -Es.

It remembers me "Perl suid", /usr/bin/sperl. Maybe we should have a
"spython" program which is like "python -I" (so it means adding
spython, spython3, spython3.5).

Does it work to pass command line options to Python in the shebang?

> Basically we're wanting to avoid locally installed items causing security
> issues or other bad behavior, without too adversely affecting users'
> abilities to work around issues or intentionally alter behavior.
>
> It would be good to get some feedback from the broader python community
> before implementing anything, so I'm asking for feedback here.

Personally, instead of having to enable a switch to have a safe
Python, I would prefer to have a safe Python by default and enable an
option to enter the unsafe mode.

But it may break backward compatibility depending on changes made in
Python if we take this way.

Victor

From barry at python.org  Wed Mar 18 17:48:25 2015
From: barry at python.org (Barry Warsaw)
Date: Wed, 18 Mar 2015 12:48:25 -0400
Subject: [Python-Dev] Use ptyhon -s as default shbang for system python
 executables/daemons
In-Reply-To: <CAMpsgwa3F2Abnptu8uFtxzCD_T-x9BCK9KtuOsVDW3kF1N88-g@mail.gmail.com>
References: <loom.20150318T164041-536@post.gmane.org>
 <CAMpsgwa3F2Abnptu8uFtxzCD_T-x9BCK9KtuOsVDW3kF1N88-g@mail.gmail.com>
Message-ID: <20150318124825.337dc1c6@anarchist.wooz.org>

On Mar 18, 2015, at 05:31 PM, Victor Stinner wrote:

>Does it work to pass command line options to Python in the shebang?

Yes, but only one "word", thus -Es or -I.

We've often mused about whether it makes sense to have two Pythons on the
system.  One for system scripts and another for users.  System Python
('/usr/bin/spython' perhaps) would be locked down and only extensible by
system packages.  On Debuntu that might mean no /usr/local on sys.path.  It
would also have a much more limited set of installed packages, i.e. only those
needed to support system functionality.

/usr/bin/python2 and /usr/bin/python3 then would be user tools, with all the
goodness they currently have.

It's never gotten much farther than musings, but protecting the system against
the weird things people install would be a good thing.  OTOH, this feels a lot
like virtual environments so maybe there's something useful to be mined there.

Cheers,
-Barry

From a.badger at gmail.com  Wed Mar 18 22:44:31 2015
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Wed, 18 Mar 2015 14:44:31 -0700
Subject: [Python-Dev] Use ptyhon -s as default shbang for system python
 executables/daemons
In-Reply-To: <20150318122203.12ff1ee3@limelight.wooz.org>
References: <loom.20150318T164041-536@post.gmane.org>
 <20150318122203.12ff1ee3@limelight.wooz.org>
Message-ID: <20150318214431.GA4050@roan.lan>

On Wed, Mar 18, 2015 at 12:22:03PM -0400, Barry Warsaw wrote:
> On Mar 18, 2015, at 03:46 PM, Orion Poplawski wrote:
> 
> >We're starting a discussion in Fedora about setting the default shbang for
> >system python executables and/or daemons to python -s or python -Es (or ?).
> 
> We've talked about this in Debian/Ubuntu land and the general consensus is
> that for Python 2, use -Es and for Python 3 use -I (which implies -Es).  I'm
> not sure we're consistent yet in making sure our build tools install these
> switches in our shebangs, but I'm hoping after Jessie we can make some
> progress on that.
> 
Interesting, I've cautiously in favor of -s in Fedora but the more I've
thought about it the less I've liked -E.  It just seems like PYTHONPATH is
analagous to LD_LIBRARY_PATH for C programs and PATH for shell scripting.
We leave both of those for local admins and users to affect the behaviour of
programs if they needed to.

Was there some discussion of -E specifically in Debian where it was
consciously decided that PYTHONPATH was not analagous or that the benefit
risk was different than for those other env vars?

-Toshio
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150318/e026e658/attachment.sig>

From donald at stufft.io  Wed Mar 18 22:51:20 2015
From: donald at stufft.io (Donald Stufft)
Date: Wed, 18 Mar 2015 17:51:20 -0400
Subject: [Python-Dev] Use ptyhon -s as default shbang for system python
	executables/daemons
In-Reply-To: <20150318124825.337dc1c6@anarchist.wooz.org>
References: <loom.20150318T164041-536@post.gmane.org>
 <CAMpsgwa3F2Abnptu8uFtxzCD_T-x9BCK9KtuOsVDW3kF1N88-g@mail.gmail.com>
 <20150318124825.337dc1c6@anarchist.wooz.org>
Message-ID: <654D3177-05D2-403F-9531-136FC98ED73F@stufft.io>


> On Mar 18, 2015, at 12:48 PM, Barry Warsaw <barry at python.org> wrote:
> 
> On Mar 18, 2015, at 05:31 PM, Victor Stinner wrote:
> 
>> Does it work to pass command line options to Python in the shebang?
> 
> Yes, but only one "word", thus -Es or -I.
> 
> We've often mused about whether it makes sense to have two Pythons on the
> system.  One for system scripts and another for users.  System Python
> ('/usr/bin/spython' perhaps) would be locked down and only extensible by
> system packages.  On Debuntu that might mean no /usr/local on sys.path.  It
> would also have a much more limited set of installed packages, i.e. only those
> needed to support system functionality.
> 
> /usr/bin/python2 and /usr/bin/python3 then would be user tools, with all the
> goodness they currently have.
> 
> It's never gotten much farther than musings, but protecting the system against
> the weird things people install would be a good thing.  OTOH, this feels a lot
> like virtual environments so maybe there's something useful to be mined there.
> 
> Cheers,
> -Barry
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

I?ve long wished that the OS had it?s own virtual environment. A lot of problems
seems to come from trying to cram the things the OS wants with the things that
the user wants into the same namespace.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150318/6ded5b92/attachment.sig>

From barry at python.org  Wed Mar 18 22:56:07 2015
From: barry at python.org (Barry Warsaw)
Date: Wed, 18 Mar 2015 17:56:07 -0400
Subject: [Python-Dev] Use ptyhon -s as default shbang for system python
 executables/daemons
In-Reply-To: <20150318214431.GA4050@roan.lan>
References: <loom.20150318T164041-536@post.gmane.org>
 <20150318122203.12ff1ee3@limelight.wooz.org>
 <20150318214431.GA4050@roan.lan>
Message-ID: <20150318175607.66a25ad3@limelight.wooz.org>

On Mar 18, 2015, at 02:44 PM, Toshio Kuratomi wrote:

>Interesting, I've cautiously in favor of -s in Fedora but the more I've
>thought about it the less I've liked -E.  It just seems like PYTHONPATH is
>analagous to LD_LIBRARY_PATH for C programs and PATH for shell scripting.
>We leave both of those for local admins and users to affect the behaviour of
>programs if they needed to.

It is, and it isn't.  It's different because you can always explicitly
override the shebang line if needed.  So if a local admin really needed to
override $PYTHONPATH (though I can't come up with a use case right now), they
could just:

$ python3 -s /usr/bin/foo

>Was there some discussion of -E specifically in Debian where it was
>consciously decided that PYTHONPATH was not analagous or that the benefit
>risk was different than for those other env vars?

I'd have to go digging around the archives.  It wasn't a recent discussion
IIRC.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150318/678ef8a2/attachment.sig>

From barry at python.org  Wed Mar 18 22:58:35 2015
From: barry at python.org (Barry Warsaw)
Date: Wed, 18 Mar 2015 17:58:35 -0400
Subject: [Python-Dev] Use ptyhon -s as default shbang for system python
 executables/daemons
In-Reply-To: <654D3177-05D2-403F-9531-136FC98ED73F@stufft.io>
References: <loom.20150318T164041-536@post.gmane.org>
 <CAMpsgwa3F2Abnptu8uFtxzCD_T-x9BCK9KtuOsVDW3kF1N88-g@mail.gmail.com>
 <20150318124825.337dc1c6@anarchist.wooz.org>
 <654D3177-05D2-403F-9531-136FC98ED73F@stufft.io>
Message-ID: <20150318175835.0fc5e10b@limelight.wooz.org>

On Mar 18, 2015, at 05:51 PM, Donald Stufft wrote:

>I?ve long wished that the OS had it?s own virtual environment. A lot of
>problems seems to come from trying to cram the things the OS wants with the
>things that the user wants into the same namespace.

Yep, and those breakages can be difficult to debug.

Comment #47 for Bug 123456: "Oh wait, you installed *what?!*"

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150318/33e6ff38/attachment.sig>

From storchaka at gmail.com  Thu Mar 19 10:28:37 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Thu, 19 Mar 2015 11:28:37 +0200
Subject: [Python-Dev] Needed reviews
Message-ID: <mee4si$rap$1@ger.gmane.org>

Here is list of my ready for review patches.  It is incomplete and 
contains only patches for which I don't expect objections or long 
discussion.  Most of them are relative easy and need only formal review. 
  Most of them wait for a review many months.


https://bugs.python.org/issue23681
     Have -b warn when directly comparing ints and bytes

https://bugs.python.org/issue23676
     Add support of UnicodeTranslateError in standard error handlers

https://bugs.python.org/issue23671
     string.Template doesn't work with the self keyword argument

https://bugs.python.org/issue23637
     Outputting warnings fails when file patch is not ASCII and message 
is unicode on 2.7.

https://bugs.python.org/issue23622
     Deprecate unrecognized backslash+letter escapes in re

https://bugs.python.org/issue23611
     Pickle nested names (e.g. unbound methods) with protocols < 4

https://bugs.python.org/issue23583
     IDLE: printing unicode subclasses broken (again)

https://bugs.python.org/issue23573
     Avoid redundant memory allocations in str.find and like

https://bugs.python.org/issue23509
     Speed up Counter operators

https://bugs.python.org/issue23502
     Tkinter doesn't support large integers (out of 32-bit range)

https://bugs.python.org/issue23488
     Random objects twice as big as necessary on 64-bit builds

https://bugs.python.org/issue23466
     PEP 461: Inconsistency between str and bytes formatting of integers

https://bugs.python.org/issue23419
     Faster default __reduce__ for classes without __init__

https://bugs.python.org/issue23290
     Faster set copying

https://bugs.python.org/issue23252
     Add support of writing to unseekable file (e.g. socket) in zipfile

https://bugs.python.org/issue23502
     pprint: added support for mapping proxy

https://bugs.python.org/issue23001
     Accept mutable bytes-like objects in builtins that for now support 
only read-only bytes-like objects

https://bugs.python.org/issue22995
     Restrict default pickleability. Fail earlier for some types instead 
of producing incorrect data.

https://bugs.python.org/issue22958
     Constructors of weakref mapping classes don't accept "self" and 
"dict" keyword arguments

https://bugs.python.org/issue22831
     Use "with" to avoid possible fd leaks. Large patch with many simple 
changes.

https://bugs.python.org/issue22826
     Support context management protocol in bkfile and simplify 
Tools/freeze/bkfile.py

https://bugs.python.org/issue22721
     pprint output for sets and dicts is not stable

https://bugs.python.org/issue22687
     horrible performance of textwrap.wrap() with a long word

https://bugs.python.org/issue22682
     Add support of KZ1048 (RK1048) encoding

https://bugs.python.org/issue22681
     Add support of KOI8-T encoding

https://bugs.python.org/issue23671
     string.Template doesn't work with the self keyword argument

https://bugs.python.org/issue23171
     Accept arbitrary iterables in cvs.writerow()

https://bugs.python.org/issue23136
     Fix inconsistency in handling week 0 in _strptime()

https://bugs.python.org/issue22557
     Speed up local import

https://bugs.python.org/issue22493
     Deprecate the use of flags not at the start of regular expression

https://bugs.python.org/issue22390
     test.regrtest should complain if a test doesn't remove temporary files

https://bugs.python.org/issue22364
     Improve some re error messages using regex for hints

https://bugs.python.org/issue22115
     Add new methods to trace Tkinter variables

https://bugs.python.org/issue22035
     Fatal error in dbm.gdbm

https://bugs.python.org/issue21802
     Reader of BufferedRWPair is not closed if writer's close() fails

https://bugs.python.org/issue21859
     Add Python implementation of FileIO

https://bugs.python.org/issue21717
     Exclusive mode for ZipFile

https://bugs.python.org/issue21708
     Deprecate nonstandard behavior of a dumbdbm database

https://bugs.python.org/issue21526
     Support new booleans in Tkinter

https://bugs.python.org/issue20168
     Derby: Convert the _tkinter module to use Argument Clinic

https://bugs.python.org/issue20159
     Derby: Convert the ElementTree module to use Argument Clinic

https://bugs.python.org/issue20148
     Derby: Convert the _sre module to use Argument Clinic

https://bugs.python.org/issue19930
     os.makedirs('dir1/dir2', 0) always fails

https://bugs.python.org/issue18684
     Pointers point out of array bound in _sre.c

https://bugs.python.org/issue18473
     Some objects pickled by Python 3.x are not unpicklable in Python 2.x

https://bugs.python.org/issue17711
     Persistent id in pickle with protocol version 0

https://bugs.python.org/issue17530
     pprint could use line continuation for long bytes literals

https://bugs.python.org/issue16314
     Support xz compression in distutils

https://bugs.python.org/issue15490
     Correct __sizeof__ support for StringIO

https://bugs.python.org/issue15133
     Make tkinter.getboolean() and BooleanVar.get() support Tcl_Obj and 
always return bool.

https://bugs.python.org/issue14904
     test_unicode_repr_oflw (in test_bigmem) crashes

https://bugs.python.org/issue14260
     Make re.groupindex non-modifiable.

https://bugs.python.org/issue13583
     Add support of slice indexes in sqlite3.Row

https://bugs.python.org/issue11344
     Add os.path.splitpath(path) function

https://bugs.python.org/issue10803
     Better support of bytearray objects in ctypes

https://bugs.python.org/issue6598
     calling email.utils.make_msgid frequently has a non-trivial 
probability of generating colliding ids

https://bugs.python.org/issue4727
     Use pickle protocol 4 in the copy module for support more types

https://bugs.python.org/issue2175
     Add support of character streams of InputSource object in SAX parsers

https://bugs.python.org/issue433028
     Add support if local flags in regular expressions. (?flag:...)


From a.badger at gmail.com  Thu Mar 19 19:15:39 2015
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Thu, 19 Mar 2015 11:15:39 -0700
Subject: [Python-Dev] Use ptyhon -s as default shbang for system python
	executables/daemons
In-Reply-To: <20150318175607.66a25ad3@limelight.wooz.org>
References: <loom.20150318T164041-536@post.gmane.org>
 <20150318122203.12ff1ee3@limelight.wooz.org>
 <20150318214431.GA4050@roan.lan>
 <20150318175607.66a25ad3@limelight.wooz.org>
Message-ID: <CABVPEKoioRy-kvRXuUpMWTZYSa0sheFDbFoUsECyg=tqyLg-6A@mail.gmail.com>

On Wed, Mar 18, 2015 at 2:56 PM, Barry Warsaw <barry at python.org> wrote:
> On Mar 18, 2015, at 02:44 PM, Toshio Kuratomi wrote:
>
>>Interesting, I've cautiously in favor of -s in Fedora but the more I've
>>thought about it the less I've liked -E.  It just seems like PYTHONPATH is
>>analagous to LD_LIBRARY_PATH for C programs and PATH for shell scripting.
>>We leave both of those for local admins and users to affect the behaviour of
>>programs if they needed to.
>
> It is, and it isn't.  It's different because you can always explicitly
> override the shebang line if needed.  So if a local admin really needed to
> override $PYTHONPATH (though I can't come up with a use case right now), they
> could just:
>
> $ python3 -s /usr/bin/foo
>
I could see that as a difference.  However, the environment variables
give users the ability to change things globally whereas overriding
the shebang line is case-by-case  so it's not a complete replacement
of the functionality.

LD_LIBRARY_PATH can be used for things like logging all calls to a
specific function, applying a bugfix to a library when you don't have
root on the box, evaluating how a potential replacement for a system
library will affect the whole system, and other things that are
supposed to affect a large number of the files on the OS.  PYTHONPATH
can be used for the same purposes as long as -E is not embedded into
the shebang lines. (I suppose you could write a "python" wrapper
script that discards -E... but you'd need root on the box to install
your wrapper [since system packages are encouraged to use the full
path to python rather than env python] and the change would affect
everyone on the box rather than just the person setting the env var).

Using -E by default for all system applications would prevent that.
The benefit of -E is that it isolates the effects of PYTHONPATH to
only specific programs.  However, that benefit doesn't seem as
striking as it first appears (or at least, as it first appeared to me
:-).  Unix env vars have their own method of isolation: if the env var
is marked for export then it is sent to child processes.  If it is not
marked for export then it only affects the current process.  So it
seems like -E isn't adding something new; it's just protecting users
from themselves.  That seems contrary to "consenting adults" (although
distributions are separate entities from python-dev ;-).

What makes -s different from -E?  If you think about it in the context
of users being able to set PYTHONPATH to add libraries that can
override system packages then I think specifying -s for system
packages establishes a default behaviour: The user can override the
defaults but only if they change the environment.  Without -s, this
expectation is violated for libraries in the user site directory.
With -s, the user would have to add the user site directory to
PYTHONPATH if they want the libraries there to override system
packages.

So I guess I'm still leaning towards -E being the wrong choice for
Fedora but Fedora lives within a broader ecosystem of python-providing
distributions.  So I'm interested in seeing whether Debian thought
about these aspects when they decided on using -E or if that would
change anyone's minds and also what other distributions think about
adding -s and/or -E to their packaged applications' shebang lines.

-Toshio

From barry at python.org  Thu Mar 19 20:27:34 2015
From: barry at python.org (Barry Warsaw)
Date: Thu, 19 Mar 2015 15:27:34 -0400
Subject: [Python-Dev] Use ptyhon -s as default shbang for system python
 executables/daemons
In-Reply-To: <CABVPEKoioRy-kvRXuUpMWTZYSa0sheFDbFoUsECyg=tqyLg-6A@mail.gmail.com>
References: <loom.20150318T164041-536@post.gmane.org>
 <20150318122203.12ff1ee3@limelight.wooz.org>
 <20150318214431.GA4050@roan.lan>
 <20150318175607.66a25ad3@limelight.wooz.org>
 <CABVPEKoioRy-kvRXuUpMWTZYSa0sheFDbFoUsECyg=tqyLg-6A@mail.gmail.com>
Message-ID: <20150319152734.5bb6cd79@limelight.wooz.org>

On Mar 19, 2015, at 11:15 AM, Toshio Kuratomi wrote:

>I could see that as a difference.  However, the environment variables
>give users the ability to change things globally whereas overriding
>the shebang line is case-by-case  so it's not a complete replacement
>of the functionality.

You make some good points.  I guess it's a trade-off between flexibility and a
known secure execution environment.  I'm not sure there's a right answer;
different admins might have valid different opinions.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150319/910a8379/attachment.sig>

From sturla.molden at gmail.com  Thu Mar 19 20:57:04 2015
From: sturla.molden at gmail.com (Sturla Molden)
Date: Thu, 19 Mar 2015 19:57:04 +0000 (UTC)
Subject: [Python-Dev] Use ptyhon -s as default shbang for system python
	executables/daemons
References: <loom.20150318T164041-536@post.gmane.org>
Message-ID: <772804275448487310.697539sturla.molden-gmail.com@news.gmane.org>

Orion Poplawski <orion at cora.nwra.com> wrote:

> It would be good to get some feedback from the broader python community
> before implementing anything, so I'm asking for feedback here.

On my systems I have /use/bin/python for the system and ~/anaconda/python
for me. Apple and Ubuntu can do whatever they want with their Python's, I
am not touching them. The main thing is to keep the system Python and the
user Python separate. That is, both the executable and the folder where
packages are installed. Whatever I install for myself can fuck up
~/anaconda, but is not allowed to mess with the system files or system
folders.

Sturla


From a.badger at gmail.com  Thu Mar 19 21:47:51 2015
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Thu, 19 Mar 2015 13:47:51 -0700
Subject: [Python-Dev] Use ptyhon -s as default shbang for system python
	executables/daemons
In-Reply-To: <20150319152734.5bb6cd79@limelight.wooz.org>
References: <loom.20150318T164041-536@post.gmane.org>
 <20150318122203.12ff1ee3@limelight.wooz.org>
 <20150318214431.GA4050@roan.lan>
 <20150318175607.66a25ad3@limelight.wooz.org>
 <CABVPEKoioRy-kvRXuUpMWTZYSa0sheFDbFoUsECyg=tqyLg-6A@mail.gmail.com>
 <20150319152734.5bb6cd79@limelight.wooz.org>
Message-ID: <CABVPEKo3edWRCLadA9_WVNvM6QZsAvToBjQGaOQXWxYC5F4qvg@mail.gmail.com>

I think I've found the Debian discussion (October 2012):

http://comments.gmane.org/gmane.linux.debian.devel.python/8188

Lack of PYTHONWARNINGS was brought up late in the discussion thread
but I think the understanding that when a particular user sets an
environment variable they want it to apply to all scripts they run was
kind of lost in the followups (It wasn't directly addressed or
mentioned again.)

-Toshio

On Thu, Mar 19, 2015 at 12:27 PM, Barry Warsaw <barry at python.org> wrote:
> On Mar 19, 2015, at 11:15 AM, Toshio Kuratomi wrote:
>
>>I could see that as a difference.  However, the environment variables
>>give users the ability to change things globally whereas overriding
>>the shebang line is case-by-case  so it's not a complete replacement
>>of the functionality.
>
> You make some good points.  I guess it's a trade-off between flexibility and a
> known secure execution environment.  I'm not sure there's a right answer;
> different admins might have valid different opinions.
>
> Cheers,
> -Barry
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/a.badger%40gmail.com
>

From cs at zip.com.au  Thu Mar 19 22:46:19 2015
From: cs at zip.com.au (Cameron Simpson)
Date: Fri, 20 Mar 2015 08:46:19 +1100
Subject: [Python-Dev] Use ptyhon -s as default shbang for system python
 executables/daemons
In-Reply-To: <772804275448487310.697539sturla.molden-gmail.com@news.gmane.org>
References: <772804275448487310.697539sturla.molden-gmail.com@news.gmane.org>
Message-ID: <20150319214619.GA12976@cskk.homeip.net>

On 19Mar2015 19:57, Sturla Molden <sturla.molden at gmail.com> wrote:
>Orion Poplawski <orion at cora.nwra.com> wrote:
>> It would be good to get some feedback from the broader python community
>> before implementing anything, so I'm asking for feedback here.
>
>On my systems I have /use/bin/python for the system and ~/anaconda/python
>for me. Apple and Ubuntu can do whatever they want with their Python's, I
>am not touching them. The main thing is to keep the system Python and the
>user Python separate. That is, both the executable and the folder where
>packages are installed. Whatever I install for myself can fuck up
>~/anaconda, but is not allowed to mess with the system files or system
>folders.

Me too. I keep my default virtualenvs in ~/var/venv, and have personal "py2" 
and "py3" scripts that invoke via the venvs.

Having a similar separation in the system would be a good thing, for the same 
reasons.

Also, it would let the OS supplier keep a _much_ smaller package/addon list for 
the "core" admin python instance, making QA etc easier and more reliable.  
Popular OSes let the local sysadmin (== "the user") pull in all sorts of stuff 
to the "system" Python, even from the supplier's repositories. Having a walled 
off "core" admin python as well seems very prudent.

Cheers,
Cameron Simpson <cs at zip.com.au>

conclude that this language as a tool is an open invitation for clever
tricks; and while exactly this may be the explanation for some of its appeal,
/viz./ to those who like to show how clever they are. I am sorry, but I must
regard this as one of the most damning things that can be said about a
programming language.
        - Edsger Dijkstra, _The Humble Programmer_ (CACM, Ocotber, 1972)

From victor.stinner at gmail.com  Thu Mar 19 23:27:26 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 19 Mar 2015 23:27:26 +0100
Subject: [Python-Dev] Use ptyhon -s as default shbang for system python
	executables/daemons
In-Reply-To: <CABVPEKo3edWRCLadA9_WVNvM6QZsAvToBjQGaOQXWxYC5F4qvg@mail.gmail.com>
References: <loom.20150318T164041-536@post.gmane.org>
 <20150318122203.12ff1ee3@limelight.wooz.org>
 <20150318214431.GA4050@roan.lan> <20150318175607.66a25ad3@limelight.wooz.org>
 <CABVPEKoioRy-kvRXuUpMWTZYSa0sheFDbFoUsECyg=tqyLg-6A@mail.gmail.com>
 <20150319152734.5bb6cd79@limelight.wooz.org>
 <CABVPEKo3edWRCLadA9_WVNvM6QZsAvToBjQGaOQXWxYC5F4qvg@mail.gmail.com>
Message-ID: <CAMpsgwYx8RdrMOTQdS8bm7Qyj1+JjEf-Oig9-gx_s1vK15MD1w@mail.gmail.com>

2015-03-19 21:47 GMT+01:00 Toshio Kuratomi <a.badger at gmail.com>:
> I think I've found the Debian discussion (October 2012):
>
> http://comments.gmane.org/gmane.linux.debian.devel.python/8188
>
> Lack of PYTHONWARNINGS was brought up late in the discussion thread

Maybe we need to modify -E or add a new option to only ignore PYTHONPATH.

Victor

From techtonik at gmail.com  Fri Mar 20 14:31:29 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Fri, 20 Mar 2015 16:31:29 +0300
Subject: [Python-Dev] 2.7.9 regression in argparse
Message-ID: <CAPkN8xLKuy+2vs69PfcQ_0MqAQozHEVvBZkaYHyxu-owROub7Q@mail.gmail.com>

Just a pointer for possible regression http://bugs.python.org/issue23058

-- 
anatoly t.

From lkb.teichmann at gmail.com  Fri Mar 20 15:03:05 2015
From: lkb.teichmann at gmail.com (Martin Teichmann)
Date: Fri, 20 Mar 2015 15:03:05 +0100
Subject: [Python-Dev] super() does not work during class initialization
Message-ID: <CAK9R32QaTyqsugF1_3-pPHfDsaEa4JEp0A_DFBboNOFwP_x82Q@mail.gmail.com>

Hi list,

while a class is being initialized in a metaclass, it is not always possible to
call classmethods of the class, as they might use super(), which in turn uses
__class__, which is not initialized yet.

I know that this is a known issue, but well, sometimes it even makes sense
to fix already known issues... so I wrote a patch that moves the initialization
of __class__ into type.__new__, so that one may use super() in a class
once it starts existing. It's issue 23722 on the issue tracker.

To illustrate what the problem is, the following code raises a RuntimeError:

class Meta(type):
    def __init__(self, name, bases, dict):
        super().__init__(name, bases, dict)
        self.f()

class A(metaclass=Meta):
    @classmethod
    def f(self):
        super()

it works fine with my patch applied.

Technically, my patch slightly changes the semantics of python if a metaclass
returns something different in its __new__ than what type.__new__ returns.
But I could not find any documentation of the current behavior, and also the
tests don't test for it, and I consider the current behavior actually buggy.
As an example let me give the following simple Singleton metaclass:

class Singleton(type):
    def __new__(cls, name, bases, dict):
        return super().__new__(cls, name, bases, dict)()

class A(metaclass=Singleton):
    def test(self):
        assert(isinstance(__class__, type))

A.test()

The current python fails the assertion, while with my patch everything is fine,
and I personally think __class__ should always actually refer to the class being
defined, which means at the minimum that it is actually, well, a class.

Greetings

Martin

From status at bugs.python.org  Fri Mar 20 18:08:19 2015
From: status at bugs.python.org (Python tracker)
Date: Fri, 20 Mar 2015 18:08:19 +0100 (CET)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20150320170819.CEDD65666D@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2015-03-13 - 2015-03-20)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    4798 (-19)
  closed 30685 (+81)
  total  35483 (+62)

Open issues with patches: 2258 


Issues opened (49)
==================

#2211: Cookie.Morsel interface needs update
http://bugs.python.org/issue2211  reopened by haypo

#22181: os.urandom() should use Linux 3.17 getrandom() syscall
http://bugs.python.org/issue22181  reopened by haypo

#22359: Remove incorrect uses of recursive make
http://bugs.python.org/issue22359  reopened by doko

#22516: Windows Installer won't - even when using "just for me"option
http://bugs.python.org/issue22516  reopened by NaCl

#23187: Segmentation fault, possibly asyncio related
http://bugs.python.org/issue23187  reopened by haypo

#23637: Warnings error with non-ascii chars.
http://bugs.python.org/issue23637  reopened by serhiy.storchaka

#23661: Setting a exception side_effect on a mock from create_autospec
http://bugs.python.org/issue23661  opened by Ignacio Rossi

#23662: Cookie.domain is undocumented
http://bugs.python.org/issue23662  opened by Malcolm Smith

#23663: Crash on failure in ctypes on Cygwin
http://bugs.python.org/issue23663  opened by elieux

#23664: Modernize HTML output of difflib.HtmlDiff.make_file()
http://bugs.python.org/issue23664  opened by berker.peksag

#23665: Provide IDLE menu option to set command-line arguments
http://bugs.python.org/issue23665  opened by rhettinger

#23666: Add shell session logging option to IDLE
http://bugs.python.org/issue23666  opened by rhettinger

#23667: IDLE to provide option for making trailing whitespace visible
http://bugs.python.org/issue23667  opened by rhettinger

#23668: Support os.[f]truncate on Windows
http://bugs.python.org/issue23668  opened by steve.dower

#23669: test_socket.NonBlockingTCPTests failing due to race condition
http://bugs.python.org/issue23669  opened by steve.dower

#23670: Modifications to support iOS as a cross-compilation target
http://bugs.python.org/issue23670  opened by freakboy3742

#23671: string.Template doesn't work with the self keyword argument
http://bugs.python.org/issue23671  opened by serhiy.storchaka

#23672: IDLE can crash if file name contains non-BMP Unicode character
http://bugs.python.org/issue23672  opened by kamisky

#23674: super() documentation isn't very clear
http://bugs.python.org/issue23674  opened by Tapani Kiiskinen

#23675: glossary entry for 'method resolution order' links to somethin
http://bugs.python.org/issue23675  opened by r.david.murray

#23676: Add support of UnicodeTranslateError in standard error handler
http://bugs.python.org/issue23676  opened by serhiy.storchaka

#23677: Mention dict and set comps in library reference
http://bugs.python.org/issue23677  opened by FrankMillman

#23680: Sporadic freeze in test_interrupted_write_retry_text
http://bugs.python.org/issue23680  opened by pitrou

#23684: urlparse() documentation does not account for default scheme
http://bugs.python.org/issue23684  opened by vadmium

#23686: Update Windows and OS X installer OpenSSL to 1.0.2a
http://bugs.python.org/issue23686  opened by alex

#23688: unnecessary copying of memoryview in gzip.GzipFile.write?
http://bugs.python.org/issue23688  opened by wolma

#23689: Memory leak in Modules/sre_lib.h
http://bugs.python.org/issue23689  opened by abacabadabacaba

#23690: re functions never release GIL
http://bugs.python.org/issue23690  opened by abacabadabacaba

#23691: re.finditer iterator is not reentrant, but doesn't protect aga
http://bugs.python.org/issue23691  opened by abacabadabacaba

#23692: Undocumented feature prevents re module from finding certain m
http://bugs.python.org/issue23692  opened by abacabadabacaba

#23693: timeit accuracy could be better
http://bugs.python.org/issue23693  opened by rbcollins

#23695: idiom for clustering a data series into n-length groups
http://bugs.python.org/issue23695  opened by Paddy McCarthy

#23697: Module level map & submit for concurrent.futures
http://bugs.python.org/issue23697  opened by ncoghlan

#23698: Fix inconsistencies in behaviour and document it correctly for
http://bugs.python.org/issue23698  opened by pythonhacker

#23699: Add a macro to ease writing rich comparisons
http://bugs.python.org/issue23699  opened by encukou

#23701: Drop extraneous comment from winreg.QueryValue's docstring
http://bugs.python.org/issue23701  opened by Claudiu.Popa

#23702: docs.python.org/3/howto/descriptor.html still refers to "unbou
http://bugs.python.org/issue23702  opened by pfalcon

#23703: urljoin() with no directory segments duplicates filename
http://bugs.python.org/issue23703  opened by vadmium

#23704: Make deques a full MutableSequence by adding index(), insert()
http://bugs.python.org/issue23704  opened by rhettinger

#23705: Speed-up deque.__contains__
http://bugs.python.org/issue23705  opened by rhettinger

#23706: pathlib.Path.write_text should include a newline argument
http://bugs.python.org/issue23706  opened by daniel.ugra

#23710: C API doc for PyObject_HEAD is outdated
http://bugs.python.org/issue23710  opened by tpievila

#23712: Experiment:  Assume that exact unicode hashes are perfect disc
http://bugs.python.org/issue23712  opened by rhettinger

#23713: intermittent failure of multiprocessing unit test test_imap_un
http://bugs.python.org/issue23713  opened by davin

#23717: strptime() with year-weekday pair can produce invalid data
http://bugs.python.org/issue23717  opened by serhiy.storchaka

#23718: strptime() can produce invalid date with negative year day
http://bugs.python.org/issue23718  opened by serhiy.storchaka

#23719: PEP 475: port test_eintr to Windows
http://bugs.python.org/issue23719  opened by haypo

#23721: Set up a daily test coverage run
http://bugs.python.org/issue23721  opened by brett.cannon

#23722: During metaclass.__init__, super() of the constructed class do
http://bugs.python.org/issue23722  opened by Martin.Teichmann



Most recent 15 issues with no replies (15)
==========================================

#23722: During metaclass.__init__, super() of the constructed class do
http://bugs.python.org/issue23722

#23721: Set up a daily test coverage run
http://bugs.python.org/issue23721

#23719: PEP 475: port test_eintr to Windows
http://bugs.python.org/issue23719

#23718: strptime() can produce invalid date with negative year day
http://bugs.python.org/issue23718

#23710: C API doc for PyObject_HEAD is outdated
http://bugs.python.org/issue23710

#23706: pathlib.Path.write_text should include a newline argument
http://bugs.python.org/issue23706

#23705: Speed-up deque.__contains__
http://bugs.python.org/issue23705

#23704: Make deques a full MutableSequence by adding index(), insert()
http://bugs.python.org/issue23704

#23702: docs.python.org/3/howto/descriptor.html still refers to "unbou
http://bugs.python.org/issue23702

#23701: Drop extraneous comment from winreg.QueryValue's docstring
http://bugs.python.org/issue23701

#23697: Module level map & submit for concurrent.futures
http://bugs.python.org/issue23697

#23692: Undocumented feature prevents re module from finding certain m
http://bugs.python.org/issue23692

#23691: re.finditer iterator is not reentrant, but doesn't protect aga
http://bugs.python.org/issue23691

#23675: glossary entry for 'method resolution order' links to somethin
http://bugs.python.org/issue23675

#23669: test_socket.NonBlockingTCPTests failing due to race condition
http://bugs.python.org/issue23669



Most recent 15 issues waiting for review (15)
=============================================

#23713: intermittent failure of multiprocessing unit test test_imap_un
http://bugs.python.org/issue23713

#23712: Experiment:  Assume that exact unicode hashes are perfect disc
http://bugs.python.org/issue23712

#23705: Speed-up deque.__contains__
http://bugs.python.org/issue23705

#23704: Make deques a full MutableSequence by adding index(), insert()
http://bugs.python.org/issue23704

#23703: urljoin() with no directory segments duplicates filename
http://bugs.python.org/issue23703

#23701: Drop extraneous comment from winreg.QueryValue's docstring
http://bugs.python.org/issue23701

#23699: Add a macro to ease writing rich comparisons
http://bugs.python.org/issue23699

#23689: Memory leak in Modules/sre_lib.h
http://bugs.python.org/issue23689

#23688: unnecessary copying of memoryview in gzip.GzipFile.write?
http://bugs.python.org/issue23688

#23677: Mention dict and set comps in library reference
http://bugs.python.org/issue23677

#23676: Add support of UnicodeTranslateError in standard error handler
http://bugs.python.org/issue23676

#23671: string.Template doesn't work with the self keyword argument
http://bugs.python.org/issue23671

#23670: Modifications to support iOS as a cross-compilation target
http://bugs.python.org/issue23670

#23668: Support os.[f]truncate on Windows
http://bugs.python.org/issue23668

#23663: Crash on failure in ctypes on Cygwin
http://bugs.python.org/issue23663



Top 10 most discussed issues (10)
=================================

#23688: unnecessary copying of memoryview in gzip.GzipFile.write?
http://bugs.python.org/issue23688  14 msgs

#23187: Segmentation fault, possibly asyncio related
http://bugs.python.org/issue23187  13 msgs

#2211: Cookie.Morsel interface needs update
http://bugs.python.org/issue2211  12 msgs

#23648: PEP 475 meta issue
http://bugs.python.org/issue23648  11 msgs

#23670: Modifications to support iOS as a cross-compilation target
http://bugs.python.org/issue23670  11 msgs

#23441: rlcompleter: tab on empty prefix => insert spaces
http://bugs.python.org/issue23441   9 msgs

#23546: Windows, 'Edit withIDLE', and multplie installed versions
http://bugs.python.org/issue23546   9 msgs

#23680: Sporadic freeze in test_interrupted_write_retry_text
http://bugs.python.org/issue23680   9 msgs

#8027: distutils fail to determine c++ linker with unixcompiler if us
http://bugs.python.org/issue8027   8 msgs

#23042: ctypes module doesn't build on FreeBSD x86
http://bugs.python.org/issue23042   8 msgs



Issues closed (78)
==================

#2052: Allow changing difflib._file_template character encoding.
http://bugs.python.org/issue2052  closed by berker.peksag

#12155: queue example doesn't stop worker threads
http://bugs.python.org/issue12155  closed by haypo

#12304: expose signalfd(2) in the signal module
http://bugs.python.org/issue12304  closed by haypo

#12500: Skip test_ssl.test_connect_ex() on connection error
http://bugs.python.org/issue12500  closed by haypo

#14555: clock_gettime/settime/getres: Add more clock identifiers
http://bugs.python.org/issue14555  closed by haypo

#15612: Rewrite StringIO to use the _PyUnicodeWriter API
http://bugs.python.org/issue15612  closed by haypo

#17515: Add sys.setasthook() to allow to use a custom AST optimizer
http://bugs.python.org/issue17515  closed by haypo

#17628: str==str: compare the first character before calling memcmp()
http://bugs.python.org/issue17628  closed by haypo

#17911: traceback: add a new thin class storing a traceback without st
http://bugs.python.org/issue17911  closed by rbcollins

#18507: import_init() should not use Py_FatalError() but return an err
http://bugs.python.org/issue18507  closed by haypo

#18983: Specify time unit for timeit CLI
http://bugs.python.org/issue18983  closed by rbcollins

#19233: test_io.test_interrupted_write_retry_text() hangs on Solaris 1
http://bugs.python.org/issue19233  closed by haypo

#19428: marshal: error cases are not documented
http://bugs.python.org/issue19428  closed by haypo

#19564: test_context() of test_multiprocessing_spawn hangs on "x86 Gen
http://bugs.python.org/issue19564  closed by haypo

#19654: test_tkinter sporadic skipped on "x86 Tiger 3.x" buildbot
http://bugs.python.org/issue19654  closed by haypo

#19748: test_time failures on AIX
http://bugs.python.org/issue19748  closed by haypo

#19816: test.regrtest: use tracemalloc to detect memory leaks?
http://bugs.python.org/issue19816  closed by haypo

#20114: Sporadic failure of test_semaphore_tracker() of test_multiproc
http://bugs.python.org/issue20114  closed by haypo

#20204: pydocs fails for some C implemented classes
http://bugs.python.org/issue20204  closed by serhiy.storchaka

#20600: test_create_server_ssl_verify_failed() failure on "PPC64 AIX 3
http://bugs.python.org/issue20600  closed by haypo

#20614: test.script_helper should copy SYSTEMROOT environment variable
http://bugs.python.org/issue20614  closed by haypo

#20680: Pickle socket enums by names
http://bugs.python.org/issue20680  closed by ethan.furman

#20717: test_4_daemon_threads() fails randomly on "x86 Windows Server 
http://bugs.python.org/issue20717  closed by haypo

#20719: testPythonOrg() of test_robotparser fails with the new www.pyt
http://bugs.python.org/issue20719  closed by haypo

#21644: Optimize bytearray(int) constructor to use calloc()
http://bugs.python.org/issue21644  closed by haypo

#21734: compilation of the _ctypes module fails on OpenIndiana: ffi_pr
http://bugs.python.org/issue21734  closed by haypo

#21735: test_threading.test_main_thread_after_fork_from_nonmain_thread
http://bugs.python.org/issue21735  closed by haypo

#21788: Rework Python finalization
http://bugs.python.org/issue21788  closed by haypo

#22333: test_threaded_import.test_parallel_meta_path() failed on x86 W
http://bugs.python.org/issue22333  closed by haypo

#22521: ctypes compilation fails on FreeBSD: Undefined symbol "ffi_cal
http://bugs.python.org/issue22521  closed by haypo

#22634: importing _ctypes failed: undefined symbol: ffi_call_win32
http://bugs.python.org/issue22634  closed by berker.peksag

#22826: Support context management protocol in bkfile
http://bugs.python.org/issue22826  closed by serhiy.storchaka

#22861: [2.7] ssl._dnsname_match() and unicode
http://bugs.python.org/issue22861  closed by haypo

#22903: unittest creates non-picklable errors
http://bugs.python.org/issue22903  closed by pitrou

#22937: unittest errors can't show locals
http://bugs.python.org/issue22937  closed by rbcollins

#22971: test_pickle: "Fatal Python error: Cannot recover from stack ov
http://bugs.python.org/issue22971  closed by haypo

#22984: test_json.test_endless_recursion(): "Fatal Python error: Canno
http://bugs.python.org/issue22984  closed by haypo

#23001: Accept mutable bytes-like objects
http://bugs.python.org/issue23001  closed by serhiy.storchaka

#23136: BUG in how _strptime() handles week 0
http://bugs.python.org/issue23136  closed by serhiy.storchaka

#23285: PEP 475 - EINTR handling
http://bugs.python.org/issue23285  closed by haypo

#23303: test_license_exists_at_url() of test_site fails on "x86 XP-4 3
http://bugs.python.org/issue23303  closed by haypo

#23333: asyncio: add a new Protocol.connection_failed() method
http://bugs.python.org/issue23333  closed by haypo

#23353: generator bug with exception: tstate->exc_value is not cleared
http://bugs.python.org/issue23353  closed by pitrou

#23419: Faster default __reduce__ for classes without __init__
http://bugs.python.org/issue23419  closed by serhiy.storchaka

#23456: asyncio: add missing @coroutine decorators
http://bugs.python.org/issue23456  closed by haypo

#23549: heapq docs should be more precise about how to access the smal
http://bugs.python.org/issue23549  closed by eli.bendersky

#23568: unittest.mock.MagicMock doesn't support __rdivmod__
http://bugs.python.org/issue23568  closed by berker.peksag

#23585: patchcheck doesn't depend on all
http://bugs.python.org/issue23585  closed by rbcollins

#23593: Update Windows and OS X installer OpenSSL to 1.0.2
http://bugs.python.org/issue23593  closed by ned.deily

#23621: Uninstalling Python 3.5 removes a "py.exe" that was installed 
http://bugs.python.org/issue23621  closed by steve.dower

#23624: str.center inconsistent with format "^"
http://bugs.python.org/issue23624  closed by rhettinger

#23628: See if os.scandir() could help speed up importlib
http://bugs.python.org/issue23628  closed by brett.cannon

#23631: 3.5 (a2) traceback regression snarls Idle
http://bugs.python.org/issue23631  closed by rbcollins

#23632: memoryview doesn't allow tuple-indexing
http://bugs.python.org/issue23632  closed by pitrou

#23644: g++ module compile fails with ???_Atomic??? does not name a ty
http://bugs.python.org/issue23644  closed by haypo

#23645: Incorrect doc for __getslice__
http://bugs.python.org/issue23645  closed by python-dev

#23646: PEP 475: handle EINTR in the time module, retry sleep()
http://bugs.python.org/issue23646  closed by haypo

#23656: shutil.copyfile (or built-in open) releases sqlite3's transact
http://bugs.python.org/issue23656  closed by h-miyajima

#23658: multiprocessing: string arg to SystemExit
http://bugs.python.org/issue23658  closed by Dan Nawrocki

#23673: IntEnum is unpicklable by previous Python versions
http://bugs.python.org/issue23673  closed by ethan.furman

#23678: imaplib status command cannot handle folder name containing wh
http://bugs.python.org/issue23678  closed by r.david.murray

#23679: SSL Ciphers RC4
http://bugs.python.org/issue23679  closed by benjamin.peterson

#23681: Have -b warn when directly comparing ints and bytes
http://bugs.python.org/issue23681  closed by serhiy.storchaka

#23682: distutils docs still talk about compatibility with Python 2.2
http://bugs.python.org/issue23682  closed by berker.peksag

#23683: allow timeit to run expensive reset code per repeats
http://bugs.python.org/issue23683  closed by rbcollins

#23685: Fix usage of PyMODINIT_FUNC
http://bugs.python.org/issue23685  closed by haypo

#23687: Stacktrace identifies wrong line in multiline list comprehensi
http://bugs.python.org/issue23687  closed by r.david.murray

#23694: PEP 475: handle EINTR in fileutils.c
http://bugs.python.org/issue23694  closed by haypo

#23696: zipimport: chain ImportError to OSError
http://bugs.python.org/issue23696  closed by haypo

#23700: tempfile.NamedTemporaryFile can close too early if used as ite
http://bugs.python.org/issue23700  closed by serhiy.storchaka

#23707: PEP 475: os.urandom() doesn't handle EINTR correctly
http://bugs.python.org/issue23707  closed by haypo

#23708: PEP 475: Add _Py_read() and _Py_write() functions
http://bugs.python.org/issue23708  closed by haypo

#23709: Refactor ossaudiodev: use _Py_read and _Py_write with the Py_b
http://bugs.python.org/issue23709  closed by haypo

#23711: ConfigParser module blames on section less ini file
http://bugs.python.org/issue23711  closed by vadmium

#23714: Set comprehension + eval doesn't work inside function definiti
http://bugs.python.org/issue23714  closed by ned.deily

#23715: PEP 475: handle EINTR in the signal module
http://bugs.python.org/issue23715  closed by haypo

#23716: test_multiprocessing_spawn hangs on x86 Ubuntu Shared 3.x
http://bugs.python.org/issue23716  closed by haypo

#23720: __del__() order is broken since 3.4.0
http://bugs.python.org/issue23720  closed by pitrou

From bcannon at gmail.com  Fri Mar 20 19:34:38 2015
From: bcannon at gmail.com (Brett Cannon)
Date: Fri, 20 Mar 2015 18:34:38 +0000
Subject: [Python-Dev] Final call for PEP 488: eliminating PYO files
Message-ID: <CAP1=2W4CejbRcjjYUoFdFReUrgZ8i_mm1vJE2aoVqqo0Q3BW=Q@mail.gmail.com>

I have decided to have the default case of no optimization levels mean that
the .pyc file name will have *no* optimization level specified in the name
and thus be just as it is today. I made this decision due to potential
backwards-compatibility issues -- although I expect them to be minutes --
and to not force other implementations like PyPy to have some bogus value
set since they don't have .pyo files to begin with (PyPy actually uses
bytecode for -O and don't bother with -OO since PyPy already uses a bunch
of memory when running).

Since this closes out the last open issue, I need either a BDFL decision or
a BDFAP to be assigned to make a decision. Guido?

======================================

PEP: 488
Title: Elimination of PYO files
Version: $Revision$
Last-Modified: $Date$
Author: Brett Cannon <brett at python.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 20-Feb-2015
Post-History:
    2015-03-06
    2015-03-13
    2015-03-20

Abstract
========

This PEP proposes eliminating the concept of PYO files from Python.
To continue the support of the separation of bytecode files based on
their optimization level, this PEP proposes extending the PYC file
name to include the optimization level in the bytecode repository
directory when it's called for (i.e., the ``__pycache__`` directory).


Rationale
=========

As of today, bytecode files come in two flavours: PYC and PYO. A PYC
file is the bytecode file generated and read from when no
optimization level is specified at interpreter startup (i.e., ``-O``
is not specified). A PYO file represents the bytecode file that is
read/written when **any** optimization level is specified (i.e., when
``-O`` **or** ``-OO`` is specified). This means that while PYC
files clearly delineate the optimization level used when they were
generated -- namely no optimizations beyond the peepholer -- the same
is not true for PYO files. To put this in terms of optimization
levels and the file extension:

  - 0: ``.pyc``
  - 1 (``-O``): ``.pyo``
  - 2 (``-OO``): ``.pyo``

The reuse of the ``.pyo`` file extension for both level 1 and 2
optimizations means that there is no clear way to tell what
optimization level was used to generate the bytecode file. In terms
of reading PYO files, this can lead to an interpreter using a mixture
of optimization levels with its code if the user was not careful to
make sure all PYO files were generated using the same optimization
level (typically done by blindly deleting all PYO files and then
using the `compileall` module to compile all-new PYO files [1]_).
This issue is only compounded when people optimize Python code beyond
what the interpreter natively supports, e.g., using the astoptimizer
project [2]_.

In terms of writing PYO files, the need to delete all PYO files
every time one either changes the optimization level they want to use
or are unsure of what optimization was used the last time PYO files
were generated leads to unnecessary file churn. The change proposed
by this PEP also allows for **all** optimization levels to be
pre-compiled for bytecode files ahead of time, something that is
currently impossible thanks to the reuse of the ``.pyo`` file
extension for multiple optimization levels.

As for distributing bytecode-only modules, having to distribute both
``.pyc`` and ``.pyo`` files is unnecessary for the common use-case
of code obfuscation and smaller file deployments. This means that
bytecode-only modules will only load from their non-optimized
``.pyc`` file name.


Proposal
========

To eliminate the ambiguity that PYO files present, this PEP proposes
eliminating the concept of PYO files and their accompanying ``.pyo``
file extension. To allow for the optimization level to be unambiguous
as well as to avoid having to regenerate optimized bytecode files
needlessly in the `__pycache__` directory, the optimization level
used to generate the bytecode file will be incorporated into the
bytecode file name. When no optimization level is specified, the
pre-PEP ``.pyc`` file name will be used (i.e., no change in file name
semantics). This increases backwards-compatibility while also being
more understanding of Python implementations which have no use for
optimization levels (e.g., PyPy[10]_).

Currently bytecode file names are created by
``importlib.util.cache_from_source()``, approximately using the
following expression defined by PEP 3147 [3]_, [4]_, [5]_::

    '{name}.{cache_tag}.pyc'.format(name=module_name,
                                    cache_tag=sys.implementation.cache_tag)

This PEP proposes to change the expression when an optimization
level is specified to::

    '{name}.{cache_tag}.opt-{optimization}.pyc'.format(
            name=module_name,
            cache_tag=sys.implementation.cache_tag,
            optimization=str(sys.flags.optimize))

The "opt-" prefix was chosen so as to provide a visual separator
from the cache tag. The placement of the optimization level after
the cache tag was chosen to preserve lexicographic sort order of
bytecode file names based on module name and cache tag which will
not vary for a single interpreter. The "opt-" prefix was chosen over
"o" so as to be somewhat self-documenting. The "opt-" prefix was
chosen over "O" so as to not have any confusion in case "0" was the
leading prefix of the optimization level.

A period was chosen over a hyphen as a separator so as to distinguish
clearly that the optimization level is not part of the interpreter
version as specified by the cache tag. It also lends to the use of
the period in the file name to delineate semantically different
concepts.

For example, if ``-OO`` had been passed to the interpreter then instead
of ``importlib.cpython-35.pyo`` the file name would be
``importlib.cpython-35.opt-2.pyc``.

It should be noted that this change in no way affects the performance
of import. Since the import system looks for a single bytecode file
based on the optimization level of the interpreter already and
generates a new bytecode file if it doesn't exist, the introduction
of potentially more bytecode files in the ``__pycache__`` directory
has no effect in terms of stat calls. The interpreter will continue
to look for only a single bytecode file based on the optimization
level and thus no increase in stat calls will occur.

The only potentially negative result of this PEP is the probable
increase in the number of ``.pyc`` files and thus increase in storage
use. But for platforms where this is an issue,
``sys.dont_write_bytecode`` exists to turn off bytecode generation so
that it can be controlled offline.


Implementation
==============

importlib
---------

As ``importlib.util.cache_from_source()`` is the API that exposes
bytecode file paths as well as being directly used by importlib, it
requires the most critical change. As of Python 3.4, the function's
signature is::

  importlib.util.cache_from_source(path, debug_override=None)

This PEP proposes changing the signature in Python 3.5 to::

  importlib.util.cache_from_source(path, debug_override=None, *,
optimization=None)

The introduced ``optimization`` keyword-only parameter will control
what optimization level is specified in the file name. If the
argument is ``None`` then the current optimization level of the
interpreter will be assumed (including no optimization). Any argument
given for ``optimization`` will be passed to ``str()`` and must have
``str.isalnum()`` be true, else ``ValueError`` will be raised (this
prevents invalid characters being used in the file name). If the
empty string is passed in for ``optimization`` then the addition of
the optimization will be suppressed, reverting to the file name
format which predates this PEP.

It is expected that beyond Python's own two optimization levels,
third-party code will use a hash of optimization names to specify the
optimization level, e.g.
``hashlib.sha256(','.join(['no dead code', 'const folding'])).hexdigest()``.
While this might lead to long file names, it is assumed that most
users never look at the contents of the __pycache__ directory and so
this won't be an issue.

The ``debug_override`` parameter will be deprecated. As the parameter
expects a boolean, the integer value of the boolean will be used as
if it had been provided as the argument to ``optimization`` (a
``None`` argument will mean the same as for ``optimization``). A
deprecation warning will be raised when ``debug_override`` is given a
value other than ``None``, but there are no plans for the complete
removal of the parameter at this time (but removal will be no later
than Python 4).

The various module attributes for importlib.machinery which relate to
bytecode file suffixes will be updated [7]_. The
``DEBUG_BYTECODE_SUFFIXES`` and ``OPTIMIZED_BYTECODE_SUFFIXES`` will
both be documented as deprecated and set to the same value as
``BYTECODE_SUFFIXES`` (removal of ``DEBUG_BYTECODE_SUFFIXES`` and
``OPTIMIZED_BYTECODE_SUFFIXES`` is not currently planned, but will be
not later than Python 4).

All various finders and loaders will also be updated as necessary,
but updating the previous mentioned parts of importlib should be all
that is required.


Rest of the standard library
----------------------------

The various functions exposed by the ``py_compile`` and
``compileall`` functions will be updated as necessary to make sure
they follow the new bytecode file name semantics [6]_, [1]_. The CLI
for the ``compileall`` module will not be directly affected (the
``-b`` flag will be implicit as it will no longer generate ``.pyo``
files when ``-O`` is specified).


Compatibility Considerations
============================

Any code directly manipulating bytecode files from Python 3.2 on
will need to consider the impact of this change on their code (prior
to Python 3.2 -- including all of Python 2 -- there was no
__pycache__ which already necessitates bifurcating bytecode file
handling support). If code was setting the ``debug_override``
argument to ``importlib.util.cache_from_source()`` then care will be
needed if they want the path to a bytecode file with an optimization
level of 2. Otherwise only code **not** using
``importlib.util.cache_from_source()`` will need updating.

As for people who distribute bytecode-only modules (i.e., use a
bytecode file instead of a source file), they will have to choose
which optimization level they want their bytecode files to be since
distributing a ``.pyo`` file with a ``.pyc`` file will no longer be
of any use. Since people typically only distribute bytecode files for
code obfuscation purposes or smaller distribution size then only
having to distribute a single ``.pyc`` should actually be beneficial
to these use-cases. And since the magic number for bytecode files
changed in Python 3.5 to support PEP 465 there is no need to support
pre-existing ``.pyo`` files [8]_.


Rejected Ideas
==============

Completely dropping optimization levels from CPython
----------------------------------------------------

Some have suggested that instead of accommodating the various
optimization levels in CPython, we should instead drop them
entirely. The argument is that significant performance gains would
occur from runtime optimizations through something like a JIT and not
through pre-execution bytecode optimizations.

This idea is rejected for this PEP as that ignores the fact that
there are people who do find the pre-existing optimization levels for
CPython useful. It also assumes that no other Python interpreter
would find what this PEP proposes useful.


Alternative formatting of the optimization level in the file name
-----------------------------------------------------------------

Using the "opt-" prefix and placing the optimization level between
the cache tag and file extension is not critical. All options which
have been considered are:

* ``importlib.cpython-35.opt-1.pyc``
* ``importlib.cpython-35.opt1.pyc``
* ``importlib.cpython-35.o1.pyc``
* ``importlib.cpython-35.O1.pyc``
* ``importlib.cpython-35.1.pyc``
* ``importlib.cpython-35-O1.pyc``
* ``importlib.O1.cpython-35.pyc``
* ``importlib.o1.cpython-35.pyc``
* ``importlib.1.cpython-35.pyc``

These were initially rejected either because they would change the
sort order of bytecode files, possible ambiguity with the cache tag,
or were not self-documenting enough. An informal poll was taken and
people clearly preferred the formatting proposed by the PEP [9]_.
Since this topic is non-technical and of personal choice, the issue
is considered solved.


Embedding the optimization level in the bytecode metadata
---------------------------------------------------------

Some have suggested that rather than embedding the optimization level
of bytecode in the file name that it be included in the file's
metadata instead. This would mean every interpreter had a single copy
of bytecode at any time. Changing the optimization level would thus
require rewriting the bytecode, but there would also only be a single
file to care about.

This has been rejected due to the fact that Python is often installed
as a root-level application and thus modifying the bytecode file for
modules in the standard library are always possible. In this
situation integrators would need to guess at what a reasonable
optimization level was for users for any/all situations. By
allowing multiple optimization levels to co-exist simultaneously it
frees integrators from having to guess what users want and allows
users to utilize the optimization level they want.


References
==========

.. [1] The compileall module
   (https://docs.python.org/3/library/compileall.html#module-compileall)

.. [2] The astoptimizer project
   (https://pypi.python.org/pypi/astoptimizer)

.. [3] ``importlib.util.cache_from_source()``
   (
https://docs.python.org/3.5/library/importlib.html#importlib.util.cache_from_source
)

.. [4] Implementation of ``importlib.util.cache_from_source()`` from
CPython 3.4.3rc1
   (
https://hg.python.org/cpython/file/038297948389/Lib/importlib/_bootstrap.py#l437
)

.. [5] PEP 3147, PYC Repository Directories, Warsaw
   (http://www.python.org/dev/peps/pep-3147)

.. [6] The py_compile module
   (https://docs.python.org/3/library/compileall.html#module-compileall)

.. [7] The importlib.machinery module
   (
https://docs.python.org/3/library/importlib.html#module-importlib.machinery)

.. [8] ``importlib.util.MAGIC_NUMBER``
   (
https://docs.python.org/3/library/importlib.html#importlib.util.MAGIC_NUMBER
)

.. [9] Informal poll of file name format options on Google+
   (https://plus.google.com/u/0/+BrettCannon/posts/fZynLNwHWGm)

.. [10] The PyPy Project
   (http://pypy.org/)


Copyright
=========

This document has been placed in the public domain.


..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End:
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150320/90740722/attachment-0001.html>

From guido at python.org  Fri Mar 20 21:41:30 2015
From: guido at python.org (Guido van Rossum)
Date: Fri, 20 Mar 2015 13:41:30 -0700
Subject: [Python-Dev] Final call for PEP 488: eliminating PYO files
In-Reply-To: <CAP1=2W4CejbRcjjYUoFdFReUrgZ8i_mm1vJE2aoVqqo0Q3BW=Q@mail.gmail.com>
References: <CAP1=2W4CejbRcjjYUoFdFReUrgZ8i_mm1vJE2aoVqqo0Q3BW=Q@mail.gmail.com>
Message-ID: <CAP7+vJLOaJAzXDq-sedZR9KsLN5kkokYxB8YFx3TPjayB7ggmA@mail.gmail.com>

I am willing to be the BDFL for this PEP. I have tried to skim the recent
discussion (only python-dev) and I don't see much remaining controversy.
HOWEVER... The PEP is not clear (or at least too subtle) about the actual
name for optimization level 0. If I have foo.py, and I compile it three
times with three different optimization levels (no optimization; -O; -OO),
and then I look in __pycache__, would I see this:

# (1)
foo.cpython-35.pyc
foo.cpython-35.opt-1.pyc
foo.cpython-35.opt-2.pyc

Or would I see this?

# (2)
foo.cpython-35.opt-0.pyc
foo.cpython-35.opt-1.pyc
foo.cpython-35.opt-2.pyc

Your lead-in ("I have decided to have the default case of no optimization
levels mean that the .pyc file name will have *no* optimization level
specified in the name and thus be just as it is today.") makes me think I
should expect (1), but I can't actually pinpoint where the language of the
PEP says this.


On Fri, Mar 20, 2015 at 11:34 AM, Brett Cannon <bcannon at gmail.com> wrote:

> I have decided to have the default case of no optimization levels mean
> that the .pyc file name will have *no* optimization level specified in
> the name and thus be just as it is today. I made this decision due to
> potential backwards-compatibility issues -- although I expect them to be
> minutes -- and to not force other implementations like PyPy to have some
> bogus value set since they don't have .pyo files to begin with (PyPy
> actually uses bytecode for -O and don't bother with -OO since PyPy already
> uses a bunch of memory when running).
>
> Since this closes out the last open issue, I need either a BDFL decision
> or a BDFAP to be assigned to make a decision. Guido?
>
> ======================================
>
> PEP: 488
> Title: Elimination of PYO files
> Version: $Revision$
> Last-Modified: $Date$
> Author: Brett Cannon <brett at python.org>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 20-Feb-2015
> Post-History:
>     2015-03-06
>     2015-03-13
>     2015-03-20
>
> Abstract
> ========
>
> This PEP proposes eliminating the concept of PYO files from Python.
> To continue the support of the separation of bytecode files based on
> their optimization level, this PEP proposes extending the PYC file
> name to include the optimization level in the bytecode repository
> directory when it's called for (i.e., the ``__pycache__`` directory).
>
>
> Rationale
> =========
>
> As of today, bytecode files come in two flavours: PYC and PYO. A PYC
> file is the bytecode file generated and read from when no
> optimization level is specified at interpreter startup (i.e., ``-O``
> is not specified). A PYO file represents the bytecode file that is
> read/written when **any** optimization level is specified (i.e., when
> ``-O`` **or** ``-OO`` is specified). This means that while PYC
> files clearly delineate the optimization level used when they were
> generated -- namely no optimizations beyond the peepholer -- the same
> is not true for PYO files. To put this in terms of optimization
> levels and the file extension:
>
>   - 0: ``.pyc``
>   - 1 (``-O``): ``.pyo``
>   - 2 (``-OO``): ``.pyo``
>
> The reuse of the ``.pyo`` file extension for both level 1 and 2
> optimizations means that there is no clear way to tell what
> optimization level was used to generate the bytecode file. In terms
> of reading PYO files, this can lead to an interpreter using a mixture
> of optimization levels with its code if the user was not careful to
> make sure all PYO files were generated using the same optimization
> level (typically done by blindly deleting all PYO files and then
> using the `compileall` module to compile all-new PYO files [1]_).
> This issue is only compounded when people optimize Python code beyond
> what the interpreter natively supports, e.g., using the astoptimizer
> project [2]_.
>
> In terms of writing PYO files, the need to delete all PYO files
> every time one either changes the optimization level they want to use
> or are unsure of what optimization was used the last time PYO files
> were generated leads to unnecessary file churn. The change proposed
> by this PEP also allows for **all** optimization levels to be
> pre-compiled for bytecode files ahead of time, something that is
> currently impossible thanks to the reuse of the ``.pyo`` file
> extension for multiple optimization levels.
>
> As for distributing bytecode-only modules, having to distribute both
> ``.pyc`` and ``.pyo`` files is unnecessary for the common use-case
> of code obfuscation and smaller file deployments. This means that
> bytecode-only modules will only load from their non-optimized
> ``.pyc`` file name.
>
>
> Proposal
> ========
>
> To eliminate the ambiguity that PYO files present, this PEP proposes
> eliminating the concept of PYO files and their accompanying ``.pyo``
> file extension. To allow for the optimization level to be unambiguous
> as well as to avoid having to regenerate optimized bytecode files
> needlessly in the `__pycache__` directory, the optimization level
> used to generate the bytecode file will be incorporated into the
> bytecode file name. When no optimization level is specified, the
> pre-PEP ``.pyc`` file name will be used (i.e., no change in file name
> semantics). This increases backwards-compatibility while also being
> more understanding of Python implementations which have no use for
> optimization levels (e.g., PyPy[10]_).
>
> Currently bytecode file names are created by
> ``importlib.util.cache_from_source()``, approximately using the
> following expression defined by PEP 3147 [3]_, [4]_, [5]_::
>
>     '{name}.{cache_tag}.pyc'.format(name=module_name,
>                                     cache_tag=sys.implementation.cache_tag)
>
> This PEP proposes to change the expression when an optimization
> level is specified to::
>
>     '{name}.{cache_tag}.opt-{optimization}.pyc'.format(
>             name=module_name,
>             cache_tag=sys.implementation.cache_tag,
>             optimization=str(sys.flags.optimize))
>
> The "opt-" prefix was chosen so as to provide a visual separator
> from the cache tag. The placement of the optimization level after
> the cache tag was chosen to preserve lexicographic sort order of
> bytecode file names based on module name and cache tag which will
> not vary for a single interpreter. The "opt-" prefix was chosen over
> "o" so as to be somewhat self-documenting. The "opt-" prefix was
> chosen over "O" so as to not have any confusion in case "0" was the
> leading prefix of the optimization level.
>
> A period was chosen over a hyphen as a separator so as to distinguish
> clearly that the optimization level is not part of the interpreter
> version as specified by the cache tag. It also lends to the use of
> the period in the file name to delineate semantically different
> concepts.
>
> For example, if ``-OO`` had been passed to the interpreter then instead
> of ``importlib.cpython-35.pyo`` the file name would be
> ``importlib.cpython-35.opt-2.pyc``.
>
> It should be noted that this change in no way affects the performance
> of import. Since the import system looks for a single bytecode file
> based on the optimization level of the interpreter already and
> generates a new bytecode file if it doesn't exist, the introduction
> of potentially more bytecode files in the ``__pycache__`` directory
> has no effect in terms of stat calls. The interpreter will continue
> to look for only a single bytecode file based on the optimization
> level and thus no increase in stat calls will occur.
>
> The only potentially negative result of this PEP is the probable
> increase in the number of ``.pyc`` files and thus increase in storage
> use. But for platforms where this is an issue,
> ``sys.dont_write_bytecode`` exists to turn off bytecode generation so
> that it can be controlled offline.
>
>
> Implementation
> ==============
>
> importlib
> ---------
>
> As ``importlib.util.cache_from_source()`` is the API that exposes
> bytecode file paths as well as being directly used by importlib, it
> requires the most critical change. As of Python 3.4, the function's
> signature is::
>
>   importlib.util.cache_from_source(path, debug_override=None)
>
> This PEP proposes changing the signature in Python 3.5 to::
>
>   importlib.util.cache_from_source(path, debug_override=None, *,
> optimization=None)
>
> The introduced ``optimization`` keyword-only parameter will control
> what optimization level is specified in the file name. If the
> argument is ``None`` then the current optimization level of the
> interpreter will be assumed (including no optimization). Any argument
> given for ``optimization`` will be passed to ``str()`` and must have
> ``str.isalnum()`` be true, else ``ValueError`` will be raised (this
> prevents invalid characters being used in the file name). If the
> empty string is passed in for ``optimization`` then the addition of
> the optimization will be suppressed, reverting to the file name
> format which predates this PEP.
>
> It is expected that beyond Python's own two optimization levels,
> third-party code will use a hash of optimization names to specify the
> optimization level, e.g.
> ``hashlib.sha256(','.join(['no dead code', 'const
> folding'])).hexdigest()``.
> While this might lead to long file names, it is assumed that most
> users never look at the contents of the __pycache__ directory and so
> this won't be an issue.
>
> The ``debug_override`` parameter will be deprecated. As the parameter
> expects a boolean, the integer value of the boolean will be used as
> if it had been provided as the argument to ``optimization`` (a
> ``None`` argument will mean the same as for ``optimization``). A
> deprecation warning will be raised when ``debug_override`` is given a
> value other than ``None``, but there are no plans for the complete
> removal of the parameter at this time (but removal will be no later
> than Python 4).
>
> The various module attributes for importlib.machinery which relate to
> bytecode file suffixes will be updated [7]_. The
> ``DEBUG_BYTECODE_SUFFIXES`` and ``OPTIMIZED_BYTECODE_SUFFIXES`` will
> both be documented as deprecated and set to the same value as
> ``BYTECODE_SUFFIXES`` (removal of ``DEBUG_BYTECODE_SUFFIXES`` and
> ``OPTIMIZED_BYTECODE_SUFFIXES`` is not currently planned, but will be
> not later than Python 4).
>
> All various finders and loaders will also be updated as necessary,
> but updating the previous mentioned parts of importlib should be all
> that is required.
>
>
> Rest of the standard library
> ----------------------------
>
> The various functions exposed by the ``py_compile`` and
> ``compileall`` functions will be updated as necessary to make sure
> they follow the new bytecode file name semantics [6]_, [1]_. The CLI
> for the ``compileall`` module will not be directly affected (the
> ``-b`` flag will be implicit as it will no longer generate ``.pyo``
> files when ``-O`` is specified).
>
>
> Compatibility Considerations
> ============================
>
> Any code directly manipulating bytecode files from Python 3.2 on
> will need to consider the impact of this change on their code (prior
> to Python 3.2 -- including all of Python 2 -- there was no
> __pycache__ which already necessitates bifurcating bytecode file
> handling support). If code was setting the ``debug_override``
> argument to ``importlib.util.cache_from_source()`` then care will be
> needed if they want the path to a bytecode file with an optimization
> level of 2. Otherwise only code **not** using
> ``importlib.util.cache_from_source()`` will need updating.
>
> As for people who distribute bytecode-only modules (i.e., use a
> bytecode file instead of a source file), they will have to choose
> which optimization level they want their bytecode files to be since
> distributing a ``.pyo`` file with a ``.pyc`` file will no longer be
> of any use. Since people typically only distribute bytecode files for
> code obfuscation purposes or smaller distribution size then only
> having to distribute a single ``.pyc`` should actually be beneficial
> to these use-cases. And since the magic number for bytecode files
> changed in Python 3.5 to support PEP 465 there is no need to support
> pre-existing ``.pyo`` files [8]_.
>
>
> Rejected Ideas
> ==============
>
> Completely dropping optimization levels from CPython
> ----------------------------------------------------
>
> Some have suggested that instead of accommodating the various
> optimization levels in CPython, we should instead drop them
> entirely. The argument is that significant performance gains would
> occur from runtime optimizations through something like a JIT and not
> through pre-execution bytecode optimizations.
>
> This idea is rejected for this PEP as that ignores the fact that
> there are people who do find the pre-existing optimization levels for
> CPython useful. It also assumes that no other Python interpreter
> would find what this PEP proposes useful.
>
>
> Alternative formatting of the optimization level in the file name
> -----------------------------------------------------------------
>
> Using the "opt-" prefix and placing the optimization level between
> the cache tag and file extension is not critical. All options which
> have been considered are:
>
> * ``importlib.cpython-35.opt-1.pyc``
> * ``importlib.cpython-35.opt1.pyc``
> * ``importlib.cpython-35.o1.pyc``
> * ``importlib.cpython-35.O1.pyc``
> * ``importlib.cpython-35.1.pyc``
> * ``importlib.cpython-35-O1.pyc``
> * ``importlib.O1.cpython-35.pyc``
> * ``importlib.o1.cpython-35.pyc``
> * ``importlib.1.cpython-35.pyc``
>
> These were initially rejected either because they would change the
> sort order of bytecode files, possible ambiguity with the cache tag,
> or were not self-documenting enough. An informal poll was taken and
> people clearly preferred the formatting proposed by the PEP [9]_.
> Since this topic is non-technical and of personal choice, the issue
> is considered solved.
>
>
> Embedding the optimization level in the bytecode metadata
> ---------------------------------------------------------
>
> Some have suggested that rather than embedding the optimization level
> of bytecode in the file name that it be included in the file's
> metadata instead. This would mean every interpreter had a single copy
> of bytecode at any time. Changing the optimization level would thus
> require rewriting the bytecode, but there would also only be a single
> file to care about.
>
> This has been rejected due to the fact that Python is often installed
> as a root-level application and thus modifying the bytecode file for
> modules in the standard library are always possible. In this
> situation integrators would need to guess at what a reasonable
> optimization level was for users for any/all situations. By
> allowing multiple optimization levels to co-exist simultaneously it
> frees integrators from having to guess what users want and allows
> users to utilize the optimization level they want.
>
>
> References
> ==========
>
> .. [1] The compileall module
>    (https://docs.python.org/3/library/compileall.html#module-compileall)
>
> .. [2] The astoptimizer project
>    (https://pypi.python.org/pypi/astoptimizer)
>
> .. [3] ``importlib.util.cache_from_source()``
>    (
> https://docs.python.org/3.5/library/importlib.html#importlib.util.cache_from_source
> )
>
> .. [4] Implementation of ``importlib.util.cache_from_source()`` from
> CPython 3.4.3rc1
>    (
> https://hg.python.org/cpython/file/038297948389/Lib/importlib/_bootstrap.py#l437
> )
>
> .. [5] PEP 3147, PYC Repository Directories, Warsaw
>    (http://www.python.org/dev/peps/pep-3147)
>
> .. [6] The py_compile module
>    (https://docs.python.org/3/library/compileall.html#module-compileall)
>
> .. [7] The importlib.machinery module
>    (
> https://docs.python.org/3/library/importlib.html#module-importlib.machinery
> )
>
> .. [8] ``importlib.util.MAGIC_NUMBER``
>    (
> https://docs.python.org/3/library/importlib.html#importlib.util.MAGIC_NUMBER
> )
>
> .. [9] Informal poll of file name format options on Google+
>    (https://plus.google.com/u/0/+BrettCannon/posts/fZynLNwHWGm)
>
> .. [10] The PyPy Project
>    (http://pypy.org/)
>
>
> Copyright
> =========
>
> This document has been placed in the public domain.
>
>
> ..
>    Local Variables:
>    mode: indented-text
>    indent-tabs-mode: nil
>    sentence-end-double-space: t
>    fill-column: 70
>    coding: utf-8
>    End:
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150320/b288eb6d/attachment.html>

From tjreedy at udel.edu  Fri Mar 20 22:35:52 2015
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 20 Mar 2015 17:35:52 -0400
Subject: [Python-Dev] 2.7.9 regression in argparse
In-Reply-To: <CAPkN8xLKuy+2vs69PfcQ_0MqAQozHEVvBZkaYHyxu-owROub7Q@mail.gmail.com>
References: <CAPkN8xLKuy+2vs69PfcQ_0MqAQozHEVvBZkaYHyxu-owROub7Q@mail.gmail.com>
Message-ID: <mei3s3$psv$1@ger.gmane.org>

On 3/20/2015 9:31 AM, anatoly techtonik wrote:
> Just a pointer for possible regression http://bugs.python.org/issue23058

I just added the argparse maintainer to the nosy list

-- 
Terry Jan Reedy


From bcannon at gmail.com  Sat Mar 21 00:00:24 2015
From: bcannon at gmail.com (Brett Cannon)
Date: Fri, 20 Mar 2015 23:00:24 +0000
Subject: [Python-Dev] Final call for PEP 488: eliminating PYO files
In-Reply-To: <CAP7+vJLOaJAzXDq-sedZR9KsLN5kkokYxB8YFx3TPjayB7ggmA@mail.gmail.com>
References: <CAP1=2W4CejbRcjjYUoFdFReUrgZ8i_mm1vJE2aoVqqo0Q3BW=Q@mail.gmail.com>
 <CAP7+vJLOaJAzXDq-sedZR9KsLN5kkokYxB8YFx3TPjayB7ggmA@mail.gmail.com>
Message-ID: <CAP1=2W5CpPdszgqrsbhgx5nQXCm5bOYUH7aXZ0vfdK3FyOk7Dg@mail.gmail.com>

On Fri, Mar 20, 2015 at 4:41 PM Guido van Rossum <guido at python.org> wrote:

> I am willing to be the BDFL for this PEP. I have tried to skim the recent
> discussion (only python-dev) and I don't see much remaining controversy.
> HOWEVER... The PEP is not clear (or at least too subtle) about the actual
> name for optimization level 0. If I have foo.py, and I compile it three
> times with three different optimization levels (no optimization; -O; -OO),
> and then I look in __pycache__, would I see this:
>
> # (1)
> foo.cpython-35.pyc
> foo.cpython-35.opt-1.pyc
> foo.cpython-35.opt-2.pyc
>
> Or would I see this?
>
> # (2)
> foo.cpython-35.opt-0.pyc
> foo.cpython-35.opt-1.pyc
> foo.cpython-35.opt-2.pyc
>

#1


>
> Your lead-in ("I have decided to have the default case of no optimization
> levels mean that the .pyc file name will have *no* optimization level
> specified in the name and thus be just as it is today.") makes me think I
> should expect (1), but I can't actually pinpoint where the language of the
> PEP says this.
>

It was meant to be explained by "When no optimization level is specified,
the pre-PEP ``.pyc`` file name will be used (i.e., no change in file name
semantics)", but obviously it's a bit too subtle. I just updated the PEP
with an explicit list of bytecode file name examples based on no -O, -O,
and -OO.

-Brett


>
>
> On Fri, Mar 20, 2015 at 11:34 AM, Brett Cannon <bcannon at gmail.com> wrote:
>
>> I have decided to have the default case of no optimization levels mean
>> that the .pyc file name will have *no* optimization level specified in
>> the name and thus be just as it is today. I made this decision due to
>> potential backwards-compatibility issues -- although I expect them to be
>> minutes -- and to not force other implementations like PyPy to have some
>> bogus value set since they don't have .pyo files to begin with (PyPy
>> actually uses bytecode for -O and don't bother with -OO since PyPy already
>> uses a bunch of memory when running).
>>
>> Since this closes out the last open issue, I need either a BDFL decision
>> or a BDFAP to be assigned to make a decision. Guido?
>>
>> ======================================
>>
>> PEP: 488
>> Title: Elimination of PYO files
>> Version: $Revision$
>> Last-Modified: $Date$
>> Author: Brett Cannon <brett at python.org>
>> Status: Draft
>> Type: Standards Track
>> Content-Type: text/x-rst
>> Created: 20-Feb-2015
>> Post-History:
>>     2015-03-06
>>     2015-03-13
>>     2015-03-20
>>
>> Abstract
>> ========
>>
>> This PEP proposes eliminating the concept of PYO files from Python.
>> To continue the support of the separation of bytecode files based on
>> their optimization level, this PEP proposes extending the PYC file
>> name to include the optimization level in the bytecode repository
>> directory when it's called for (i.e., the ``__pycache__`` directory).
>>
>>
>> Rationale
>> =========
>>
>> As of today, bytecode files come in two flavours: PYC and PYO. A PYC
>> file is the bytecode file generated and read from when no
>> optimization level is specified at interpreter startup (i.e., ``-O``
>> is not specified). A PYO file represents the bytecode file that is
>> read/written when **any** optimization level is specified (i.e., when
>> ``-O`` **or** ``-OO`` is specified). This means that while PYC
>> files clearly delineate the optimization level used when they were
>> generated -- namely no optimizations beyond the peepholer -- the same
>> is not true for PYO files. To put this in terms of optimization
>> levels and the file extension:
>>
>>   - 0: ``.pyc``
>>   - 1 (``-O``): ``.pyo``
>>   - 2 (``-OO``): ``.pyo``
>>
>> The reuse of the ``.pyo`` file extension for both level 1 and 2
>> optimizations means that there is no clear way to tell what
>> optimization level was used to generate the bytecode file. In terms
>> of reading PYO files, this can lead to an interpreter using a mixture
>> of optimization levels with its code if the user was not careful to
>> make sure all PYO files were generated using the same optimization
>> level (typically done by blindly deleting all PYO files and then
>> using the `compileall` module to compile all-new PYO files [1]_).
>> This issue is only compounded when people optimize Python code beyond
>> what the interpreter natively supports, e.g., using the astoptimizer
>> project [2]_.
>>
>> In terms of writing PYO files, the need to delete all PYO files
>> every time one either changes the optimization level they want to use
>> or are unsure of what optimization was used the last time PYO files
>> were generated leads to unnecessary file churn. The change proposed
>> by this PEP also allows for **all** optimization levels to be
>> pre-compiled for bytecode files ahead of time, something that is
>> currently impossible thanks to the reuse of the ``.pyo`` file
>> extension for multiple optimization levels.
>>
>> As for distributing bytecode-only modules, having to distribute both
>> ``.pyc`` and ``.pyo`` files is unnecessary for the common use-case
>> of code obfuscation and smaller file deployments. This means that
>> bytecode-only modules will only load from their non-optimized
>> ``.pyc`` file name.
>>
>>
>> Proposal
>> ========
>>
>> To eliminate the ambiguity that PYO files present, this PEP proposes
>> eliminating the concept of PYO files and their accompanying ``.pyo``
>> file extension. To allow for the optimization level to be unambiguous
>> as well as to avoid having to regenerate optimized bytecode files
>> needlessly in the `__pycache__` directory, the optimization level
>> used to generate the bytecode file will be incorporated into the
>> bytecode file name. When no optimization level is specified, the
>> pre-PEP ``.pyc`` file name will be used (i.e., no change in file name
>> semantics). This increases backwards-compatibility while also being
>> more understanding of Python implementations which have no use for
>> optimization levels (e.g., PyPy[10]_).
>>
>> Currently bytecode file names are created by
>> ``importlib.util.cache_from_source()``, approximately using the
>> following expression defined by PEP 3147 [3]_, [4]_, [5]_::
>>
>>     '{name}.{cache_tag}.pyc'.format(name=module_name,
>>
>> cache_tag=sys.implementation.cache_tag)
>>
>> This PEP proposes to change the expression when an optimization
>> level is specified to::
>>
>>     '{name}.{cache_tag}.opt-{optimization}.pyc'.format(
>>             name=module_name,
>>             cache_tag=sys.implementation.cache_tag,
>>             optimization=str(sys.flags.optimize))
>>
>> The "opt-" prefix was chosen so as to provide a visual separator
>> from the cache tag. The placement of the optimization level after
>> the cache tag was chosen to preserve lexicographic sort order of
>> bytecode file names based on module name and cache tag which will
>> not vary for a single interpreter. The "opt-" prefix was chosen over
>> "o" so as to be somewhat self-documenting. The "opt-" prefix was
>> chosen over "O" so as to not have any confusion in case "0" was the
>> leading prefix of the optimization level.
>>
>> A period was chosen over a hyphen as a separator so as to distinguish
>> clearly that the optimization level is not part of the interpreter
>> version as specified by the cache tag. It also lends to the use of
>> the period in the file name to delineate semantically different
>> concepts.
>>
>> For example, if ``-OO`` had been passed to the interpreter then instead
>> of ``importlib.cpython-35.pyo`` the file name would be
>> ``importlib.cpython-35.opt-2.pyc``.
>>
>> It should be noted that this change in no way affects the performance
>> of import. Since the import system looks for a single bytecode file
>> based on the optimization level of the interpreter already and
>> generates a new bytecode file if it doesn't exist, the introduction
>> of potentially more bytecode files in the ``__pycache__`` directory
>> has no effect in terms of stat calls. The interpreter will continue
>> to look for only a single bytecode file based on the optimization
>> level and thus no increase in stat calls will occur.
>>
>> The only potentially negative result of this PEP is the probable
>> increase in the number of ``.pyc`` files and thus increase in storage
>> use. But for platforms where this is an issue,
>> ``sys.dont_write_bytecode`` exists to turn off bytecode generation so
>> that it can be controlled offline.
>>
>>
>> Implementation
>> ==============
>>
>> importlib
>> ---------
>>
>> As ``importlib.util.cache_from_source()`` is the API that exposes
>> bytecode file paths as well as being directly used by importlib, it
>> requires the most critical change. As of Python 3.4, the function's
>> signature is::
>>
>>   importlib.util.cache_from_source(path, debug_override=None)
>>
>> This PEP proposes changing the signature in Python 3.5 to::
>>
>>   importlib.util.cache_from_source(path, debug_override=None, *,
>> optimization=None)
>>
>> The introduced ``optimization`` keyword-only parameter will control
>> what optimization level is specified in the file name. If the
>> argument is ``None`` then the current optimization level of the
>> interpreter will be assumed (including no optimization). Any argument
>> given for ``optimization`` will be passed to ``str()`` and must have
>> ``str.isalnum()`` be true, else ``ValueError`` will be raised (this
>> prevents invalid characters being used in the file name). If the
>> empty string is passed in for ``optimization`` then the addition of
>> the optimization will be suppressed, reverting to the file name
>> format which predates this PEP.
>>
>> It is expected that beyond Python's own two optimization levels,
>> third-party code will use a hash of optimization names to specify the
>> optimization level, e.g.
>> ``hashlib.sha256(','.join(['no dead code', 'const
>> folding'])).hexdigest()``.
>> While this might lead to long file names, it is assumed that most
>> users never look at the contents of the __pycache__ directory and so
>> this won't be an issue.
>>
>> The ``debug_override`` parameter will be deprecated. As the parameter
>> expects a boolean, the integer value of the boolean will be used as
>> if it had been provided as the argument to ``optimization`` (a
>> ``None`` argument will mean the same as for ``optimization``). A
>> deprecation warning will be raised when ``debug_override`` is given a
>> value other than ``None``, but there are no plans for the complete
>> removal of the parameter at this time (but removal will be no later
>> than Python 4).
>>
>> The various module attributes for importlib.machinery which relate to
>> bytecode file suffixes will be updated [7]_. The
>> ``DEBUG_BYTECODE_SUFFIXES`` and ``OPTIMIZED_BYTECODE_SUFFIXES`` will
>> both be documented as deprecated and set to the same value as
>> ``BYTECODE_SUFFIXES`` (removal of ``DEBUG_BYTECODE_SUFFIXES`` and
>> ``OPTIMIZED_BYTECODE_SUFFIXES`` is not currently planned, but will be
>> not later than Python 4).
>>
>> All various finders and loaders will also be updated as necessary,
>> but updating the previous mentioned parts of importlib should be all
>> that is required.
>>
>>
>> Rest of the standard library
>> ----------------------------
>>
>> The various functions exposed by the ``py_compile`` and
>> ``compileall`` functions will be updated as necessary to make sure
>> they follow the new bytecode file name semantics [6]_, [1]_. The CLI
>> for the ``compileall`` module will not be directly affected (the
>> ``-b`` flag will be implicit as it will no longer generate ``.pyo``
>> files when ``-O`` is specified).
>>
>>
>> Compatibility Considerations
>> ============================
>>
>> Any code directly manipulating bytecode files from Python 3.2 on
>> will need to consider the impact of this change on their code (prior
>> to Python 3.2 -- including all of Python 2 -- there was no
>> __pycache__ which already necessitates bifurcating bytecode file
>> handling support). If code was setting the ``debug_override``
>> argument to ``importlib.util.cache_from_source()`` then care will be
>> needed if they want the path to a bytecode file with an optimization
>> level of 2. Otherwise only code **not** using
>> ``importlib.util.cache_from_source()`` will need updating.
>>
>> As for people who distribute bytecode-only modules (i.e., use a
>> bytecode file instead of a source file), they will have to choose
>> which optimization level they want their bytecode files to be since
>> distributing a ``.pyo`` file with a ``.pyc`` file will no longer be
>> of any use. Since people typically only distribute bytecode files for
>> code obfuscation purposes or smaller distribution size then only
>> having to distribute a single ``.pyc`` should actually be beneficial
>> to these use-cases. And since the magic number for bytecode files
>> changed in Python 3.5 to support PEP 465 there is no need to support
>> pre-existing ``.pyo`` files [8]_.
>>
>>
>> Rejected Ideas
>> ==============
>>
>> Completely dropping optimization levels from CPython
>> ----------------------------------------------------
>>
>> Some have suggested that instead of accommodating the various
>> optimization levels in CPython, we should instead drop them
>> entirely. The argument is that significant performance gains would
>> occur from runtime optimizations through something like a JIT and not
>> through pre-execution bytecode optimizations.
>>
>> This idea is rejected for this PEP as that ignores the fact that
>> there are people who do find the pre-existing optimization levels for
>> CPython useful. It also assumes that no other Python interpreter
>> would find what this PEP proposes useful.
>>
>>
>> Alternative formatting of the optimization level in the file name
>> -----------------------------------------------------------------
>>
>> Using the "opt-" prefix and placing the optimization level between
>> the cache tag and file extension is not critical. All options which
>> have been considered are:
>>
>> * ``importlib.cpython-35.opt-1.pyc``
>> * ``importlib.cpython-35.opt1.pyc``
>> * ``importlib.cpython-35.o1.pyc``
>> * ``importlib.cpython-35.O1.pyc``
>> * ``importlib.cpython-35.1.pyc``
>> * ``importlib.cpython-35-O1.pyc``
>> * ``importlib.O1.cpython-35.pyc``
>> * ``importlib.o1.cpython-35.pyc``
>> * ``importlib.1.cpython-35.pyc``
>>
>> These were initially rejected either because they would change the
>> sort order of bytecode files, possible ambiguity with the cache tag,
>> or were not self-documenting enough. An informal poll was taken and
>> people clearly preferred the formatting proposed by the PEP [9]_.
>> Since this topic is non-technical and of personal choice, the issue
>> is considered solved.
>>
>>
>> Embedding the optimization level in the bytecode metadata
>> ---------------------------------------------------------
>>
>> Some have suggested that rather than embedding the optimization level
>> of bytecode in the file name that it be included in the file's
>> metadata instead. This would mean every interpreter had a single copy
>> of bytecode at any time. Changing the optimization level would thus
>> require rewriting the bytecode, but there would also only be a single
>> file to care about.
>>
>> This has been rejected due to the fact that Python is often installed
>> as a root-level application and thus modifying the bytecode file for
>> modules in the standard library are always possible. In this
>> situation integrators would need to guess at what a reasonable
>> optimization level was for users for any/all situations. By
>> allowing multiple optimization levels to co-exist simultaneously it
>> frees integrators from having to guess what users want and allows
>> users to utilize the optimization level they want.
>>
>>
>> References
>> ==========
>>
>> .. [1] The compileall module
>>    (https://docs.python.org/3/library/compileall.html#module-compileall)
>>
>> .. [2] The astoptimizer project
>>    (https://pypi.python.org/pypi/astoptimizer)
>>
>> .. [3] ``importlib.util.cache_from_source()``
>>    (
>> https://docs.python.org/3.5/library/importlib.html#importlib.util.cache_from_source
>> )
>>
>> .. [4] Implementation of ``importlib.util.cache_from_source()`` from
>> CPython 3.4.3rc1
>>    (
>> https://hg.python.org/cpython/file/038297948389/Lib/importlib/_bootstrap.py#l437
>> )
>>
>> .. [5] PEP 3147, PYC Repository Directories, Warsaw
>>    (http://www.python.org/dev/peps/pep-3147)
>>
>> .. [6] The py_compile module
>>    (https://docs.python.org/3/library/compileall.html#module-compileall)
>>
>> .. [7] The importlib.machinery module
>>    (
>> https://docs.python.org/3/library/importlib.html#module-importlib.machinery
>> )
>>
>> .. [8] ``importlib.util.MAGIC_NUMBER``
>>    (
>> https://docs.python.org/3/library/importlib.html#importlib.util.MAGIC_NUMBER
>> )
>>
>> .. [9] Informal poll of file name format options on Google+
>>    (https://plus.google.com/u/0/+BrettCannon/posts/fZynLNwHWGm)
>>
>> .. [10] The PyPy Project
>>    (http://pypy.org/)
>>
>>
>> Copyright
>> =========
>>
>> This document has been placed in the public domain.
>>
>>
>> ..
>>    Local Variables:
>>    mode: indented-text
>>    indent-tabs-mode: nil
>>    sentence-end-double-space: t
>>    fill-column: 70
>>    coding: utf-8
>>    End:
>>
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150320/5900ef53/attachment-0001.html>

From mistersheik at gmail.com  Sat Mar 21 00:16:07 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Fri, 20 Mar 2015 19:16:07 -0400
Subject: [Python-Dev] Missing *-unpacking generalizations (issue 2292)
In-Reply-To: <20150319175353.26159.19083@psf.upfronthosting.co.za>
References: <20150319175353.26159.19083@psf.upfronthosting.co.za>
Message-ID: <CAA68w_kp998DcJt70s7w1C0mUifwb1YWKF7oDrH4QjY0NAAV-A@mail.gmail.com>

Wow, this is an excellent review.  Thank you.

My only question is with respect to this:

I think there ought to be two opcodes; one for unpacking maps in
function calls and another for literals. The whole function location
thing is rather hideous.

What are the two opcodes?  BUILD_MAP_UNPACK and BUILD_MAP_UNPACK_WITH_CALL?

The first takes (n) a number of maps that it will merge, and the second
does the same but also accepts (function_call_location) for the purpose of
error reporting?

Thanks,

Neil



On Thu, Mar 19, 2015 at 1:53 PM, <benjamin at python.org> wrote:

> It's a start.
>
> There need to be documentation updates.
>
> There are still unrelated style changes in compile.c that should be
> reverted.
>
>
> http://bugs.python.org/review/2292/diff/14152/Include/opcode.h
> File Include/opcode.h (right):
>
> http://bugs.python.org/review/2292/diff/14152/Include/opcode.h#newcode71
> Include/opcode.h:71: #define DELETE_GLOBAL              98
> This file should not be manually changed. Rather,
> Tools/scripts/generate_opcode_h.py should be run.
>
> http://bugs.python.org/review/2292/diff/14152/Lib/importlib/_bootstrap.py
> File Lib/importlib/_bootstrap.py (right):
>
>
> http://bugs.python.org/review/2292/diff/14152/Lib/importlib/_bootstrap.py#newcode428
> Lib/importlib/_bootstrap.py:428: MAGIC_NUMBER = (3321).to_bytes(2,
> 'little') + b'\r\n'
> As the comment above indicates, the magic value should be incremented by
> 10 not 1. Also, the comment needs to be updated.
>
> http://bugs.python.org/review/2292/diff/14152/Lib/test/test_ast.py
> File Lib/test/test_ast.py (right):
>
>
> http://bugs.python.org/review/2292/diff/14152/Lib/test/test_ast.py#newcode937
> Lib/test/test_ast.py:937: main()
> Why is this here?
>
> http://bugs.python.org/review/2292/diff/14152/Lib/test/test_parser.py
> File Lib/test/test_parser.py (right):
>
>
> http://bugs.python.org/review/2292/diff/14152/Lib/test/test_parser.py#newcode334
> Lib/test/test_parser.py:334: self.check_expr('{**{}, 3:4, **{5:6,
> 7:8}}')
> Should there be tests for the new function call syntax, too?
>
> http://bugs.python.org/review/2292/diff/14152/Python/ast.c
> File Python/ast.c (right):
>
> http://bugs.python.org/review/2292/diff/14152/Python/ast.c#newcode1746
> Python/ast.c:1746: ast_error(c, n, "iterable unpacking cannot be used in
> comprehension");
> |n| should probably be |ch|
>
> http://bugs.python.org/review/2292/diff/14152/Python/ast.c#newcode2022
> Python/ast.c:2022: if (TYPE(CHILD(ch, 0)) == DOUBLESTAR)
> int is_dict = TYPE(CHILD(ch, 0)) == DOUBLESTAR;
>
> http://bugs.python.org/review/2292/diff/14152/Python/ast.c#newcode2026
> Python/ast.c:2026: && TYPE(CHILD(ch, 1)) == COMMA)) {
> boolean operators should be on the previous line
>
> http://bugs.python.org/review/2292/diff/14152/Python/ast.c#newcode2031
> Python/ast.c:2031: && TYPE(CHILD(ch, 1)) == comp_for) {
> ditto
>
> http://bugs.python.org/review/2292/diff/14152/Python/ast.c#newcode2036
> Python/ast.c:2036: && TYPE(CHILD(ch, 3 - is_dict)) == comp_for) {
> ditto
>
> http://bugs.python.org/review/2292/diff/14152/Python/ceval.c
> File Python/ceval.c (right):
>
> http://bugs.python.org/review/2292/diff/14152/Python/ceval.c#newcode2403
> Python/ceval.c:2403: as_tuple = PyObject_CallFunctionObjArgs(
> Use PyList_AsTuple.
>
> http://bugs.python.org/review/2292/diff/14152/Python/ceval.c#newcode2498
> Python/ceval.c:2498: TARGET(BUILD_MAP_UNPACK) {
> I think there ought to be two opcodes; one for unpacking maps in
> function calls and another for literals. The whole function location
> thing is rather hideous.
>
> http://bugs.python.org/review/2292/diff/14152/Python/ceval.c#newcode2526
> Python/ceval.c:2526: if (PySet_Size(intersection)) {
> Probably this would be faster with PySet_GET_SIZE(so).
>
> http://bugs.python.org/review/2292/diff/14152/Python/compile.c
> File Python/compile.c (right):
>
> http://bugs.python.org/review/2292/diff/14152/Python/compile.c#newcode2931
> Python/compile.c:2931: asdl_seq_SET(elts, i, elt->v.Starred.value);
> The compiler should not be mutating the AST.
>
> http://bugs.python.org/review/2292/diff/14152/Python/compile.c#newcode3088
> Python/compile.c:3088: int i, nseen, nkw = 0;
> Many of these should probably be Py_ssize_t.
>
> http://bugs.python.org/review/2292/diff/14152/Python/symtable.c
> File Python/symtable.c (right):
>
> http://bugs.python.org/review/2292/diff/14152/Python/symtable.c#newcode1372
> Python/symtable.c:1372: if (e != NULL) {
> Please fix the callers, so they don't pass NULL here.
>
> http://bugs.python.org/review/2292/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150320/38c3a1bf/attachment.html>

From guido at python.org  Sat Mar 21 01:06:03 2015
From: guido at python.org (Guido van Rossum)
Date: Fri, 20 Mar 2015 17:06:03 -0700
Subject: [Python-Dev] Final call for PEP 488: eliminating PYO files
In-Reply-To: <CAP1=2W5CpPdszgqrsbhgx5nQXCm5bOYUH7aXZ0vfdK3FyOk7Dg@mail.gmail.com>
References: <CAP1=2W4CejbRcjjYUoFdFReUrgZ8i_mm1vJE2aoVqqo0Q3BW=Q@mail.gmail.com>
 <CAP7+vJLOaJAzXDq-sedZR9KsLN5kkokYxB8YFx3TPjayB7ggmA@mail.gmail.com>
 <CAP1=2W5CpPdszgqrsbhgx5nQXCm5bOYUH7aXZ0vfdK3FyOk7Dg@mail.gmail.com>
Message-ID: <CAP7+vJLyGoxhfWE+AED1CKX-v_FRZm+ijDy7gw9hD07TG_azpQ@mail.gmail.com>

Awesome, that's what I was hoping. Accepted! Congrats and thank you very
much for writing the PEP and guiding the discussion.

On Fri, Mar 20, 2015 at 4:00 PM, Brett Cannon <bcannon at gmail.com> wrote:

>
>
> On Fri, Mar 20, 2015 at 4:41 PM Guido van Rossum <guido at python.org> wrote:
>
>> I am willing to be the BDFL for this PEP. I have tried to skim the recent
>> discussion (only python-dev) and I don't see much remaining controversy.
>> HOWEVER... The PEP is not clear (or at least too subtle) about the actual
>> name for optimization level 0. If I have foo.py, and I compile it three
>> times with three different optimization levels (no optimization; -O; -OO),
>> and then I look in __pycache__, would I see this:
>>
>> # (1)
>> foo.cpython-35.pyc
>> foo.cpython-35.opt-1.pyc
>> foo.cpython-35.opt-2.pyc
>>
>> Or would I see this?
>>
>> # (2)
>> foo.cpython-35.opt-0.pyc
>> foo.cpython-35.opt-1.pyc
>> foo.cpython-35.opt-2.pyc
>>
>
> #1
>
>
>>
>> Your lead-in ("I have decided to have the default case of no optimization
>> levels mean that the .pyc file name will have *no* optimization level
>> specified in the name and thus be just as it is today.") makes me think I
>> should expect (1), but I can't actually pinpoint where the language of the
>> PEP says this.
>>
>
> It was meant to be explained by "When no optimization level is specified,
> the pre-PEP ``.pyc`` file name will be used (i.e., no change in file name
> semantics)", but obviously it's a bit too subtle. I just updated the PEP
> with an explicit list of bytecode file name examples based on no -O, -O,
> and -OO.
>
> -Brett
>
>
>>
>>
>> On Fri, Mar 20, 2015 at 11:34 AM, Brett Cannon <bcannon at gmail.com> wrote:
>>
>>> I have decided to have the default case of no optimization levels mean
>>> that the .pyc file name will have *no* optimization level specified in
>>> the name and thus be just as it is today. I made this decision due to
>>> potential backwards-compatibility issues -- although I expect them to be
>>> minutes -- and to not force other implementations like PyPy to have some
>>> bogus value set since they don't have .pyo files to begin with (PyPy
>>> actually uses bytecode for -O and don't bother with -OO since PyPy already
>>> uses a bunch of memory when running).
>>>
>>> Since this closes out the last open issue, I need either a BDFL decision
>>> or a BDFAP to be assigned to make a decision. Guido?
>>>
>>> ======================================
>>>
>>> PEP: 488
>>> Title: Elimination of PYO files
>>> Version: $Revision$
>>> Last-Modified: $Date$
>>> Author: Brett Cannon <brett at python.org>
>>> Status: Draft
>>> Type: Standards Track
>>> Content-Type: text/x-rst
>>> Created: 20-Feb-2015
>>> Post-History:
>>>     2015-03-06
>>>     2015-03-13
>>>     2015-03-20
>>>
>>> Abstract
>>> ========
>>>
>>> This PEP proposes eliminating the concept of PYO files from Python.
>>> To continue the support of the separation of bytecode files based on
>>> their optimization level, this PEP proposes extending the PYC file
>>> name to include the optimization level in the bytecode repository
>>> directory when it's called for (i.e., the ``__pycache__`` directory).
>>>
>>>
>>> Rationale
>>> =========
>>>
>>> As of today, bytecode files come in two flavours: PYC and PYO. A PYC
>>> file is the bytecode file generated and read from when no
>>> optimization level is specified at interpreter startup (i.e., ``-O``
>>> is not specified). A PYO file represents the bytecode file that is
>>> read/written when **any** optimization level is specified (i.e., when
>>> ``-O`` **or** ``-OO`` is specified). This means that while PYC
>>> files clearly delineate the optimization level used when they were
>>> generated -- namely no optimizations beyond the peepholer -- the same
>>> is not true for PYO files. To put this in terms of optimization
>>> levels and the file extension:
>>>
>>>   - 0: ``.pyc``
>>>   - 1 (``-O``): ``.pyo``
>>>   - 2 (``-OO``): ``.pyo``
>>>
>>> The reuse of the ``.pyo`` file extension for both level 1 and 2
>>> optimizations means that there is no clear way to tell what
>>> optimization level was used to generate the bytecode file. In terms
>>> of reading PYO files, this can lead to an interpreter using a mixture
>>> of optimization levels with its code if the user was not careful to
>>> make sure all PYO files were generated using the same optimization
>>> level (typically done by blindly deleting all PYO files and then
>>> using the `compileall` module to compile all-new PYO files [1]_).
>>> This issue is only compounded when people optimize Python code beyond
>>> what the interpreter natively supports, e.g., using the astoptimizer
>>> project [2]_.
>>>
>>> In terms of writing PYO files, the need to delete all PYO files
>>> every time one either changes the optimization level they want to use
>>> or are unsure of what optimization was used the last time PYO files
>>> were generated leads to unnecessary file churn. The change proposed
>>> by this PEP also allows for **all** optimization levels to be
>>> pre-compiled for bytecode files ahead of time, something that is
>>> currently impossible thanks to the reuse of the ``.pyo`` file
>>> extension for multiple optimization levels.
>>>
>>> As for distributing bytecode-only modules, having to distribute both
>>> ``.pyc`` and ``.pyo`` files is unnecessary for the common use-case
>>> of code obfuscation and smaller file deployments. This means that
>>> bytecode-only modules will only load from their non-optimized
>>> ``.pyc`` file name.
>>>
>>>
>>> Proposal
>>> ========
>>>
>>> To eliminate the ambiguity that PYO files present, this PEP proposes
>>> eliminating the concept of PYO files and their accompanying ``.pyo``
>>> file extension. To allow for the optimization level to be unambiguous
>>> as well as to avoid having to regenerate optimized bytecode files
>>> needlessly in the `__pycache__` directory, the optimization level
>>> used to generate the bytecode file will be incorporated into the
>>> bytecode file name. When no optimization level is specified, the
>>> pre-PEP ``.pyc`` file name will be used (i.e., no change in file name
>>> semantics). This increases backwards-compatibility while also being
>>> more understanding of Python implementations which have no use for
>>> optimization levels (e.g., PyPy[10]_).
>>>
>>> Currently bytecode file names are created by
>>> ``importlib.util.cache_from_source()``, approximately using the
>>> following expression defined by PEP 3147 [3]_, [4]_, [5]_::
>>>
>>>     '{name}.{cache_tag}.pyc'.format(name=module_name,
>>>
>>> cache_tag=sys.implementation.cache_tag)
>>>
>>> This PEP proposes to change the expression when an optimization
>>> level is specified to::
>>>
>>>     '{name}.{cache_tag}.opt-{optimization}.pyc'.format(
>>>             name=module_name,
>>>             cache_tag=sys.implementation.cache_tag,
>>>             optimization=str(sys.flags.optimize))
>>>
>>> The "opt-" prefix was chosen so as to provide a visual separator
>>> from the cache tag. The placement of the optimization level after
>>> the cache tag was chosen to preserve lexicographic sort order of
>>> bytecode file names based on module name and cache tag which will
>>> not vary for a single interpreter. The "opt-" prefix was chosen over
>>> "o" so as to be somewhat self-documenting. The "opt-" prefix was
>>> chosen over "O" so as to not have any confusion in case "0" was the
>>> leading prefix of the optimization level.
>>>
>>> A period was chosen over a hyphen as a separator so as to distinguish
>>> clearly that the optimization level is not part of the interpreter
>>> version as specified by the cache tag. It also lends to the use of
>>> the period in the file name to delineate semantically different
>>> concepts.
>>>
>>> For example, if ``-OO`` had been passed to the interpreter then instead
>>> of ``importlib.cpython-35.pyo`` the file name would be
>>> ``importlib.cpython-35.opt-2.pyc``.
>>>
>>> It should be noted that this change in no way affects the performance
>>> of import. Since the import system looks for a single bytecode file
>>> based on the optimization level of the interpreter already and
>>> generates a new bytecode file if it doesn't exist, the introduction
>>> of potentially more bytecode files in the ``__pycache__`` directory
>>> has no effect in terms of stat calls. The interpreter will continue
>>> to look for only a single bytecode file based on the optimization
>>> level and thus no increase in stat calls will occur.
>>>
>>> The only potentially negative result of this PEP is the probable
>>> increase in the number of ``.pyc`` files and thus increase in storage
>>> use. But for platforms where this is an issue,
>>> ``sys.dont_write_bytecode`` exists to turn off bytecode generation so
>>> that it can be controlled offline.
>>>
>>>
>>> Implementation
>>> ==============
>>>
>>> importlib
>>> ---------
>>>
>>> As ``importlib.util.cache_from_source()`` is the API that exposes
>>> bytecode file paths as well as being directly used by importlib, it
>>> requires the most critical change. As of Python 3.4, the function's
>>> signature is::
>>>
>>>   importlib.util.cache_from_source(path, debug_override=None)
>>>
>>> This PEP proposes changing the signature in Python 3.5 to::
>>>
>>>   importlib.util.cache_from_source(path, debug_override=None, *,
>>> optimization=None)
>>>
>>> The introduced ``optimization`` keyword-only parameter will control
>>> what optimization level is specified in the file name. If the
>>> argument is ``None`` then the current optimization level of the
>>> interpreter will be assumed (including no optimization). Any argument
>>> given for ``optimization`` will be passed to ``str()`` and must have
>>> ``str.isalnum()`` be true, else ``ValueError`` will be raised (this
>>> prevents invalid characters being used in the file name). If the
>>> empty string is passed in for ``optimization`` then the addition of
>>> the optimization will be suppressed, reverting to the file name
>>> format which predates this PEP.
>>>
>>> It is expected that beyond Python's own two optimization levels,
>>> third-party code will use a hash of optimization names to specify the
>>> optimization level, e.g.
>>> ``hashlib.sha256(','.join(['no dead code', 'const
>>> folding'])).hexdigest()``.
>>> While this might lead to long file names, it is assumed that most
>>> users never look at the contents of the __pycache__ directory and so
>>> this won't be an issue.
>>>
>>> The ``debug_override`` parameter will be deprecated. As the parameter
>>> expects a boolean, the integer value of the boolean will be used as
>>> if it had been provided as the argument to ``optimization`` (a
>>> ``None`` argument will mean the same as for ``optimization``). A
>>> deprecation warning will be raised when ``debug_override`` is given a
>>> value other than ``None``, but there are no plans for the complete
>>> removal of the parameter at this time (but removal will be no later
>>> than Python 4).
>>>
>>> The various module attributes for importlib.machinery which relate to
>>> bytecode file suffixes will be updated [7]_. The
>>> ``DEBUG_BYTECODE_SUFFIXES`` and ``OPTIMIZED_BYTECODE_SUFFIXES`` will
>>> both be documented as deprecated and set to the same value as
>>> ``BYTECODE_SUFFIXES`` (removal of ``DEBUG_BYTECODE_SUFFIXES`` and
>>> ``OPTIMIZED_BYTECODE_SUFFIXES`` is not currently planned, but will be
>>> not later than Python 4).
>>>
>>> All various finders and loaders will also be updated as necessary,
>>> but updating the previous mentioned parts of importlib should be all
>>> that is required.
>>>
>>>
>>> Rest of the standard library
>>> ----------------------------
>>>
>>> The various functions exposed by the ``py_compile`` and
>>> ``compileall`` functions will be updated as necessary to make sure
>>> they follow the new bytecode file name semantics [6]_, [1]_. The CLI
>>> for the ``compileall`` module will not be directly affected (the
>>> ``-b`` flag will be implicit as it will no longer generate ``.pyo``
>>> files when ``-O`` is specified).
>>>
>>>
>>> Compatibility Considerations
>>> ============================
>>>
>>> Any code directly manipulating bytecode files from Python 3.2 on
>>> will need to consider the impact of this change on their code (prior
>>> to Python 3.2 -- including all of Python 2 -- there was no
>>> __pycache__ which already necessitates bifurcating bytecode file
>>> handling support). If code was setting the ``debug_override``
>>> argument to ``importlib.util.cache_from_source()`` then care will be
>>> needed if they want the path to a bytecode file with an optimization
>>> level of 2. Otherwise only code **not** using
>>> ``importlib.util.cache_from_source()`` will need updating.
>>>
>>> As for people who distribute bytecode-only modules (i.e., use a
>>> bytecode file instead of a source file), they will have to choose
>>> which optimization level they want their bytecode files to be since
>>> distributing a ``.pyo`` file with a ``.pyc`` file will no longer be
>>> of any use. Since people typically only distribute bytecode files for
>>> code obfuscation purposes or smaller distribution size then only
>>> having to distribute a single ``.pyc`` should actually be beneficial
>>> to these use-cases. And since the magic number for bytecode files
>>> changed in Python 3.5 to support PEP 465 there is no need to support
>>> pre-existing ``.pyo`` files [8]_.
>>>
>>>
>>> Rejected Ideas
>>> ==============
>>>
>>> Completely dropping optimization levels from CPython
>>> ----------------------------------------------------
>>>
>>> Some have suggested that instead of accommodating the various
>>> optimization levels in CPython, we should instead drop them
>>> entirely. The argument is that significant performance gains would
>>> occur from runtime optimizations through something like a JIT and not
>>> through pre-execution bytecode optimizations.
>>>
>>> This idea is rejected for this PEP as that ignores the fact that
>>> there are people who do find the pre-existing optimization levels for
>>> CPython useful. It also assumes that no other Python interpreter
>>> would find what this PEP proposes useful.
>>>
>>>
>>> Alternative formatting of the optimization level in the file name
>>> -----------------------------------------------------------------
>>>
>>> Using the "opt-" prefix and placing the optimization level between
>>> the cache tag and file extension is not critical. All options which
>>> have been considered are:
>>>
>>> * ``importlib.cpython-35.opt-1.pyc``
>>> * ``importlib.cpython-35.opt1.pyc``
>>> * ``importlib.cpython-35.o1.pyc``
>>> * ``importlib.cpython-35.O1.pyc``
>>> * ``importlib.cpython-35.1.pyc``
>>> * ``importlib.cpython-35-O1.pyc``
>>> * ``importlib.O1.cpython-35.pyc``
>>> * ``importlib.o1.cpython-35.pyc``
>>> * ``importlib.1.cpython-35.pyc``
>>>
>>> These were initially rejected either because they would change the
>>> sort order of bytecode files, possible ambiguity with the cache tag,
>>> or were not self-documenting enough. An informal poll was taken and
>>> people clearly preferred the formatting proposed by the PEP [9]_.
>>> Since this topic is non-technical and of personal choice, the issue
>>> is considered solved.
>>>
>>>
>>> Embedding the optimization level in the bytecode metadata
>>> ---------------------------------------------------------
>>>
>>> Some have suggested that rather than embedding the optimization level
>>> of bytecode in the file name that it be included in the file's
>>> metadata instead. This would mean every interpreter had a single copy
>>> of bytecode at any time. Changing the optimization level would thus
>>> require rewriting the bytecode, but there would also only be a single
>>> file to care about.
>>>
>>> This has been rejected due to the fact that Python is often installed
>>> as a root-level application and thus modifying the bytecode file for
>>> modules in the standard library are always possible. In this
>>> situation integrators would need to guess at what a reasonable
>>> optimization level was for users for any/all situations. By
>>> allowing multiple optimization levels to co-exist simultaneously it
>>> frees integrators from having to guess what users want and allows
>>> users to utilize the optimization level they want.
>>>
>>>
>>> References
>>> ==========
>>>
>>> .. [1] The compileall module
>>>    (https://docs.python.org/3/library/compileall.html#module-compileall)
>>>
>>> .. [2] The astoptimizer project
>>>    (https://pypi.python.org/pypi/astoptimizer)
>>>
>>> .. [3] ``importlib.util.cache_from_source()``
>>>    (
>>> https://docs.python.org/3.5/library/importlib.html#importlib.util.cache_from_source
>>> )
>>>
>>> .. [4] Implementation of ``importlib.util.cache_from_source()`` from
>>> CPython 3.4.3rc1
>>>    (
>>> https://hg.python.org/cpython/file/038297948389/Lib/importlib/_bootstrap.py#l437
>>> )
>>>
>>> .. [5] PEP 3147, PYC Repository Directories, Warsaw
>>>    (http://www.python.org/dev/peps/pep-3147)
>>>
>>> .. [6] The py_compile module
>>>    (https://docs.python.org/3/library/compileall.html#module-compileall)
>>>
>>> .. [7] The importlib.machinery module
>>>    (
>>> https://docs.python.org/3/library/importlib.html#module-importlib.machinery
>>> )
>>>
>>> .. [8] ``importlib.util.MAGIC_NUMBER``
>>>    (
>>> https://docs.python.org/3/library/importlib.html#importlib.util.MAGIC_NUMBER
>>> )
>>>
>>> .. [9] Informal poll of file name format options on Google+
>>>    (https://plus.google.com/u/0/+BrettCannon/posts/fZynLNwHWGm)
>>>
>>> .. [10] The PyPy Project
>>>    (http://pypy.org/)
>>>
>>>
>>> Copyright
>>> =========
>>>
>>> This document has been placed in the public domain.
>>>
>>>
>>> ..
>>>    Local Variables:
>>>    mode: indented-text
>>>    indent-tabs-mode: nil
>>>    sentence-end-double-space: t
>>>    fill-column: 70
>>>    coding: utf-8
>>>    End:
>>>
>>>
>>> _______________________________________________
>>> Python-Dev mailing list
>>> Python-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe:
>>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>>
>>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150320/e56a7f3e/attachment-0001.html>

From luoyonggang at gmail.com  Sat Mar 21 01:15:29 2015
From: luoyonggang at gmail.com (=?UTF-8?B?572X5YuH5YiaKFlvbmdnYW5nIEx1bykg?=)
Date: Sat, 21 Mar 2015 08:15:29 +0800
Subject: [Python-Dev] Fwd: class os.DirEntry is confusing,
In-Reply-To: <CAE2XoE_YZCgZSQJqdQgZ1kkT_g9U2Dxqq2HwNAfotAXdHtJHfQ@mail.gmail.com>
References: <CAE2XoE_YZCgZSQJqdQgZ1kkT_g9U2Dxqq2HwNAfotAXdHtJHfQ@mail.gmail.com>
Message-ID: <CAE2XoE-fuzz4Uu3ZcTq1FrenmEY5avC4AYFK3ANAZVATrVaPyA@mail.gmail.com>

---------- Forwarded message ----------
From: ???(Yonggang Luo) <luoyonggang at gmail.com>
Date: 2015-03-21 2:24 GMT+08:00
Subject: class os.DirEntry is confusing,
To: Python <python-ideas at python.org>


Should be replaced with os.FsEntry or os.ScanEntry

--
         ??
?
???
Yours
    sincerely,
Yonggang Luo


-- 
         ??
?
???
Yours
    sincerely,
Yonggang Luo

From mistersheik at gmail.com  Sat Mar 21 01:54:39 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Fri, 20 Mar 2015 20:54:39 -0400
Subject: [Python-Dev] pep 7
Message-ID: <CAA68w_nYX0kZo7y0+=k4w92kuEO3X5vYFMk9F9hfjzLvE+94BQ@mail.gmail.com>

If ever someone wants to clean up the repository to conform to PEP 7, I
wrote a program that catches a couple hundred PEP 7 violations in ./Python
alone (1400 in the whole codebase):

import os
import re

def grep(path, regex):
    reg_obj = re.compile(regex, re.M)
    res = []
    for root, dirs, fnames in os.walk(path):
        for fname in fnames:
            if fname.endswith('.c'):
                path = os.path.join(root, fname)
                with open(path) as f:
                    data = f.read()
                    for m in reg_obj.finditer(data):
                        line_number = sum(c == '\n'
                                          for c in data[:m.start()]) + 1
                        res.append("{}: {}".format(path, line_number))
    return res

for pattern in [
        r'^\s*\|\|',
        r'^\s*\&\&',
        r'} else {',
        r'\<return\s*\(',
]:
    print("Searching for", pattern)
    print("\n".join(grep('.', pattern)))

In my experience, it was hard to write PEP 7 conforming code when the
surrounding code is inconsistent.

Best,

Neil
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150320/bf7cae78/attachment.html>

From lewisc at pdx.edu  Sat Mar 21 02:30:06 2015
From: lewisc at pdx.edu (Lewis Coates)
Date: Fri, 20 Mar 2015 18:30:06 -0700
Subject: [Python-Dev] some minor questions about pep8
Message-ID: <CAFmnkTBYpaJRaeo_ff6KKVwLTszfF1Zah+yGvg7_yq82bSTmfw@mail.gmail.com>

In pep8 there are two conflicting statements, both

https://www.python.org/dev/peps/pep-0008/#version-bookkeeping
https://www.python.org/dev/peps/pep-0008/#imports

Stipulate that they should be "at the top of the file after any module
comments and docstrings." Which of these takes precedence?

Secondly, we also have an "__author__", and "__project__" variables, I
assume these would be put with the version information as well?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150320/466c84f3/attachment.html>

From ben+python at benfinney.id.au  Sat Mar 21 02:38:55 2015
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sat, 21 Mar 2015 12:38:55 +1100
Subject: [Python-Dev] some minor questions about pep8
References: <CAFmnkTBYpaJRaeo_ff6KKVwLTszfF1Zah+yGvg7_yq82bSTmfw@mail.gmail.com>
Message-ID: <85lhir6seo.fsf@benfinney.id.au>

Lewis Coates <lewisc at pdx.edu> writes:

> In pep8 there are two conflicting statements, both
>
> https://www.python.org/dev/peps/pep-0008/#version-bookkeeping
> https://www.python.org/dev/peps/pep-0008/#imports
>
> Stipulate that they should be "at the top of the file after any module
> comments and docstrings." Which of these takes precedence?

I don't know an official answer. The convention I've observed is
overwhelmingly in one direction: import statements come before any
assignment statements.

> Secondly, we also have an "__author__", and "__project__" variables, I
> assume these would be put with the version information as well?

Yes.

-- 
 \     ?Welchen Teil von ?Gestalt? verstehen Sie nicht?  [What part of |
  `\                ?gestalt? don't you understand?]? ?Karsten M. Self |
_o__)                                                                  |
Ben Finney


From brian at python.org  Sat Mar 21 03:35:47 2015
From: brian at python.org (Brian Curtin)
Date: Fri, 20 Mar 2015 21:35:47 -0500
Subject: [Python-Dev] pep 7
In-Reply-To: <CAA68w_nYX0kZo7y0+=k4w92kuEO3X5vYFMk9F9hfjzLvE+94BQ@mail.gmail.com>
References: <CAA68w_nYX0kZo7y0+=k4w92kuEO3X5vYFMk9F9hfjzLvE+94BQ@mail.gmail.com>
Message-ID: <CAD+XWwpi36R37ESLHJ-gY14yfUxkNjLVh7P3L75bTA8_AW6VCg@mail.gmail.com>

On Fri, Mar 20, 2015 at 7:54 PM, Neil Girdhar <mistersheik at gmail.com> wrote:
> If ever someone wants to clean up the repository to conform to PEP 7, I
> wrote a program that catches a couple hundred PEP 7 violations in ./Python
> alone (1400 in the whole codebase):
>
> import os
> import re
>
> def grep(path, regex):
>     reg_obj = re.compile(regex, re.M)
>     res = []
>     for root, dirs, fnames in os.walk(path):
>         for fname in fnames:
>             if fname.endswith('.c'):
>                 path = os.path.join(root, fname)
>                 with open(path) as f:
>                     data = f.read()
>                     for m in reg_obj.finditer(data):
>                         line_number = sum(c == '\n'
>                                           for c in data[:m.start()]) + 1
>                         res.append("{}: {}".format(path, line_number))
>     return res
>
> for pattern in [
>         r'^\s*\|\|',
>         r'^\s*\&\&',
>         r'} else {',
>         r'\<return\s*\(',
> ]:
>     print("Searching for", pattern)
>     print("\n".join(grep('.', pattern)))
>
> In my experience, it was hard to write PEP 7 conforming code when the
> surrounding code is inconsistent.

You can usually change surrounding code within reason if you want to
add conforming code of your own, but there's little value and high
risk in any mass change just to apply the style guidelines.

From mistersheik at gmail.com  Sat Mar 21 04:02:34 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Fri, 20 Mar 2015 23:02:34 -0400
Subject: [Python-Dev] pep 7
In-Reply-To: <CAD+XWwpi36R37ESLHJ-gY14yfUxkNjLVh7P3L75bTA8_AW6VCg@mail.gmail.com>
References: <CAA68w_nYX0kZo7y0+=k4w92kuEO3X5vYFMk9F9hfjzLvE+94BQ@mail.gmail.com>
 <CAD+XWwpi36R37ESLHJ-gY14yfUxkNjLVh7P3L75bTA8_AW6VCg@mail.gmail.com>
Message-ID: <CAA68w_mbzcdykXeJ438e-8XAROrc-yvzFnOzysOEdv_Sm82PuA@mail.gmail.com>

The code reviews I got asked me to revert PEP 7 changes.  I can understand
that, but then logically someone should go ahead and clean up the code.
It's not "high risk" if you just check for whitespace equivalence of the
source code and binary equivalence of the compiled code.  The value is for
people who are new to the codebase.

Best,

Neil

On Fri, Mar 20, 2015 at 10:35 PM, Brian Curtin <brian at python.org> wrote:

> On Fri, Mar 20, 2015 at 7:54 PM, Neil Girdhar <mistersheik at gmail.com>
> wrote:
> > If ever someone wants to clean up the repository to conform to PEP 7, I
> > wrote a program that catches a couple hundred PEP 7 violations in
> ./Python
> > alone (1400 in the whole codebase):
> >
> > import os
> > import re
> >
> > def grep(path, regex):
> >     reg_obj = re.compile(regex, re.M)
> >     res = []
> >     for root, dirs, fnames in os.walk(path):
> >         for fname in fnames:
> >             if fname.endswith('.c'):
> >                 path = os.path.join(root, fname)
> >                 with open(path) as f:
> >                     data = f.read()
> >                     for m in reg_obj.finditer(data):
> >                         line_number = sum(c == '\n'
> >                                           for c in data[:m.start()]) + 1
> >                         res.append("{}: {}".format(path, line_number))
> >     return res
> >
> > for pattern in [
> >         r'^\s*\|\|',
> >         r'^\s*\&\&',
> >         r'} else {',
> >         r'\<return\s*\(',
> > ]:
> >     print("Searching for", pattern)
> >     print("\n".join(grep('.', pattern)))
> >
> > In my experience, it was hard to write PEP 7 conforming code when the
> > surrounding code is inconsistent.
>
> You can usually change surrounding code within reason if you want to
> add conforming code of your own, but there's little value and high
> risk in any mass change just to apply the style guidelines.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150320/8ed96573/attachment.html>

From brian at python.org  Sat Mar 21 04:35:49 2015
From: brian at python.org (Brian Curtin)
Date: Fri, 20 Mar 2015 22:35:49 -0500
Subject: [Python-Dev] pep 7
In-Reply-To: <CAA68w_mbzcdykXeJ438e-8XAROrc-yvzFnOzysOEdv_Sm82PuA@mail.gmail.com>
References: <CAA68w_nYX0kZo7y0+=k4w92kuEO3X5vYFMk9F9hfjzLvE+94BQ@mail.gmail.com>
 <CAD+XWwpi36R37ESLHJ-gY14yfUxkNjLVh7P3L75bTA8_AW6VCg@mail.gmail.com>
 <CAA68w_mbzcdykXeJ438e-8XAROrc-yvzFnOzysOEdv_Sm82PuA@mail.gmail.com>
Message-ID: <CAD+XWwoUnqg9ANWWPOjSCdp=hJ4fgqkRvksp5Z-e7N-G-dO+TQ@mail.gmail.com>

On Fri, Mar 20, 2015 at 10:02 PM, Neil Girdhar <mistersheik at gmail.com> wrote:
> The code reviews I got asked me to revert PEP 7 changes.  I can understand
> that, but then logically someone should go ahead and clean up the code.
> It's not "high risk" if you just check for whitespace equivalence of the
> source code and binary equivalence of the compiled code.  The value is for
> people who are new to the codebase.

There are a lot of areas of the C code that aren't explicitly or
directly tested, so yes, a lot of changes are high risk, especially in
bulk. While a one time change while checking binary equivalence would
do it, it's also a huge amount of churn just to follow a guideline.
Without an automated checker for the guidelines, if things get in they
just get in, and sometimes you can modify them while making
improvements to the code, but sometimes it depends on what exactly
you're doing as well. On top of this, we already disallow mass PEP 8
changes to avoid the churn there as well, and it took a good bit of
convincing for another semi-recent mass change (although I can't
remember the topic, but it was deemed safe enough to make).

Another common issue with mass code churn like this is that it affects
tooling, such as `hg blame`

From guido at python.org  Sat Mar 21 04:53:10 2015
From: guido at python.org (Guido van Rossum)
Date: Fri, 20 Mar 2015 20:53:10 -0700
Subject: [Python-Dev] some minor questions about pep8
In-Reply-To: <85lhir6seo.fsf@benfinney.id.au>
References: <CAFmnkTBYpaJRaeo_ff6KKVwLTszfF1Zah+yGvg7_yq82bSTmfw@mail.gmail.com>
 <85lhir6seo.fsf@benfinney.id.au>
Message-ID: <CAP7+vJJPnqUMA=fpz+arV1ZZrT6dGU5d+KNJM5ZLmrRGumkLKQ@mail.gmail.com>

FWIW, I think __version__, __author__ etc. were bad ideas. Almost nobody
manages these correctly. Note that the PEP 8 section starts with less than
an endorsement: "If you *have* to have Subversion, CVS, or RCS crud in your
source file, do it as follows."

That said, if an official answer is required, common sense would suggest
that __version__ should go before the imports. (I would put it before the
docstring too, except then the docstring wouldn't be a docstring any more.
Go figure.)

On Fri, Mar 20, 2015 at 6:38 PM, Ben Finney <ben+python at benfinney.id.au>
wrote:

> Lewis Coates <lewisc at pdx.edu> writes:
>
> > In pep8 there are two conflicting statements, both
> >
> > https://www.python.org/dev/peps/pep-0008/#version-bookkeeping
> > https://www.python.org/dev/peps/pep-0008/#imports
> >
> > Stipulate that they should be "at the top of the file after any module
> > comments and docstrings." Which of these takes precedence?
>
> I don't know an official answer. The convention I've observed is
> overwhelmingly in one direction: import statements come before any
> assignment statements.
>
> > Secondly, we also have an "__author__", and "__project__" variables, I
> > assume these would be put with the version information as well?
>
> Yes.
>
> --
>  \     ?Welchen Teil von ?Gestalt? verstehen Sie nicht?  [What part of |
>   `\                ?gestalt? don't you understand?]? ?Karsten M. Self |
> _o__)                                                                  |
> Ben Finney
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150320/ac40c7e4/attachment.html>

From guido at python.org  Sat Mar 21 04:57:02 2015
From: guido at python.org (Guido van Rossum)
Date: Fri, 20 Mar 2015 20:57:02 -0700
Subject: [Python-Dev] pep 7
In-Reply-To: <CAA68w_mbzcdykXeJ438e-8XAROrc-yvzFnOzysOEdv_Sm82PuA@mail.gmail.com>
References: <CAA68w_nYX0kZo7y0+=k4w92kuEO3X5vYFMk9F9hfjzLvE+94BQ@mail.gmail.com>
 <CAD+XWwpi36R37ESLHJ-gY14yfUxkNjLVh7P3L75bTA8_AW6VCg@mail.gmail.com>
 <CAA68w_mbzcdykXeJ438e-8XAROrc-yvzFnOzysOEdv_Sm82PuA@mail.gmail.com>
Message-ID: <CAP7+vJKxaHUGQF-8gT5vDpHRfMyt6pgtmSaOh3ApgFY-RN=8=w@mail.gmail.com>

Neil, you have no idea. Please back off.

On Fri, Mar 20, 2015 at 8:02 PM, Neil Girdhar <mistersheik at gmail.com> wrote:

> The code reviews I got asked me to revert PEP 7 changes.  I can understand
> that, but then logically someone should go ahead and clean up the code.
> It's not "high risk" if you just check for whitespace equivalence of the
> source code and binary equivalence of the compiled code.  The value is for
> people who are new to the codebase.
>
> Best,
>
> Neil
>
> On Fri, Mar 20, 2015 at 10:35 PM, Brian Curtin <brian at python.org> wrote:
>
>> On Fri, Mar 20, 2015 at 7:54 PM, Neil Girdhar <mistersheik at gmail.com>
>> wrote:
>> > If ever someone wants to clean up the repository to conform to PEP 7, I
>> > wrote a program that catches a couple hundred PEP 7 violations in
>> ./Python
>> > alone (1400 in the whole codebase):
>> >
>> > import os
>> > import re
>> >
>> > def grep(path, regex):
>> >     reg_obj = re.compile(regex, re.M)
>> >     res = []
>> >     for root, dirs, fnames in os.walk(path):
>> >         for fname in fnames:
>> >             if fname.endswith('.c'):
>> >                 path = os.path.join(root, fname)
>> >                 with open(path) as f:
>> >                     data = f.read()
>> >                     for m in reg_obj.finditer(data):
>> >                         line_number = sum(c == '\n'
>> >                                           for c in data[:m.start()]) + 1
>> >                         res.append("{}: {}".format(path, line_number))
>> >     return res
>> >
>> > for pattern in [
>> >         r'^\s*\|\|',
>> >         r'^\s*\&\&',
>> >         r'} else {',
>> >         r'\<return\s*\(',
>> > ]:
>> >     print("Searching for", pattern)
>> >     print("\n".join(grep('.', pattern)))
>> >
>> > In my experience, it was hard to write PEP 7 conforming code when the
>> > surrounding code is inconsistent.
>>
>> You can usually change surrounding code within reason if you want to
>> add conforming code of your own, but there's little value and high
>> risk in any mass change just to apply the style guidelines.
>>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150320/262df9bb/attachment.html>

From ianlee1521 at gmail.com  Sat Mar 21 05:24:28 2015
From: ianlee1521 at gmail.com (Ian Lee)
Date: Fri, 20 Mar 2015 21:24:28 -0700
Subject: [Python-Dev] some minor questions about pep8
In-Reply-To: <CAP7+vJJPnqUMA=fpz+arV1ZZrT6dGU5d+KNJM5ZLmrRGumkLKQ@mail.gmail.com>
References: <CAFmnkTBYpaJRaeo_ff6KKVwLTszfF1Zah+yGvg7_yq82bSTmfw@mail.gmail.com>
 <85lhir6seo.fsf@benfinney.id.au>
 <CAP7+vJJPnqUMA=fpz+arV1ZZrT6dGU5d+KNJM5ZLmrRGumkLKQ@mail.gmail.com>
Message-ID: <CA+GyjMntcKaRUjfOzZg82q=13kjeCkKAYPT=ovhQUokqnPaXng@mail.gmail.com>

Guido,

In that case would you be open to a patch to update the PEP accordingly?

Additionally, does that official statement cover other dunder assignments
(e.g. "__author__"?). If so I'll update the PEP8 tool accordingly.

Thanks,

~ Ian Lee
On Mar 20, 2015 8:55 PM, "Guido van Rossum" <guido at python.org> wrote:

> FWIW, I think __version__, __author__ etc. were bad ideas. Almost nobody
> manages these correctly. Note that the PEP 8 section starts with less than
> an endorsement: "If you *have* to have Subversion, CVS, or RCS crud in
> your source file, do it as follows."
>
> That said, if an official answer is required, common sense would suggest
> that __version__ should go before the imports. (I would put it before the
> docstring too, except then the docstring wouldn't be a docstring any more.
> Go figure.)
>
> On Fri, Mar 20, 2015 at 6:38 PM, Ben Finney <ben+python at benfinney.id.au>
> wrote:
>
>> Lewis Coates <lewisc at pdx.edu> writes:
>>
>> > In pep8 there are two conflicting statements, both
>> >
>> > https://www.python.org/dev/peps/pep-0008/#version-bookkeeping
>> > https://www.python.org/dev/peps/pep-0008/#imports
>> >
>> > Stipulate that they should be "at the top of the file after any module
>> > comments and docstrings." Which of these takes precedence?
>>
>> I don't know an official answer. The convention I've observed is
>> overwhelmingly in one direction: import statements come before any
>> assignment statements.
>>
>> > Secondly, we also have an "__author__", and "__project__" variables, I
>> > assume these would be put with the version information as well?
>>
>> Yes.
>>
>> --
>>  \     ?Welchen Teil von ?Gestalt? verstehen Sie nicht?  [What part of |
>>   `\                ?gestalt? don't you understand?]? ?Karsten M. Self |
>> _o__)                                                                  |
>> Ben Finney
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/ianlee1521%40gmail.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150320/14f60958/attachment-0001.html>

From brian at python.org  Sat Mar 21 05:29:51 2015
From: brian at python.org (Brian Curtin)
Date: Fri, 20 Mar 2015 23:29:51 -0500
Subject: [Python-Dev] pep 7
In-Reply-To: <CAP7+vJKxaHUGQF-8gT5vDpHRfMyt6pgtmSaOh3ApgFY-RN=8=w@mail.gmail.com>
References: <CAA68w_nYX0kZo7y0+=k4w92kuEO3X5vYFMk9F9hfjzLvE+94BQ@mail.gmail.com>
 <CAD+XWwpi36R37ESLHJ-gY14yfUxkNjLVh7P3L75bTA8_AW6VCg@mail.gmail.com>
 <CAA68w_mbzcdykXeJ438e-8XAROrc-yvzFnOzysOEdv_Sm82PuA@mail.gmail.com>
 <CAP7+vJKxaHUGQF-8gT5vDpHRfMyt6pgtmSaOh3ApgFY-RN=8=w@mail.gmail.com>
Message-ID: <CAD+XWwoc0HXWXH5G9g3pe2MsmBbmr0C-_67PbCw3BhrO=i5Tkg@mail.gmail.com>

On Fri, Mar 20, 2015 at 10:57 PM, Guido van Rossum <guido at python.org> wrote:
> Neil, you have no idea. Please back off.

I wouldn't go that far. Wanting a quality code base certainly isn't a
bad thing, but there's a lot more progress to be made by working with
what's there and being as mindful as possible of the guidelines moving
forward. After all, they're guidelines, not rules.

From storchaka at gmail.com  Sat Mar 21 05:41:18 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 21 Mar 2015 06:41:18 +0200
Subject: [Python-Dev] How to document functions with optional positional
	parameters?
Message-ID: <meispe$vth$1@ger.gmane.org>

How to document functions with optional positional parameters?

For example binascii.crc32(). It has two positional parameters, one is 
mandatory, and one is optional with default value 0. With Argument 
Clinic its signature is crc32(data, crc=0, /). In the documentation it 
is written as crc32(data[, crc]) (and noted that the default value of 
the second parameter is 0). Both are not valid Python syntax. Can the 
documentation be change to crc32(data, crc=0)?

Related issues:

https://bugs.python.org/issue21488 (changed encode(obj, 
encoding='ascii', errors='strict') to encode(obj, [encoding[, errors]]))

https://bugs.python.org/issue22832 (changed ioctl(fd, op[, arg[, 
mutate_flag]]) to ioctl(fd, request, arg=0, mutate_flag=True))

https://bugs.python.org/issue22341 (discussed changing crc32(data[, 
crc]) to crc32(data, crc=0))


From ianlee1521 at gmail.com  Sat Mar 21 05:59:27 2015
From: ianlee1521 at gmail.com (Ian Lee)
Date: Fri, 20 Mar 2015 21:59:27 -0700
Subject: [Python-Dev] some minor questions about pep8
In-Reply-To: <CA+GyjMntcKaRUjfOzZg82q=13kjeCkKAYPT=ovhQUokqnPaXng@mail.gmail.com>
References: <CAFmnkTBYpaJRaeo_ff6KKVwLTszfF1Zah+yGvg7_yq82bSTmfw@mail.gmail.com>
 <85lhir6seo.fsf@benfinney.id.au>
 <CAP7+vJJPnqUMA=fpz+arV1ZZrT6dGU5d+KNJM5ZLmrRGumkLKQ@mail.gmail.com>
 <CA+GyjMntcKaRUjfOzZg82q=13kjeCkKAYPT=ovhQUokqnPaXng@mail.gmail.com>
Message-ID: <CA+GyjMksfgG3mWbPCXhJ=y_etPHZzuoNWgWNru2oEzvagyPiOA@mail.gmail.com>

FWIW, I would vote for the "__version__", "__author__", etc assignments
being after the imports. Reason being cases where the "__version__" is not
from VCS, but is calculated from pkg_resources:

from pkg_resources import get_distribution
__version__ = get_distribution('mypackage').version

Also, then more useful things like "__all__" (which can very reasonably
rely on imports), can be together with "__version__" and "__author__"
assignments.

I would vote:

shebang
docstring
imports
dunder assignments
other code...


</2 cents>


~ Ian Lee

On Fri, Mar 20, 2015 at 9:24 PM, Ian Lee <ianlee1521 at gmail.com> wrote:

> Guido,
>
> In that case would you be open to a patch to update the PEP accordingly?
>
> Additionally, does that official statement cover other dunder assignments
> (e.g. "__author__"?). If so I'll update the PEP8 tool accordingly.
>
> Thanks,
>
> ~ Ian Lee
> On Mar 20, 2015 8:55 PM, "Guido van Rossum" <guido at python.org> wrote:
>
>> FWIW, I think __version__, __author__ etc. were bad ideas. Almost nobody
>> manages these correctly. Note that the PEP 8 section starts with less than
>> an endorsement: "If you *have* to have Subversion, CVS, or RCS crud in
>> your source file, do it as follows."
>>
>> That said, if an official answer is required, common sense would suggest
>> that __version__ should go before the imports. (I would put it before the
>> docstring too, except then the docstring wouldn't be a docstring any more.
>> Go figure.)
>>
>> On Fri, Mar 20, 2015 at 6:38 PM, Ben Finney <ben+python at benfinney.id.au>
>> wrote:
>>
>>> Lewis Coates <lewisc at pdx.edu> writes:
>>>
>>> > In pep8 there are two conflicting statements, both
>>> >
>>> > https://www.python.org/dev/peps/pep-0008/#version-bookkeeping
>>> > https://www.python.org/dev/peps/pep-0008/#imports
>>> >
>>> > Stipulate that they should be "at the top of the file after any module
>>> > comments and docstrings." Which of these takes precedence?
>>>
>>> I don't know an official answer. The convention I've observed is
>>> overwhelmingly in one direction: import statements come before any
>>> assignment statements.
>>>
>>> > Secondly, we also have an "__author__", and "__project__" variables, I
>>> > assume these would be put with the version information as well?
>>>
>>> Yes.
>>>
>>> --
>>>  \     ?Welchen Teil von ?Gestalt? verstehen Sie nicht?  [What part of |
>>>   `\                ?gestalt? don't you understand?]? ?Karsten M. Self |
>>> _o__)                                                                  |
>>> Ben Finney
>>>
>>> _______________________________________________
>>> Python-Dev mailing list
>>> Python-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe:
>>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>>
>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/ianlee1521%40gmail.com
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150320/408843e4/attachment.html>

From victor.stinner at gmail.com  Sat Mar 21 12:03:02 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sat, 21 Mar 2015 12:03:02 +0100
Subject: [Python-Dev] How to document functions with optional positional
	parameters?
In-Reply-To: <meispe$vth$1@ger.gmane.org>
References: <meispe$vth$1@ger.gmane.org>
Message-ID: <CAMpsgwYmee13iUPryqXG9_gFgR0CeAk7DKvPvBJQ7QBj=t+jYA@mail.gmail.com>

Le samedi 21 mars 2015, Serhiy Storchaka <storchaka at gmail.com> a ?crit :


>
> For example binascii.crc32(). It has two positional parameters, one is
mandatory, and one is optional with default value 0. With Argument Clinic
its signature is crc32(data, crc=0, /).

 The \ is useful, it indicates that you cannot use keywords. If you want to
drop \, modify the function to accept keywords.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150321/4a8df146/attachment.html>

From ncoghlan at gmail.com  Sat Mar 21 12:46:14 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 21 Mar 2015 21:46:14 +1000
Subject: [Python-Dev] Needed reviews
In-Reply-To: <mee4si$rap$1@ger.gmane.org>
References: <mee4si$rap$1@ger.gmane.org>
Message-ID: <CADiSq7dvKuKZ-OVqni-=st1+bYp5DVn8O0w9w=K+agELSUOQMA@mail.gmail.com>

On 19 March 2015 at 19:28, Serhiy Storchaka <storchaka at gmail.com> wrote:
> Here is list of my ready for review patches.  It is incomplete and contains
> only patches for which I don't expect objections or long discussion.  Most
> of them are relative easy and need only formal review.  Most of them wait
> for a review many months.

It's worth noting that If there are changes you feel are genuinely low
risk, you can go ahead and merge them based on your own commit review
(even if you also wrote the original patch). It's far from ideal, but
also a pragmatic recognition of how low the available amount of core
developer time we have is relative to the amount of work that *could*
be done :(

The only time that guideline doesn't apply is if the release manager
declares mandatory commit reviews while preparing a release, and I
think Larry uses a release branch for that these days instead.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat Mar 21 12:50:04 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 21 Mar 2015 21:50:04 +1000
Subject: [Python-Dev] Use ptyhon -s as default shbang for system python
	executables/daemons
In-Reply-To: <20150318124825.337dc1c6@anarchist.wooz.org>
References: <loom.20150318T164041-536@post.gmane.org>
 <CAMpsgwa3F2Abnptu8uFtxzCD_T-x9BCK9KtuOsVDW3kF1N88-g@mail.gmail.com>
 <20150318124825.337dc1c6@anarchist.wooz.org>
Message-ID: <CADiSq7dpxnaq4F7HRTAP5e8XzyRxvEpRDdFx+RtHSmtTDmH20w@mail.gmail.com>

On 19 March 2015 at 02:48, Barry Warsaw <barry at python.org> wrote:
> It's never gotten much farther than musings, but protecting the system against
> the weird things people install would be a good thing.  OTOH, this feels a lot
> like virtual environments so maybe there's something useful to be mined there.

I took it a fair bit further than musings:
https://www.python.org/dev/peps/pep-0432/#a-system-python-executable

Unfortunately PEP 432's a big project with very high barriers to entry
(you need to come to grips with the current arcane startup sequence
first), so I've never managed to hand it over after running out of
time to pursue it myself.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat Mar 21 12:52:34 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 21 Mar 2015 21:52:34 +1000
Subject: [Python-Dev] Use ptyhon -s as default shbang for system python
	executables/daemons
In-Reply-To: <654D3177-05D2-403F-9531-136FC98ED73F@stufft.io>
References: <loom.20150318T164041-536@post.gmane.org>
 <CAMpsgwa3F2Abnptu8uFtxzCD_T-x9BCK9KtuOsVDW3kF1N88-g@mail.gmail.com>
 <20150318124825.337dc1c6@anarchist.wooz.org>
 <654D3177-05D2-403F-9531-136FC98ED73F@stufft.io>
Message-ID: <CADiSq7dEUFeVGLKV3ft3+jLYZqPrnp9K+QSs9q4J1eg3QZysNQ@mail.gmail.com>

On 19 March 2015 at 07:51, Donald Stufft <donald at stufft.io> wrote:
> I?ve long wished that the OS had it?s own virtual environment. A lot of problems
> seems to come from trying to cram the things the OS wants with the things that
> the user wants into the same namespace.

I'm more wanting to go in the other direction and suggest to folks
that if they're not *scripting the OS*, then the system Python isn't
what they want, and they should be using at least a virtual
environment, preferably an entirely separate software collection that
they can upgrade on their own timeline, independently of what they
system does.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat Mar 21 13:11:41 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 21 Mar 2015 22:11:41 +1000
Subject: [Python-Dev] super() does not work during class initialization
In-Reply-To: <CAK9R32QaTyqsugF1_3-pPHfDsaEa4JEp0A_DFBboNOFwP_x82Q@mail.gmail.com>
References: <CAK9R32QaTyqsugF1_3-pPHfDsaEa4JEp0A_DFBboNOFwP_x82Q@mail.gmail.com>
Message-ID: <CADiSq7fxRDGg5kE8gjX-q1=CkMi0t_Q7UwigoqNUZqi43Nw9FA@mail.gmail.com>

On 21 March 2015 at 00:03, Martin Teichmann <lkb.teichmann at gmail.com> wrote:
> The current python fails the assertion, while with my patch everything is fine,
> and I personally think __class__ should always actually refer to the class being
> defined, which means at the minimum that it is actually, well, a class.

For folks that haven't looked at the tracker issue: I personally like
the change, but it does involve storing the cell object in a
dunder-variable in the class namespace while it's being defined (until
type.__new__ executes and both populates it and removes it from the
class namespace).

Since it introduces new behaviour that's visible to Python level code,
I've suggested that Martin roll the suggestion into his current PEP
487 (which adds __init_subclass__ to similarly tidy up a few
challenges with the way classes are currently initialised).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat Mar 21 13:14:45 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 21 Mar 2015 22:14:45 +1000
Subject: [Python-Dev] Fwd: class os.DirEntry is confusing,
In-Reply-To: <CAE2XoE-fuzz4Uu3ZcTq1FrenmEY5avC4AYFK3ANAZVATrVaPyA@mail.gmail.com>
References: <CAE2XoE_YZCgZSQJqdQgZ1kkT_g9U2Dxqq2HwNAfotAXdHtJHfQ@mail.gmail.com>
 <CAE2XoE-fuzz4Uu3ZcTq1FrenmEY5avC4AYFK3ANAZVATrVaPyA@mail.gmail.com>
Message-ID: <CADiSq7d6a97LnSKN_yhCaU-FYmu6Zod1Q1Q+zyCLUpUhEWiCVw@mail.gmail.com>

On 21 March 2015 at 10:15, ???(Yonggang Luo) <luoyonggang at gmail.com> wrote:
> ---------- Forwarded message ----------
> From: ???(Yonggang Luo) <luoyonggang at gmail.com>
> Date: 2015-03-21 2:24 GMT+08:00
> Subject: class os.DirEntry is confusing,
> To: Python <python-ideas at python.org>
>
>
> Should be replaced with os.FsEntry or os.ScanEntry

The name comes from the underlying C/POSIX concept (dirent in this
case), which is fairly common for the low level APIs in the os module.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From solipsis at pitrou.net  Sat Mar 21 13:19:40 2015
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 21 Mar 2015 13:19:40 +0100
Subject: [Python-Dev] Use ptyhon -s as default shbang for system python
 executables/daemons
References: <loom.20150318T164041-536@post.gmane.org>
 <CAMpsgwa3F2Abnptu8uFtxzCD_T-x9BCK9KtuOsVDW3kF1N88-g@mail.gmail.com>
 <20150318124825.337dc1c6@anarchist.wooz.org>
 <654D3177-05D2-403F-9531-136FC98ED73F@stufft.io>
 <CADiSq7dEUFeVGLKV3ft3+jLYZqPrnp9K+QSs9q4J1eg3QZysNQ@mail.gmail.com>
Message-ID: <20150321131940.6291fdaa@fsol>

On Sat, 21 Mar 2015 21:52:34 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 19 March 2015 at 07:51, Donald Stufft <donald at stufft.io> wrote:
> > I?ve long wished that the OS had it?s own virtual environment. A lot of problems
> > seems to come from trying to cram the things the OS wants with the things that
> > the user wants into the same namespace.
> 
> I'm more wanting to go in the other direction and suggest to folks
> that if they're not *scripting the OS*, then the system Python isn't
> what they want, and they should be using at least a virtual
> environment,

So that defeats the whole point of having a system distribution.
I don't have any stakes in Debian / RedHat etc., but that sounds
a little bit ridiculous to me ;-) On the other hand, perhaps it would
allow ditching any requests to keep code compatible with old versions
such as 2.6.

Regards

Antoine.



From bcannon at gmail.com  Sat Mar 21 14:34:19 2015
From: bcannon at gmail.com (Brett Cannon)
Date: Sat, 21 Mar 2015 13:34:19 +0000
Subject: [Python-Dev] Final call for PEP 488: eliminating PYO files
In-Reply-To: <CAP7+vJLyGoxhfWE+AED1CKX-v_FRZm+ijDy7gw9hD07TG_azpQ@mail.gmail.com>
References: <CAP1=2W4CejbRcjjYUoFdFReUrgZ8i_mm1vJE2aoVqqo0Q3BW=Q@mail.gmail.com>
 <CAP7+vJLOaJAzXDq-sedZR9KsLN5kkokYxB8YFx3TPjayB7ggmA@mail.gmail.com>
 <CAP1=2W5CpPdszgqrsbhgx5nQXCm5bOYUH7aXZ0vfdK3FyOk7Dg@mail.gmail.com>
 <CAP7+vJLyGoxhfWE+AED1CKX-v_FRZm+ijDy7gw9hD07TG_azpQ@mail.gmail.com>
Message-ID: <CAP1=2W4AqxhKM+Rfy0e_qVWPG6Wdn2kiqFDut588+JwySWeqGQ@mail.gmail.com>

Thanks! PEP 488 is now marked as accepted. I expect I will have PEP 488
implemented before the PyCon sprints are over (work will be tracked in
http://bugs.python.org/issue23731).

On Fri, Mar 20, 2015 at 8:06 PM Guido van Rossum <guido at python.org> wrote:

> Awesome, that's what I was hoping. Accepted! Congrats and thank you very
> much for writing the PEP and guiding the discussion.
>
> On Fri, Mar 20, 2015 at 4:00 PM, Brett Cannon <bcannon at gmail.com> wrote:
>
>>
>>
>> On Fri, Mar 20, 2015 at 4:41 PM Guido van Rossum <guido at python.org>
>> wrote:
>>
>>> I am willing to be the BDFL for this PEP. I have tried to skim the
>>> recent discussion (only python-dev) and I don't see much remaining
>>> controversy. HOWEVER... The PEP is not clear (or at least too subtle) about
>>> the actual name for optimization level 0. If I have foo.py, and I compile
>>> it three times with three different optimization levels (no optimization;
>>> -O; -OO), and then I look in __pycache__, would I see this:
>>>
>>> # (1)
>>> foo.cpython-35.pyc
>>> foo.cpython-35.opt-1.pyc
>>> foo.cpython-35.opt-2.pyc
>>>
>>> Or would I see this?
>>>
>>> # (2)
>>> foo.cpython-35.opt-0.pyc
>>> foo.cpython-35.opt-1.pyc
>>> foo.cpython-35.opt-2.pyc
>>>
>>
>> #1
>>
>>
>>>
>>> Your lead-in ("I have decided to have the default case of no
>>> optimization levels mean that the .pyc file name will have *no* optimization
>>> level specified in the name and thus be just as it is today.") makes me
>>> think I should expect (1), but I can't actually pinpoint where the language
>>> of the PEP says this.
>>>
>>
>> It was meant to be explained by "When no optimization level is
>> specified, the pre-PEP ``.pyc`` file name will be used (i.e., no change
>> in file name
>> semantics)", but obviously it's a bit too subtle. I just updated the PEP
>> with an explicit list of bytecode file name examples based on no -O, -O,
>> and -OO.
>>
>> -Brett
>>
>>
>>>
>>>
>>> On Fri, Mar 20, 2015 at 11:34 AM, Brett Cannon <bcannon at gmail.com>
>>> wrote:
>>>
>>>> I have decided to have the default case of no optimization levels mean
>>>> that the .pyc file name will have *no* optimization level specified in
>>>> the name and thus be just as it is today. I made this decision due to
>>>> potential backwards-compatibility issues -- although I expect them to be
>>>> minutes -- and to not force other implementations like PyPy to have some
>>>> bogus value set since they don't have .pyo files to begin with (PyPy
>>>> actually uses bytecode for -O and don't bother with -OO since PyPy already
>>>> uses a bunch of memory when running).
>>>>
>>>> Since this closes out the last open issue, I need either a BDFL
>>>> decision or a BDFAP to be assigned to make a decision. Guido?
>>>>
>>>> ======================================
>>>>
>>>> PEP: 488
>>>> Title: Elimination of PYO files
>>>> Version: $Revision$
>>>> Last-Modified: $Date$
>>>> Author: Brett Cannon <brett at python.org>
>>>> Status: Draft
>>>> Type: Standards Track
>>>> Content-Type: text/x-rst
>>>> Created: 20-Feb-2015
>>>> Post-History:
>>>>     2015-03-06
>>>>     2015-03-13
>>>>     2015-03-20
>>>>
>>>> Abstract
>>>> ========
>>>>
>>>> This PEP proposes eliminating the concept of PYO files from Python.
>>>> To continue the support of the separation of bytecode files based on
>>>> their optimization level, this PEP proposes extending the PYC file
>>>> name to include the optimization level in the bytecode repository
>>>> directory when it's called for (i.e., the ``__pycache__`` directory).
>>>>
>>>>
>>>> Rationale
>>>> =========
>>>>
>>>> As of today, bytecode files come in two flavours: PYC and PYO. A PYC
>>>> file is the bytecode file generated and read from when no
>>>> optimization level is specified at interpreter startup (i.e., ``-O``
>>>> is not specified). A PYO file represents the bytecode file that is
>>>> read/written when **any** optimization level is specified (i.e., when
>>>> ``-O`` **or** ``-OO`` is specified). This means that while PYC
>>>> files clearly delineate the optimization level used when they were
>>>> generated -- namely no optimizations beyond the peepholer -- the same
>>>> is not true for PYO files. To put this in terms of optimization
>>>> levels and the file extension:
>>>>
>>>>   - 0: ``.pyc``
>>>>   - 1 (``-O``): ``.pyo``
>>>>   - 2 (``-OO``): ``.pyo``
>>>>
>>>> The reuse of the ``.pyo`` file extension for both level 1 and 2
>>>> optimizations means that there is no clear way to tell what
>>>> optimization level was used to generate the bytecode file. In terms
>>>> of reading PYO files, this can lead to an interpreter using a mixture
>>>> of optimization levels with its code if the user was not careful to
>>>> make sure all PYO files were generated using the same optimization
>>>> level (typically done by blindly deleting all PYO files and then
>>>> using the `compileall` module to compile all-new PYO files [1]_).
>>>> This issue is only compounded when people optimize Python code beyond
>>>> what the interpreter natively supports, e.g., using the astoptimizer
>>>> project [2]_.
>>>>
>>>> In terms of writing PYO files, the need to delete all PYO files
>>>> every time one either changes the optimization level they want to use
>>>> or are unsure of what optimization was used the last time PYO files
>>>> were generated leads to unnecessary file churn. The change proposed
>>>> by this PEP also allows for **all** optimization levels to be
>>>> pre-compiled for bytecode files ahead of time, something that is
>>>> currently impossible thanks to the reuse of the ``.pyo`` file
>>>> extension for multiple optimization levels.
>>>>
>>>> As for distributing bytecode-only modules, having to distribute both
>>>> ``.pyc`` and ``.pyo`` files is unnecessary for the common use-case
>>>> of code obfuscation and smaller file deployments. This means that
>>>> bytecode-only modules will only load from their non-optimized
>>>> ``.pyc`` file name.
>>>>
>>>>
>>>> Proposal
>>>> ========
>>>>
>>>> To eliminate the ambiguity that PYO files present, this PEP proposes
>>>> eliminating the concept of PYO files and their accompanying ``.pyo``
>>>> file extension. To allow for the optimization level to be unambiguous
>>>> as well as to avoid having to regenerate optimized bytecode files
>>>> needlessly in the `__pycache__` directory, the optimization level
>>>> used to generate the bytecode file will be incorporated into the
>>>> bytecode file name. When no optimization level is specified, the
>>>> pre-PEP ``.pyc`` file name will be used (i.e., no change in file name
>>>> semantics). This increases backwards-compatibility while also being
>>>> more understanding of Python implementations which have no use for
>>>> optimization levels (e.g., PyPy[10]_).
>>>>
>>>> Currently bytecode file names are created by
>>>> ``importlib.util.cache_from_source()``, approximately using the
>>>> following expression defined by PEP 3147 [3]_, [4]_, [5]_::
>>>>
>>>>     '{name}.{cache_tag}.pyc'.format(name=module_name,
>>>>
>>>> cache_tag=sys.implementation.cache_tag)
>>>>
>>>> This PEP proposes to change the expression when an optimization
>>>> level is specified to::
>>>>
>>>>     '{name}.{cache_tag}.opt-{optimization}.pyc'.format(
>>>>             name=module_name,
>>>>             cache_tag=sys.implementation.cache_tag,
>>>>             optimization=str(sys.flags.optimize))
>>>>
>>>> The "opt-" prefix was chosen so as to provide a visual separator
>>>> from the cache tag. The placement of the optimization level after
>>>> the cache tag was chosen to preserve lexicographic sort order of
>>>> bytecode file names based on module name and cache tag which will
>>>> not vary for a single interpreter. The "opt-" prefix was chosen over
>>>> "o" so as to be somewhat self-documenting. The "opt-" prefix was
>>>> chosen over "O" so as to not have any confusion in case "0" was the
>>>> leading prefix of the optimization level.
>>>>
>>>> A period was chosen over a hyphen as a separator so as to distinguish
>>>> clearly that the optimization level is not part of the interpreter
>>>> version as specified by the cache tag. It also lends to the use of
>>>> the period in the file name to delineate semantically different
>>>> concepts.
>>>>
>>>> For example, if ``-OO`` had been passed to the interpreter then instead
>>>> of ``importlib.cpython-35.pyo`` the file name would be
>>>> ``importlib.cpython-35.opt-2.pyc``.
>>>>
>>>> It should be noted that this change in no way affects the performance
>>>> of import. Since the import system looks for a single bytecode file
>>>> based on the optimization level of the interpreter already and
>>>> generates a new bytecode file if it doesn't exist, the introduction
>>>> of potentially more bytecode files in the ``__pycache__`` directory
>>>> has no effect in terms of stat calls. The interpreter will continue
>>>> to look for only a single bytecode file based on the optimization
>>>> level and thus no increase in stat calls will occur.
>>>>
>>>> The only potentially negative result of this PEP is the probable
>>>> increase in the number of ``.pyc`` files and thus increase in storage
>>>> use. But for platforms where this is an issue,
>>>> ``sys.dont_write_bytecode`` exists to turn off bytecode generation so
>>>> that it can be controlled offline.
>>>>
>>>>
>>>> Implementation
>>>> ==============
>>>>
>>>> importlib
>>>> ---------
>>>>
>>>> As ``importlib.util.cache_from_source()`` is the API that exposes
>>>> bytecode file paths as well as being directly used by importlib, it
>>>> requires the most critical change. As of Python 3.4, the function's
>>>> signature is::
>>>>
>>>>   importlib.util.cache_from_source(path, debug_override=None)
>>>>
>>>> This PEP proposes changing the signature in Python 3.5 to::
>>>>
>>>>   importlib.util.cache_from_source(path, debug_override=None, *,
>>>> optimization=None)
>>>>
>>>> The introduced ``optimization`` keyword-only parameter will control
>>>> what optimization level is specified in the file name. If the
>>>> argument is ``None`` then the current optimization level of the
>>>> interpreter will be assumed (including no optimization). Any argument
>>>> given for ``optimization`` will be passed to ``str()`` and must have
>>>> ``str.isalnum()`` be true, else ``ValueError`` will be raised (this
>>>> prevents invalid characters being used in the file name). If the
>>>> empty string is passed in for ``optimization`` then the addition of
>>>> the optimization will be suppressed, reverting to the file name
>>>> format which predates this PEP.
>>>>
>>>> It is expected that beyond Python's own two optimization levels,
>>>> third-party code will use a hash of optimization names to specify the
>>>> optimization level, e.g.
>>>> ``hashlib.sha256(','.join(['no dead code', 'const
>>>> folding'])).hexdigest()``.
>>>> While this might lead to long file names, it is assumed that most
>>>> users never look at the contents of the __pycache__ directory and so
>>>> this won't be an issue.
>>>>
>>>> The ``debug_override`` parameter will be deprecated. As the parameter
>>>> expects a boolean, the integer value of the boolean will be used as
>>>> if it had been provided as the argument to ``optimization`` (a
>>>> ``None`` argument will mean the same as for ``optimization``). A
>>>> deprecation warning will be raised when ``debug_override`` is given a
>>>> value other than ``None``, but there are no plans for the complete
>>>> removal of the parameter at this time (but removal will be no later
>>>> than Python 4).
>>>>
>>>> The various module attributes for importlib.machinery which relate to
>>>> bytecode file suffixes will be updated [7]_. The
>>>> ``DEBUG_BYTECODE_SUFFIXES`` and ``OPTIMIZED_BYTECODE_SUFFIXES`` will
>>>> both be documented as deprecated and set to the same value as
>>>> ``BYTECODE_SUFFIXES`` (removal of ``DEBUG_BYTECODE_SUFFIXES`` and
>>>> ``OPTIMIZED_BYTECODE_SUFFIXES`` is not currently planned, but will be
>>>> not later than Python 4).
>>>>
>>>> All various finders and loaders will also be updated as necessary,
>>>> but updating the previous mentioned parts of importlib should be all
>>>> that is required.
>>>>
>>>>
>>>> Rest of the standard library
>>>> ----------------------------
>>>>
>>>> The various functions exposed by the ``py_compile`` and
>>>> ``compileall`` functions will be updated as necessary to make sure
>>>> they follow the new bytecode file name semantics [6]_, [1]_. The CLI
>>>> for the ``compileall`` module will not be directly affected (the
>>>> ``-b`` flag will be implicit as it will no longer generate ``.pyo``
>>>> files when ``-O`` is specified).
>>>>
>>>>
>>>> Compatibility Considerations
>>>> ============================
>>>>
>>>> Any code directly manipulating bytecode files from Python 3.2 on
>>>> will need to consider the impact of this change on their code (prior
>>>> to Python 3.2 -- including all of Python 2 -- there was no
>>>> __pycache__ which already necessitates bifurcating bytecode file
>>>> handling support). If code was setting the ``debug_override``
>>>> argument to ``importlib.util.cache_from_source()`` then care will be
>>>> needed if they want the path to a bytecode file with an optimization
>>>> level of 2. Otherwise only code **not** using
>>>> ``importlib.util.cache_from_source()`` will need updating.
>>>>
>>>> As for people who distribute bytecode-only modules (i.e., use a
>>>> bytecode file instead of a source file), they will have to choose
>>>> which optimization level they want their bytecode files to be since
>>>> distributing a ``.pyo`` file with a ``.pyc`` file will no longer be
>>>> of any use. Since people typically only distribute bytecode files for
>>>> code obfuscation purposes or smaller distribution size then only
>>>> having to distribute a single ``.pyc`` should actually be beneficial
>>>> to these use-cases. And since the magic number for bytecode files
>>>> changed in Python 3.5 to support PEP 465 there is no need to support
>>>> pre-existing ``.pyo`` files [8]_.
>>>>
>>>>
>>>> Rejected Ideas
>>>> ==============
>>>>
>>>> Completely dropping optimization levels from CPython
>>>> ----------------------------------------------------
>>>>
>>>> Some have suggested that instead of accommodating the various
>>>> optimization levels in CPython, we should instead drop them
>>>> entirely. The argument is that significant performance gains would
>>>> occur from runtime optimizations through something like a JIT and not
>>>> through pre-execution bytecode optimizations.
>>>>
>>>> This idea is rejected for this PEP as that ignores the fact that
>>>> there are people who do find the pre-existing optimization levels for
>>>> CPython useful. It also assumes that no other Python interpreter
>>>> would find what this PEP proposes useful.
>>>>
>>>>
>>>> Alternative formatting of the optimization level in the file name
>>>> -----------------------------------------------------------------
>>>>
>>>> Using the "opt-" prefix and placing the optimization level between
>>>> the cache tag and file extension is not critical. All options which
>>>> have been considered are:
>>>>
>>>> * ``importlib.cpython-35.opt-1.pyc``
>>>> * ``importlib.cpython-35.opt1.pyc``
>>>> * ``importlib.cpython-35.o1.pyc``
>>>> * ``importlib.cpython-35.O1.pyc``
>>>> * ``importlib.cpython-35.1.pyc``
>>>> * ``importlib.cpython-35-O1.pyc``
>>>> * ``importlib.O1.cpython-35.pyc``
>>>> * ``importlib.o1.cpython-35.pyc``
>>>> * ``importlib.1.cpython-35.pyc``
>>>>
>>>> These were initially rejected either because they would change the
>>>> sort order of bytecode files, possible ambiguity with the cache tag,
>>>> or were not self-documenting enough. An informal poll was taken and
>>>> people clearly preferred the formatting proposed by the PEP [9]_.
>>>> Since this topic is non-technical and of personal choice, the issue
>>>> is considered solved.
>>>>
>>>>
>>>> Embedding the optimization level in the bytecode metadata
>>>> ---------------------------------------------------------
>>>>
>>>> Some have suggested that rather than embedding the optimization level
>>>> of bytecode in the file name that it be included in the file's
>>>> metadata instead. This would mean every interpreter had a single copy
>>>> of bytecode at any time. Changing the optimization level would thus
>>>> require rewriting the bytecode, but there would also only be a single
>>>> file to care about.
>>>>
>>>> This has been rejected due to the fact that Python is often installed
>>>> as a root-level application and thus modifying the bytecode file for
>>>> modules in the standard library are always possible. In this
>>>> situation integrators would need to guess at what a reasonable
>>>> optimization level was for users for any/all situations. By
>>>> allowing multiple optimization levels to co-exist simultaneously it
>>>> frees integrators from having to guess what users want and allows
>>>> users to utilize the optimization level they want.
>>>>
>>>>
>>>> References
>>>> ==========
>>>>
>>>> .. [1] The compileall module
>>>>    (https://docs.python.org/3/library/compileall.html#module-compileall
>>>> )
>>>>
>>>> .. [2] The astoptimizer project
>>>>    (https://pypi.python.org/pypi/astoptimizer)
>>>>
>>>> .. [3] ``importlib.util.cache_from_source()``
>>>>    (
>>>> https://docs.python.org/3.5/library/importlib.html#importlib.util.cache_from_source
>>>> )
>>>>
>>>> .. [4] Implementation of ``importlib.util.cache_from_source()`` from
>>>> CPython 3.4.3rc1
>>>>    (
>>>> https://hg.python.org/cpython/file/038297948389/Lib/importlib/_bootstrap.py#l437
>>>> )
>>>>
>>>> .. [5] PEP 3147, PYC Repository Directories, Warsaw
>>>>    (http://www.python.org/dev/peps/pep-3147)
>>>>
>>>> .. [6] The py_compile module
>>>>    (https://docs.python.org/3/library/compileall.html#module-compileall
>>>> )
>>>>
>>>> .. [7] The importlib.machinery module
>>>>    (
>>>> https://docs.python.org/3/library/importlib.html#module-importlib.machinery
>>>> )
>>>>
>>>> .. [8] ``importlib.util.MAGIC_NUMBER``
>>>>    (
>>>> https://docs.python.org/3/library/importlib.html#importlib.util.MAGIC_NUMBER
>>>> )
>>>>
>>>> .. [9] Informal poll of file name format options on Google+
>>>>    (https://plus.google.com/u/0/+BrettCannon/posts/fZynLNwHWGm)
>>>>
>>>> .. [10] The PyPy Project
>>>>    (http://pypy.org/)
>>>>
>>>>
>>>> Copyright
>>>> =========
>>>>
>>>> This document has been placed in the public domain.
>>>>
>>>>
>>>> ..
>>>>    Local Variables:
>>>>    mode: indented-text
>>>>    indent-tabs-mode: nil
>>>>    sentence-end-double-space: t
>>>>    fill-column: 70
>>>>    coding: utf-8
>>>>    End:
>>>>
>>>>
>>>> _______________________________________________
>>>> Python-Dev mailing list
>>>> Python-Dev at python.org
>>>> https://mail.python.org/mailman/listinfo/python-dev
>>>> Unsubscribe:
>>>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>>>
>>>>
>>>
>>>
>>> --
>>> --Guido van Rossum (python.org/~guido)
>>>
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150321/f26d8a0f/attachment-0001.html>

From ncoghlan at gmail.com  Sat Mar 21 15:16:21 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 22 Mar 2015 00:16:21 +1000
Subject: [Python-Dev] Use ptyhon -s as default shbang for system python
	executables/daemons
In-Reply-To: <20150321131940.6291fdaa@fsol>
References: <loom.20150318T164041-536@post.gmane.org>
 <CAMpsgwa3F2Abnptu8uFtxzCD_T-x9BCK9KtuOsVDW3kF1N88-g@mail.gmail.com>
 <20150318124825.337dc1c6@anarchist.wooz.org>
 <654D3177-05D2-403F-9531-136FC98ED73F@stufft.io>
 <CADiSq7dEUFeVGLKV3ft3+jLYZqPrnp9K+QSs9q4J1eg3QZysNQ@mail.gmail.com>
 <20150321131940.6291fdaa@fsol>
Message-ID: <CADiSq7e2ojFbsvZXQfcdGqwgsSgPBNGDWdfzjQwouvh5Hc5RnA@mail.gmail.com>

On 21 March 2015 at 22:19, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sat, 21 Mar 2015 21:52:34 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
>> On 19 March 2015 at 07:51, Donald Stufft <donald at stufft.io> wrote:
>> > I?ve long wished that the OS had it?s own virtual environment. A lot of problems
>> > seems to come from trying to cram the things the OS wants with the things that
>> > the user wants into the same namespace.
>>
>> I'm more wanting to go in the other direction and suggest to folks
>> that if they're not *scripting the OS*, then the system Python isn't
>> what they want, and they should be using at least a virtual
>> environment,
>
> So that defeats the whole point of having a system distribution.

Given how many of the operating system packages are written in Python,
it really doesn't - system administrators getting to use Python to
script the OS is a secondary purpose, the system Python installation
is mostly there to make the OS itself work (if it wasn't for that,
migrating the system Python to Python 3 would have been a lot
simpler!).

In my view, running system independent applications in the system
Python is at best a tertiary purpose - many of those apps (especially
web services) *don't want* to be using system modules at all, which is
why virtualenv switched to disabling the global site-packages by
default.

> I don't have any stakes in Debian / RedHat etc., but that sounds
> a little bit ridiculous to me ;-) On the other hand, perhaps it would
> allow ditching any requests to keep code compatible with old versions
> such as 2.6.

That's exactly the reason for the preference - unless they're
specifically wanting to script RHEL/CentOS 6 or a similar long term
support version at this point, ideally folks should be able to use
Python 2.7 as their baseline version by now. RHEL/CentOS 7 are both on
2.7, while Python 2.7 has been available for RHEL/CentOS 6 by way of
software collections for a couple of years now.

When Python 2 goes EOL upstream in 2020, the standard 10 years of
support on RHEL 7 will still have 4 years left to run - I'd prefer to
see most folks migrated to a Python 3 software collection by then (or
failing that, at least to the Python 3 stack Slavek is aiming to get
into EPEL 7).

Unfortunately, as far as I am aware, Debian Stable and Ubuntu LTS
don't yet have a distro-endorsed equivalent of
softwarecollections.org, and it's not yet clear if running on a faster
moving base distro in a Docker or Rocket container will be an adequate
alternative.

I believe SCLs moved over to using Linux environment modules as the
backend tech in 2.0 (for compatibility with existing HPC
environments), so it should likely be possible to adapt them to work
with APT in addition to RPM, but that requires finding folks both
interested in doing the technical work, as well as in making the case
for SCLs as a feature in the relevant distros (they're a large enough
departure from the traditional mode of operation of a Linux distro
that it took a long time to find a stable upstream home for the
ongoing maintenance of SCLs for the Fedora/RHEL/CentOS ecosystem -
primary maintenance responsibility finally landed earlier this year as
a CentOS SIG)

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ericsnowcurrently at gmail.com  Sat Mar 21 18:18:12 2015
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Sat, 21 Mar 2015 11:18:12 -0600
Subject: [Python-Dev] Final call for PEP 488: eliminating PYO files
In-Reply-To: <CAP1=2W4AqxhKM+Rfy0e_qVWPG6Wdn2kiqFDut588+JwySWeqGQ@mail.gmail.com>
References: <CAP1=2W4CejbRcjjYUoFdFReUrgZ8i_mm1vJE2aoVqqo0Q3BW=Q@mail.gmail.com>
 <CAP7+vJLOaJAzXDq-sedZR9KsLN5kkokYxB8YFx3TPjayB7ggmA@mail.gmail.com>
 <CAP1=2W5CpPdszgqrsbhgx5nQXCm5bOYUH7aXZ0vfdK3FyOk7Dg@mail.gmail.com>
 <CAP7+vJLyGoxhfWE+AED1CKX-v_FRZm+ijDy7gw9hD07TG_azpQ@mail.gmail.com>
 <CAP1=2W4AqxhKM+Rfy0e_qVWPG6Wdn2kiqFDut588+JwySWeqGQ@mail.gmail.com>
Message-ID: <CALFfu7DdiSw2xkADocwiyquvkt9wq=qWPSN17YJXxJfsihtKug@mail.gmail.com>

On Mar 21, 2015 7:44 AM, "Brett Cannon" <bcannon at gmail.com> wrote:
>
> Thanks! PEP 488 is now marked as accepted. I expect I will have PEP 488
implemented before the PyCon sprints are over (work will be tracked in
http://bugs.python.org/issue23731).
>
> On Fri, Mar 20, 2015 at 8:06 PM Guido van Rossum <guido at python.org> wrote:
>>
>> Awesome, that's what I was hoping. Accepted! Congrats and thank you very
much for writing the PEP and guiding the discussion.

Congratulations Brett!  This is a welcome change.  I'll be sure to give you
a review.

-eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150321/d0a4c49f/attachment.html>

From donald at stufft.io  Sat Mar 21 19:29:13 2015
From: donald at stufft.io (Donald Stufft)
Date: Sat, 21 Mar 2015 14:29:13 -0400
Subject: [Python-Dev] Use ptyhon -s as default shbang for system python
	executables/daemons
In-Reply-To: <CADiSq7dEUFeVGLKV3ft3+jLYZqPrnp9K+QSs9q4J1eg3QZysNQ@mail.gmail.com>
References: <loom.20150318T164041-536@post.gmane.org>
 <CAMpsgwa3F2Abnptu8uFtxzCD_T-x9BCK9KtuOsVDW3kF1N88-g@mail.gmail.com>
 <20150318124825.337dc1c6@anarchist.wooz.org>
 <654D3177-05D2-403F-9531-136FC98ED73F@stufft.io>
 <CADiSq7dEUFeVGLKV3ft3+jLYZqPrnp9K+QSs9q4J1eg3QZysNQ@mail.gmail.com>
Message-ID: <A5D52925-8F0C-49FC-9A11-AE12BBD8404C@stufft.io>


> On Mar 21, 2015, at 7:52 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> On 19 March 2015 at 07:51, Donald Stufft <donald at stufft.io> wrote:
>> I?ve long wished that the OS had it?s own virtual environment. A lot of problems
>> seems to come from trying to cram the things the OS wants with the things that
>> the user wants into the same namespace.
> 
> I'm more wanting to go in the other direction and suggest to folks
> that if they're not *scripting the OS*, then the system Python isn't
> what they want, and they should be using at least a virtual
> environment, preferably an entirely separate software collection that
> they can upgrade on their own timeline, independently of what they
> system does.
> 

It?s likely easier to get the OS to move it?s own things to a virtual
environment than it is to convince every single person who uses an OS
to never install globally.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150321/2333bb7d/attachment.sig>

From tjreedy at udel.edu  Sat Mar 21 19:47:42 2015
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 21 Mar 2015 14:47:42 -0400
Subject: [Python-Dev] How to document functions with optional positional
	parameters?
In-Reply-To: <meispe$vth$1@ger.gmane.org>
References: <meispe$vth$1@ger.gmane.org>
Message-ID: <mekecr$ra$1@ger.gmane.org>

On 3/21/2015 12:41 AM, Serhiy Storchaka wrote:
> How to document functions with optional positional parameters?
>
> For example binascii.crc32(). It has two positional parameters, one is
> mandatory, and one is optional with default value 0. With Argument
> Clinic its signature is crc32(data, crc=0, /). In the documentation it
> is written as crc32(data[, crc]) (and noted that the default value of
> the second parameter is 0). Both are not valid Python syntax. Can the
> documentation be change to crc32(data, crc=0)?

I think the docs should be using / as well.

> Related issues:
>
> https://bugs.python.org/issue21488 (changed encode(obj,
> encoding='ascii', errors='strict') to encode(obj, [encoding[, errors]]))
>
> https://bugs.python.org/issue22832 (changed ioctl(fd, op[, arg[,
> mutate_flag]]) to ioctl(fd, request, arg=0, mutate_flag=True))
>
> https://bugs.python.org/issue22341 (discussed changing crc32(data[,
> crc]) to crc32(data, crc=0))
>


-- 
Terry Jan Reedy


From barry at python.org  Sat Mar 21 21:53:50 2015
From: barry at python.org (Barry Warsaw)
Date: Sat, 21 Mar 2015 16:53:50 -0400
Subject: [Python-Dev] some minor questions about pep8
In-Reply-To: <CAP7+vJJPnqUMA=fpz+arV1ZZrT6dGU5d+KNJM5ZLmrRGumkLKQ@mail.gmail.com>
References: <CAFmnkTBYpaJRaeo_ff6KKVwLTszfF1Zah+yGvg7_yq82bSTmfw@mail.gmail.com>
 <85lhir6seo.fsf@benfinney.id.au>
 <CAP7+vJJPnqUMA=fpz+arV1ZZrT6dGU5d+KNJM5ZLmrRGumkLKQ@mail.gmail.com>
Message-ID: <20150321165350.726db103@anarchist.wooz.org>

On Mar 20, 2015, at 08:53 PM, Guido van Rossum wrote:

>FWIW, I think __version__, __author__ etc. were bad ideas. Almost nobody
>manages these correctly. Note that the PEP 8 section starts with less than
>an endorsement: "If you *have* to have Subversion, CVS, or RCS crud in your
>source file, do it as follows."

I tend to agree.  Individual module version numbers are mostly useless,
especially with today's hash-based vcses.

That's different than package versions, and for which I really need to
resurrect and update PEP 396.

>That said, if an official answer is required, common sense would suggest
>that __version__ should go before the imports. (I would put it before the
>docstring too, except then the docstring wouldn't be a docstring any more.
>Go figure.)

And after __future__ imports too, right?

Cheers,
-Barry

From jim.baker at python.org  Sat Mar 21 21:14:03 2015
From: jim.baker at python.org (Jim Baker)
Date: Sat, 21 Mar 2015 14:14:03 -0600
Subject: [Python-Dev] Google Summer of Code
Message-ID: <CAOhO=aNQrXNwC2W2BTUG9VCKYj9+gJateq6iySUcJdT0BYhDww@mail.gmail.com>

Jython plans to participate in the Google Summer of Code for 2015. If you
are interested, I have outlined a number of projects on our ideas page that
students could work on:

   - Work on JyNI, which adds C extension API support to Jython
   - Performance optimizations, including startup time
   - Python bytecode compiler for Python source (must be written in Java!)

See https://github.com/jythontools/gsoc2015 for the ideas page. Pull
requests and comments gladly accepted to expand this list of ideas.

The ideal student must be a very strong Java developer, with an existing
track record of projects on such sites as GitHub and BitBucket. Knowledge
of C and especially of the CPython source code invariably helps. Ideally
you will also become a Jython committer as a result of your intense
participation this summer.

Note that the deadline for student proposals is very soon, Friday March 27.
I apologize we are just getting our participation for GSOC together, but we
have also been busy finalizing the Jython 2.7.0 release candidate.


-- 
- Jim

jim.baker@{colorado.edu|python.org|rackspace.com|zyasoft.com}
twitter.com/jimbaker
github.com/jimbaker
bitbucket.com/jimbaker
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150321/81389584/attachment.html>

From barry at python.org  Sat Mar 21 21:56:30 2015
From: barry at python.org (Barry Warsaw)
Date: Sat, 21 Mar 2015 16:56:30 -0400
Subject: [Python-Dev] some minor questions about pep8
In-Reply-To: <CA+GyjMksfgG3mWbPCXhJ=y_etPHZzuoNWgWNru2oEzvagyPiOA@mail.gmail.com>
References: <CAFmnkTBYpaJRaeo_ff6KKVwLTszfF1Zah+yGvg7_yq82bSTmfw@mail.gmail.com>
 <85lhir6seo.fsf@benfinney.id.au>
 <CAP7+vJJPnqUMA=fpz+arV1ZZrT6dGU5d+KNJM5ZLmrRGumkLKQ@mail.gmail.com>
 <CA+GyjMntcKaRUjfOzZg82q=13kjeCkKAYPT=ovhQUokqnPaXng@mail.gmail.com>
 <CA+GyjMksfgG3mWbPCXhJ=y_etPHZzuoNWgWNru2oEzvagyPiOA@mail.gmail.com>
Message-ID: <20150321165630.3d6bd1a7@anarchist.wooz.org>

On Mar 20, 2015, at 09:59 PM, Ian Lee wrote:

>shebang
>docstring
>imports
>dunder assignments
>other code...

I generally put imports after dunders, except for __future__.  I want the
first screenful of code to contain them, and imports can be a few dozen lines
in some modules.

-Barry

From benjamin at python.org  Sat Mar 21 21:57:22 2015
From: benjamin at python.org (Benjamin Peterson)
Date: Sat, 21 Mar 2015 16:57:22 -0400
Subject: [Python-Dev] Missing *-unpacking generalizations (issue 2292)
In-Reply-To: <CAA68w_kp998DcJt70s7w1C0mUifwb1YWKF7oDrH4QjY0NAAV-A@mail.gmail.com>
References: <20150319175353.26159.19083@psf.upfronthosting.co.za>
 <CAA68w_kp998DcJt70s7w1C0mUifwb1YWKF7oDrH4QjY0NAAV-A@mail.gmail.com>
Message-ID: <1426971442.2005452.243475606.2CF0A0A3@webmail.messagingengine.com>



On Fri, Mar 20, 2015, at 19:16, Neil Girdhar wrote:
> Wow, this is an excellent review.  Thank you.
> 
> My only question is with respect to this:
> 
> I think there ought to be two opcodes; one for unpacking maps in
> function calls and another for literals. The whole function location
> thing is rather hideous.
> 
> What are the two opcodes?  BUILD_MAP_UNPACK and
> BUILD_MAP_UNPACK_WITH_CALL?
> 
> The first takes (n) a number of maps that it will merge, and the second
> does the same but also accepts (function_call_location) for the purpose
> of
> error reporting?

Exactly.

From donald at stufft.io  Sat Mar 21 22:00:11 2015
From: donald at stufft.io (Donald Stufft)
Date: Sat, 21 Mar 2015 17:00:11 -0400
Subject: [Python-Dev] some minor questions about pep8
In-Reply-To: <20150321165350.726db103@anarchist.wooz.org>
References: <CAFmnkTBYpaJRaeo_ff6KKVwLTszfF1Zah+yGvg7_yq82bSTmfw@mail.gmail.com>
 <85lhir6seo.fsf@benfinney.id.au>
 <CAP7+vJJPnqUMA=fpz+arV1ZZrT6dGU5d+KNJM5ZLmrRGumkLKQ@mail.gmail.com>
 <20150321165350.726db103@anarchist.wooz.org>
Message-ID: <B08A3E43-C525-43CC-AC37-6906C9082F89@stufft.io>


> On Mar 21, 2015, at 4:53 PM, Barry Warsaw <barry at python.org> wrote:
> 
> On Mar 20, 2015, at 08:53 PM, Guido van Rossum wrote:
> 
>> FWIW, I think __version__, __author__ etc. were bad ideas. Almost nobody
>> manages these correctly. Note that the PEP 8 section starts with less than
>> an endorsement: "If you *have* to have Subversion, CVS, or RCS crud in your
>> source file, do it as follows."
> 
> I tend to agree.  Individual module version numbers are mostly useless,
> especially with today's hash-based vcses.
> 
> That's different than package versions, and for which I really need to
> resurrect and update PEP 396.

I sort of think (though I haven?t completely convinced myself) that adding
something like __version__ to a package is a work around to the fact that
we don?t have an API that lets someone ask ?what is the distribution and
version that this module/import-package came from?.

Something like:

    >>> import theorticalapi
    >>> import mymodule
    >>>
    >>> d = theorticalapi.module_dist(mymodule)
    >>> print(d.name)
    >>> mycustommodules
    >>> print(d.version)
    >>> 1.0

Since we (theoretically) should have all of this information already inside
of the packaging metadata, we just don?t have a way to say ?given a particular
import, give me the information I want).

> 
>> That said, if an official answer is required, common sense would suggest
>> that __version__ should go before the imports. (I would put it before the
>> docstring too, except then the docstring wouldn't be a docstring any more.
>> Go figure.)
> 
> And after __future__ imports too, right?
> 
> Cheers,
> -Barry
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150321/50269253/attachment.sig>

From python at mrabarnett.plus.com  Sat Mar 21 22:17:26 2015
From: python at mrabarnett.plus.com (MRAB)
Date: Sat, 21 Mar 2015 21:17:26 +0000
Subject: [Python-Dev] some minor questions about pep8
In-Reply-To: <B08A3E43-C525-43CC-AC37-6906C9082F89@stufft.io>
References: <CAFmnkTBYpaJRaeo_ff6KKVwLTszfF1Zah+yGvg7_yq82bSTmfw@mail.gmail.com>
 <85lhir6seo.fsf@benfinney.id.au>
 <CAP7+vJJPnqUMA=fpz+arV1ZZrT6dGU5d+KNJM5ZLmrRGumkLKQ@mail.gmail.com>
 <20150321165350.726db103@anarchist.wooz.org>
 <B08A3E43-C525-43CC-AC37-6906C9082F89@stufft.io>
Message-ID: <550DDFE6.2020801@mrabarnett.plus.com>

On 2015-03-21 21:00, Donald Stufft wrote:
>
>> On Mar 21, 2015, at 4:53 PM, Barry Warsaw <barry at python.org> wrote:
>>
>> On Mar 20, 2015, at 08:53 PM, Guido van Rossum wrote:
>>
>>> FWIW, I think __version__, __author__ etc. were bad ideas. Almost nobody
>>> manages these correctly. Note that the PEP 8 section starts with less than
>>> an endorsement: "If you *have* to have Subversion, CVS, or RCS crud in your
>>> source file, do it as follows."
>>
>> I tend to agree.  Individual module version numbers are mostly useless,
>> especially with today's hash-based vcses.
>>
>> That's different than package versions, and for which I really need to
>> resurrect and update PEP 396.
>
> I sort of think (though I haven?t completely convinced myself) that adding
> something like __version__ to a package is a work around to the fact that
> we don?t have an API that lets someone ask ?what is the distribution and
> version that this module/import-package came from?.
>
> Something like:
>
>      >>> import theorticalapi
>      >>> import mymodule
>      >>>
>      >>> d = theorticalapi.module_dist(mymodule)
>      >>> print(d.name)
>      >>> mycustommodules
>      >>> print(d.version)
>      >>> 1.0
>
> Since we (theoretically) should have all of this information already inside
> of the packaging metadata, we just don?t have a way to say ?given a particular
> import, give me the information I want).
>
A bit OT (and should probably be in python-ideas), but I've come to 
think that it might be more helpful to have double version numbers.

What do I mean by that?

Well, an "addition" version that's increased when something is added, 
and a "deletion" version that's increased when something is removed.

Suppose there's a new release of a module.

If it has a higher "addition version", then something has been added, 
which won't affect existing code.

If it has a higher "deletion version", then something has been removed, 
which _might_ affect existing code, so you'll need to check. If the code 
doesn't use what was removed, then you can accept the higher deletion 
version.

>>
>>> That said, if an official answer is required, common sense would suggest
>>> that __version__ should go before the imports. (I would put it before the
>>> docstring too, except then the docstring wouldn't be a docstring any more.
>>> Go figure.)
>>
>> And after __future__ imports too, right?
>>


From donald at stufft.io  Sat Mar 21 22:21:13 2015
From: donald at stufft.io (Donald Stufft)
Date: Sat, 21 Mar 2015 17:21:13 -0400
Subject: [Python-Dev] some minor questions about pep8
In-Reply-To: <550DDFE6.2020801@mrabarnett.plus.com>
References: <CAFmnkTBYpaJRaeo_ff6KKVwLTszfF1Zah+yGvg7_yq82bSTmfw@mail.gmail.com>
 <85lhir6seo.fsf@benfinney.id.au>
 <CAP7+vJJPnqUMA=fpz+arV1ZZrT6dGU5d+KNJM5ZLmrRGumkLKQ@mail.gmail.com>
 <20150321165350.726db103@anarchist.wooz.org>
 <B08A3E43-C525-43CC-AC37-6906C9082F89@stufft.io>
 <550DDFE6.2020801@mrabarnett.plus.com>
Message-ID: <70DB1BB2-201D-4C89-BF4D-53368877D2A8@stufft.io>


> On Mar 21, 2015, at 5:17 PM, MRAB <python at mrabarnett.plus.com> wrote:
> 
> On 2015-03-21 21:00, Donald Stufft wrote:
>> 
>>> On Mar 21, 2015, at 4:53 PM, Barry Warsaw <barry at python.org> wrote:
>>> 
>>> On Mar 20, 2015, at 08:53 PM, Guido van Rossum wrote:
>>> 
>>>> FWIW, I think __version__, __author__ etc. were bad ideas. Almost nobody
>>>> manages these correctly. Note that the PEP 8 section starts with less than
>>>> an endorsement: "If you *have* to have Subversion, CVS, or RCS crud in your
>>>> source file, do it as follows."
>>> 
>>> I tend to agree.  Individual module version numbers are mostly useless,
>>> especially with today's hash-based vcses.
>>> 
>>> That's different than package versions, and for which I really need to
>>> resurrect and update PEP 396.
>> 
>> I sort of think (though I haven?t completely convinced myself) that adding
>> something like __version__ to a package is a work around to the fact that
>> we don?t have an API that lets someone ask ?what is the distribution and
>> version that this module/import-package came from?.
>> 
>> Something like:
>> 
>>     >>> import theorticalapi
>>     >>> import mymodule
>>     >>>
>>     >>> d = theorticalapi.module_dist(mymodule)
>>     >>> print(d.name)
>>     >>> mycustommodules
>>     >>> print(d.version)
>>     >>> 1.0
>> 
>> Since we (theoretically) should have all of this information already inside
>> of the packaging metadata, we just don?t have a way to say ?given a particular
>> import, give me the information I want).
>> 
> A bit OT (and should probably be in python-ideas), but I've come to think that it might be more helpful to have double version numbers.
> 
> What do I mean by that?
> 
> Well, an "addition" version that's increased when something is added, and a "deletion" version that's increased when something is removed.
> 
> Suppose there's a new release of a module.
> 
> If it has a higher "addition version", then something has been added, which won't affect existing code.
> 
> If it has a higher "deletion version", then something has been removed, which _might_ affect existing code, so you'll need to check. If the code doesn't use what was removed, then you can accept the higher deletion version.

This is likely better solved by something like https://semver.org/ which is similar to what many people do already:

X.Y.Z+1 == Bug fixes only, should be backwards compatible.
X.Y+1.0 == New features, should be backwards compatible but comes with new things.
X+1.0.0 == Backwards Compatibility Break (Something deleted or changed)

> 
>>> 
>>>> That said, if an official answer is required, common sense would suggest
>>>> that __version__ should go before the imports. (I would put it before the
>>>> docstring too, except then the docstring wouldn't be a docstring any more.
>>>> Go figure.)
>>> 
>>> And after __future__ imports too, right?
>>> 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150321/be201c29/attachment.sig>

From barry at python.org  Sat Mar 21 22:46:10 2015
From: barry at python.org (Barry Warsaw)
Date: Sat, 21 Mar 2015 17:46:10 -0400
Subject: [Python-Dev] some minor questions about pep8
In-Reply-To: <B08A3E43-C525-43CC-AC37-6906C9082F89@stufft.io>
References: <CAFmnkTBYpaJRaeo_ff6KKVwLTszfF1Zah+yGvg7_yq82bSTmfw@mail.gmail.com>
 <85lhir6seo.fsf@benfinney.id.au>
 <CAP7+vJJPnqUMA=fpz+arV1ZZrT6dGU5d+KNJM5ZLmrRGumkLKQ@mail.gmail.com>
 <20150321165350.726db103@anarchist.wooz.org>
 <B08A3E43-C525-43CC-AC37-6906C9082F89@stufft.io>
Message-ID: <20150321174610.1e310a80@limelight.wooz.org>

On Mar 21, 2015, at 05:00 PM, Donald Stufft wrote:

>I sort of think (though I haven?t completely convinced myself) that adding
>something like __version__ to a package is a work around to the fact that
>we don?t have an API that lets someone ask ?what is the distribution and
>version that this module/import-package came from?.

I tend to agree.  Having that would solve one of the big problems that lead to
PEP 394 thinking, that of having to update version numbers in more than one
place.  ISTM the best place to do it is once in setup.py and let the metadata
flow.  The only downside is for doing in-tree development without 'installing'
the package (e.g. absence of venv, tox, or similar).

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150321/e6577ecc/attachment.sig>

From probiner at gmail.com  Sun Mar 22 06:46:43 2015
From: probiner at gmail.com (pedro santos)
Date: Sun, 22 Mar 2015 05:46:43 +0000
Subject: [Python-Dev] 0-base and 1-base indexed iterables? Custom slicing
	rules?
Message-ID: <CAMrmD0q7puV0j_cPqqudws4qQqMd7aPv56yY7NLXepzN9AxyQg@mail.gmail.com>

Hi, I'm an Art and CG student learning Python and today's exercise was
about positions in a tiled room. The fact that I had to check if a position
was inside the room and given that in a 1x1 room, 0.0 was considered in and
1.0 was considered out, it kept me thinking about 0-base indexing iterables
and slicing.

Read some articles and discussions, some pros and cons to each 0-base and
1-base, concerns about slicing, etc. But ultimately the question that got
stuck in me and didn't found an answer was:
Why can't both 0-base and 1-base indexing exist in the same language, and
why can't slicing be customized?



If I'm indexing the ruler marks, intervals, boundaries, dots, makes sense
to start of at 0; rul=[0,1,2,3,4,5,6] would index every mark on my ruler so
that accordingly rul[0]=0, rul[5]=5.
If I'm indexing the blue circles, natural number quantities, objects,
spans, makes sense to start at 1; cir= [1,2,3,4,5] so that cir[1]=1 and
cir[5]=5.

Now, a lot of the discussion was to do with slicing coupled with the
indexing and I don't totally understand why.

a ? x < b is not so intuitive when dealing with objects ("I want balls 1 up
to the the one before 3"), so on one side, you put the finger on what you
want and on the other, on what you don't want. But this method does have
the neat property of producing neighbor selections that border perfectly,
as in [:a][a:b][b:c]. Although in inverse order(-1), the results can be
unexpected as it returns values off-by-one from its counterpart like;
L=[0,1,2,3,4,5] so that L[1:3]=[1,2] and L[3:1:-1]=[3:2]. So it's
consistent with the rule a ? x < b, grabbing the lower limit item, but it
can feel strange by not producing the same selection with inverse order.

a ? x ? b is a natural way to select objets ("I want the balls 1 up to 3"),
so you're putting the finger on the things you want. If you inverse the
order(-1) it's still very easy to grasp what are you picking because
whatever you select it's included like: L=[0,1,2,3,4,5] so that
L[1:3]=[1,2,3] and L[3:1:-1]=[3,2,1]. Problems seem to arrive though, when
trying to do neighbor selections, where one would have to do
[:a][a+1:b][b+1:c] to have the border perfectly. That terrible?

Even though one could see a ? x < b to be more adept to 0-base, and a ? x ?
b to be more adept to 1-base, the way I see it, index base and slicing
rules could be somehow independent. And one would index and slice the way
it would fit the rationale of the problem or the data, because even slicing
a 1-base indexed array with a ? x < b, would still produce an expected
outcome, as in cir=[1,2,3,4,5] so that cir[1:3]=[1,2] or cir[:3]=[1,2].
Same thing applying a ? x ? b to a 0-base indexed array, as in
rul[0,1,2,3,4,5] so that rul[:2]=[0,1,2] or rul[0:2]=[0,1,2].

Given that python is an example of human friendly code language,
emphasizing readability, wouldn't having 0 and 1 base indexing and custom
slicing methods, improve the thought process when writing and reading the
code, by fitting them better to specific contexts or data?
Is there some language that provides both or each language picks only one?

Cheers


-- 




*------------------------------[image:
http://i153.photobucket.com/albums/s202/animatics/probiner-sig.gif]Pedro
Alpiar?a dos Santos Animator  3DModeler  Illustrator >>
http://probiner.x10.mx/ <http://probiner.x10.mx/>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150322/df61b1ef/attachment.html>

From ncoghlan at gmail.com  Sun Mar 22 07:12:51 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 22 Mar 2015 16:12:51 +1000
Subject: [Python-Dev] How to document functions with optional positional
	parameters?
In-Reply-To: <mekecr$ra$1@ger.gmane.org>
References: <meispe$vth$1@ger.gmane.org>
	<mekecr$ra$1@ger.gmane.org>
Message-ID: <CADiSq7dpAg-tZA8e=nBM4KCTvju_gv+atXynUJNdRDiprN-j6A@mail.gmail.com>

On 22 March 2015 at 04:47, Terry Reedy <tjreedy at udel.edu> wrote:
> On 3/21/2015 12:41 AM, Serhiy Storchaka wrote:
>>
>> How to document functions with optional positional parameters?
>>
>> For example binascii.crc32(). It has two positional parameters, one is
>> mandatory, and one is optional with default value 0. With Argument
>> Clinic its signature is crc32(data, crc=0, /). In the documentation it
>> is written as crc32(data[, crc]) (and noted that the default value of
>> the second parameter is 0). Both are not valid Python syntax. Can the
>> documentation be change to crc32(data, crc=0)?
>
>
> I think the docs should be using / as well.

Right, even though the Python level parser doesn't allow this syntax,
it's the one used by Argument Clinic to express positional-only
parameters for functions defined in C, and as a result the inspect
module uses it as well:

>>> import inspect
>>> print(inspect.signature(hex))
(number, /)

This is the specific problem Larry was aiming to address in
https://www.python.org/dev/peps/pep-0457/ - getting the docs, Argument
Clinic and the inspect module to align in their notation, even though
he wasn't proposing to change Python itself to allow positional-only
parameter definitions.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From v+python at g.nevcal.com  Sun Mar 22 07:17:09 2015
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Sat, 21 Mar 2015 23:17:09 -0700
Subject: [Python-Dev] 0-base and 1-base indexed iterables? Custom
 slicing rules?
In-Reply-To: <CAMrmD0q7puV0j_cPqqudws4qQqMd7aPv56yY7NLXepzN9AxyQg@mail.gmail.com>
References: <CAMrmD0q7puV0j_cPqqudws4qQqMd7aPv56yY7NLXepzN9AxyQg@mail.gmail.com>
Message-ID: <550E5E65.2020404@g.nevcal.com>

On 3/21/2015 10:46 PM, pedro santos wrote:
>
> Hi, I'm an Art and CG student learning Python and today's exercise was 
> about positions in a tiled room. The fact that I had to check if a 
> position was inside the room and given that in a 1x1 room, 0.0 was 
> considered in and 1.0 was considered out, it kept me thinking about 
> 0-base indexing iterables and slicing.
>
This sort of discussion is better held on python-list at python.org which 
is for learning and using the current and past versions of Python.

This mailing list is for coordinating the development of future versions 
of Python.

If your idea is to start a discussion to attempt to affect future 
versions of Python, that sort of discussion belongs on 
python-ideas at python.org, and ideas that are appropriate for 
implementation in future versions of Python eventually come to 
python-dev when they are accepted for implementation.

There are lots of friendly people on all these lists, but it is best to 
use the correct one, so that people can focus on the intended topics of 
the lists when reading each of them.

Cheers!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150321/80576611/attachment.html>

From ncoghlan at gmail.com  Sun Mar 22 07:26:48 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 22 Mar 2015 16:26:48 +1000
Subject: [Python-Dev] some minor questions about pep8
In-Reply-To: <20150321174610.1e310a80@limelight.wooz.org>
References: <CAFmnkTBYpaJRaeo_ff6KKVwLTszfF1Zah+yGvg7_yq82bSTmfw@mail.gmail.com>
 <85lhir6seo.fsf@benfinney.id.au>
 <CAP7+vJJPnqUMA=fpz+arV1ZZrT6dGU5d+KNJM5ZLmrRGumkLKQ@mail.gmail.com>
 <20150321165350.726db103@anarchist.wooz.org>
 <B08A3E43-C525-43CC-AC37-6906C9082F89@stufft.io>
 <20150321174610.1e310a80@limelight.wooz.org>
Message-ID: <CADiSq7fUZaLrjrzmcDjo6vYRJXWH3A-3n_gFDsjJew7s88fi+Q@mail.gmail.com>

On 22 March 2015 at 07:46, Barry Warsaw <barry at python.org> wrote:
> On Mar 21, 2015, at 05:00 PM, Donald Stufft wrote:
>
>>I sort of think (though I haven?t completely convinced myself) that adding
>>something like __version__ to a package is a work around to the fact that
>>we don?t have an API that lets someone ask ?what is the distribution and
>>version that this module/import-package came from?.
>
> I tend to agree.  Having that would solve one of the big problems that lead to
> PEP 394 thinking, that of having to update version numbers in more than one
> place.  ISTM the best place to do it is once in setup.py and let the metadata
> flow.  The only downside is for doing in-tree development without 'installing'
> the package (e.g. absence of venv, tox, or similar).

We don't *quite* track enough distribution metadata currently to
reliably build the reverse mapping from module names to the packages
that installed them. The draft "python.exports" extension in PEP 459
is supposed to provide that missing piece:
https://www.python.org/dev/peps/pep-0459/#the-python-exports-extension

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From storchaka at gmail.com  Sun Mar 22 09:41:27 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sun, 22 Mar 2015 10:41:27 +0200
Subject: [Python-Dev] How to document functions with optional positional
	parameters?
In-Reply-To: <CAMpsgwYmee13iUPryqXG9_gFgR0CeAk7DKvPvBJQ7QBj=t+jYA@mail.gmail.com>
References: <meispe$vth$1@ger.gmane.org>
 <CAMpsgwYmee13iUPryqXG9_gFgR0CeAk7DKvPvBJQ7QBj=t+jYA@mail.gmail.com>
Message-ID: <melv7n$rtq$1@ger.gmane.org>

On 21.03.15 13:03, Victor Stinner wrote:
> The \ is useful, it indicates that you cannot use keywords.

Wouldn't it confuse users?

> If you want to drop \, modify the function to accept keywords.

Yes, this is a solution. But parsing keyword arguments is slower than 
parsing positional arguments. And I'm working on patches that optimizes 
parsing code generated by Argument Clinic. First my patches will handle 
only positional parameters, with keywords it is harder.


From storchaka at gmail.com  Sun Mar 22 10:20:13 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sun, 22 Mar 2015 11:20:13 +0200
Subject: [Python-Dev] Needed reviews
In-Reply-To: <CADiSq7dvKuKZ-OVqni-=st1+bYp5DVn8O0w9w=K+agELSUOQMA@mail.gmail.com>
References: <mee4si$rap$1@ger.gmane.org>
 <CADiSq7dvKuKZ-OVqni-=st1+bYp5DVn8O0w9w=K+agELSUOQMA@mail.gmail.com>
Message-ID: <mem1gd$rtb$1@ger.gmane.org>

On 21.03.15 13:46, Nick Coghlan wrote:
> On 19 March 2015 at 19:28, Serhiy Storchaka <storchaka at gmail.com> wrote:
>> Here is list of my ready for review patches.  It is incomplete and contains
>> only patches for which I don't expect objections or long discussion.  Most
>> of them are relative easy and need only formal review.  Most of them wait
>> for a review many months.
>
> It's worth noting that If there are changes you feel are genuinely low
> risk, you can go ahead and merge them based on your own commit review
> (even if you also wrote the original patch).

Yes, but four eyes are better than two eyes. I make mistakes. In some 
issues I hesitate about documentation part. In some issues (issue14260 
and issue22721) I provided two alternative solutions and need a tip to 
choose from them. While I am mainly sure about the correctness of the 
patch, I'm often hesitate about the direction. Is the bug worth fixing? 
Is the new feature worth to be added to Python?

Thanks Alexander, Amaury, Benjamin, Berker, Demian, ?ric, Ethan, Martin, 
Paul, Victor and others that responded on my request.


From Stefan.Richthofer at gmx.de  Sun Mar 22 11:44:59 2015
From: Stefan.Richthofer at gmx.de (Stefan Richthofer)
Date: Sun, 22 Mar 2015 11:44:59 +0100
Subject: [Python-Dev] 0-base and 1-base indexed iterables? Custom
 slicing rules?
In-Reply-To: <CAMrmD0q7puV0j_cPqqudws4qQqMd7aPv56yY7NLXepzN9AxyQg@mail.gmail.com>
References: <CAMrmD0q7puV0j_cPqqudws4qQqMd7aPv56yY7NLXepzN9AxyQg@mail.gmail.com>
Message-ID: <trinity-ab2ea393-007d-4d0c-b3ce-e54800337e9c-1427021099529@3capp-gmx-bs11>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150322/8913f4dd/attachment-0001.html>

From p.f.moore at gmail.com  Sun Mar 22 15:44:12 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Sun, 22 Mar 2015 14:44:12 +0000
Subject: [Python-Dev] Installing Python to non-ASCII paths
Message-ID: <CACac1F92WX9e31D53K8fgwC0hkDN6=VxAwg1duPW+HHbA-sF8w@mail.gmail.com>

Something that hit me today, which might become a more common issue
when the Windows installers move towards installing to the user
directory, is that there appear to be some bugs in handling of
non-ASCII paths.

Two that I spotted are a failure of the "script wrappers" installed by
pip to work with a non-ASCII interpreter path (reported to distlib)
and a possible issue with the py.exe launcher when a script has
non-ASCII in the shebang line (not reported yet because I'm not clear
on what's going on).

I've only seen Windows-specific issues - I don't know how common
non-ASCII paths for the python interpreter are on Unix or OSX, or
whether the more or less universal use of UTF-8 on Unix makes such
issues less common. But if anyone has an environment that makes
testing on non-ASCII install paths easy, it might be worth doing some
checks just so we can catch any major ones before 3.5 is released.

On which note, I'm assuming neither of the issues I've found are major
blockers. "pip.exe doesn't work if Python is installed in a directory
with non-ASCII characters in the name" can be worked around by using
python -m pip, and the launcher issue by using a generic shebang like
#!/usr/bin/python3.5.

Paul

From mail at timgolden.me.uk  Sun Mar 22 16:12:43 2015
From: mail at timgolden.me.uk (Tim Golden)
Date: Sun, 22 Mar 2015 15:12:43 +0000
Subject: [Python-Dev] Installing Python to non-ASCII paths
In-Reply-To: <CACac1F92WX9e31D53K8fgwC0hkDN6=VxAwg1duPW+HHbA-sF8w@mail.gmail.com>
References: <CACac1F92WX9e31D53K8fgwC0hkDN6=VxAwg1duPW+HHbA-sF8w@mail.gmail.com>
Message-ID: <550EDBEB.3010004@timgolden.me.uk>

On 22/03/2015 14:44, Paul Moore wrote:
> On which note, I'm assuming neither of the issues I've found are major
> blockers. "pip.exe doesn't work if Python is installed in a directory
> with non-ASCII characters in the name" can be worked around by using
> python -m pip, and the launcher issue by using a generic shebang like
> #!/usr/bin/python3.5.

That can become a more major blocker if "pip doesn't work" == "ensurepip 
doesn't work and blocks thus the installer crashes" as was the case with 
a mimetypes issue a little while back.

I'll create a ??? user (which is the easiest non-ASCII name to create on 
a UK keyboard) to see how cleanly the latest installer works.

TJG


From mail at timgolden.me.uk  Sun Mar 22 16:29:54 2015
From: mail at timgolden.me.uk (Tim Golden)
Date: Sun, 22 Mar 2015 15:29:54 +0000
Subject: [Python-Dev] Installing Python to non-ASCII paths
In-Reply-To: <550EDBEB.3010004@timgolden.me.uk>
References: <CACac1F92WX9e31D53K8fgwC0hkDN6=VxAwg1duPW+HHbA-sF8w@mail.gmail.com>
 <550EDBEB.3010004@timgolden.me.uk>
Message-ID: <550EDFF2.3090601@timgolden.me.uk>

On 22/03/2015 15:12, Tim Golden wrote:
> On 22/03/2015 14:44, Paul Moore wrote:
>> On which note, I'm assuming neither of the issues I've found are major
>> blockers. "pip.exe doesn't work if Python is installed in a directory
>> with non-ASCII characters in the name" can be worked around by using
>> python -m pip, and the launcher issue by using a generic shebang like
>> #!/usr/bin/python3.5.
>
> That can become a more major blocker if "pip doesn't work" == "ensurepip
> doesn't work and blocks thus the installer crashes" as was the case with
> a mimetypes issue a little while back.
>
> I'll create a ??? user (which is the easiest non-ASCII name to create on
> a UK keyboard) to see how cleanly the latest installer works.

Tried with "Mr ???". The installer's fine but the installed pip.exe 
fails while "py -3 -mpip" succeeeds as Paul notes.

TJG

From kwi at kwi.dk  Sun Mar 22 18:56:09 2015
From: kwi at kwi.dk (=?UTF-8?B?U8O4cmVuIEzDuHZib3Jn?=)
Date: Sun, 22 Mar 2015 18:56:09 +0100
Subject: [Python-Dev] Request for comments: [issue22941] IPv4Interface
 arithmetic changes subnet mask
Message-ID: <550F0239.4040609@kwi.dk>

Thanks for the feedback, and apologies for my late reply. I have to
say, I'm not entirely sold on the argument for raising an exception on 
subnet "overflow".

First, I'll note what happens if we overflow an IPv4Address:

     >>> ipaddress.IPv4Address('255.255.255.255') + 1
     Traceback (most recent call last):
     ...
     ipaddress.AddressValueError: 4294967296 (>= 2**32) is not permitted
     as an IPv4 address

Now, I used "IPv4Interface() + 1" to mean "Give me the IP next to the
current one in the current subnet", knowing from the context that the
address would be valid and available.

     >>> host = ipaddress.IPv4Interface('10.0.0.2/24')
     >>> peer = host + 1

In this context, I would welcome an exception, as it would certainly be 
an error if I overflowed the subnet.

However, there are also situations in which overflowing would be valid 
and expected, e.g. as a way to skip to the "same" IP in the next subnet:

     >>> ip = ipaddress.IPv4Interface('10.0.0.42/24')
     >>> ip + ip.network.num_addresses
     IPv4Interface('10.0.1.42/24')

It's not even a hypothetical example; I've been working on a distributed 
embedded system where all the hosts have two (redundant) addresses 
differing only by their subnet; this could be a convenient way calculate 
one address from the other.

There's an additional issue with raising an exception, and that is that
it still won't catch overflow errors in my example use case:

     >>> host = ipaddress.IPv4Interface('10.0.0.254/24')
     >>> peer = host + 1
     >>> peer
     IPv4Interface('10.0.0.255/24')

This doesn't overflow and does not trigger an exception, but the
resulting peer address is still invalid (it's the subnet broadcast
address, not a host address). As such, the exception isn't even useful
as an error detection tool. (I'll not suggest raising an exception when 
hitting the broadcast or network address; that way lies madness.)

As for consistency with IPv4Address, I can argue either way:

Pro-exception:

"Overflowing an IPv4Interface raises AddressValueError just like with
IPv4Address."

Contra-exception:

"An IPv4Interface behaves exactly like an IPv4Address, except that it
also has an associated subnet mask." (This is essentially how the type
is currently documented).

Thoughts?

Best regards,
S?ren L?vborg

From victor.stinner at gmail.com  Sun Mar 22 20:34:51 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sun, 22 Mar 2015 20:34:51 +0100
Subject: [Python-Dev] Installing Python to non-ASCII paths
In-Reply-To: <CACac1F92WX9e31D53K8fgwC0hkDN6=VxAwg1duPW+HHbA-sF8w@mail.gmail.com>
References: <CACac1F92WX9e31D53K8fgwC0hkDN6=VxAwg1duPW+HHbA-sF8w@mail.gmail.com>
Message-ID: <CAMpsgwaUJanYTo9hE3E4YEZUDY=7BOr9-jB=yvgsz_TzEgR+9A@mail.gmail.com>

Hi Paul,

Please open an issue, I can take a look. Please describe a scenario to
reproduce the issue.

Victor

2015-03-22 15:44 GMT+01:00 Paul Moore <p.f.moore at gmail.com>:
> Something that hit me today, which might become a more common issue
> when the Windows installers move towards installing to the user
> directory, is that there appear to be some bugs in handling of
> non-ASCII paths.
>
> Two that I spotted are a failure of the "script wrappers" installed by
> pip to work with a non-ASCII interpreter path (reported to distlib)
> and a possible issue with the py.exe launcher when a script has
> non-ASCII in the shebang line (not reported yet because I'm not clear
> on what's going on).
>
> I've only seen Windows-specific issues - I don't know how common
> non-ASCII paths for the python interpreter are on Unix or OSX, or
> whether the more or less universal use of UTF-8 on Unix makes such
> issues less common. But if anyone has an environment that makes
> testing on non-ASCII install paths easy, it might be worth doing some
> checks just so we can catch any major ones before 3.5 is released.
>
> On which note, I'm assuming neither of the issues I've found are major
> blockers. "pip.exe doesn't work if Python is installed in a directory
> with non-ASCII characters in the name" can be worked around by using
> python -m pip, and the launcher issue by using a generic shebang like
> #!/usr/bin/python3.5.
>
> Paul
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com

From p.f.moore at gmail.com  Sun Mar 22 20:43:51 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Sun, 22 Mar 2015 19:43:51 +0000
Subject: [Python-Dev] Installing Python to non-ASCII paths
In-Reply-To: <CAMpsgwaUJanYTo9hE3E4YEZUDY=7BOr9-jB=yvgsz_TzEgR+9A@mail.gmail.com>
References: <CACac1F92WX9e31D53K8fgwC0hkDN6=VxAwg1duPW+HHbA-sF8w@mail.gmail.com>
 <CAMpsgwaUJanYTo9hE3E4YEZUDY=7BOr9-jB=yvgsz_TzEgR+9A@mail.gmail.com>
Message-ID: <CACac1F8hhWGOiLnRQaFj16rLMo+gmKc34HaC5=xDwY5zyC3LrQ@mail.gmail.com>

On 22 March 2015 at 19:34, Victor Stinner <victor.stinner at gmail.com> wrote:
> Please open an issue, I can take a look. Please describe a scenario to
> reproduce the issue.

The "issue" with the launcher seems to have bee a red herring. When I
set up a test case properly, it worked. I suspect I messed up writing
the shebang line in UTF8. Thanks anyway. (And the pip.exe issue is a
distlib issue, which I have reported).

Paul

From ncoghlan at gmail.com  Sun Mar 22 23:25:07 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 23 Mar 2015 08:25:07 +1000
Subject: [Python-Dev] Needed reviews
In-Reply-To: <mem1gd$rtb$1@ger.gmane.org>
References: <mee4si$rap$1@ger.gmane.org>
 <CADiSq7dvKuKZ-OVqni-=st1+bYp5DVn8O0w9w=K+agELSUOQMA@mail.gmail.com>
 <mem1gd$rtb$1@ger.gmane.org>
Message-ID: <CADiSq7digTp-Gj5-LZ2Lc+H_Wj39cCnuLgkYNnhFe_FzLUx8Og@mail.gmail.com>

On 22 Mar 2015 19:22, "Serhiy Storchaka" <storchaka at gmail.com> wrote:
>
> On 21.03.15 13:46, Nick Coghlan wrote:
>>
>> On 19 March 2015 at 19:28, Serhiy Storchaka <storchaka at gmail.com> wrote:
>>>
>>> Here is list of my ready for review patches.  It is incomplete and
contains
>>> only patches for which I don't expect objections or long discussion.
Most
>>> of them are relative easy and need only formal review.  Most of them
wait
>>> for a review many months.
>>
>>
>> It's worth noting that If there are changes you feel are genuinely low
>> risk, you can go ahead and merge them based on your own commit review
>> (even if you also wrote the original patch).
>
>
> Yes, but four eyes are better than two eyes. I make mistakes. In some
issues I hesitate about documentation part. In some issues (issue14260 and
issue22721) I provided two alternative solutions and need a tip to choose
from them. While I am mainly sure about the correctness of the patch, I'm
often hesitate about the direction. Is the bug worth fixing? Is the new
feature worth to be added to Python?

Aye, agreed - those are the kinds of cases where I'd nudge folks for a
review as well. Committing directly is something I'd only do where I'm
already entirely confident the change is an improvement over the status quo.

However, now that I think about it further, it's very rare for me to be
completely confident in a change without another core dev at least giving a
+1 on the general idea.

> Thanks Alexander, Amaury, Benjamin, Berker, Demian, ?ric, Ethan, Martin,
Paul, Victor and others that responded on my request.

Indeed! One of the things I'm actually hoping to achieve through PEPs like
474 & 462 is to get to a point where we can better acknowledge and thank
folks for all the work that goes into patch reviews :)

Cheers,
Nick.

>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150323/a6677be0/attachment.html>

From guido at python.org  Sun Mar 22 23:42:05 2015
From: guido at python.org (Guido van Rossum)
Date: Sun, 22 Mar 2015 15:42:05 -0700
Subject: [Python-Dev] some minor questions about pep8
In-Reply-To: <CA+GyjMntcKaRUjfOzZg82q=13kjeCkKAYPT=ovhQUokqnPaXng@mail.gmail.com>
References: <CAFmnkTBYpaJRaeo_ff6KKVwLTszfF1Zah+yGvg7_yq82bSTmfw@mail.gmail.com>
 <85lhir6seo.fsf@benfinney.id.au>
 <CAP7+vJJPnqUMA=fpz+arV1ZZrT6dGU5d+KNJM5ZLmrRGumkLKQ@mail.gmail.com>
 <CA+GyjMntcKaRUjfOzZg82q=13kjeCkKAYPT=ovhQUokqnPaXng@mail.gmail.com>
Message-ID: <CAP7+vJJiLBhH3m23sn2ZWnsABXwAZCRJeEC46BCj0Gp5x-eeDA@mail.gmail.com>

On Fri, Mar 20, 2015 at 9:24 PM, Ian Lee <ianlee1521 at gmail.com> wrote:

> Guido,
>
> In that case would you be open to a patch to update the PEP accordingly?
>
Only if it totally eradicate __version__ and similar from the PEP.


> Additionally, does that official statement cover other dunder assignments
> (e.g. "__author__"?). If so I'll update the PEP8 tool accordingly.


What official statement? I'm not the Pope. :-)

--Guido

> Thanks,
>
> ~ Ian Lee
> On Mar 20, 2015 8:55 PM, "Guido van Rossum" <guido at python.org> wrote:
>
>> FWIW, I think __version__, __author__ etc. were bad ideas. Almost nobody
>> manages these correctly. Note that the PEP 8 section starts with less than
>> an endorsement: "If you *have* to have Subversion, CVS, or RCS crud in
>> your source file, do it as follows."
>>
>> That said, if an official answer is required, common sense would suggest
>> that __version__ should go before the imports. (I would put it before the
>> docstring too, except then the docstring wouldn't be a docstring any more.
>> Go figure.)
>>
>> On Fri, Mar 20, 2015 at 6:38 PM, Ben Finney <ben+python at benfinney.id.au>
>> wrote:
>>
>>> Lewis Coates <lewisc at pdx.edu> writes:
>>>
>>> > In pep8 there are two conflicting statements, both
>>> >
>>> > https://www.python.org/dev/peps/pep-0008/#version-bookkeeping
>>> > https://www.python.org/dev/peps/pep-0008/#imports
>>> >
>>> > Stipulate that they should be "at the top of the file after any module
>>> > comments and docstrings." Which of these takes precedence?
>>>
>>> I don't know an official answer. The convention I've observed is
>>> overwhelmingly in one direction: import statements come before any
>>> assignment statements.
>>>
>>> > Secondly, we also have an "__author__", and "__project__" variables, I
>>> > assume these would be put with the version information as well?
>>>
>>> Yes.
>>>
>>> --
>>>  \     ?Welchen Teil von ?Gestalt? verstehen Sie nicht?  [What part of |
>>>   `\                ?gestalt? don't you understand?]? ?Karsten M. Self |
>>> _o__)                                                                  |
>>> Ben Finney
>>>
>>> _______________________________________________
>>> Python-Dev mailing list
>>> Python-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe:
>>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>>
>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/ianlee1521%40gmail.com
>>
>>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150322/28480af4/attachment.html>

From ncoghlan at gmail.com  Sun Mar 22 23:51:23 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 23 Mar 2015 08:51:23 +1000
Subject: [Python-Dev] Installing Python to non-ASCII paths
In-Reply-To: <CACac1F92WX9e31D53K8fgwC0hkDN6=VxAwg1duPW+HHbA-sF8w@mail.gmail.com>
References: <CACac1F92WX9e31D53K8fgwC0hkDN6=VxAwg1duPW+HHbA-sF8w@mail.gmail.com>
Message-ID: <CADiSq7e8pxx_cneN0JE+Xyygcc_kvShsfO5LLg-dB6Gt7cXeeA@mail.gmail.com>

On 23 Mar 2015 00:45, "Paul Moore" <p.f.moore at gmail.com> wrote:
>
> Something that hit me today, which might become a more common issue
> when the Windows installers move towards installing to the user
> directory, is that there appear to be some bugs in handling of
> non-ASCII paths.
>
> Two that I spotted are a failure of the "script wrappers" installed by
> pip to work with a non-ASCII interpreter path (reported to distlib)
> and a possible issue with the py.exe launcher when a script has
> non-ASCII in the shebang line (not reported yet because I'm not clear
> on what's going on).
>
> I've only seen Windows-specific issues - I don't know how common
> non-ASCII paths for the python interpreter are on Unix or OSX, or
> whether the more or less universal use of UTF-8 on Unix makes such
> issues less common.

POSIX is fine if the locale encoding is correct, but can go fairly wrong if
it isn't. Last major complaints I heard related to upstart sometimes
getting it wrong in cron and for some daemonized setups (systemd appears to
be more robust in setting it correctly as it pulls the expected setting
from a system wide config file).

"LANG=C" also doesn't work well, as that tells CPython to use ASCII instead
of UTF-8 or whatever the actual system encoding is. Armin Ronacher pointed
out "LANG=C.UTF-8" as a good alternative, but whether that's available or
not is currently distro-specific. I filed an upstream bug with the glibc
devs asking for that to be made standard, and they seemed amenable to the
idea, but I haven't checked back in on its progress recently.

> But if anyone has an environment that makes
> testing on non-ASCII install paths easy, it might be worth doing some
> checks just so we can catch any major ones before 3.5 is released.

I'd suggest looking at the venv tests and using them as inspiration to
create a separate "test_venv_nonascii" test file that checks:

* creating a venv containing non-ASCII characters
* copying the Python binary to a temporary directory with non-ASCII
characters in the name and using that to create a venv

More generally, we should likely enhance the venv tests to actually *run*
the installed pip binary to list the installed packages. That will
automatically test the distlib script wrappers, as well as checking the
installed package set matches what we're currently bundling.

With those changes, the buildbots would go a long way towards ensuring that
non-ASCII installation paths always work correctly, as well as making it
relatively straightforward for other implementations to adopt the same
checks.

Cheers,
Nick.

>
> On which note, I'm assuming neither of the issues I've found are major
> blockers. "pip.exe doesn't work if Python is installed in a directory
> with non-ASCII characters in the name" can be worked around by using
> python -m pip, and the launcher issue by using a generic shebang like
> #!/usr/bin/python3.5.
>
> Paul
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150323/d4754591/attachment-0001.html>

From v+python at g.nevcal.com  Mon Mar 23 02:46:15 2015
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Sun, 22 Mar 2015 18:46:15 -0700
Subject: [Python-Dev] Installing Python to non-ASCII paths
In-Reply-To: <550EDBEB.3010004@timgolden.me.uk>
References: <CACac1F92WX9e31D53K8fgwC0hkDN6=VxAwg1duPW+HHbA-sF8w@mail.gmail.com>
 <550EDBEB.3010004@timgolden.me.uk>
Message-ID: <550F7067.4020902@g.nevcal.com>

On 3/22/2015 8:12 AM, Tim Golden wrote:
> I'll create a ??? user (which is the easiest non-ASCII name to create 
> on a UK keyboard) to see how cleanly the latest installer works. 

You can also copy/paste.  A path with a Cyrillic, Greek, Chinese, 
Tibetan, Japanese, Armenian, and Romanian character, none of which are 
in the "Windows ANSI" character set, should suffice...  Here ya go...

????????

In my work with Windows, I've certainly seen that ? is much more 
acceptable to more programs than ? or these other ones.
<http://ar.wikipedia.org/wiki/%D8%A3%D9%84%D9%81%D8%A8%D8%A7%D8%A6%D9%8A%D8%A9_%D9%8A%D9%88%D9%86%D8%A7%D9%86%D9%8A%D8%A9> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150322/73e5c9e9/attachment.html>

From cs at zip.com.au  Mon Mar 23 07:10:42 2015
From: cs at zip.com.au (Cameron Simpson)
Date: Mon, 23 Mar 2015 17:10:42 +1100
Subject: [Python-Dev] Use ptyhon -s as default shbang for system python
 executables/daemons
In-Reply-To: <A5D52925-8F0C-49FC-9A11-AE12BBD8404C@stufft.io>
References: <A5D52925-8F0C-49FC-9A11-AE12BBD8404C@stufft.io>
Message-ID: <20150323061042.GA71955@cskk.homeip.net>

On 21Mar2015 14:29, Donald Stufft <donald at stufft.io> wrote:
>> On Mar 21, 2015, at 7:52 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> On 19 March 2015 at 07:51, Donald Stufft <donald at stufft.io> wrote:
>>> I?ve long wished that the OS had it?s own virtual environment. A lot of problems
>>> seems to come from trying to cram the things the OS wants with the things that
>>> the user wants into the same namespace.
>>
>> I'm more wanting to go in the other direction and suggest to folks
>> that if they're not *scripting the OS*, then the system Python isn't
>> what they want, and they should be using at least a virtual
>> environment, preferably an entirely separate software collection that
>> they can upgrade on their own timeline, independently of what they
>> system does.
>
>It?s likely easier to get the OS to move it?s own things to a virtual
>environment than it is to convince every single person who uses an OS
>to never install globally.

I agree.

And just as a data point, this cropped up on the Fedora list yesterday:

  I broke Yum (by messing with Python libs)
  http://www.spinics.net/linux/fedora/fedora-users/msg458069.html

TL;DR: OP used pip on his system python. Yum broke. Probably hampered his 
attempts to repair, too.

Cheers,
Cameron Simpson <cs at zip.com.au>

The mind reigns, but does not govern.   - Paul Valery

From mail at timgolden.me.uk  Mon Mar 23 09:25:04 2015
From: mail at timgolden.me.uk (Tim Golden)
Date: Mon, 23 Mar 2015 08:25:04 +0000
Subject: [Python-Dev] Installing Python to non-ASCII paths
In-Reply-To: <550F7067.4020902@g.nevcal.com>
References: <CACac1F92WX9e31D53K8fgwC0hkDN6=VxAwg1duPW+HHbA-sF8w@mail.gmail.com>
 <550EDBEB.3010004@timgolden.me.uk> <550F7067.4020902@g.nevcal.com>
Message-ID: <550FCDE0.8030601@timgolden.me.uk>

On 23/03/2015 01:46, Glenn Linderman wrote:
> On 3/22/2015 8:12 AM, Tim Golden wrote:
>> I'll create a ??? user (which is the easiest non-ASCII name to create
>> on a UK keyboard) to see how cleanly the latest installer works. 
> 
> You can also copy/paste.  A path with a Cyrillic, Greek, Chinese,
> Tibetan, Japanese, Armenian, and Romanian character, none of which are
> in the "Windows ANSI" character set, should suffice...  Here ya go...
> 
> ????????
> 
> In my work with Windows, I've certainly seen that ? is much more
> acceptable to more programs than ? or these other ones.
> <http://ar.wikipedia.org/wiki/%D8%A3%D9%84%D9%81%D8%A8%D8%A7%D8%A6%D9%8A%D8%A9_%D9%8A%D9%88%D9%86%D8%A7%D9%86%D9%8A%D8%A9>

Thanks, Glenn. Good point.

TJG

From a.badger at gmail.com  Mon Mar 23 15:22:56 2015
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Mon, 23 Mar 2015 07:22:56 -0700
Subject: [Python-Dev] Use ptyhon -s as default shbang for system python
	executables/daemons
In-Reply-To: <CAMpsgwYx8RdrMOTQdS8bm7Qyj1+JjEf-Oig9-gx_s1vK15MD1w@mail.gmail.com>
References: <loom.20150318T164041-536@post.gmane.org>
 <20150318122203.12ff1ee3@limelight.wooz.org>
 <20150318214431.GA4050@roan.lan>
 <20150318175607.66a25ad3@limelight.wooz.org>
 <CABVPEKoioRy-kvRXuUpMWTZYSa0sheFDbFoUsECyg=tqyLg-6A@mail.gmail.com>
 <20150319152734.5bb6cd79@limelight.wooz.org>
 <CABVPEKo3edWRCLadA9_WVNvM6QZsAvToBjQGaOQXWxYC5F4qvg@mail.gmail.com>
 <CAMpsgwYx8RdrMOTQdS8bm7Qyj1+JjEf-Oig9-gx_s1vK15MD1w@mail.gmail.com>
Message-ID: <CABVPEKrL3_yUKtQV4cvjzJ+woMe=kVOsCnU0+hPYYx85W01rKQ@mail.gmail.com>

-Toshio
On Mar 19, 2015 3:27 PM, "Victor Stinner" <victor.stinner at gmail.com> wrote:
>
> 2015-03-19 21:47 GMT+01:00 Toshio Kuratomi <a.badger at gmail.com>:
> > I think I've found the Debian discussion (October 2012):
> >
> > http://comments.gmane.org/gmane.linux.debian.devel.python/8188
> >
> > Lack of PYTHONWARNINGS was brought up late in the discussion thread
>
> Maybe we need to modify -E or add a new option to only ignore PYTHONPATH.
>
I think pythonpath is still useful on its own.

Building off Nick's idea of a system python vs a python for users to use, I
would see a more useful modification to be able to specify SPYTHONPATH (and
other env vars) to go along with /usr/bin/spython .  That way the user
maintains the capability to override specific libraries globally just like
with LD_LIBRARY_PATH, PATH, and similar but you won't accidentally
configure your own python to use one set of paths for your five python apps
and then have that leak over and affect system tools.

-Toshio
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150323/54bb4186/attachment.html>

From solipsis at pitrou.net  Mon Mar 23 15:30:23 2015
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 23 Mar 2015 15:30:23 +0100
Subject: [Python-Dev] Use ptyhon -s as default shbang for system python
 executables/daemons
References: <loom.20150318T164041-536@post.gmane.org>
 <20150318122203.12ff1ee3@limelight.wooz.org>
 <20150318214431.GA4050@roan.lan>
 <20150318175607.66a25ad3@limelight.wooz.org>
 <CABVPEKoioRy-kvRXuUpMWTZYSa0sheFDbFoUsECyg=tqyLg-6A@mail.gmail.com>
 <20150319152734.5bb6cd79@limelight.wooz.org>
 <CABVPEKo3edWRCLadA9_WVNvM6QZsAvToBjQGaOQXWxYC5F4qvg@mail.gmail.com>
 <CAMpsgwYx8RdrMOTQdS8bm7Qyj1+JjEf-Oig9-gx_s1vK15MD1w@mail.gmail.com>
 <CABVPEKrL3_yUKtQV4cvjzJ+woMe=kVOsCnU0+hPYYx85W01rKQ@mail.gmail.com>
Message-ID: <20150323153023.6fa0466e@fsol>

On Mon, 23 Mar 2015 07:22:56 -0700
Toshio Kuratomi <a.badger at gmail.com> wrote:
> 
> Building off Nick's idea of a system python vs a python for users to use, I
> would see a more useful modification to be able to specify SPYTHONPATH (and
> other env vars) to go along with /usr/bin/spython .  That way the user
> maintains the capability to override specific libraries globally just like
> with LD_LIBRARY_PATH, PATH, and similar but you won't accidentally
> configure your own python to use one set of paths for your five python apps
> and then have that leak over and affect system tools.

I really think Donald has a good point when he suggests a specific
virtualenv for system programs using Python.

Regards

Antoine.



From lkb.teichmann at gmail.com  Mon Mar 23 10:38:30 2015
From: lkb.teichmann at gmail.com (Martin Teichmann)
Date: Mon, 23 Mar 2015 10:38:30 +0100
Subject: [Python-Dev] super() does not work during class initialization
In-Reply-To: <CADiSq7fxRDGg5kE8gjX-q1=CkMi0t_Q7UwigoqNUZqi43Nw9FA@mail.gmail.com>
References: <CAK9R32QaTyqsugF1_3-pPHfDsaEa4JEp0A_DFBboNOFwP_x82Q@mail.gmail.com>
 <CADiSq7fxRDGg5kE8gjX-q1=CkMi0t_Q7UwigoqNUZqi43Nw9FA@mail.gmail.com>
Message-ID: <CAK9R32QmnjAWJkWR0WY93Xgr+G1aQmyqtcG4gG-C1nrjV17dNQ@mail.gmail.com>

> For folks that haven't looked at the tracker issue: I personally like
> the change, but it does involve storing the cell object in a
> dunder-variable in the class namespace while it's being defined (until
> type.__new__ executes and both populates it and removes it from the
> class namespace).

I think I had the same weird feelings like you when writing the code. It feels
a bit dodgy to abuse the class namespace to transport information from the
compiler to type.__new__. Only when I saw that __qualname__ already does
it, I decided it was probably not such a bad idea. The current implementation,
setting the __class__ cell at the end of __build_class__ doesn't feel dodgy to
me, it simply feels wrong. Nothing of real importance should happen there, as
it is just an implementation detail.

If we are fearing that we clutter namespace too much, we might call the
entries @qualname and @cell, then we are sure they will never mask
a user's class member.

Most of my bad feelings actually only come from the fact that we don't know
what we actually put our entries into, as __prepare__ might return something
weird that doesn't do what we expect. Given Eric Snow's comment that
class namespaces will be ordered by default soon anyways, we might deprecate
__prepare__ altogether, eliminating most of the bad feelings.

Speaking of which, instead of having OrderedDict re-implemented in C, maybe
we could just change the compiler to leave the order in which things are defined
in a class in the class namespace, say as a member __order__? Then we could
use plain-old dicts for the class namespace, and we would not slow down class
creation (not that it matters much), as determining the order would happen at
compile time.

> Since it introduces new behaviour that's visible to Python level code,
> I've suggested that Martin roll the suggestion into his current PEP
> 487 (which adds __init_subclass__ to similarly tidy up a few
> challenges with the way classes are currently initialised).

I thought about that, but while I stumbled over this issue while working
on PEP 487, it is actually unrelated. I didn't want people to think that
this change is needed to implement PEP 487. And while
"and now for something completely different" might be a nice
Monty Python reference, it is mostly frowned upon in PEPs as per PEP 1.

Alternatively, I could write a new PEP. But I actually think that
the change is not worth a PEP, because the changes will move
the CPython implementation actually closer to what is documented,
so I feel it is more a bugfix than an extension.

I have no problem to pursue any of the above paths,
how do other people feel about it?

Greetings

Martin

From a.badger at gmail.com  Mon Mar 23 16:06:13 2015
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Mon, 23 Mar 2015 08:06:13 -0700
Subject: [Python-Dev] Use ptyhon -s as default shbang for system python
 executables/daemons
In-Reply-To: <20150323153023.6fa0466e@fsol>
References: <loom.20150318T164041-536@post.gmane.org>
 <20150318122203.12ff1ee3@limelight.wooz.org>
 <20150318214431.GA4050@roan.lan>
 <20150318175607.66a25ad3@limelight.wooz.org>
 <CABVPEKoioRy-kvRXuUpMWTZYSa0sheFDbFoUsECyg=tqyLg-6A@mail.gmail.com>
 <20150319152734.5bb6cd79@limelight.wooz.org>
 <CABVPEKo3edWRCLadA9_WVNvM6QZsAvToBjQGaOQXWxYC5F4qvg@mail.gmail.com>
 <CAMpsgwYx8RdrMOTQdS8bm7Qyj1+JjEf-Oig9-gx_s1vK15MD1w@mail.gmail.com>
 <CABVPEKrL3_yUKtQV4cvjzJ+woMe=kVOsCnU0+hPYYx85W01rKQ@mail.gmail.com>
 <20150323153023.6fa0466e@fsol>
Message-ID: <20150323150613.GB4050@roan.lan>

On Mon, Mar 23, 2015 at 03:30:23PM +0100, Antoine Pitrou wrote:
> On Mon, 23 Mar 2015 07:22:56 -0700
> Toshio Kuratomi <a.badger at gmail.com> wrote:
> > 
> > Building off Nick's idea of a system python vs a python for users to use, I
> > would see a more useful modification to be able to specify SPYTHONPATH (and
> > other env vars) to go along with /usr/bin/spython .  That way the user
> > maintains the capability to override specific libraries globally just like
> > with LD_LIBRARY_PATH, PATH, and similar but you won't accidentally
> > configure your own python to use one set of paths for your five python apps
> > and then have that leak over and affect system tools.
> 
> I really think Donald has a good point when he suggests a specific
> virtualenv for system programs using Python.
> 
The isolation is what we're seeking but I think the amount of work required
and the added complexity for the distributions will make that hard to get
distributions to sign up for.

If someone had the time to write a front end to install packages into
a single "system-wide isolation unit" whose backend was a virtualenv we
might be able to get distributions on-board with using that.

The front end would need to install software so that you can still invoke
/usr/bin/system-application and "system-application" would take care of
activating the virtualenv.  It would need to be about as simple to build
as the present python2 setup.py build/install with the flexibility in
options that the distros need to install into FHS approved paths.  Some
things like man pages, locale files, config files, and possibly other data
files might need to be installed outside of the virtualenv directory.  Many
setup.py's already punt on some of those, though, letting the user choose
to install them manually.  So this might be similar.  It would need to be able
to handle 32bit and 64bit versions of the same library installed on the same
system.  It would need to be able to handle different versions of the same
library installed on the same system (as few of those as possible but it's
an unfortunate cornercase that can't be entirely ignored even for just
system packages).  It would need a mode where it doesn't use the network at
all; only operates with the packages and sources that are present on the
box.

And remember these two things: (1) we'd be asking the distros to do
a tremendous amount of work changing their packages to install into
a virtualenv instead of the python setup.py way that is well documented and
everyone's been using for ages.  it'll be a tough sell even with good
tooling.  (2) this theoretical front-end would have to appeal to the distro
maintainers so there would have to be a lot of talk to understand what
capabilities the distro maintainers would need from it.

-Toshio
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150323/66af199f/attachment.sig>

From solipsis at pitrou.net  Mon Mar 23 16:14:52 2015
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 23 Mar 2015 16:14:52 +0100
Subject: [Python-Dev] Use ptyhon -s as default shbang for system python
 executables/daemons
In-Reply-To: <20150323150613.GB4050@roan.lan>
References: <loom.20150318T164041-536@post.gmane.org>
 <20150318122203.12ff1ee3@limelight.wooz.org>
 <20150318214431.GA4050@roan.lan>
 <20150318175607.66a25ad3@limelight.wooz.org>
 <CABVPEKoioRy-kvRXuUpMWTZYSa0sheFDbFoUsECyg=tqyLg-6A@mail.gmail.com>
 <20150319152734.5bb6cd79@limelight.wooz.org>
 <CABVPEKo3edWRCLadA9_WVNvM6QZsAvToBjQGaOQXWxYC5F4qvg@mail.gmail.com>
 <CAMpsgwYx8RdrMOTQdS8bm7Qyj1+JjEf-Oig9-gx_s1vK15MD1w@mail.gmail.com>
 <CABVPEKrL3_yUKtQV4cvjzJ+woMe=kVOsCnU0+hPYYx85W01rKQ@mail.gmail.com>
 <20150323153023.6fa0466e@fsol> <20150323150613.GB4050@roan.lan>
Message-ID: <20150323161452.01c7189c@fsol>

On Mon, 23 Mar 2015 08:06:13 -0700
Toshio Kuratomi <a.badger at gmail.com> wrote:
> > 
> > I really think Donald has a good point when he suggests a specific
> > virtualenv for system programs using Python.
> > 
> The isolation is what we're seeking but I think the amount of work required
> and the added complexity for the distributions will make that hard to get
> distributions to sign up for.
> 
> If someone had the time to write a front end to install packages into
> a single "system-wide isolation unit" whose backend was a virtualenv we
> might be able to get distributions on-board with using that.

I don't think we're asking distributions anything. We're suggesting a
possible path, but it's not python-dev's job to dictate distributions
how they should package Python.

The virtualenv solution has the virtue that any improvement we might
put in it to help system packagers would automatically benefit everyone.
A specific "system Python" would not.

> The front end would need to install software so that you can still invoke
> /usr/bin/system-application and "system-application" would take care of
> activating the virtualenv.  It would need to be about as simple to build
> as the present python2 setup.py build/install with the flexibility in
> options that the distros need to install into FHS approved paths.  Some
> things like man pages, locale files, config files, and possibly other data
> files might need to be installed outside of the virtualenv directory.

Well, I don't understand what difference a virtualenv would make.
Using a virtualenv amounts to invoking a different interpreter path.
The rest of the filesystem (man pages locations, etc.) is still
accessible in the same way. But I may miss something :-)

Regards

Antoine.

From agriff at tin.it  Mon Mar 23 18:59:10 2015
From: agriff at tin.it (Andrea Griffini)
Date: Mon, 23 Mar 2015 18:59:10 +0100
Subject: [Python-Dev] --with-valgrind and --enable-shared
Message-ID: <CALy7ODsAn29_sPvOWn=O0Yu=XWuSMdvJCW5aR1eHpt8=idsu2Q@mail.gmail.com>

Hello,

does it have any sense for a linux distribution (arch to be specific) to
provide default Python package compiled with valgrind support? I thought
this flag was just about silencing false positives generated by valgrind
(in other words a workaround for "bugs" of another software) and useful
only when developing Python itself or C extensions.

The same distribution also compiles by default to a shared library and this
has a quite noticeable impact on performance on x64 (surprisingly for me)
for CPU bound processing; in a few test cases I measured valgrind+shared
Python running at 66% the speed of plain ./configure && make Python on my
system. Is this setting reasonable for general users?

If they are good defaults, why aren't them the default?

Andrea Griffini
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150323/b62342f1/attachment.html>

From brett at python.org  Mon Mar 23 19:08:23 2015
From: brett at python.org (Brett Cannon)
Date: Mon, 23 Mar 2015 18:08:23 +0000
Subject: [Python-Dev] --with-valgrind and --enable-shared
In-Reply-To: <CALy7ODsAn29_sPvOWn=O0Yu=XWuSMdvJCW5aR1eHpt8=idsu2Q@mail.gmail.com>
References: <CALy7ODsAn29_sPvOWn=O0Yu=XWuSMdvJCW5aR1eHpt8=idsu2Q@mail.gmail.com>
Message-ID: <CAP1=2W5N3s9APfx_KW7BcaomG1ba4+OxkFtHMPOmeD6jCAx4Jw@mail.gmail.com>

On Mon, Mar 23, 2015 at 1:59 PM Andrea Griffini <agriff at tin.it> wrote:

> Hello,
>
> does it have any sense for a linux distribution (arch to be specific) to
> provide default Python package compiled with valgrind support? I thought
> this flag was just about silencing false positives generated by valgrind
> (in other words a workaround for "bugs" of another software) and useful
> only when developing Python itself or C extensions.
>

It's not really our place to say if it makes sense for Arch to compile with
valgrind flags turned on. It really depends on how they use Python in their
Linux distribution and what their own goals are.


>
> The same distribution also compiles by default to a shared library and
> this has a quite noticeable impact on performance on x64 (surprisingly for
> me) for CPU bound processing; in a few test cases I measured
> valgrind+shared Python running at 66% the speed of plain ./configure &&
> make Python on my system. Is this setting reasonable for general users?
>

Once again, it all depends on how Arch uses Python.


>
> If they are good defaults, why aren't them the default?
>

That's a question for Arch Linux and not us.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150323/051dd8ec/attachment.html>

From agriff at tin.it  Mon Mar 23 19:16:17 2015
From: agriff at tin.it (Andrea Griffini)
Date: Mon, 23 Mar 2015 19:16:17 +0100
Subject: [Python-Dev] --with-valgrind and --enable-shared
In-Reply-To: <CAP1=2W5N3s9APfx_KW7BcaomG1ba4+OxkFtHMPOmeD6jCAx4Jw@mail.gmail.com>
References: <CALy7ODsAn29_sPvOWn=O0Yu=XWuSMdvJCW5aR1eHpt8=idsu2Q@mail.gmail.com>
 <CAP1=2W5N3s9APfx_KW7BcaomG1ba4+OxkFtHMPOmeD6jCAx4Jw@mail.gmail.com>
Message-ID: <CALy7ODu5z0vVOf0Xr_A+9hwLGhPWkDT-1F7b-DmVe6HBybzEaw@mail.gmail.com>

On Mon, Mar 23, 2015 at 7:08 PM, Brett Cannon <brett at python.org> wrote:
>
>
>
> It's not really our place to say if it makes sense for Arch to compile
> with valgrind flags turned on. It really depends on how they use Python in
> their Linux distribution and what their own goals are.
>

I already asked the package maintainers about this, I just wanted to know
if my understanding about what --with-valgrind means is correct or if there
are good reason to turn it on (except debugging Python).


>
>
>>
>> If they are good defaults, why aren't them the default?
>>
>
> That's a question for Arch Linux and not us.
>

I probably didn't explain myself correctly: I was asking why they're not
the default values for Python configure script...

Andrea
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150323/4c18274e/attachment.html>

From solipsis at pitrou.net  Mon Mar 23 19:23:51 2015
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 23 Mar 2015 19:23:51 +0100
Subject: [Python-Dev] --with-valgrind and --enable-shared
References: <CALy7ODsAn29_sPvOWn=O0Yu=XWuSMdvJCW5aR1eHpt8=idsu2Q@mail.gmail.com>
 <CAP1=2W5N3s9APfx_KW7BcaomG1ba4+OxkFtHMPOmeD6jCAx4Jw@mail.gmail.com>
 <CALy7ODu5z0vVOf0Xr_A+9hwLGhPWkDT-1F7b-DmVe6HBybzEaw@mail.gmail.com>
Message-ID: <20150323192351.7ae5ce29@fsol>

On Mon, 23 Mar 2015 19:16:17 +0100
Andrea Griffini <agriff at tin.it> wrote:
> On Mon, Mar 23, 2015 at 7:08 PM, Brett Cannon <brett at python.org> wrote:
> >
> > It's not really our place to say if it makes sense for Arch to compile
> > with valgrind flags turned on. It really depends on how they use Python in
> > their Linux distribution and what their own goals are.
> >
> 
> I already asked the package maintainers about this, I just wanted to know
> if my understanding about what --with-valgrind means is correct or if there
> are good reason to turn it on (except debugging Python).

I think your understanding is correct. It's the first time I hear of
Python being compiled --with-valgrind in a Linux distribution.

As for --enable-shared, it's more of a distro policy and isn't unheard
of. The RedHat world (Mageia, etc.) enables it as well. Perhaps it
makes updates easier if there are many installed programs embedding the
Python interpreter.

Regards

Antoine.



From njs at pobox.com  Mon Mar 23 20:39:56 2015
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 23 Mar 2015 12:39:56 -0700
Subject: [Python-Dev] Use ptyhon -s as default shbang for system python
	executables/daemons
In-Reply-To: <20150323161452.01c7189c@fsol>
References: <loom.20150318T164041-536@post.gmane.org>
 <20150318122203.12ff1ee3@limelight.wooz.org>
 <20150318214431.GA4050@roan.lan>
 <20150318175607.66a25ad3@limelight.wooz.org>
 <CABVPEKoioRy-kvRXuUpMWTZYSa0sheFDbFoUsECyg=tqyLg-6A@mail.gmail.com>
 <20150319152734.5bb6cd79@limelight.wooz.org>
 <CABVPEKo3edWRCLadA9_WVNvM6QZsAvToBjQGaOQXWxYC5F4qvg@mail.gmail.com>
 <CAMpsgwYx8RdrMOTQdS8bm7Qyj1+JjEf-Oig9-gx_s1vK15MD1w@mail.gmail.com>
 <CABVPEKrL3_yUKtQV4cvjzJ+woMe=kVOsCnU0+hPYYx85W01rKQ@mail.gmail.com>
 <20150323153023.6fa0466e@fsol> <20150323150613.GB4050@roan.lan>
 <20150323161452.01c7189c@fsol>
Message-ID: <CAPJVwBnTmVDdAeob-MdPgF+UGJjJZt8NP-60cw834ahr5em4cQ@mail.gmail.com>

On Mar 23, 2015 8:15 AM, "Antoine Pitrou" <solipsis at pitrou.net> wrote:
>
> On Mon, 23 Mar 2015 08:06:13 -0700
> Toshio Kuratomi <a.badger at gmail.com> wrote:
> > >
> > > I really think Donald has a good point when he suggests a specific
> > > virtualenv for system programs using Python.
> > >
> > The isolation is what we're seeking but I think the amount of work
required
> > and the added complexity for the distributions will make that hard to
get
> > distributions to sign up for.
> >
> > If someone had the time to write a front end to install packages into
> > a single "system-wide isolation unit" whose backend was a virtualenv we
> > might be able to get distributions on-board with using that.
>
> I don't think we're asking distributions anything. We're suggesting a
> possible path, but it's not python-dev's job to dictate distributions
> how they should package Python.
>
> The virtualenv solution has the virtue that any improvement we might
> put in it to help system packagers would automatically benefit everyone.
> A specific "system Python" would not.
>
> > The front end would need to install software so that you can still
invoke
> > /usr/bin/system-application and "system-application" would take care of
> > activating the virtualenv.  It would need to be about as simple to build
> > as the present python2 setup.py build/install with the flexibility in
> > options that the distros need to install into FHS approved paths.  Some
> > things like man pages, locale files, config files, and possibly other
data
> > files might need to be installed outside of the virtualenv directory.
>
> Well, I don't understand what difference a virtualenv would make.
> Using a virtualenv amounts to invoking a different interpreter path.
> The rest of the filesystem (man pages locations, etc.) is still
> accessible in the same way. But I may miss something :-)

The main issue that jumps to my mind is that 'yum/apt-get install
some-python-package' should install it into both the base python
interpreter and the system virtualenv, but that 'sudo pip install
some-python-package' should install into only the base interpreter but not
the system virtualenv. (Even if those two commands are run in sequence with
different versions of some-python-package.) This seems potentially complex.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150323/9758601c/attachment-0001.html>

From a.badger at gmail.com  Mon Mar 23 20:47:07 2015
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Mon, 23 Mar 2015 12:47:07 -0700
Subject: [Python-Dev] Use ptyhon -s as default shbang for system python
 executables/daemons
In-Reply-To: <20150323161452.01c7189c@fsol>
References: <20150318214431.GA4050@roan.lan>
 <20150318175607.66a25ad3@limelight.wooz.org>
 <CABVPEKoioRy-kvRXuUpMWTZYSa0sheFDbFoUsECyg=tqyLg-6A@mail.gmail.com>
 <20150319152734.5bb6cd79@limelight.wooz.org>
 <CABVPEKo3edWRCLadA9_WVNvM6QZsAvToBjQGaOQXWxYC5F4qvg@mail.gmail.com>
 <CAMpsgwYx8RdrMOTQdS8bm7Qyj1+JjEf-Oig9-gx_s1vK15MD1w@mail.gmail.com>
 <CABVPEKrL3_yUKtQV4cvjzJ+woMe=kVOsCnU0+hPYYx85W01rKQ@mail.gmail.com>
 <20150323153023.6fa0466e@fsol> <20150323150613.GB4050@roan.lan>
 <20150323161452.01c7189c@fsol>
Message-ID: <20150323194707.GC4050@roan.lan>

On Mon, Mar 23, 2015 at 04:14:52PM +0100, Antoine Pitrou wrote:
> On Mon, 23 Mar 2015 08:06:13 -0700
> Toshio Kuratomi <a.badger at gmail.com> wrote:
> > > 
> > > I really think Donald has a good point when he suggests a specific
> > > virtualenv for system programs using Python.
> > > 
> > The isolation is what we're seeking but I think the amount of work required
> > and the added complexity for the distributions will make that hard to get
> > distributions to sign up for.
> > 
> > If someone had the time to write a front end to install packages into
> > a single "system-wide isolation unit" whose backend was a virtualenv we
> > might be able to get distributions on-board with using that.
> 
> I don't think we're asking distributions anything. We're suggesting a
> possible path, but it's not python-dev's job to dictate distributions
> how they should package Python.
> 
> The virtualenv solution has the virtue that any improvement we might
> put in it to help system packagers would automatically benefit everyone.
> A specific "system Python" would not.
> 
> > The front end would need to install software so that you can still invoke
> > /usr/bin/system-application and "system-application" would take care of
> > activating the virtualenv.  It would need to be about as simple to build
> > as the present python2 setup.py build/install with the flexibility in
> > options that the distros need to install into FHS approved paths.  Some
> > things like man pages, locale files, config files, and possibly other data
> > files might need to be installed outside of the virtualenv directory.
> 
> Well, I don't understand what difference a virtualenv would make.
> Using a virtualenv amounts to invoking a different interpreter path.
> The rest of the filesystem (man pages locations, etc.) is still
> accessible in the same way. But I may miss something :-)
> 
<nod>  I think people who are saying "The system should just use
virtualenv" aren't realizing all of the reasons that's not as simple as it
sounds for distributions to implement.  thus the work required to implement
alternate solutions like a system python may seem less to the distros 
unless those issues are partially addressed at the virtualenv and
python-packaging level.

-Toshio
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150323/faf9f9c7/attachment.sig>

From greg at krypto.org  Mon Mar 23 22:58:06 2015
From: greg at krypto.org (Gregory P. Smith)
Date: Mon, 23 Mar 2015 21:58:06 +0000
Subject: [Python-Dev] Use ptyhon -s as default shbang for system python
	executables/daemons
In-Reply-To: <20150318124825.337dc1c6@anarchist.wooz.org>
References: <loom.20150318T164041-536@post.gmane.org>
 <CAMpsgwa3F2Abnptu8uFtxzCD_T-x9BCK9KtuOsVDW3kF1N88-g@mail.gmail.com>
 <20150318124825.337dc1c6@anarchist.wooz.org>
Message-ID: <CAGE7PNKPybgipHmyrEZih+WKSsUxjyLDWj0-bd-a9vDRACJXNw@mail.gmail.com>

On Wed, Mar 18, 2015 at 9:48 AM Barry Warsaw <barry at python.org> wrote:

> On Mar 18, 2015, at 05:31 PM, Victor Stinner wrote:
>
> >Does it work to pass command line options to Python in the shebang?
>
> Yes, but only one "word", thus -Es or -I.
>
> We've often mused about whether it makes sense to have two Pythons on the
> system.  One for system scripts and another for users.  System Python
> ('/usr/bin/spython' perhaps) would be locked down and only extensible by
> system packages.  On Debuntu that might mean no /usr/local on sys.path.  It
> would also have a much more limited set of installed packages, i.e. only
> those
> needed to support system functionality.
>

I recommend this. It'd be nice to see a common OS distro actually do it.

For a system-only Python lockdown you should remove distutils and pip to
prevent anyone from easily installing anything that isn't a distro package
into it. Possibly even removing its site-packages directory and sys.path
entry all together (things your distro needs to install could mix directly
with the stdlib packages)


> /usr/bin/python2 and /usr/bin/python3 then would be user tools, with all
> the
> goodness they currently have.
>
> It's never gotten much farther than musings, but protecting the system
> against
> the weird things people install would be a good thing.  OTOH, this feels a
> lot
> like virtual environments so maybe there's something useful to be mined
> there.
>

While people sometimes suggest virtualenv as a solution for this. It isn't
really the same thing. It isn't a hermetic clone of the original
interpreter.  It copies the main binary but symlinks back to much of the
stdlib.  So any existing virtualenv can be an out of date unsupported mix
of both after the original interpreter is updated. This has caused users
problems in the past with some minor version updates where an internal
non-public API used between some binary and a stdlib module was updated as
part of a bugfix. Suddenly they didn't match up for existing virtualenvs.

virtualenv is an amazing hack that I promote to most anyone for their own
applications use with the occasional known caveats (solvable by regurly
rebuilding your virtualenvs)...  But I wouldn't want to see it used for the
core OS itself even though it sounds better at first glance.

-gps
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150323/fd7ee20d/attachment.html>

From solipsis at pitrou.net  Mon Mar 23 23:06:57 2015
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 23 Mar 2015 23:06:57 +0100
Subject: [Python-Dev] Use ptyhon -s as default shbang for system python
 executables/daemons
References: <loom.20150318T164041-536@post.gmane.org>
 <CAMpsgwa3F2Abnptu8uFtxzCD_T-x9BCK9KtuOsVDW3kF1N88-g@mail.gmail.com>
 <20150318124825.337dc1c6@anarchist.wooz.org>
 <CAGE7PNKPybgipHmyrEZih+WKSsUxjyLDWj0-bd-a9vDRACJXNw@mail.gmail.com>
Message-ID: <20150323230657.294e3b6a@fsol>

On Mon, 23 Mar 2015 21:58:06 +0000
"Gregory P. Smith" <greg at krypto.org> wrote:
> 
> virtualenv is an amazing hack that I promote to most anyone for their own
> applications use with the occasional known caveats (solvable by regurly
> rebuilding your virtualenvs)...  But I wouldn't want to see it used for the
> core OS itself even though it sounds better at first glance.

pyvenv is not a hack, it mostly creates a couple symlinks and adds a
pip install into the environment.

Regards

Antoine.



From cs at zip.com.au  Mon Mar 23 23:10:30 2015
From: cs at zip.com.au (Cameron Simpson)
Date: Tue, 24 Mar 2015 09:10:30 +1100
Subject: [Python-Dev] Use ptyhon -s as default shbang for system python
 executables/daemons
In-Reply-To: <CAGE7PNKPybgipHmyrEZih+WKSsUxjyLDWj0-bd-a9vDRACJXNw@mail.gmail.com>
References: <CAGE7PNKPybgipHmyrEZih+WKSsUxjyLDWj0-bd-a9vDRACJXNw@mail.gmail.com>
Message-ID: <20150323221030.GA76846@cskk.homeip.net>

On 23Mar2015 21:58, Gregory P. Smith <greg at krypto.org> wrote:
>While people sometimes suggest virtualenv as a solution for this. It isn't
>really the same thing. It isn't a hermetic clone of the original
>interpreter.  It copies the main binary but symlinks back to much of the
>stdlib.

Oh.

I had thought a non-standalone venv arranged sys.path to fall back to the 
source interpreter.  Clearly I have not paid attention.

Cheers,
Cameron Simpson <cs at zip.com.au>

Yes, sometimes Perl looks like line-noise to the uninitiated, but to the
seasoned Perl programmer, it looks like checksummed line-noise with a mission
in life.        - The Llama Book

From storchaka at gmail.com  Mon Mar 23 23:29:38 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 24 Mar 2015 00:29:38 +0200
Subject: [Python-Dev] cpython: #23657 Don't explicitly do an isinstance
 check for str in zipapp
In-Reply-To: <20150322153314.70038.25460@psf.io>
References: <20150322153314.70038.25460@psf.io>
Message-ID: <meq44i$8rg$1@ger.gmane.org>

On 22.03.15 17:33, paul.moore wrote:
> https://hg.python.org/cpython/rev/0b2993742650
> changeset:   95126:0b2993742650
> user:        Paul Moore <p.f.moore at gmail.com>
> date:        Sun Mar 22 15:32:36 2015 +0000
> summary:
>    #23657 Don't explicitly do an isinstance check for str in zipapp
>
> As a result, explicitly support pathlib.Path objects as arguments.
> Also added tests for the CLI interface.

Congratulate with your first commit Paul!



From p.f.moore at gmail.com  Mon Mar 23 23:33:53 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 23 Mar 2015 22:33:53 +0000
Subject: [Python-Dev] cpython: #23657 Don't explicitly do an isinstance
 check for str in zipapp
In-Reply-To: <meq44i$8rg$1@ger.gmane.org>
References: <20150322153314.70038.25460@psf.io>
	<meq44i$8rg$1@ger.gmane.org>
Message-ID: <CACac1F8FqzQTG4oj6C6fgbq7OiHnwuYsQr+ro0c3jKTGqXHc-g@mail.gmail.com>

On 23 March 2015 at 22:29, Serhiy Storchaka <storchaka at gmail.com> wrote:
>> As a result, explicitly support pathlib.Path objects as arguments.
>> Also added tests for the CLI interface.
>
> Congratulate with your first commit Paul!

Thanks :-)
Paul

From greg.ewing at canterbury.ac.nz  Mon Mar 23 23:22:08 2015
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 24 Mar 2015 11:22:08 +1300
Subject: [Python-Dev] super() does not work during class initialization
In-Reply-To: <CAK9R32QmnjAWJkWR0WY93Xgr+G1aQmyqtcG4gG-C1nrjV17dNQ@mail.gmail.com>
References: <CAK9R32QaTyqsugF1_3-pPHfDsaEa4JEp0A_DFBboNOFwP_x82Q@mail.gmail.com>
 <CADiSq7fxRDGg5kE8gjX-q1=CkMi0t_Q7UwigoqNUZqi43Nw9FA@mail.gmail.com>
 <CAK9R32QmnjAWJkWR0WY93Xgr+G1aQmyqtcG4gG-C1nrjV17dNQ@mail.gmail.com>
Message-ID: <55109210.9020901@canterbury.ac.nz>

Martin Teichmann wrote:
> maybe
> we could just change the compiler to leave the order in which things are defined
> in a class in the class namespace, say as a member __order__? Then we could
> use plain-old dicts for the class namespace, and we would not slow down class
> creation (not that it matters much), as determining the order would happen at
> compile time.

I don't think the compiler can determine the order in
all cases. Consider:

   class Spam:

     if moon_is_full:
       alpha = 1
       beta = 2
     else:
       beta = 2
       alpha = 1

-- 
Greg

From kjp at uchicago.edu  Tue Mar 24 15:28:52 2015
From: kjp at uchicago.edu (Karl Pickett)
Date: Tue, 24 Mar 2015 09:28:52 -0500
Subject: [Python-Dev] How is obmalloc safe with "Invalid read of size 4" ?
Message-ID: <CAKo58-diZJj295Pv5j8nyk==Zp5aUZbP_=_DThJGAiaOaTo3GA@mail.gmail.com>

We are having random, rare, nonreproducible segfaults/hangs with python2 on
ubuntu 14.04 in EC2.  I've managed to attach GDB to some hung ones and
there looks like clear memory corruption in the 'interned' hash table,
causing lookdict_string() to spin forever because all remaining slots have
a garbage 'key' pointer.  This happens just loading the 'site' module
dependencies, like 're' or 'codecs', before any of our code even gets run.

So we then tried running it under valgrind, and we got a lot of nasty
errors.  Even after reading the Misc/README.valgrind, which talks about
*uninitialized* reads being ok, I still don't see how reading from *freed*
memory would ever be safe, and why the suppression file thinks thats ok:

$ valgrind   ./pymd79/bin/python -c ""
==14651== Memcheck, a memory error detector
==14651== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==14651== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright info
==14651== Command: ./pymd79/bin/python -c
==14651==
==14651== Invalid read of size 4
==14651==    at 0x461E40: Py_ADDRESS_IN_RANGE (obmalloc.c:1911)
==14651==    by 0x461EA3: PyObject_Free (obmalloc.c:994)
==14651==    by 0x4789AB: tupledealloc (tupleobject.c:235)
==14651==    by 0x5225BA: code_dealloc (codeobject.c:309)
==14651==    by 0x4CFFC3: load_source_module (import.c:1100)
==14651==    by 0x4D0E16: import_submodule (import.c:2700)
==14651==    by 0x4D1E19: PyImport_ImportModuleLevel (import.c:2515)
==14651==    by 0x4AE49A: builtin___import__ (bltinmodule.c:49)
==14651==    by 0x422C89: PyObject_Call (abstract.c:2529)
==14651==    by 0x4B12E5: PyEval_EvalFrameEx (ceval.c:3902)
==14651==    by 0x4B6A47: PyEval_EvalCodeEx (ceval.c:3265)
==14651==    by 0x4B6B71: PyEval_EvalCode (ceval.c:667)
==14651==  Address 0x5bcd020 is 2,256 bytes inside a block of size 2,801
free'd
==14651==    at 0x4C28577: free (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==14651==    by 0x4DB2B0: PyMarshal_ReadLastObjectFromFile (marshal.c:1145)
==14651==    by 0x4CFE71: load_source_module (import.c:801)

- Karl
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150324/76115737/attachment.html>

From hrvoje.niksic at avl.com  Tue Mar 24 16:22:31 2015
From: hrvoje.niksic at avl.com (Hrvoje Niksic)
Date: Tue, 24 Mar 2015 16:22:31 +0100
Subject: [Python-Dev] How is obmalloc safe with "Invalid read of size 4"
 ?
In-Reply-To: <CAKo58-diZJj295Pv5j8nyk==Zp5aUZbP_=_DThJGAiaOaTo3GA@mail.gmail.com>
References: <CAKo58-diZJj295Pv5j8nyk==Zp5aUZbP_=_DThJGAiaOaTo3GA@mail.gmail.com>
Message-ID: <55118137.60108@avl.com>

On 03/24/2015 03:28 PM, Karl Pickett wrote:
> So we then tried running it under valgrind, and we got a lot of nasty
> errors.  Even after reading the Misc/README.valgrind, which talks about
> *uninitialized* reads being ok, I still don't see how reading from
> *freed* memory would ever be safe, and why the suppression file thinks
> thats ok:

PyObject_Free() is not reading *freed* memory, it is reading memory 
outside (right before) the allocated range. This is, of course, 
undefined behavior as far as C is concerned and an invalid read in the 
eyes of valgrind. Still, it's the kind of thing you can get away with if 
you are writing a heavily optimized allocator (and if your name starts 
with "Tim" and ends with "Peters").

README.valgrind explains in quite some detail why this is done. In 
short, it allows for a very fast check whether the memory passed to 
PyObject_Free() was originally allocated by system malloc or by Python's 
pool allocator.


From sunfish7 at gmail.com  Tue Mar 24 16:53:18 2015
From: sunfish7 at gmail.com (=?utf-8?B?z4A=?=)
Date: Tue, 24 Mar 2015 15:53:18 +0000
Subject: [Python-Dev] PiCxx
Message-ID: <2E1837FF-9A40-40E6-BB27-FB57BA9D8E9C@gmail.com>

Hello PythonDevvers,

I apologise in advance. This is slightly off topic. This will be my only post on the subject.

PyCXX (http://sourceforge.net/projects/cxx/ <http://sourceforge.net/projects/cxx/>) accomplishes roughly the same as Boost::Python (C++ wrapper for CPython), only without requiring Boost.

The original code has become tangled and bloated.  
Over the winter I've rewritten it using C++11.  New C++11 features have allowed for more direct/compact/concise/readable code.

I've called my rewrite PiCxx and put it up here: https://github.com/p-i-/PiCxx <https://github.com/p-i-/PiCxx>

PiCxx only supports Python3 currently, as that is all I need for my own use.  It wouldn't be too much work to add 2.x support.  Also I've only tested it on my OSX system (although it is designed to be cross-platform).  It offers an alternative to Boost::Python that doesn't require Boost.

Improvements, suggestions, bug fixes, etc are all welcome.

Well, that's all. I originally subscribed to this list because I thought I might need some help navigating some of the dark corners of the CPython API, but thanks to a handful of Samurai on SO and IRC I seem to have scraped through.

I will unsubscribe in due course; it's been charming to spend awhile in the belly of the snake, and humbling to witness how hard you guys work.

Happy spring everyone,

?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150324/c5b92676/attachment.html>

From tjreedy at udel.edu  Tue Mar 24 20:17:17 2015
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 24 Mar 2015 15:17:17 -0400
Subject: [Python-Dev] PiCxx
In-Reply-To: <2E1837FF-9A40-40E6-BB27-FB57BA9D8E9C@gmail.com>
References: <2E1837FF-9A40-40E6-BB27-FB57BA9D8E9C@gmail.com>
Message-ID: <mesd8e$m1p$1@ger.gmane.org>

On 3/24/2015 11:53 AM, ? wrote:
> Hello PythonDevvers,
>
> I apologise in advance. This is slightly off topic. This will be my only
> post on the subject.

pydev is about development *of* Python the language and CPython the 
implementation.  python-list is about development *with* Python.  This 
post would be on-topic for python-list and would have fit better there.

> PyCXX (http://sourceforge.net/projects/cxx/) accomplishes roughly the
> same as Boost::Python (C++ wrapper for CPython), only without requiring
> Boost.
>
> The original code has become tangled and bloated.
> Over the winter I've rewritten it using C++11.  New C++11 features have
> allowed for more direct/compact/concise/readable code.
>
> I've called my rewrite PiCxx and put it up here:
> https://github.com/p-i-/PiCxx
> Improvements, suggestions, bug fixes, etc are all welcome.

Announcements and requests like this are fairly routine on python-list. 
  Sometimes they get responses and discussion on the list, sometimes not.

One can participate on python-list via news/web mirror 
gmane.comp.python.general at news.gmane.org.

-- 
Terry Jan Reedy



From p.f.moore at gmail.com  Tue Mar 24 22:31:29 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 24 Mar 2015 21:31:29 +0000
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <CACac1F86h7qQ=ggKS5Z8FCR4w1L1k-smoqP0362TpqK23autJw@mail.gmail.com>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAKJDb-N1ewaSsWyEycVHWwDASx8OjXG9LAWw3k3ntCZtw6oJSQ@mail.gmail.com>
 <CAP1=2W7Cx5JpQV_pr61RQs1uBUSJF5f6KG0cD-QCwR2+9ijegg@mail.gmail.com>
 <CACac1F86h7qQ=ggKS5Z8FCR4w1L1k-smoqP0362TpqK23autJw@mail.gmail.com>
Message-ID: <CACac1F_xb+0YMCxV2RB_DmPUGZia6HJDMYFLTBaOfDq7oqC9AA@mail.gmail.com>

On 12 March 2015 at 17:44, Paul Moore <p.f.moore at gmail.com> wrote:
> On 12 March 2015 at 17:26, Brett Cannon <brett at python.org> wrote:
>>> I'm all for ditching our 'libffi_msvc' in favor of adding libffi as
>>> another 'external' for the Windows build.  I have managed to get
>>> _ctypes to build on Windows using vanilla libffi sources, prepared
>>> using their configure script from within Git Bash and built with our
>>> usual Windows build system (properly patched).  Unfortunately, making
>>> things usable will take some work on ctypes itself, which I'm not
>>> qualified to do. I'm happy to pass on my procedure and patches for
>>> getting to the point of successful compilation to anyone who feels up
>>> to fixing the things that are broken.
>>
>>
>> So it seems possible to use upstream libffi but will require some work.
>
> I'd be willing to contemplate helping out on the Windows side of
> things, if nobody else steps up (with the proviso that I have little
> free time, and I'm saying this without much idea of what's involved
> :-)) If Zachary can give a bit more detail on what the work on ctypes
> is, and/or put what he has somewhere that I could have a look at, that
> might help.

One thing that seems to be an issue. On Windows, ctypes detects if the
FFI call used the wrong number of arguments off the stack, and raises
a ValueError if it does. The tests rely on that behaviour. But it's
based on ffi_call() returning a value, which upstream libffi doesn't
do. As far as I can tell (not that the libffi docs are exactly
comprehensive...) there's no way of getting that information from
upstream libffi.

What does Unix ctypes do when faced with a call being made with the
wrong number of arguments? On Windows, using upstream libffi and
omitting the existing check, it seems to crash the Python process,
which obviously isn't good. But the test that fails is
Windows-specific, and short of going through all the tests looking for
one that checks passing the wrong number of arguments and isn't
platform-specific, I don't know how Unix handles this.

Can anyone on Unix tell me if a ctypes call with the wrong number of
arguments returns ValueError on Unix? Something like strcmp() (with no
args) should do as a test, I guess...

If there's a way Unix handles this, I can see about replicating it on
Windows. But if there isn't, I fear we could always need a patched
libffi to maintain the interface we currently have...

Thanks,
Paul

From fijall at gmail.com  Wed Mar 25 09:05:52 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 25 Mar 2015 10:05:52 +0200
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <CACac1F_xb+0YMCxV2RB_DmPUGZia6HJDMYFLTBaOfDq7oqC9AA@mail.gmail.com>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAKJDb-N1ewaSsWyEycVHWwDASx8OjXG9LAWw3k3ntCZtw6oJSQ@mail.gmail.com>
 <CAP1=2W7Cx5JpQV_pr61RQs1uBUSJF5f6KG0cD-QCwR2+9ijegg@mail.gmail.com>
 <CACac1F86h7qQ=ggKS5Z8FCR4w1L1k-smoqP0362TpqK23autJw@mail.gmail.com>
 <CACac1F_xb+0YMCxV2RB_DmPUGZia6HJDMYFLTBaOfDq7oqC9AA@mail.gmail.com>
Message-ID: <CAK5idxTH--Hsw2tF4cU+yVXUfV1y7jDN=KKneut9sZu_sjQf1A@mail.gmail.com>

On Tue, Mar 24, 2015 at 11:31 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 12 March 2015 at 17:44, Paul Moore <p.f.moore at gmail.com> wrote:
>> On 12 March 2015 at 17:26, Brett Cannon <brett at python.org> wrote:
>>>> I'm all for ditching our 'libffi_msvc' in favor of adding libffi as
>>>> another 'external' for the Windows build.  I have managed to get
>>>> _ctypes to build on Windows using vanilla libffi sources, prepared
>>>> using their configure script from within Git Bash and built with our
>>>> usual Windows build system (properly patched).  Unfortunately, making
>>>> things usable will take some work on ctypes itself, which I'm not
>>>> qualified to do. I'm happy to pass on my procedure and patches for
>>>> getting to the point of successful compilation to anyone who feels up
>>>> to fixing the things that are broken.
>>>
>>>
>>> So it seems possible to use upstream libffi but will require some work.
>>
>> I'd be willing to contemplate helping out on the Windows side of
>> things, if nobody else steps up (with the proviso that I have little
>> free time, and I'm saying this without much idea of what's involved
>> :-)) If Zachary can give a bit more detail on what the work on ctypes
>> is, and/or put what he has somewhere that I could have a look at, that
>> might help.
>
> One thing that seems to be an issue. On Windows, ctypes detects if the
> FFI call used the wrong number of arguments off the stack, and raises
> a ValueError if it does. The tests rely on that behaviour. But it's
> based on ffi_call() returning a value, which upstream libffi doesn't
> do. As far as I can tell (not that the libffi docs are exactly
> comprehensive...) there's no way of getting that information from
> upstream libffi.
>
> What does Unix ctypes do when faced with a call being made with the
> wrong number of arguments? On Windows, using upstream libffi and
> omitting the existing check, it seems to crash the Python process,
> which obviously isn't good. But the test that fails is
> Windows-specific, and short of going through all the tests looking for
> one that checks passing the wrong number of arguments and isn't
> platform-specific, I don't know how Unix handles this.
>
> Can anyone on Unix tell me if a ctypes call with the wrong number of
> arguments returns ValueError on Unix? Something like strcmp() (with no
> args) should do as a test, I guess...
>
> If there's a way Unix handles this, I can see about replicating it on
> Windows. But if there isn't, I fear we could always need a patched
> libffi to maintain the interface we currently have...
>
> Thanks,
> Paul

Linux crashes. The mechanism for detecting the number of arguments is
only available on windows (note that this is a band-aid anyway, since
if your arguments are of the wrong kind you segfault anyway). We do
have two copies of libffi one for windows one for unix anyway, don't
we?

From p.f.moore at gmail.com  Wed Mar 25 09:15:18 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 25 Mar 2015 08:15:18 +0000
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <CAK5idxTH--Hsw2tF4cU+yVXUfV1y7jDN=KKneut9sZu_sjQf1A@mail.gmail.com>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAKJDb-N1ewaSsWyEycVHWwDASx8OjXG9LAWw3k3ntCZtw6oJSQ@mail.gmail.com>
 <CAP1=2W7Cx5JpQV_pr61RQs1uBUSJF5f6KG0cD-QCwR2+9ijegg@mail.gmail.com>
 <CACac1F86h7qQ=ggKS5Z8FCR4w1L1k-smoqP0362TpqK23autJw@mail.gmail.com>
 <CACac1F_xb+0YMCxV2RB_DmPUGZia6HJDMYFLTBaOfDq7oqC9AA@mail.gmail.com>
 <CAK5idxTH--Hsw2tF4cU+yVXUfV1y7jDN=KKneut9sZu_sjQf1A@mail.gmail.com>
Message-ID: <CACac1F80q8MZ3Q7fGxYXkvGhQZ4cxWXHgmP0H-7wqedw-DR5WQ@mail.gmail.com>

On 25 March 2015 at 08:05, Maciej Fijalkowski <fijall at gmail.com> wrote:
> Linux crashes. The mechanism for detecting the number of arguments is
> only available on windows (note that this is a band-aid anyway, since
> if your arguments are of the wrong kind you segfault anyway). We do
> have two copies of libffi one for windows one for unix anyway, don't
> we?

Yes, this is in relation to looking at whether we can just use the
upstream libffi directly on Windows. It looks like we guarantee a
ValueError for the wrong number of arguments on Windows and upstream
libffi doesn't give a way to detect that, so there may be a backward
compatibility issue in doing so.

Thanks for checking for me.
Paul

From ncoghlan at gmail.com  Wed Mar 25 09:54:28 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 25 Mar 2015 18:54:28 +1000
Subject: [Python-Dev] Use ptyhon -s as default shbang for system python
	executables/daemons
In-Reply-To: <CAGE7PNKPybgipHmyrEZih+WKSsUxjyLDWj0-bd-a9vDRACJXNw@mail.gmail.com>
References: <loom.20150318T164041-536@post.gmane.org>
 <CAMpsgwa3F2Abnptu8uFtxzCD_T-x9BCK9KtuOsVDW3kF1N88-g@mail.gmail.com>
 <20150318124825.337dc1c6@anarchist.wooz.org>
 <CAGE7PNKPybgipHmyrEZih+WKSsUxjyLDWj0-bd-a9vDRACJXNw@mail.gmail.com>
Message-ID: <CADiSq7e7cDXW9AOK=8VaFOWVPs37eF=JbyTTkp7uqXMrMgtusg@mail.gmail.com>

On 24 Mar 2015 07:59, "Gregory P. Smith" <greg at krypto.org> wrote:
>
>
> On Wed, Mar 18, 2015 at 9:48 AM Barry Warsaw <barry at python.org> wrote:
>>
>> On Mar 18, 2015, at 05:31 PM, Victor Stinner wrote:
>>
>> >Does it work to pass command line options to Python in the shebang?
>>
>> Yes, but only one "word", thus -Es or -I.
>>
>> We've often mused about whether it makes sense to have two Pythons on the
>> system.  One for system scripts and another for users.  System Python
>> ('/usr/bin/spython' perhaps) would be locked down and only extensible by
>> system packages.  On Debuntu that might mean no /usr/local on sys.path.
It
>> would also have a much more limited set of installed packages, i.e. only
those
>> needed to support system functionality.
>
>
> I recommend this. It'd be nice to see a common OS distro actually do it.
>
> For a system-only Python lockdown you should remove distutils and pip to
prevent anyone from easily installing anything that isn't a distro package
into it. Possibly even removing its site-packages directory and sys.path
entry all together (things your distro needs to install could mix directly
with the stdlib packages)

The hard part of this is the complexity in getpath.c and the current
difficulty of testing the interaction of all the different configuration
settings properly. Completely overriding the standard path setup isn't too
bad, it's *partially* overriding it that's hard.

That's really one of the main goals of the proposed startup refactoring in
PEP 432 - separating things into distinct phases so the path configuration
is easier to customise and so the current behaviour becomes easier to test.

That's a fairly big project with high barriers to entry though, as even
just wrapping your head around the current accumulated complexity in the
way the default sys.path gets constructed is a time consuming task.

That approach has the virtue of not being a "system Python" specific
solution, it's a "make embedding the CPython runtime much easier" solution,
with "system Python with different default settings" as the reference use
case.

>> /usr/bin/python2 and /usr/bin/python3 then would be user tools, with all
the
>> goodness they currently have.
>>
>> It's never gotten much farther than musings, but protecting the system
against
>> the weird things people install would be a good thing.  OTOH, this feels
a lot
>> like virtual environments so maybe there's something useful to be mined
there.

For distro level work it would be closer to the environment modules used in
high performance computing work (which are also the basis of
softwarecollections.org)

Where things get really messy is on the package management side. Either we
end up with yet another copy of modules that are often already duplicated
between the Python 2 & 3 stacks, or else the relevant user Python needs to
be customised to also be able to see the system level libraries for things
like Kerberos, interfacing with the package manager, dbus, etc.

The former approach = "ugh, no, just no" as far as I'm concerned, while the
latter would be a messy patch to try to carry given the current difficulty
of customising and testing the startup sequence.

>
>
> While people sometimes suggest virtualenv as a solution for this. It
isn't really the same thing. It isn't a hermetic clone of the original
interpreter.  It copies the main binary but symlinks back to much of the
stdlib.  So any existing virtualenv can be an out of date unsupported mix
of both after the original interpreter is updated. This has caused users
problems in the past with some minor version updates where an internal
non-public API used between some binary and a stdlib module was updated as
part of a bugfix. Suddenly they didn't match up for existing virtualenvs.
>
> virtualenv is an amazing hack that I promote to most anyone for their own
applications use with the occasional known caveats (solvable by regurly
rebuilding your virtualenvs)...  But I wouldn't want to see it used for the
core OS itself even though it sounds better at first glance.

Right, this is why Fedora/RHEL/CentOS have software collections instead,
and those have their own "gotchas" that currently make them unsuitable for
use as the main system Python (for example: the Fedora installer & package
manager are both written in Python, so having the main system Python be an
SCL would bring SCL support into the minimal package set, which we wouldn't
want to do. The typical aim is to find ways to get things *out* of that
set, not add new ones)

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150325/4afaf6c4/attachment-0001.html>

From solipsis at pitrou.net  Wed Mar 25 10:09:10 2015
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 25 Mar 2015 10:09:10 +0100
Subject: [Python-Dev] libffi embedded in CPython
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAKJDb-N1ewaSsWyEycVHWwDASx8OjXG9LAWw3k3ntCZtw6oJSQ@mail.gmail.com>
 <CAP1=2W7Cx5JpQV_pr61RQs1uBUSJF5f6KG0cD-QCwR2+9ijegg@mail.gmail.com>
 <CACac1F86h7qQ=ggKS5Z8FCR4w1L1k-smoqP0362TpqK23autJw@mail.gmail.com>
 <CACac1F_xb+0YMCxV2RB_DmPUGZia6HJDMYFLTBaOfDq7oqC9AA@mail.gmail.com>
 <CAK5idxTH--Hsw2tF4cU+yVXUfV1y7jDN=KKneut9sZu_sjQf1A@mail.gmail.com>
 <CACac1F80q8MZ3Q7fGxYXkvGhQZ4cxWXHgmP0H-7wqedw-DR5WQ@mail.gmail.com>
Message-ID: <20150325100910.10e29a5c@fsol>

On Wed, 25 Mar 2015 08:15:18 +0000
Paul Moore <p.f.moore at gmail.com> wrote:
> On 25 March 2015 at 08:05, Maciej Fijalkowski <fijall at gmail.com> wrote:
> > Linux crashes. The mechanism for detecting the number of arguments is
> > only available on windows (note that this is a band-aid anyway, since
> > if your arguments are of the wrong kind you segfault anyway). We do
> > have two copies of libffi one for windows one for unix anyway, don't
> > we?
> 
> Yes, this is in relation to looking at whether we can just use the
> upstream libffi directly on Windows. It looks like we guarantee a
> ValueError for the wrong number of arguments on Windows and upstream
> libffi doesn't give a way to detect that, so there may be a backward
> compatibility issue in doing so.

I'm not sure we guarantee anything. In any case, it's only a small
proportion of the kind of crashes you can get by messing the signature.

Regards

Antoine.



From p.f.moore at gmail.com  Wed Mar 25 10:22:01 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 25 Mar 2015 09:22:01 +0000
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <20150325100910.10e29a5c@fsol>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAKJDb-N1ewaSsWyEycVHWwDASx8OjXG9LAWw3k3ntCZtw6oJSQ@mail.gmail.com>
 <CAP1=2W7Cx5JpQV_pr61RQs1uBUSJF5f6KG0cD-QCwR2+9ijegg@mail.gmail.com>
 <CACac1F86h7qQ=ggKS5Z8FCR4w1L1k-smoqP0362TpqK23autJw@mail.gmail.com>
 <CACac1F_xb+0YMCxV2RB_DmPUGZia6HJDMYFLTBaOfDq7oqC9AA@mail.gmail.com>
 <CAK5idxTH--Hsw2tF4cU+yVXUfV1y7jDN=KKneut9sZu_sjQf1A@mail.gmail.com>
 <CACac1F80q8MZ3Q7fGxYXkvGhQZ4cxWXHgmP0H-7wqedw-DR5WQ@mail.gmail.com>
 <20150325100910.10e29a5c@fsol>
Message-ID: <CACac1F88pcjQg7J4E8gn-Gy9aJMKrVWc1_zDhHcYpbgJ49JJfA@mail.gmail.com>

On 25 March 2015 at 09:09, Antoine Pitrou <solipsis at pitrou.net> wrote:
> I'm not sure we guarantee anything. In any case, it's only a small
> proportion of the kind of crashes you can get by messing the signature.

Fair point. I guess what I'm asking is, would it be OK to remove the
code that checks for a stack size discrepancy and raises ValueError,
and the tests that verify this behaviour, as part of switching to
using upstream libffi directly?

On a related note, is there any information available on how the
"externals" repo is maintained? In particular, should things in there
be exact copies of upstream, or is it OK to include extra data (in
this case, the results of running "configure" for the Windows build)?
It works for me either way, it's just a matter of how the build
process would be structured and maintained.

Thanks,
Paul

Paul

From solipsis at pitrou.net  Wed Mar 25 10:30:30 2015
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 25 Mar 2015 10:30:30 +0100
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <CACac1F88pcjQg7J4E8gn-Gy9aJMKrVWc1_zDhHcYpbgJ49JJfA@mail.gmail.com>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAKJDb-N1ewaSsWyEycVHWwDASx8OjXG9LAWw3k3ntCZtw6oJSQ@mail.gmail.com>
 <CAP1=2W7Cx5JpQV_pr61RQs1uBUSJF5f6KG0cD-QCwR2+9ijegg@mail.gmail.com>
 <CACac1F86h7qQ=ggKS5Z8FCR4w1L1k-smoqP0362TpqK23autJw@mail.gmail.com>
 <CACac1F_xb+0YMCxV2RB_DmPUGZia6HJDMYFLTBaOfDq7oqC9AA@mail.gmail.com>
 <CAK5idxTH--Hsw2tF4cU+yVXUfV1y7jDN=KKneut9sZu_sjQf1A@mail.gmail.com>
 <CACac1F80q8MZ3Q7fGxYXkvGhQZ4cxWXHgmP0H-7wqedw-DR5WQ@mail.gmail.com>
 <20150325100910.10e29a5c@fsol>
 <CACac1F88pcjQg7J4E8gn-Gy9aJMKrVWc1_zDhHcYpbgJ49JJfA@mail.gmail.com>
Message-ID: <20150325103030.22842700@fsol>

On Wed, 25 Mar 2015 09:22:01 +0000
Paul Moore <p.f.moore at gmail.com> wrote:
> On 25 March 2015 at 09:09, Antoine Pitrou <solipsis at pitrou.net> wrote:
> > I'm not sure we guarantee anything. In any case, it's only a small
> > proportion of the kind of crashes you can get by messing the signature.
> 
> Fair point. I guess what I'm asking is, would it be OK to remove the
> code that checks for a stack size discrepancy and raises ValueError,
> and the tests that verify this behaviour, as part of switching to
> using upstream libffi directly?

IMHO yes. Of course it's still a behaviour change, but relying on it
sounds seriously broken.

> On a related note, is there any information available on how the
> "externals" repo is maintained? In particular, should things in there
> be exact copies of upstream, or is it OK to include extra data (in
> this case, the results of running "configure" for the Windows build)?
> It works for me either way, it's just a matter of how the build
> process would be structured and maintained.

Zachary or Steve could probably answer you on that one.

Regards

Antoine.

From ncoghlan at gmail.com  Wed Mar 25 13:32:03 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 25 Mar 2015 22:32:03 +1000
Subject: [Python-Dev] super() does not work during class initialization
In-Reply-To: <55109210.9020901@canterbury.ac.nz>
References: <CAK9R32QaTyqsugF1_3-pPHfDsaEa4JEp0A_DFBboNOFwP_x82Q@mail.gmail.com>
 <CADiSq7fxRDGg5kE8gjX-q1=CkMi0t_Q7UwigoqNUZqi43Nw9FA@mail.gmail.com>
 <CAK9R32QmnjAWJkWR0WY93Xgr+G1aQmyqtcG4gG-C1nrjV17dNQ@mail.gmail.com>
 <55109210.9020901@canterbury.ac.nz>
Message-ID: <CADiSq7cBFG7KUMub2-BeCog+OHJqYecbZ+599YWMiJY6Ug14Pw@mail.gmail.com>

On 24 March 2015 at 08:22, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Martin Teichmann wrote:
>>
>> maybe
>> we could just change the compiler to leave the order in which things are
>> defined
>> in a class in the class namespace, say as a member __order__? Then we
>> could
>> use plain-old dicts for the class namespace, and we would not slow down
>> class
>> creation (not that it matters much), as determining the order would happen
>> at
>> compile time.
>
>
> I don't think the compiler can determine the order in
> all cases. Consider:
>
>   class Spam:
>
>     if moon_is_full:
>       alpha = 1
>       beta = 2
>     else:
>       beta = 2
>       alpha = 1

This is also expected to work in class namespaces:

    locals()["alpha"] = 1

The language reference suggests it isn't, there's an open tracker
issue I filed some time ago to suggest clarifying it but haven't found
the time to actually sit down and come up with readable wording:
http://bugs.python.org/issue17960

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From zachary.ware at gmail.com  Wed Mar 25 14:36:35 2015
From: zachary.ware at gmail.com (Zachary Ware)
Date: Wed, 25 Mar 2015 08:36:35 -0500
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <CACac1F88pcjQg7J4E8gn-Gy9aJMKrVWc1_zDhHcYpbgJ49JJfA@mail.gmail.com>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAKJDb-N1ewaSsWyEycVHWwDASx8OjXG9LAWw3k3ntCZtw6oJSQ@mail.gmail.com>
 <CAP1=2W7Cx5JpQV_pr61RQs1uBUSJF5f6KG0cD-QCwR2+9ijegg@mail.gmail.com>
 <CACac1F86h7qQ=ggKS5Z8FCR4w1L1k-smoqP0362TpqK23autJw@mail.gmail.com>
 <CACac1F_xb+0YMCxV2RB_DmPUGZia6HJDMYFLTBaOfDq7oqC9AA@mail.gmail.com>
 <CAK5idxTH--Hsw2tF4cU+yVXUfV1y7jDN=KKneut9sZu_sjQf1A@mail.gmail.com>
 <CACac1F80q8MZ3Q7fGxYXkvGhQZ4cxWXHgmP0H-7wqedw-DR5WQ@mail.gmail.com>
 <20150325100910.10e29a5c@fsol>
 <CACac1F88pcjQg7J4E8gn-Gy9aJMKrVWc1_zDhHcYpbgJ49JJfA@mail.gmail.com>
Message-ID: <CAKJDb-Pnasx0HG76iAjdNE=Uskwe27wPgg-a--q+AkGFrsU8Ow@mail.gmail.com>

On Mar 25, 2015 4:22 AM, "Paul Moore" <p.f.moore at gmail.com> wrote:
>
> On 25 March 2015 at 09:09, Antoine Pitrou <solipsis at pitrou.net> wrote:
> > I'm not sure we guarantee anything. In any case, it's only a small
> > proportion of the kind of crashes you can get by messing the signature.
>
> Fair point. I guess what I'm asking is, would it be OK to remove the
> code that checks for a stack size discrepancy and raises ValueError,
> and the tests that verify this behaviour, as part of switching to
> using upstream libffi directly?
>
> On a related note, is there any information available on how the
> "externals" repo is maintained? In particular, should things in there
> be exact copies of upstream, or is it OK to include extra data (in
> this case, the results of running "configure" for the Windows build)?
> It works for me either way, it's just a matter of how the build
> process would be structured and maintained.

Its not extremely clear how it's "supposed to be" done; look at the
differences between how we handle OpenSSL and Tcl/Tk, for example. One way
or the other, though, we will store the configure output so that our build
doesn't depend on any more than it absolutely has to.

--
Zach
On a phone
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150325/39865c39/attachment.html>

From lkb.teichmann at gmail.com  Wed Mar 25 15:29:24 2015
From: lkb.teichmann at gmail.com (Martin Teichmann)
Date: Wed, 25 Mar 2015 15:29:24 +0100
Subject: [Python-Dev] super() does not work during class initialization
In-Reply-To: <CADiSq7cBFG7KUMub2-BeCog+OHJqYecbZ+599YWMiJY6Ug14Pw@mail.gmail.com>
References: <CAK9R32QaTyqsugF1_3-pPHfDsaEa4JEp0A_DFBboNOFwP_x82Q@mail.gmail.com>
 <CADiSq7fxRDGg5kE8gjX-q1=CkMi0t_Q7UwigoqNUZqi43Nw9FA@mail.gmail.com>
 <CAK9R32QmnjAWJkWR0WY93Xgr+G1aQmyqtcG4gG-C1nrjV17dNQ@mail.gmail.com>
 <55109210.9020901@canterbury.ac.nz>
 <CADiSq7cBFG7KUMub2-BeCog+OHJqYecbZ+599YWMiJY6Ug14Pw@mail.gmail.com>
Message-ID: <CAK9R32QLqPgffffSr=ysx6LGthXCjNJ0gr=e0PGCHFi+3X1jZg@mail.gmail.com>

>> I don't think the compiler can determine the order in
>> all cases. Consider:
>>
>>   class Spam:
>>
>>     if moon_is_full:
>>       alpha = 1
>>       beta = 2
>>     else:
>>       beta = 2
>>       alpha = 1
>
> This is also expected to work in class namespaces:
>
>     locals()["alpha"] = 1
>
> The language reference suggests it isn't, there's an open tracker
> issue I filed some time ago to suggest clarifying it but haven't found
> the time to actually sit down and come up with readable wording:
> http://bugs.python.org/issue17960

Well, for sure the compiler cannot deduce things happening at runtime,
but it still has an order in which things appear at compile time. And
for nearly all use cases that's by far enough. We cannot stop people
from doing weird things, but unless there is a use case for those
weird things, we don't need to support them.

And I think that tampering with locals() is not really a good idea. In
your issue you mention that you want that so that you can create
enums programatically. This is already doable by writing:

    from enum import Enum
    from types import new_class

    def cb(ns):
        ns.update({"a{}".format(i): i for i in range(100)})
        return ns

    Calc = new_class("Calc", (Enum,), None, cb)

I think this is simple enough that we don't need another way
of doing it.

Btw, going on-topic again, what should I do to get my patch
to make super() work during class initialization into python?

Greetings

Martin

From Steve.Dower at microsoft.com  Wed Mar 25 16:57:53 2015
From: Steve.Dower at microsoft.com (Steve Dower)
Date: Wed, 25 Mar 2015 15:57:53 +0000
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <CAKJDb-Pnasx0HG76iAjdNE=Uskwe27wPgg-a--q+AkGFrsU8Ow@mail.gmail.com>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAKJDb-N1ewaSsWyEycVHWwDASx8OjXG9LAWw3k3ntCZtw6oJSQ@mail.gmail.com>
 <CAP1=2W7Cx5JpQV_pr61RQs1uBUSJF5f6KG0cD-QCwR2+9ijegg@mail.gmail.com>
 <CACac1F86h7qQ=ggKS5Z8FCR4w1L1k-smoqP0362TpqK23autJw@mail.gmail.com>
 <CACac1F_xb+0YMCxV2RB_DmPUGZia6HJDMYFLTBaOfDq7oqC9AA@mail.gmail.com>
 <CAK5idxTH--Hsw2tF4cU+yVXUfV1y7jDN=KKneut9sZu_sjQf1A@mail.gmail.com>
 <CACac1F80q8MZ3Q7fGxYXkvGhQZ4cxWXHgmP0H-7wqedw-DR5WQ@mail.gmail.com>
 <20150325100910.10e29a5c@fsol>
 <CACac1F88pcjQg7J4E8gn-Gy9aJMKrVWc1_zDhHcYpbgJ49JJfA@mail.gmail.com>
 <CAKJDb-Pnasx0HG76iAjdNE=Uskwe27wPgg-a--q+AkGFrsU8Ow@mail.gmail.com>
Message-ID: <BY1PR03MB146674354E3CED584235BAE6F50B0@BY1PR03MB1466.namprd03.prod.outlook.com>

Zachary Ware wrote:
> On Mar 25, 2015 4:22 AM, "Paul Moore" <p.f.moore at gmail.com> wrote:
>> On a related note, is there any information available on how the
>> "externals" repo is maintained? In particular, should things in there
>> be exact copies of upstream, or is it OK to include extra data (in
>> this case, the results of running "configure" for the Windows build)?
>> It works for me either way, it's just a matter of how the build
>> process would be structured and maintained.
> 
> Its not extremely clear how it's "supposed to be" done; look at the differences
> between how we handle OpenSSL and Tcl/Tk, for example. One way or the other,
> though, we will store the configure output so that our build doesn't depend on
> any more than it absolutely has to.

It would be nice for at least one other person to know how to do it. I can't see the SVN logs, but I feel like you're the only person who's touched it in a while :)

As well as the configure output, we sometimes include very light patching (for example, Tcl/Tk currently has a patch to their makefile to work around a VC14 bug). I'm not sure whether that's considered good practice or not, but I prefer only needing to do it once vs. building it into the build system.

Cheers,
Steve

From ncoghlan at gmail.com  Thu Mar 26 03:28:15 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 26 Mar 2015 12:28:15 +1000
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <BY1PR03MB146674354E3CED584235BAE6F50B0@BY1PR03MB1466.namprd03.prod.outlook.com>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAKJDb-N1ewaSsWyEycVHWwDASx8OjXG9LAWw3k3ntCZtw6oJSQ@mail.gmail.com>
 <CAP1=2W7Cx5JpQV_pr61RQs1uBUSJF5f6KG0cD-QCwR2+9ijegg@mail.gmail.com>
 <CACac1F86h7qQ=ggKS5Z8FCR4w1L1k-smoqP0362TpqK23autJw@mail.gmail.com>
 <CACac1F_xb+0YMCxV2RB_DmPUGZia6HJDMYFLTBaOfDq7oqC9AA@mail.gmail.com>
 <CAK5idxTH--Hsw2tF4cU+yVXUfV1y7jDN=KKneut9sZu_sjQf1A@mail.gmail.com>
 <CACac1F80q8MZ3Q7fGxYXkvGhQZ4cxWXHgmP0H-7wqedw-DR5WQ@mail.gmail.com>
 <20150325100910.10e29a5c@fsol>
 <CACac1F88pcjQg7J4E8gn-Gy9aJMKrVWc1_zDhHcYpbgJ49JJfA@mail.gmail.com>
 <CAKJDb-Pnasx0HG76iAjdNE=Uskwe27wPgg-a--q+AkGFrsU8Ow@mail.gmail.com>
 <BY1PR03MB146674354E3CED584235BAE6F50B0@BY1PR03MB1466.namprd03.prod.outlook.com>
Message-ID: <CADiSq7c+6GzDaoKxxBMc0=0hUzwMdEZYNzp2Apz0C6DqGzNo2A@mail.gmail.com>

On 26 March 2015 at 01:57, Steve Dower <Steve.Dower at microsoft.com> wrote:
> Zachary Ware wrote:
>> On Mar 25, 2015 4:22 AM, "Paul Moore" <p.f.moore at gmail.com> wrote:
>>> On a related note, is there any information available on how the
>>> "externals" repo is maintained? In particular, should things in there
>>> be exact copies of upstream, or is it OK to include extra data (in
>>> this case, the results of running "configure" for the Windows build)?
>>> It works for me either way, it's just a matter of how the build
>>> process would be structured and maintained.
>>
>> Its not extremely clear how it's "supposed to be" done; look at the differences
>> between how we handle OpenSSL and Tcl/Tk, for example. One way or the other,
>> though, we will store the configure output so that our build doesn't depend on
>> any more than it absolutely has to.
>
> It would be nice for at least one other person to know how to do it. I can't see the SVN logs, but I feel like you're the only person who's touched it in a while :)
>
> As well as the configure output, we sometimes include very light patching (for example, Tcl/Tk currently has a patch to their makefile to work around a VC14 bug). I'm not sure whether that's considered good practice or not, but I prefer only needing to do it once vs. building it into the build system.

It likely makes sense to try to at least write down the status quo
regarding the externals repo as a new top level section in the
Developer's Guide. That way you'll at least have a shared
understanding of the starting point for any further improvements.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From zachary.ware+pydev at gmail.com  Thu Mar 26 03:40:08 2015
From: zachary.ware+pydev at gmail.com (Zachary Ware)
Date: Wed, 25 Mar 2015 21:40:08 -0500
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <CADiSq7c+6GzDaoKxxBMc0=0hUzwMdEZYNzp2Apz0C6DqGzNo2A@mail.gmail.com>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAKJDb-N1ewaSsWyEycVHWwDASx8OjXG9LAWw3k3ntCZtw6oJSQ@mail.gmail.com>
 <CAP1=2W7Cx5JpQV_pr61RQs1uBUSJF5f6KG0cD-QCwR2+9ijegg@mail.gmail.com>
 <CACac1F86h7qQ=ggKS5Z8FCR4w1L1k-smoqP0362TpqK23autJw@mail.gmail.com>
 <CACac1F_xb+0YMCxV2RB_DmPUGZia6HJDMYFLTBaOfDq7oqC9AA@mail.gmail.com>
 <CAK5idxTH--Hsw2tF4cU+yVXUfV1y7jDN=KKneut9sZu_sjQf1A@mail.gmail.com>
 <CACac1F80q8MZ3Q7fGxYXkvGhQZ4cxWXHgmP0H-7wqedw-DR5WQ@mail.gmail.com>
 <20150325100910.10e29a5c@fsol>
 <CACac1F88pcjQg7J4E8gn-Gy9aJMKrVWc1_zDhHcYpbgJ49JJfA@mail.gmail.com>
 <CAKJDb-Pnasx0HG76iAjdNE=Uskwe27wPgg-a--q+AkGFrsU8Ow@mail.gmail.com>
 <BY1PR03MB146674354E3CED584235BAE6F50B0@BY1PR03MB1466.namprd03.prod.outlook.com>
 <CADiSq7c+6GzDaoKxxBMc0=0hUzwMdEZYNzp2Apz0C6DqGzNo2A@mail.gmail.com>
Message-ID: <CAKJDb-PCT5tEic=LFbha_FUm+=8GCB0ktj3=4J+JpxLV+S=B2Q@mail.gmail.com>

On Mar 25, 2015 9:28 PM, "Nick Coghlan" <ncoghlan at gmail.com> wrote:
>
> On 26 March 2015 at 01:57, Steve Dower <Steve.Dower at microsoft.com> wrote:
> > Zachary Ware wrote:
> >> On Mar 25, 2015 4:22 AM, "Paul Moore" <p.f.moore at gmail.com> wrote:
> >>> On a related note, is there any information available on how the
> >>> "externals" repo is maintained? In particular, should things in there
> >>> be exact copies of upstream, or is it OK to include extra data (in
> >>> this case, the results of running "configure" for the Windows build)?
> >>> It works for me either way, it's just a matter of how the build
> >>> process would be structured and maintained.
> >>
> >> Its not extremely clear how it's "supposed to be" done; look at the
differences
> >> between how we handle OpenSSL and Tcl/Tk, for example. One way or the
other,
> >> though, we will store the configure output so that our build doesn't
depend on
> >> any more than it absolutely has to.
> >
> > It would be nice for at least one other person to know how to do it. I
can't see the SVN logs, but I feel like you're the only person who's
touched it in a while :)
> >
> > As well as the configure output, we sometimes include very light
patching (for example, Tcl/Tk currently has a patch to their makefile to
work around a VC14 bug). I'm not sure whether that's considered good
practice or not, but I prefer only needing to do it once vs. building it
into the build system.
>
> It likely makes sense to try to at least write down the status quo
> regarding the externals repo as a new top level section in the
> Developer's Guide. That way you'll at least have a shared
> understanding of the starting point for any further improvements.

I'll try to get something thrown together at least by the time I leave
PyCon.  Unfortunately I can't guarantee anything sooner than that.

--
Zach
On a phone
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150325/a7afe23f/attachment.html>

From ncoghlan at gmail.com  Thu Mar 26 04:18:53 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 26 Mar 2015 13:18:53 +1000
Subject: [Python-Dev] libffi embedded in CPython
In-Reply-To: <CAKJDb-PCT5tEic=LFbha_FUm+=8GCB0ktj3=4J+JpxLV+S=B2Q@mail.gmail.com>
References: <m70uth$s5g$1@ger.gmane.org>
 <549891df.c332e00a.54c7.34f8@mx.google.com>
 <CAKJDb-N1ewaSsWyEycVHWwDASx8OjXG9LAWw3k3ntCZtw6oJSQ@mail.gmail.com>
 <CAP1=2W7Cx5JpQV_pr61RQs1uBUSJF5f6KG0cD-QCwR2+9ijegg@mail.gmail.com>
 <CACac1F86h7qQ=ggKS5Z8FCR4w1L1k-smoqP0362TpqK23autJw@mail.gmail.com>
 <CACac1F_xb+0YMCxV2RB_DmPUGZia6HJDMYFLTBaOfDq7oqC9AA@mail.gmail.com>
 <CAK5idxTH--Hsw2tF4cU+yVXUfV1y7jDN=KKneut9sZu_sjQf1A@mail.gmail.com>
 <CACac1F80q8MZ3Q7fGxYXkvGhQZ4cxWXHgmP0H-7wqedw-DR5WQ@mail.gmail.com>
 <20150325100910.10e29a5c@fsol>
 <CACac1F88pcjQg7J4E8gn-Gy9aJMKrVWc1_zDhHcYpbgJ49JJfA@mail.gmail.com>
 <CAKJDb-Pnasx0HG76iAjdNE=Uskwe27wPgg-a--q+AkGFrsU8Ow@mail.gmail.com>
 <BY1PR03MB146674354E3CED584235BAE6F50B0@BY1PR03MB1466.namprd03.prod.outlook.com>
 <CADiSq7c+6GzDaoKxxBMc0=0hUzwMdEZYNzp2Apz0C6DqGzNo2A@mail.gmail.com>
 <CAKJDb-PCT5tEic=LFbha_FUm+=8GCB0ktj3=4J+JpxLV+S=B2Q@mail.gmail.com>
Message-ID: <CADiSq7d-TowA8SUYTJqbnTKBm4=fvFkxE1MTKcXT5qWgc+Mh2w@mail.gmail.com>

On 26 March 2015 at 12:40, Zachary Ware <zachary.ware+pydev at gmail.com> wrote:
> On Mar 25, 2015 9:28 PM, "Nick Coghlan" <ncoghlan at gmail.com> wrote:
>>
>> On 26 March 2015 at 01:57, Steve Dower <Steve.Dower at microsoft.com> wrote:
>> > Zachary Ware wrote:
>> >> On Mar 25, 2015 4:22 AM, "Paul Moore" <p.f.moore at gmail.com> wrote:
>> >>> On a related note, is there any information available on how the
>> >>> "externals" repo is maintained? In particular, should things in there
>> >>> be exact copies of upstream, or is it OK to include extra data (in
>> >>> this case, the results of running "configure" for the Windows build)?
>> >>> It works for me either way, it's just a matter of how the build
>> >>> process would be structured and maintained.
>> >>
>> >> Its not extremely clear how it's "supposed to be" done; look at the
>> >> differences
>> >> between how we handle OpenSSL and Tcl/Tk, for example. One way or the
>> >> other,
>> >> though, we will store the configure output so that our build doesn't
>> >> depend on
>> >> any more than it absolutely has to.
>> >
>> > It would be nice for at least one other person to know how to do it. I
>> > can't see the SVN logs, but I feel like you're the only person who's touched
>> > it in a while :)
>> >
>> > As well as the configure output, we sometimes include very light
>> > patching (for example, Tcl/Tk currently has a patch to their makefile to
>> > work around a VC14 bug). I'm not sure whether that's considered good
>> > practice or not, but I prefer only needing to do it once vs. building it
>> > into the build system.
>>
>> It likely makes sense to try to at least write down the status quo
>> regarding the externals repo as a new top level section in the
>> Developer's Guide. That way you'll at least have a shared
>> understanding of the starting point for any further improvements.
>
> I'll try to get something thrown together at least by the time I leave
> PyCon.  Unfortunately I can't guarantee anything sooner than that.

That sounds plenty soon enough to me - my own list of "I should really
do something about that some day" items stretches back years, as the
issue tracker occasionally reminds me :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From storchaka at gmail.com  Thu Mar 26 10:11:51 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Thu, 26 Mar 2015 11:11:51 +0200
Subject: [Python-Dev] peps: New PEP 490: Chain exceptions at C level
In-Reply-To: <20150326080820.24234.33840@psf.io>
References: <20150326080820.24234.33840@psf.io>
Message-ID: <mf0igo$pst$1@ger.gmane.org>

On 26.03.15 10:08, victor.stinner wrote:
> https://hg.python.org/peps/rev/7daf3bfd9586
> changeset:   5741:7daf3bfd9586
> user:        Victor Stinner <victor.stinner at gmail.com>
> date:        Thu Mar 26 09:08:08 2015 +0100
> summary:
>    New PEP 490: Chain exceptions at C level

> +Python 3.5 introduces a new private ``_PyErr_ChainExceptions()`` function which
> +is enough to chain manually exceptions.

It also was added in Python 3.4.3.

I meditar about adding _PyErr_ReplaceException() in 2.7 for simpler 
backporting patches from 3.x.

> +Functions like ``PyErr_SetString()`` don't chain automatically exceptions. To
> +make usage of ``_PyErr_ChainExceptions()`` easier, new functions are added:
> +
> +* PyErr_SetStringChain(exc_type, message)
> +* PyErr_FormatChaine(exc_type, format, ...)

Typo.

> +* PyErr_SetNoneChain(exc_type)
> +* PyErr_SetObjectChain(exc_type, exc_value)

I would first make these functions private, as _PyErr_ChainExceptions(). 
After proofing their usefulness in the stdlib, they can be made public.


From andrew.svetlov at gmail.com  Thu Mar 26 11:52:14 2015
From: andrew.svetlov at gmail.com (Andrew Svetlov)
Date: Thu, 26 Mar 2015 12:52:14 +0200
Subject: [Python-Dev] peps: New PEP 490: Chain exceptions at C level
In-Reply-To: <mf0igo$pst$1@ger.gmane.org>
References: <20150326080820.24234.33840@psf.io>
	<mf0igo$pst$1@ger.gmane.org>
Message-ID: <CAL3CFcW2rGR=J1aroq1H5GU5pk_w6HtsgNR9VWh+v7UBBzYwgQ@mail.gmail.com>

There is another issue: exception chain is not set up on exception
creation in python code, only on throwing.
Thus I have to assign `__context__` and `__cause__` attributes
manually if I want to call `future.set_exception(exc)` instead of
`raise exc`.

See aiohttp code for usage example:
https://github.com/KeepSafe/aiohttp/blob/931efbd518d0d99522d1cd36b43620657cd07304/aiohttp/client.py#L522

On Thu, Mar 26, 2015 at 11:11 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> On 26.03.15 10:08, victor.stinner wrote:
>>
>> https://hg.python.org/peps/rev/7daf3bfd9586
>> changeset:   5741:7daf3bfd9586
>> user:        Victor Stinner <victor.stinner at gmail.com>
>> date:        Thu Mar 26 09:08:08 2015 +0100
>> summary:
>>    New PEP 490: Chain exceptions at C level
>
>
>> +Python 3.5 introduces a new private ``_PyErr_ChainExceptions()`` function
>> which
>> +is enough to chain manually exceptions.
>
>
> It also was added in Python 3.4.3.
>
> I meditar about adding _PyErr_ReplaceException() in 2.7 for simpler
> backporting patches from 3.x.
>
>> +Functions like ``PyErr_SetString()`` don't chain automatically
>> exceptions. To
>> +make usage of ``_PyErr_ChainExceptions()`` easier, new functions are
>> added:
>> +
>> +* PyErr_SetStringChain(exc_type, message)
>> +* PyErr_FormatChaine(exc_type, format, ...)
>
>
> Typo.
>
>> +* PyErr_SetNoneChain(exc_type)
>> +* PyErr_SetObjectChain(exc_type, exc_value)
>
>
> I would first make these functions private, as _PyErr_ChainExceptions().
> After proofing their usefulness in the stdlib, they can be made public.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com



-- 
Thanks,
Andrew Svetlov

From victor.stinner at gmail.com  Thu Mar 26 12:36:07 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 26 Mar 2015 12:36:07 +0100
Subject: [Python-Dev] peps: New PEP 490: Chain exceptions at C level
In-Reply-To: <CAL3CFcW2rGR=J1aroq1H5GU5pk_w6HtsgNR9VWh+v7UBBzYwgQ@mail.gmail.com>
References: <20150326080820.24234.33840@psf.io> <mf0igo$pst$1@ger.gmane.org>
 <CAL3CFcW2rGR=J1aroq1H5GU5pk_w6HtsgNR9VWh+v7UBBzYwgQ@mail.gmail.com>
Message-ID: <CAMpsgwZsD=hgodRzDPL1voPj4-=_CSGF76yTwTB7bcACmkiJLg@mail.gmail.com>

2015-03-26 11:52 GMT+01:00 Andrew Svetlov <andrew.svetlov at gmail.com>:
> There is another issue: exception chain is not set up on exception
> creation in python code, only on throwing.

I'm not suprised of that.

> Thus I have to assign `__context__` and `__cause__` attributes
> manually if I want to call `future.set_exception(exc)` instead of
> `raise exc`.

Do you mean that we need an helper to make this task even simpler? Or
do you suggest to set them automatically in the constructor?

Victor

From victor.stinner at gmail.com  Thu Mar 26 13:30:18 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 26 Mar 2015 13:30:18 +0100
Subject: [Python-Dev] RFC: PEP 490 - Chain exceptions at C level
Message-ID: <CAMpsgwY23Vze5oGPu97cBMxHumu93Ss-Pi1Gtq95cuZvwgP5Uw@mail.gmail.com>

Hi,

Here is the first draft of my PEP to chain automatically exceptions at
C level in Python 3.5. Plain text version of the PEP below. HTML
version of the PEP:

   https://www.python.org/dev/peps/pep-0490/

It has already an implementation, the pyerr_chain.patch file attached to
http://bugs.python.org/issue23763

The patch is very short.

We may modify more functions to hide the previous exception when the
new exception contains as much or more information. Keeping the
previous exception waste memory, make reference cycles more likely and
is useless when displaying the full exception chain.

I wrote a huge patch modifying a lot of functions to explicitly *not*
chain exceptions: pyerr_match_clear-2.patch of the issue #23763. I
wrote the patch to show where exceptions will be chained with
pyerr_chain.patch applied. But we might apply it if we want to keep
exactly the same behaviour between Python 3.4 and 3.5 for C modules
the stdlib.

pyerr_match_clear-2.patch adds also checks on the type of the previous
exception to not replace an "unexpected" exception. For example, we
can replace an expected TypeError with a ValueError (without chaining
them), but we should not raise a ValueError if we got a
KeyboardInterrupt, a MemoryError or other unexpected exceptions. But
such changes are unrelated to the PEP and may be done in a separated
issue.

The starting point of my PEP is my work on CPython done with the
pyfailmalloc module. This module injects MemoryError. I fixed a ton of
issues in the code handling errors in CPython (see the issue #18408).
I noticed that Python didn't complain when exceptions are lost. I
added many assertions to detect these mistakes. Recently, I modified
Python 3.5 to raise an exception even in release mode (issue #23571)
if a C function returns NULL without an exception set, or if a C
function returns a result with an exception set.


Victor


PEP: 490
Title: Chain exceptions at C level
Version: $Revision$
Last-Modified: $Date$
Author: Victor Stinner <victor.stinner at gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 25-March-2015
Python-Version: 3.5


Abstract
========

Chain exceptions at C level, as already done at Python level.


Rationale
=========

Python 3 introduced a new killer feature: exceptions are chained by
default, PEP 3134.

Example::

    try:
        raise TypeError("err1")
    except TypeError:
        raise ValueError("err2")

Output::

    Traceback (most recent call last):
      File "test.py", line 2, in <module>
        raise TypeError("err1")
    TypeError: err1

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "test.py", line 4, in <module>
        raise ValueError("err2")
    ValueError: err2

Exceptions are chained by default in Python code, but not in
extensions written in C.

A new private ``_PyErr_ChainExceptions()`` function was introduced in
Python 3.4.3 and 3.5 to chain exceptions. Currently, it must be called
explicitly to chain exceptions and its usage is not trivial.

Example of ``_PyErr_ChainExceptions()`` usage from the ``zipimport``
module to chain the previous ``OSError`` to a new ``ZipImportError``
exception::

    PyObject *exc, *val, *tb;
    PyErr_Fetch(&exc, &val, &tb);
    PyErr_Format(ZipImportError, "can't open Zip file: %R", archive);
    _PyErr_ChainExceptions(exc, val, tb);

This PEP proposes to also chain exceptions automatically at C level to
stay consistent and give more information on failures to help
debugging. The previous example becomes simply::

    PyErr_Format(ZipImportError, "can't open Zip file: %R", archive);


Proposal
========

Modify PyErr_*() functions to chain exceptions
----------------------------------------------

Modify C functions raising exceptions of the Python C API to
automatically chain exceptions: modify ``PyErr_SetString()``,
``PyErr_Format()``, ``PyErr_SetNone()``, etc.


Modify functions to not chain exceptions
----------------------------------------

Keeping the previous exception is not always interesting when the new
exception contains information of the previous exception or even more
information, especially when the two exceptions have the same type.

Example of an useless exception chain with ``int(str)``::

    TypeError: a bytes-like object is required, not 'type'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: int() argument must be a string, a bytes-like object or
a number, not 'type'

The new ``TypeError`` exception contains more information than the
previous exception. The previous exception should be hidden.

The ``PyErr_Clear()`` function can be called to clear the current
exception before raising a new exception, to not chain the current
exception with a new exception.


Modify functions to chain exceptions
------------------------------------

Some functions save and then restore the current exception. If a new
exception is raised, the exception is currently displayed into
sys.stderr or ignored depending on the function.  Some of these
functions should be modified to chain exceptions instead.

Examples of function ignoring the new exception(s):

* ``ptrace_enter_call()``: ignore exception
* ``subprocess_fork_exec()``: ignore exception raised by enable_gc()
* ``t_bootstrap()`` of the ``_thread`` module: ignore exception raised
  by trying to display the bootstrap function to ``sys.stderr``
* ``PyDict_GetItem()``, ``_PyDict_GetItem_KnownHash()``: ignore
  exception raised by looking for a key in the dictionary
* ``_PyErr_TrySetFromCause()``: ignore exception
* ``PyFrame_LocalsToFast()``: ignore exception raised by
  ``dict_to_map()``
* ``_PyObject_Dump()``: ignore exception. ``_PyObject_Dump()`` is used
  to debug, to inspect a running process, it should not modify the
  Python state.
* ``Py_ReprLeave()``: ignore exception "because there is no way to
  report them"
* ``type_dealloc()``: ignore exception raised by
  ``remove_all_subclasses()``
* ``PyObject_ClearWeakRefs()``: ignore exception?
* ``call_exc_trace()``, ``call_trace_protected()``: ignore exception
* ``remove_importlib_frames()``: ignore exception
* ``do_mktuple()``, helper used by ``Py_BuildValue()`` for example:
  ignore exception?
* ``flush_io()``: ignore exception
* ``sys_write()``, ``sys_format()``: ignore exception
* ``_PyTraceback_Add()``: ignore exception
* ``PyTraceBack_Print()``: ignore exception

Examples of function displaying the new exception to ``sys.stderr``:

* ``atexit_callfuncs()``: display exceptions with
  ``PyErr_Display()`` and return the latest exception, the function
  calls multiple callbacks and only returns the latest exception
* ``sock_dealloc()``: log the ``ResourceWarning`` exception with
  ``PyErr_WriteUnraisable()``
* ``slot_tp_del()``: display exception with
  ``PyErr_WriteUnraisable()``
* ``_PyGen_Finalize()``: display ``gen_close()`` exception with
  ``PyErr_WriteUnraisable()``
* ``slot_tp_finalize()``: display exception raised by the
  ``__del__()`` method with ``PyErr_WriteUnraisable()``
* ``PyErr_GivenExceptionMatches()``: display exception raised by
  ``PyType_IsSubtype()`` with ``PyErr_WriteUnraisable()``


Backward compatibility
======================

A side effect of chaining exceptions is that exceptions store
traceback objects which store frame objects which store local
variables.  Local variables are kept alive by exceptions. A common
issue is a reference cycle between local variables and exceptions: an
exception is stored in a local variable and the frame indirectly
stored in the exception. The cycle only impacts applications storing
exceptions.

The reference cycle can now be fixed with the new
``traceback.TracebackException`` object introduced in Python 3.5. It
stores informations required to format a full textual traceback without
storing local variables.

The ``asyncio`` is impacted by the reference cycle issue. This module
is also maintained outside Python standard library to release a
version for Python 3.3.  ``traceback.TracebackException`` will maybe
be backported in a private ``asyncio`` module to fix reference cycle
issues.


Alternatives
============

No change
---------

A new private ``_PyErr_ChainExceptions()`` function is enough to chain
manually exceptions.

Exceptions will only be chained explicitly where it makes sense.


New helpers to chain exceptions
-------------------------------

Functions like ``PyErr_SetString()`` don't chain automatically
exceptions. To make the usage of ``_PyErr_ChainExceptions()`` easier,
new private functions are added:

* ``_PyErr_SetStringChain(exc_type, message)``
* ``_PyErr_FormatChain(exc_type, format, ...)``
* ``_PyErr_SetNoneChain(exc_type)``
* ``_PyErr_SetObjectChain(exc_type, exc_value)``

Helper functions to raise specific exceptions like
``_PyErr_SetKeyError(key)`` or ``PyErr_SetImportError(message, name,
path)`` don't chain exceptions.  The generic
``_PyErr_ChainExceptions(exc_type, exc_value, exc_tb)`` should be used
to chain exceptions with these helper functions.


Appendix
========

PEPs
----

* `PEP 3134 -- Exception Chaining and Embedded Tracebacks
  <https://www.python.org/dev/peps/pep-3134/>`_ (Python 3.0):
  new ``__context__`` and ``__cause__`` attributes for exceptions
* `PEP 415 - Implement context suppression with exception attributes
  <https://www.python.org/dev/peps/pep-0415/>`_ (Python 3.3):
  ``raise exc from None``
* `PEP 409 - Suppressing exception context
  <https://www.python.org/dev/peps/pep-0409/>`_
  (superseded by the PEP 415)


Python C API
------------

The header file ``Include/pyerror.h`` declares functions related to
exceptions.

Functions raising exceptions:

* ``PyErr_SetNone(exc_type)``
* ``PyErr_SetObject(exc_type, exc_value)``
* ``PyErr_SetString(exc_type, message)``
* ``PyErr_Format(exc, format, ...)``

Helpers to raise specific exceptions:

* ``PyErr_BadArgument()``
* ``PyErr_BadInternalCall()``
* ``PyErr_NoMemory()``
* ``PyErr_SetFromErrno(exc)``
* ``PyErr_SetFromWindowsErr(err)``
* ``PyErr_SetImportError(message, name, path)``
* ``_PyErr_SetKeyError(key)``
* ``_PyErr_TrySetFromCause(prefix_format, ...)``

Manage the current exception:

* ``PyErr_Clear()``: clear the current exception,
  like ``except: pass``
* ``PyErr_Fetch(exc_type, exc_value, exc_tb)``
* ``PyErr_Restore(exc_type, exc_value, exc_tb)``
* ``PyErr_GetExcInfo(exc_type, exc_value, exc_tb)``
* ``PyErr_SetExcInfo(exc_type, exc_value, exc_tb)``

Others function to handle exceptions:

* ``PyErr_ExceptionMatches(exc)``: check to implement
  ``except exc:  ...``
* ``PyErr_GivenExceptionMatches(exc1, exc2)``
* ``PyErr_NormalizeException(exc_type, exc_value, exc_tb)``
* ``_PyErr_ChainExceptions(exc_type, exc_value, exc_tb)``


Python Issues
-------------

Chain exceptions:

* `Issue #23763: Chain exceptions in C
  <http://bugs.python.org/issue23763>`_
* `Issue #23696: zipimport: chain ImportError to OSError
  <http://bugs.python.org/issue23696>`_
* `Issue #21715: Chaining exceptions at C level
  <http://bugs.python.org/issue21715>`_: added
  ``_PyErr_ChainExceptions()``
* `Issue #18488: sqlite: finalize() method of user function may be
  called with an exception set if a call to step() method failed
  <http://bugs.python.org/issue18488>`_
* `Issue #23781: Add private _PyErr_ReplaceException() in 2.7
  <http://bugs.python.org/issue23781>`_
* `Issue #23782: Leak in _PyTraceback_Add
  <http://bugs.python.org/issue23782>`_

Changes preventing to loose exceptions:

* `Issue #23571: Raise SystemError if a function returns a result with an
  exception set <http://bugs.python.org/issue23571>`_
* `Issue #18408: Fixes crashes found by pyfailmalloc
  <http://bugs.python.org/issue18408>`_


Copyright
=========

This document has been placed in the public domain.



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End:

From victor.stinner at gmail.com  Thu Mar 26 13:31:56 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 26 Mar 2015 13:31:56 +0100
Subject: [Python-Dev] peps: New PEP 490: Chain exceptions at C level
In-Reply-To: <mf0igo$pst$1@ger.gmane.org>
References: <20150326080820.24234.33840@psf.io> <mf0igo$pst$1@ger.gmane.org>
Message-ID: <CAMpsgwZ-i4e7SG9fp1jeBWoyhKsRRt40RZtaKmagoMNoysoV8A@mail.gmail.com>

Thanks Serhiy for your review, I modified the PEP. I just sent a first
draft (including your suggestions) to python-dev.

Victor

From andrew.svetlov at gmail.com  Thu Mar 26 13:45:54 2015
From: andrew.svetlov at gmail.com (Andrew Svetlov)
Date: Thu, 26 Mar 2015 14:45:54 +0200
Subject: [Python-Dev] peps: New PEP 490: Chain exceptions at C level
In-Reply-To: <CAMpsgwZsD=hgodRzDPL1voPj4-=_CSGF76yTwTB7bcACmkiJLg@mail.gmail.com>
References: <20150326080820.24234.33840@psf.io> <mf0igo$pst$1@ger.gmane.org>
 <CAL3CFcW2rGR=J1aroq1H5GU5pk_w6HtsgNR9VWh+v7UBBzYwgQ@mail.gmail.com>
 <CAMpsgwZsD=hgodRzDPL1voPj4-=_CSGF76yTwTB7bcACmkiJLg@mail.gmail.com>
Message-ID: <CAL3CFcXeZvpc6Ec3hhTxmaZ5dL4HmbAX3iA+uyAZ4zXSRqFwEg@mail.gmail.com>

I think setting context in exception constructor would be fine.

On Thu, Mar 26, 2015 at 1:36 PM, Victor Stinner
<victor.stinner at gmail.com> wrote:
> 2015-03-26 11:52 GMT+01:00 Andrew Svetlov <andrew.svetlov at gmail.com>:
>> There is another issue: exception chain is not set up on exception
>> creation in python code, only on throwing.
>
> I'm not suprised of that.
>
>> Thus I have to assign `__context__` and `__cause__` attributes
>> manually if I want to call `future.set_exception(exc)` instead of
>> `raise exc`.
>
> Do you mean that we need an helper to make this task even simpler? Or
> do you suggest to set them automatically in the constructor?
>
> Victor



-- 
Thanks,
Andrew Svetlov

From victor.stinner at gmail.com  Fri Mar 27 01:16:37 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Fri, 27 Mar 2015 01:16:37 +0100
Subject: [Python-Dev] Needed reviews
In-Reply-To: <mee4si$rap$1@ger.gmane.org>
References: <mee4si$rap$1@ger.gmane.org>
Message-ID: <CAMpsgwa6T3QzrJJRVrjsoNv7bjWeC7mzBA4hJdX9a1w+Ynd+3Q@mail.gmail.com>

2015-03-19 10:28 GMT+01:00 Serhiy Storchaka <storchaka at gmail.com>:
> https://bugs.python.org/issue23681
>     Have -b warn when directly comparing ints and bytes

closed

> https://bugs.python.org/issue23676
>     Add support of UnicodeTranslateError in standard error handlers

commented

> https://bugs.python.org/issue23671
>     string.Template doesn't work with the self keyword argument

closed

> https://bugs.python.org/issue23637
>     Outputting warnings fails when file patch is not ASCII and message is
> unicode on 2.7.

commented

> https://bugs.python.org/issue23622
>     Deprecate unrecognized backslash+letter escapes in re

closed

> https://bugs.python.org/issue23611
>     Pickle nested names (e.g. unbound methods) with protocols < 4

commented

> https://bugs.python.org/issue23583
>     IDLE: printing unicode subclasses broken (again)

closed

> https://bugs.python.org/issue23573
>     Avoid redundant memory allocations in str.find and like

closed and reopened, already commented

> https://bugs.python.org/issue23509
>     Speed up Counter operators

reviewed

> https://bugs.python.org/issue23502
>     Tkinter doesn't support large integers (out of 32-bit range)

closed
(note: the title was different, "pprint: added support for mapping proxy")

> https://bugs.python.org/issue23488
>     Random objects twice as big as necessary on 64-bit builds

reviewed

> https://bugs.python.org/issue23466
>     PEP 461: Inconsistency between str and bytes formatting of integers

already reviewed

> https://bugs.python.org/issue23419
>     Faster default __reduce__ for classes without __init__

closed (rejected)

> https://bugs.python.org/issue23290
>     Faster set copying

commented and reviewed

I stop here for tonight.

Victor

From storchaka at gmail.com  Fri Mar 27 09:56:40 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Fri, 27 Mar 2015 10:56:40 +0200
Subject: [Python-Dev] Needed reviews
In-Reply-To: <CAMpsgwa6T3QzrJJRVrjsoNv7bjWeC7mzBA4hJdX9a1w+Ynd+3Q@mail.gmail.com>
References: <mee4si$rap$1@ger.gmane.org>
 <CAMpsgwa6T3QzrJJRVrjsoNv7bjWeC7mzBA4hJdX9a1w+Ynd+3Q@mail.gmail.com>
Message-ID: <mf3609$b02$1@ger.gmane.org>

On 27.03.15 02:16, Victor Stinner wrote:
> 2015-03-19 10:28 GMT+01:00 Serhiy Storchaka <storchaka at gmail.com>:

>> https://bugs.python.org/issue23502
>>      Tkinter doesn't support large integers (out of 32-bit range)
>
> closed
> (note: the title was different, "pprint: added support for mapping proxy")

My fault. The correct issue is https://bugs.python.org/issue16840.

> I stop here for tonight.

Many thanks Victor!



From status at bugs.python.org  Fri Mar 27 18:08:22 2015
From: status at bugs.python.org (Python tracker)
Date: Fri, 27 Mar 2015 18:08:22 +0100 (CET)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20150327170822.4130A566AE@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2015-03-20 - 2015-03-27)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    4820 (+22)
  closed 30732 (+47)
  total  35552 (+69)

Open issues with patches: 2248 


Issues opened (55)
==================

#23571: Raise SystemError if a function returns a result with an excep
http://bugs.python.org/issue23571  reopened by berker.peksag

#23573: Avoid redundant allocations in str.find and like
http://bugs.python.org/issue23573  reopened by serhiy.storchaka

#23720: __del__() order is broken since 3.4.0
http://bugs.python.org/issue23720  reopened by Alexey Kazantsev

#23723: Provide a way to disable bytecode staleness checks
http://bugs.python.org/issue23723  opened by brett.cannon

#23725: update tempfile docs to say that TemporaryFile is secure
http://bugs.python.org/issue23725  opened by zbysz

#23726: Don't enable GC for classes that don't add new fields
http://bugs.python.org/issue23726  opened by eltoder

#23727: rfc2231 line continuations result in an additional newline
http://bugs.python.org/issue23727  opened by Brian Peterson

#23728: binascii.crc_hqx() can return negative integer
http://bugs.python.org/issue23728  opened by serhiy.storchaka

#23730: Document return value for ZipFile.extract()
http://bugs.python.org/issue23730  opened by mjpieters

#23731: Implement PEP 488
http://bugs.python.org/issue23731  opened by brett.cannon

#23732: Update porting HOWTO about new -b semantics
http://bugs.python.org/issue23732  opened by brett.cannon

#23733: Update porting HOWTO for bytes interpolation
http://bugs.python.org/issue23733  opened by brett.cannon

#23734: zipimport should not check pyc timestamps against zipped py fi
http://bugs.python.org/issue23734  opened by gregory.p.smith

#23735: Readline not adjusting width after resize with 6.3
http://bugs.python.org/issue23735  opened by Carlos Pita

#23738: Clarify documentation of positional-only default values
http://bugs.python.org/issue23738  opened by vadmium

#23739: locale.setlocale checks locale name for string incorrectly
http://bugs.python.org/issue23739  opened by Saulius ??emaitaitis

#23740: http.client request and send method have some datatype issues
http://bugs.python.org/issue23740  opened by r.david.murray

#23743: Python crashes upon exit if importing g++ compiled mod after i
http://bugs.python.org/issue23743  opened by matham

#23745: Exception when parsing an email using email.parser.BytesParser
http://bugs.python.org/issue23745  opened by Elmer

#23746: sysconfg.is_python_build() is buggy
http://bugs.python.org/issue23746  opened by pythonhacker

#23747: platform module exposes win32_ver function on posix systems
http://bugs.python.org/issue23747  opened by pythonhacker

#23749: asyncio missing wrap_socket
http://bugs.python.org/issue23749  opened by gc

#23750: Clarify difference between os.system/subprocess.call in sectio
http://bugs.python.org/issue23750  opened by Andreas Sommer

#23752: Cleanup io.FileIO
http://bugs.python.org/issue23752  opened by haypo

#23754: Add a new os.read_into() function to avoid memory copies
http://bugs.python.org/issue23754  opened by haypo

#23755: tempfile.NamedTemporaryFile should be able to toggle "delete"
http://bugs.python.org/issue23755  opened by Stephen Gallagher

#23756: Tighten definition of bytes-like objects
http://bugs.python.org/issue23756  opened by vadmium

#23758: Improve documenation about num_params in sqlite3 create_functi
http://bugs.python.org/issue23758  opened by ced

#23759: urllib.parse: make coap:// known
http://bugs.python.org/issue23759  opened by chrysn

#23760: Tkinter in Python 3.4 on Windows don't post internal clipboard
http://bugs.python.org/issue23760  opened by Victor Korolkevich

#23761: test_socket fails on Mac OSX 10.9.5
http://bugs.python.org/issue23761  opened by willingc

#23762: python.org page on new-style classes should be updated
http://bugs.python.org/issue23762  opened by joelthelion

#23763: Chain exceptions in C
http://bugs.python.org/issue23763  opened by haypo

#23764: Reference inspect.Signature.bind from functools.wraps document
http://bugs.python.org/issue23764  opened by productivememberofsociety666

#23765: Remove IsBadStringPtr calls in ctypes
http://bugs.python.org/issue23765  opened by steve.dower

#23767: Library and include paths not added when cross compiling on lo
http://bugs.python.org/issue23767  opened by kernevil

#23768: assert on getting the representation of a thread in atexit fun
http://bugs.python.org/issue23768  opened by xdegaye

#23769: valgrind reports leaks for test_zipimport
http://bugs.python.org/issue23769  opened by rkuska

#23770: Rework of exceptions are handled in the parser module (in vali
http://bugs.python.org/issue23770  opened by haypo

#23771: Timeouts on "x86 Ubuntu Shared 3.x" buildbot
http://bugs.python.org/issue23771  opened by haypo

#23772: pymonotonic() gone backward on "AMD64 Debian root 3.x" buildbo
http://bugs.python.org/issue23772  opened by haypo

#23773: importlib does not properly remove frames when invoking extern
http://bugs.python.org/issue23773  opened by shiz

#23774: Test failures when unable to write to install location
http://bugs.python.org/issue23774  opened by steve.dower

#23779: imaplib authenticate raises TypeError if authenticator tries t
http://bugs.python.org/issue23779  opened by craigh

#23780: Surprising behaviour when passing list to os.path.join.
http://bugs.python.org/issue23780  opened by The Compiler

#23781: Add private _PyErr_ReplaceException() in 2.7
http://bugs.python.org/issue23781  opened by serhiy.storchaka

#23782: Leak in _PyTraceback_Add
http://bugs.python.org/issue23782  opened by serhiy.storchaka

#23783: Leak in PyObject_ClearWeakRefs
http://bugs.python.org/issue23783  opened by serhiy.storchaka

#23785: Leak in TextIOWrapper.tell()
http://bugs.python.org/issue23785  opened by serhiy.storchaka

#23786: test_unaligned_buffers (test.test_hash.HashEqualityTestCase) .
http://bugs.python.org/issue23786  opened by petriborg

#23787: sum() function docstring lists arguments incorrectly
http://bugs.python.org/issue23787  opened by vsinitsyn

#23788: test_urllib2_localnet.test_bad_address fails: OSError not rais
http://bugs.python.org/issue23788  opened by raniere

#23789: decimal.Decimal: __format__ behaviour
http://bugs.python.org/issue23789  opened by MonsieurPaul

#23790: When xdrlib.Packer().pack_string() fails, the Packer is corrup
http://bugs.python.org/issue23790  opened by ostcar

#23791: Identify Moved Lines with difflib
http://bugs.python.org/issue23791  opened by bkiefer



Most recent 15 issues with no replies (15)
==========================================

#23791: Identify Moved Lines with difflib
http://bugs.python.org/issue23791

#23779: imaplib authenticate raises TypeError if authenticator tries t
http://bugs.python.org/issue23779

#23773: importlib does not properly remove frames when invoking extern
http://bugs.python.org/issue23773

#23770: Rework of exceptions are handled in the parser module (in vali
http://bugs.python.org/issue23770

#23768: assert on getting the representation of a thread in atexit fun
http://bugs.python.org/issue23768

#23767: Library and include paths not added when cross compiling on lo
http://bugs.python.org/issue23767

#23762: python.org page on new-style classes should be updated
http://bugs.python.org/issue23762

#23760: Tkinter in Python 3.4 on Windows don't post internal clipboard
http://bugs.python.org/issue23760

#23758: Improve documenation about num_params in sqlite3 create_functi
http://bugs.python.org/issue23758

#23745: Exception when parsing an email using email.parser.BytesParser
http://bugs.python.org/issue23745

#23734: zipimport should not check pyc timestamps against zipped py fi
http://bugs.python.org/issue23734

#23733: Update porting HOWTO for bytes interpolation
http://bugs.python.org/issue23733

#23732: Update porting HOWTO about new -b semantics
http://bugs.python.org/issue23732

#23730: Document return value for ZipFile.extract()
http://bugs.python.org/issue23730

#23728: binascii.crc_hqx() can return negative integer
http://bugs.python.org/issue23728



Most recent 15 issues waiting for review (15)
=============================================

#23790: When xdrlib.Packer().pack_string() fails, the Packer is corrup
http://bugs.python.org/issue23790

#23786: test_unaligned_buffers (test.test_hash.HashEqualityTestCase) .
http://bugs.python.org/issue23786

#23785: Leak in TextIOWrapper.tell()
http://bugs.python.org/issue23785

#23783: Leak in PyObject_ClearWeakRefs
http://bugs.python.org/issue23783

#23781: Add private _PyErr_ReplaceException() in 2.7
http://bugs.python.org/issue23781

#23780: Surprising behaviour when passing list to os.path.join.
http://bugs.python.org/issue23780

#23779: imaplib authenticate raises TypeError if authenticator tries t
http://bugs.python.org/issue23779

#23773: importlib does not properly remove frames when invoking extern
http://bugs.python.org/issue23773

#23770: Rework of exceptions are handled in the parser module (in vali
http://bugs.python.org/issue23770

#23767: Library and include paths not added when cross compiling on lo
http://bugs.python.org/issue23767

#23763: Chain exceptions in C
http://bugs.python.org/issue23763

#23758: Improve documenation about num_params in sqlite3 create_functi
http://bugs.python.org/issue23758

#23752: Cleanup io.FileIO
http://bugs.python.org/issue23752

#23738: Clarify documentation of positional-only default values
http://bugs.python.org/issue23738

#23726: Don't enable GC for classes that don't add new fields
http://bugs.python.org/issue23726



Top 10 most discussed issues (10)
=================================

#22516: Windows Installer won't - even when using "just for me"option
http://bugs.python.org/issue22516  32 msgs

#23571: Raise SystemError if a function returns a result with an excep
http://bugs.python.org/issue23571  19 msgs

#23764: Reference inspect.Signature.bind from functools.wraps document
http://bugs.python.org/issue23764  15 msgs

#16328: win_add2path.py sets wrong user path
http://bugs.python.org/issue16328  13 msgs

#23786: test_unaligned_buffers (test.test_hash.HashEqualityTestCase) .
http://bugs.python.org/issue23786  13 msgs

#23763: Chain exceptions in C
http://bugs.python.org/issue23763  12 msgs

#23546: Windows, 'Edit withIDLE', and multiple installed versions
http://bugs.python.org/issue23546  10 msgs

#2211: Cookie.Morsel interface needs update
http://bugs.python.org/issue2211   9 msgs

#23529: Limit decompressed data when reading from LZMAFile and BZ2File
http://bugs.python.org/issue23529   8 msgs

#23720: __del__() order is broken since 3.4.0
http://bugs.python.org/issue23720   8 msgs



Issues closed (48)
==================

#4727: copyreg doesn't support keyword only arguments in __new__
http://bugs.python.org/issue4727  closed by serhiy.storchaka

#11468: Improve unittest basic example in the doc
http://bugs.python.org/issue11468  closed by ezio.melotti

#17530: pprint could use line continuation for long bytes literals
http://bugs.python.org/issue17530  closed by serhiy.storchaka

#21526: Support new booleans in Tkinter
http://bugs.python.org/issue21526  closed by serhiy.storchaka

#21560: gzip.write changes trailer ISIZE field before type checking - 
http://bugs.python.org/issue21560  closed by serhiy.storchaka

#21717: Exclusive mode for ZipFile and TarFile
http://bugs.python.org/issue21717  closed by serhiy.storchaka

#21802: Reader of BufferedRWPair is not closed if writer's close() fai
http://bugs.python.org/issue21802  closed by serhiy.storchaka

#22163: max_wbits set incorrectly to -zlib.MAX_WBITS in tarfile, shoul
http://bugs.python.org/issue22163  closed by vadmium

#22289: support.transient_internet() doesn't catch timeout on FTP test
http://bugs.python.org/issue22289  closed by ned.deily

#22351: NNTP constructor exception leaves socket for garbage collector
http://bugs.python.org/issue22351  closed by serhiy.storchaka

#22364: Improve some re error messages using regex for hints
http://bugs.python.org/issue22364  closed by serhiy.storchaka

#22687: horrible performance of textwrap.wrap() with a long word
http://bugs.python.org/issue22687  closed by serhiy.storchaka

#22832: Tweak parameter names for fcntl module
http://bugs.python.org/issue22832  closed by serhiy.storchaka

#22933: Misleading sentence in doc for shutil.move
http://bugs.python.org/issue22933  closed by python-dev

#23075: Mock backport in 2.7 relies on implementation defined behavior
http://bugs.python.org/issue23075  closed by serhiy.storchaka

#23252: Add support of writing to unseekable file in zipfile
http://bugs.python.org/issue23252  closed by serhiy.storchaka

#23314: Disabling CRT asserts in debug build
http://bugs.python.org/issue23314  closed by steve.dower

#23465: Implement PEP 486 - Make the Python Launcher aware of virtual 
http://bugs.python.org/issue23465  closed by steve.dower

#23502: pprint: added support for mapping proxy
http://bugs.python.org/issue23502  closed by serhiy.storchaka

#23512: The list of built-in functions is not alphabetical on https://
http://bugs.python.org/issue23512  closed by ezio.melotti

#23539: Content-length not set for HTTP methods expecting body when bo
http://bugs.python.org/issue23539  closed by r.david.murray

#23583: IDLE: printing unicode subclasses broken (again)
http://bugs.python.org/issue23583  closed by serhiy.storchaka

#23619: Python 3.5.0a2 installer fails on Windows
http://bugs.python.org/issue23619  closed by steve.dower

#23622: Deprecate unrecognized backslash+letter escapes in re
http://bugs.python.org/issue23622  closed by serhiy.storchaka

#23654: infinite loop in faulthandler._stack_overflow
http://bugs.python.org/issue23654  closed by haypo

#23657: Don't do isinstance checks in zipapp
http://bugs.python.org/issue23657  closed by paul.moore

#23665: Provide IDLE menu option to set command-line arguments
http://bugs.python.org/issue23665  closed by terry.reedy

#23671: string.Template doesn't work with the self keyword argument
http://bugs.python.org/issue23671  closed by serhiy.storchaka

#23688: unnecessary copying of memoryview in gzip.GzipFile.write?
http://bugs.python.org/issue23688  closed by serhiy.storchaka

#23704: Make deques a full MutableSequence by adding index(), insert()
http://bugs.python.org/issue23704  closed by rhettinger

#23705: Speed-up deque.__contains__
http://bugs.python.org/issue23705  closed by rhettinger

#23724: Stack Trace should show argument value passed into a function,
http://bugs.python.org/issue23724  closed by r.david.murray

#23729: Import ElementTree documentation for namespaces and XPath
http://bugs.python.org/issue23729  closed by rhettinger

#23736: "make test" on clean py3 install on CentOS 6.2 - 2 tests fail
http://bugs.python.org/issue23736  closed by r.david.murray

#23737: spam
http://bugs.python.org/issue23737  closed by ezio.melotti

#23741: Small pprint refactoring
http://bugs.python.org/issue23741  closed by serhiy.storchaka

#23742: expandvars removes single quotes ( ' )
http://bugs.python.org/issue23742  closed by serhiy.storchaka

#23744: Speed-up deque.__bool__
http://bugs.python.org/issue23744  closed by rhettinger

#23748: platform._uname_cache is writeable
http://bugs.python.org/issue23748  closed by haypo

#23751: Allow reverse lookup (value-to-member) for Enum sublcasses dec
http://bugs.python.org/issue23751  closed by mmuddasani

#23753: Drop HAVE_FSTAT: require fstat() to compile/use Python
http://bugs.python.org/issue23753  closed by haypo

#23757: tuple function gives wrong answer when called on list subclass
http://bugs.python.org/issue23757  closed by r.david.murray

#23766: json.dumps: solidus ("/") should be escaped
http://bugs.python.org/issue23766  closed by r.david.murray

#23775: Fix pprint of OrderedDict
http://bugs.python.org/issue23775  closed by serhiy.storchaka

#23776: Don't use assert for checking arguments in pprint
http://bugs.python.org/issue23776  closed by serhiy.storchaka

#23777: argparse subcommand action can't be specified
http://bugs.python.org/issue23777  closed by berker.peksag

#23778: Add a visual distinction to multiline versionchanged blocks
http://bugs.python.org/issue23778  closed by r.david.murray

#23784: Reloading tokenize module leads to error
http://bugs.python.org/issue23784  closed by serhiy.storchaka

From victor.stinner at gmail.com  Sat Mar 28 10:39:30 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sat, 28 Mar 2015 10:39:30 +0100
Subject: [Python-Dev] Sporadic failures of test_subprocess and
	test_multiprocessing_spawn
Message-ID: <CAMpsgwbNZJyD_ekNoyxMtSg7oqxzKU-=LGbPkzQW8hsdFwQh7g@mail.gmail.com>

Hi,

Can you please take a look at the following issue and try to reproduce it?
http://bugs.python.org/issue23771

The following tests sometimes hang on "x86 Ubuntu Shared 3.x" and
"AMD64 Debian root 3.x" buildbots:

- test_notify_all() of test_multiprocessing_spawn
- test_double_close_on_error() of test_subprocess
- other sporadic failures of test_subprocess

I'm quite sure that they are regressions, maybe related to the
implementation of the PEP 475. In the middle of all PEP 475 changes, I
changed some functions to release the GIL on I/O, it wasn't the case
before. I may be related.

Are you able to reproduce these issues? I'm unable to reproduce them
on Fedora 21. Maybe they are more likely on Debian-like operating
systems?

Victor

From victor.stinner at gmail.com  Sat Mar 28 10:48:32 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sat, 28 Mar 2015 10:48:32 +0100
Subject: [Python-Dev] MemoryError and other bugs on AMD64 OpenIndiana 3.x
Message-ID: <CAMpsgwY1DN-SzRv7NkakLoaLkUrP+ewsXKUWUk9T7iezZ7F-ng@mail.gmail.com>

Hi,

The buildbot AMD64 OpenIndiana 3.x is always red since at least 6
months. Almost always, tests fail because the buildbot has not enough
memory. Well, it's fun to see how Python handles MemoryError (it's
pretty good, there is no hard crash! my work on failmalloc was maybe
useful ;-)), but it doesn't help to detect regressions.

Would it be possible to enhance this buildbot?

I copy this email to the owner of buildbot. But I also sent it to
python-dev because MemoryError is not the only error, please take care
of our buildbots!

Thanks,
Victor

From victor.stinner at gmail.com  Sat Mar 28 10:51:09 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sat, 28 Mar 2015 10:51:09 +0100
Subject: [Python-Dev] Buildbot x86 XP-4 3.x doesn't compile anymore: drop it?
Message-ID: <CAMpsgwYb8s5aSZb-wjh9po5kr7=p3w5NFbCvrq3h_R8Hd_nn=A@mail.gmail.com>

Hi,

The buildbot x86 XP-4 3.x doesn't compile anymore since 3 months or
more (maybe when Steve upgraded the Visual Studio project to VS 2015?
I don't know).

Would it be possible to fix this buildbot, or to turn it off?

By the way, do we seriously want to support Windows XP? I mean, *who*
will maintain it (no me sorry!). I saw recent changes to explicitly
*drop* support for Windows older than Visa (stop using GetTickCount,
always call GetTickCount64, for time.monotonic).

Victor

From victor.stinner at gmail.com  Sat Mar 28 10:53:50 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sat, 28 Mar 2015 10:53:50 +0100
Subject: [Python-Dev] OpenBSD buildbot has many failures
Message-ID: <CAMpsgwan3rctQyEcQxXbYRFMDYVhbCkqR+Zcp9DC=DbsRxk8zQ@mail.gmail.com>

Hi,

The OpenBSD buildbot always fail with the same failures since many
months (ex: test_socket). Is someone interested to fix them? If no,
would it be possible to turn it off to get a better view of
regressions? Currently, they are too many red buildbots to quickly see
regressions introduced by recent commits.

http://buildbot.python.org/all/builders/x86%20OpenBSD%205.5%203.x

Victor

From victor.stinner at gmail.com  Sat Mar 28 10:57:40 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sat, 28 Mar 2015 10:57:40 +0100
Subject: [Python-Dev] Buildbot PPC64 AIX 3.x failures
Message-ID: <CAMpsgwbYEo7kYzsvAcaUhiyGZfaJyHOpdzuw3xbc5TrOcL=DUQ@mail.gmail.com>

Hi,

There are many failures on the AIX buildbot. Can someone try to fix
them? Or would it be possible to turn off the buildbot to quickly see
regressions?

http://buildbot.python.org/all/builders/PPC64%20AIX%203.x/builds/3426/steps/test/logs/stdio

The buildbot has not enough memory, and some tests are failing since
more than 3 months.

Victor

From mail at timgolden.me.uk  Sat Mar 28 11:19:16 2015
From: mail at timgolden.me.uk (Tim Golden)
Date: Sat, 28 Mar 2015 10:19:16 +0000
Subject: [Python-Dev] Buildbot x86 XP-4 3.x doesn't compile anymore:
 drop it?
In-Reply-To: <CAMpsgwYb8s5aSZb-wjh9po5kr7=p3w5NFbCvrq3h_R8Hd_nn=A@mail.gmail.com>
References: <CAMpsgwYb8s5aSZb-wjh9po5kr7=p3w5NFbCvrq3h_R8Hd_nn=A@mail.gmail.com>
Message-ID: <55168024.7030705@timgolden.me.uk>

On 28/03/2015 09:51, Victor Stinner wrote:
> Hi,
>
> The buildbot x86 XP-4 3.x doesn't compile anymore since 3 months or
> more (maybe when Steve upgraded the Visual Studio project to VS 2015?
> I don't know).
>
> Would it be possible to fix this buildbot, or to turn it off?
>
> By the way, do we seriously want to support Windows XP? I mean, *who*
> will maintain it (no me sorry!). I saw recent changes to explicitly
> *drop* support for Windows older than Visa (stop using GetTickCount,
> always call GetTickCount64, for time.monotonic).

Per PEP 11, we track Microsoft lifecycle support, so we no longer 
support WinXP on 3.5 and newer. We do support XP on 3.4 and older, 
including 2.7. At least, that's my understanding. Given that, the only 
XP buildbot which makes sense is one running VS 2008 (2.7, 3.1, 3.2) or 
VS 2010 (3.3, 3.4).

I think -- and I'm responding here to the wider set of emails Victor has 
just posted re failing buildbots -- that the difficulty with red 
buildbots is finding someone at the intersection of understanding the 
buildbot platform and understanding the code which has just broken it. 
(Or at least, having the wherewithal to come to understand both).

And that someone might well not be the person who's hosting the 
buildbot: at its minimum they run up a VM and install the OS and the 
required tools. It's can be a way of offering help to the Python 
community without even being involved in core development at all.

I concur with what Victor's touched on: that buildbots are not useful if 
they're always red. But finding the right person to turn them green is 
not necessarily easy.

For myself I'm quite willing for someone to ping me with: "there's a 
Windows buildbot gone red; can you have a look?" But I have precious 
little time, and if the failure requires me to garner an understanding 
of a deep-level code change, a version-specific issue with the VS 
compiler, and some odd interaction with VM on which the buildbot is 
running, I'm unlikely to have enough time or energy to invest in it.

Willing-but-not-always-able-ly-yours,

TJG

From rosuav at gmail.com  Sat Mar 28 11:10:17 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 28 Mar 2015 21:10:17 +1100
Subject: [Python-Dev] Sporadic failures of test_subprocess and
	test_multiprocessing_spawn
In-Reply-To: <CAMpsgwbNZJyD_ekNoyxMtSg7oqxzKU-=LGbPkzQW8hsdFwQh7g@mail.gmail.com>
References: <CAMpsgwbNZJyD_ekNoyxMtSg7oqxzKU-=LGbPkzQW8hsdFwQh7g@mail.gmail.com>
Message-ID: <CAPTjJmrw0tKpJrksfv3nxayZFB=ZJn4Z=kgayHrCr12ObFLH=w@mail.gmail.com>

On Sat, Mar 28, 2015 at 8:39 PM, Victor Stinner
<victor.stinner at gmail.com> wrote:
> Are you able to reproduce these issues? I'm unable to reproduce them
> on Fedora 21. Maybe they are more likely on Debian-like operating
> systems?

I just tried it on my Debian Wheezy AMD64, not running as root. (The
same computer the buildbot runs on, but the host OS rather than the
virtual machine.) Unable to reproduce. Ran a full 'make test' and
still no problems.

Am now trying it on the buildbot itself, to see if I can recreate it.

ChrisA

From victor.stinner at gmail.com  Sat Mar 28 11:36:57 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sat, 28 Mar 2015 11:36:57 +0100
Subject: [Python-Dev] Buildbot x86 XP-4 3.x doesn't compile anymore:
	drop it?
In-Reply-To: <55168024.7030705@timgolden.me.uk>
References: <CAMpsgwYb8s5aSZb-wjh9po5kr7=p3w5NFbCvrq3h_R8Hd_nn=A@mail.gmail.com>
 <55168024.7030705@timgolden.me.uk>
Message-ID: <CAMpsgwa=9Wa7gh3+q8L2FPSrW_n5htQ2jzKLm3JvTDLWOikCDw@mail.gmail.com>

Le samedi 28 mars 2015, Tim Golden <mail at timgolden.me.uk> a ?crit :
>
> For myself I'm quite willing for someone to ping me with: "there's a
> Windows buildbot gone red; can you have a look?" But I have precious little
> time, and if the failure requires me to garner an understanding of a
> deep-level code change, a version-specific issue with the VS compiler, and
> some odd interaction with VM on which the buildbot is running, I'm unlikely
> to have enough time or energy to invest in it.
>

You don't need to monitor buildobts everydays. Last days I checked
buildbots because I made chnages which behave differently on each platform.

Usually, I try to check buildbots each month and I only try to fix obvious
bugs and a few of the most annoying issues.

The problem is more on platforms which less interesged developers like AIX,
OpenBSD, old FreeBSD, etc.

Victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150328/07ea731a/attachment.html>

From rosuav at gmail.com  Sat Mar 28 12:26:08 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 28 Mar 2015 22:26:08 +1100
Subject: [Python-Dev] Sporadic failures of test_subprocess and
	test_multiprocessing_spawn
In-Reply-To: <CAPTjJmrw0tKpJrksfv3nxayZFB=ZJn4Z=kgayHrCr12ObFLH=w@mail.gmail.com>
References: <CAMpsgwbNZJyD_ekNoyxMtSg7oqxzKU-=LGbPkzQW8hsdFwQh7g@mail.gmail.com>
 <CAPTjJmrw0tKpJrksfv3nxayZFB=ZJn4Z=kgayHrCr12ObFLH=w@mail.gmail.com>
Message-ID: <CAPTjJmqLGfDs8St5FRMm7eMV+w7=WAyJyVdjDheiKMeUCOHqvw@mail.gmail.com>

On Sat, Mar 28, 2015 at 9:10 PM, Chris Angelico <rosuav at gmail.com> wrote:
> On Sat, Mar 28, 2015 at 8:39 PM, Victor Stinner
> <victor.stinner at gmail.com> wrote:
>> Are you able to reproduce these issues? I'm unable to reproduce them
>> on Fedora 21. Maybe they are more likely on Debian-like operating
>> systems?
>
> I just tried it on my Debian Wheezy AMD64, not running as root. (The
> same computer the buildbot runs on, but the host OS rather than the
> virtual machine.) Unable to reproduce. Ran a full 'make test' and
> still no problems.
>
> Am now trying it on the buildbot itself, to see if I can recreate it.

It seems to be stalling out. I'm not sure exactly what's happening
here. Running just that one file doesn't do it, but running the full
test suite results in a stall.

ChrisA

From victor.stinner at gmail.com  Sat Mar 28 13:50:07 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sat, 28 Mar 2015 13:50:07 +0100
Subject: [Python-Dev] Sporadic failures of test_subprocess and
	test_multiprocessing_spawn
In-Reply-To: <CAPTjJmqLGfDs8St5FRMm7eMV+w7=WAyJyVdjDheiKMeUCOHqvw@mail.gmail.com>
References: <CAMpsgwbNZJyD_ekNoyxMtSg7oqxzKU-=LGbPkzQW8hsdFwQh7g@mail.gmail.com>
 <CAPTjJmrw0tKpJrksfv3nxayZFB=ZJn4Z=kgayHrCr12ObFLH=w@mail.gmail.com>
 <CAPTjJmqLGfDs8St5FRMm7eMV+w7=WAyJyVdjDheiKMeUCOHqvw@mail.gmail.com>
Message-ID: <CAMpsgwZJSCUB2b641QSSjBRMXiP+OCPd6u10mcyXaOdw9Aidog@mail.gmail.com>

Good, you are able to reproduce it. The next step is to identify the
sequence of test to reproduce it. How do you run the test suite? Are you
using -j1?

Victor

Le samedi 28 mars 2015, Chris Angelico <rosuav at gmail.com> a ?crit :

> On Sat, Mar 28, 2015 at 9:10 PM, Chris Angelico <rosuav at gmail.com
> <javascript:;>> wrote:
> > On Sat, Mar 28, 2015 at 8:39 PM, Victor Stinner
> > <victor.stinner at gmail.com <javascript:;>> wrote:
> >> Are you able to reproduce these issues? I'm unable to reproduce them
> >> on Fedora 21. Maybe they are more likely on Debian-like operating
> >> systems?
> >
> > I just tried it on my Debian Wheezy AMD64, not running as root. (The
> > same computer the buildbot runs on, but the host OS rather than the
> > virtual machine.) Unable to reproduce. Ran a full 'make test' and
> > still no problems.
> >
> > Am now trying it on the buildbot itself, to see if I can recreate it.
>
> It seems to be stalling out. I'm not sure exactly what's happening
> here. Running just that one file doesn't do it, but running the full
> test suite results in a stall.
>
> ChrisA
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org <javascript:;>
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150328/bb8ddfde/attachment.html>

From rosuav at gmail.com  Sat Mar 28 14:21:34 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 29 Mar 2015 00:21:34 +1100
Subject: [Python-Dev] Sporadic failures of test_subprocess and
	test_multiprocessing_spawn
In-Reply-To: <CAMpsgwZJSCUB2b641QSSjBRMXiP+OCPd6u10mcyXaOdw9Aidog@mail.gmail.com>
References: <CAMpsgwbNZJyD_ekNoyxMtSg7oqxzKU-=LGbPkzQW8hsdFwQh7g@mail.gmail.com>
 <CAPTjJmrw0tKpJrksfv3nxayZFB=ZJn4Z=kgayHrCr12ObFLH=w@mail.gmail.com>
 <CAPTjJmqLGfDs8St5FRMm7eMV+w7=WAyJyVdjDheiKMeUCOHqvw@mail.gmail.com>
 <CAMpsgwZJSCUB2b641QSSjBRMXiP+OCPd6u10mcyXaOdw9Aidog@mail.gmail.com>
Message-ID: <CAPTjJmrCUA-HRW3rDeCADGR6X0XHQFP4Eeim2df4p4nq1bNkCg@mail.gmail.com>

On Sat, Mar 28, 2015 at 11:50 PM, Victor Stinner
<victor.stinner at gmail.com> wrote:
> Good, you are able to reproduce it. The next step is to identify the
> sequence of test to reproduce it. How do you run the test suite? Are you
> using -j1?

I just ran 'make test'. Early in the output are these lines:

./python  ./Tools/scripts/run_tests.py
/root/build/python -W default -bb -E -R -W error::BytesWarning -m test
-r -w -j 0 -u all,-largefile,-audio,-gui

(The build was done in /root/build, fwiw)

So it seems to be -j0.

ChrisA

From dje.gcc at gmail.com  Sat Mar 28 16:28:54 2015
From: dje.gcc at gmail.com (David Edelsohn)
Date: Sat, 28 Mar 2015 11:28:54 -0400
Subject: [Python-Dev] Buildbot PPC64 AIX 3.x failures
In-Reply-To: <CAMpsgwbYEo7kYzsvAcaUhiyGZfaJyHOpdzuw3xbc5TrOcL=DUQ@mail.gmail.com>
References: <CAMpsgwbYEo7kYzsvAcaUhiyGZfaJyHOpdzuw3xbc5TrOcL=DUQ@mail.gmail.com>
Message-ID: <CAGWvnynzX0zAcqZXN8o7uW=irqRQi6sKM-JNWBqdvLfZKduTzA@mail.gmail.com>

On Sat, Mar 28, 2015 at 5:57 AM, Victor Stinner
<victor.stinner at gmail.com> wrote:
> Hi,
>
> There are many failures on the AIX buildbot. Can someone try to fix
> them? Or would it be possible to turn off the buildbot to quickly see
> regressions?
>
> http://buildbot.python.org/all/builders/PPC64%20AIX%203.x/builds/3426/steps/test/logs/stdio
>
> The buildbot has not enough memory, and some tests are failing since
> more than 3 months.

I have cleaned out the build directories and restarted the buildslave.

There is more than enough memory and most memory process limits are
unlimited.  One of the tests presumably crashed in a bad way that
prevented normal cleanup by the buildslave.

- David

From storchaka at gmail.com  Sat Mar 28 17:40:37 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 28 Mar 2015 18:40:37 +0200
Subject: [Python-Dev] Sporadic failures of test_subprocess and
	test_multiprocessing_spawn
In-Reply-To: <CAMpsgwbNZJyD_ekNoyxMtSg7oqxzKU-=LGbPkzQW8hsdFwQh7g@mail.gmail.com>
References: <CAMpsgwbNZJyD_ekNoyxMtSg7oqxzKU-=LGbPkzQW8hsdFwQh7g@mail.gmail.com>
Message-ID: <mf6li5$6tq$1@ger.gmane.org>

On 28.03.15 11:39, Victor Stinner wrote:
> Can you please take a look at the following issue and try to reproduce it?
> http://bugs.python.org/issue23771
>
> The following tests sometimes hang on "x86 Ubuntu Shared 3.x" and
> "AMD64 Debian root 3.x" buildbots:
>
> - test_notify_all() of test_multiprocessing_spawn
> - test_double_close_on_error() of test_subprocess
> - other sporadic failures of test_subprocess
>
> I'm quite sure that they are regressions, maybe related to the
> implementation of the PEP 475. In the middle of all PEP 475 changes, I
> changed some functions to release the GIL on I/O, it wasn't the case
> before. I may be related.
>
> Are you able to reproduce these issues? I'm unable to reproduce them
> on Fedora 21. Maybe they are more likely on Debian-like operating
> systems?

Just run tests with low memory limit.

(ulimit -v 60000; ./python -m test.regrtest -uall -v 
test_multiprocessing_spawn;)

test_io also hangs.



From db3l.net at gmail.com  Sat Mar 28 21:14:29 2015
From: db3l.net at gmail.com (David Bolen)
Date: Sat, 28 Mar 2015 16:14:29 -0400
Subject: [Python-Dev] Buildbot x86 XP-4 3.x doesn't compile anymore:
	drop it?
References: <CAMpsgwYb8s5aSZb-wjh9po5kr7=p3w5NFbCvrq3h_R8Hd_nn=A@mail.gmail.com>
Message-ID: <m2twx4anh6.fsf@valheru.db3l.homeip.net>

Victor Stinner <victor.stinner at gmail.com> writes:

> Would it be possible to fix this buildbot, or to turn it off?
(...)
> By the way, do we seriously want to support Windows XP? I mean, *who*
> will maintain it (no me sorry!).  I saw recent changes to explicitly
> *drop* support for Windows older than Visa (stop using GetTickCount,
> always call GetTickCount64, for time.monotonic).

Disabling the 3.x branch (or 3.5+ going forward) on the XP buildbot
seems plausible to me, assuming the issue is unlikely to be fixed (or
someone found with time to look into it).

If, as Tim suggests in his response, 3.5+ need not be officially
supported on XP anyway, it may not be worth much effort to address.
At some point (such as switching to VS 2015), we aren't going to be
able to compile newer releases under XP anyway.

I'm assuming you aren't suggesting turning off the XP buildbot
entirely, correct?  As long as any fixes are still getting applied to
2.x or earlier 3.x releases, I do think having a buildbot covering
those branches for XP would be useful, if only to catch possible
issues in back-ported fixes.  There's still an amazing number of XP
systems in service.  While I doubt very many of those running XP also
need the latest 3.x release of Python, I expect there are quite a few
active 2.x users.

-- David


From victor.stinner at gmail.com  Sun Mar 29 00:13:55 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sun, 29 Mar 2015 00:13:55 +0100
Subject: [Python-Dev] Buildbot x86 XP-4 3.x doesn't compile anymore:
	drop it?
In-Reply-To: <m2twx4anh6.fsf@valheru.db3l.homeip.net>
References: <CAMpsgwYb8s5aSZb-wjh9po5kr7=p3w5NFbCvrq3h_R8Hd_nn=A@mail.gmail.com>
 <m2twx4anh6.fsf@valheru.db3l.homeip.net>
Message-ID: <CAMpsgwaTmMRdyYT5p2OaxWGU8m49o2DBgdbMCDNZs+JpcdV9Eg@mail.gmail.com>

Le 28 mars 2015 21:15, "David Bolen" <db3l.net at gmail.com> a ?crit :

> I'm assuming you aren't suggesting turning off the XP buildbot
> entirely, correct?

I'm talking about the 3.x slave, so only python 3.5.

Would it be possible to specify in What's New In Python 3.5 which Windows
versions are no more supported?

Victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150329/4148f55b/attachment.html>

From Steve.Dower at microsoft.com  Sun Mar 29 05:16:19 2015
From: Steve.Dower at microsoft.com (Steve Dower)
Date: Sun, 29 Mar 2015 03:16:19 +0000
Subject: [Python-Dev] Buildbot x86 XP-4 3.x doesn't compile
 anymore:	drop it?
In-Reply-To: <CAMpsgwaTmMRdyYT5p2OaxWGU8m49o2DBgdbMCDNZs+JpcdV9Eg@mail.gmail.com>
References: <CAMpsgwYb8s5aSZb-wjh9po5kr7=p3w5NFbCvrq3h_R8Hd_nn=A@mail.gmail.com>
 <m2twx4anh6.fsf@valheru.db3l.homeip.net>,
 <CAMpsgwaTmMRdyYT5p2OaxWGU8m49o2DBgdbMCDNZs+JpcdV9Eg@mail.gmail.com>
Message-ID: <BY1PR03MB1466D62DCB5A552F55DA85FCF5F60@BY1PR03MB1466.namprd03.prod.outlook.com>

We discussed that a while back and decided the PEP was sufficient, but I'm not entirely opposed to a reminder.

Probably best to repurpose this buildbot for 2.7. David has a Windows 8.1 one that should be marked as stable for 3.x (though it's not quite stable yet... AFAIK it's the only machine with VS 2015 other than mine).

Cheers,
Steve

Top-posted from my Windows Phone
________________________________
From: Victor Stinner<mailto:victor.stinner at gmail.com>
Sent: ?3/?28/?2015 16:14
To: David Bolen<mailto:db3l.net at gmail.com>
Cc: python-dev at python.org<mailto:python-dev at python.org>
Subject: Re: [Python-Dev] Buildbot x86 XP-4 3.x doesn't compile anymore: drop it?


Le 28 mars 2015 21:15, "David Bolen" <db3l.net at gmail.com<mailto:db3l.net at gmail.com>> a ?crit :

> I'm assuming you aren't suggesting turning off the XP buildbot
> entirely, correct?

I'm talking about the 3.x slave, so only python 3.5.

Would it be possible to specify in What's New In Python 3.5 which Windows versions are no more supported?

Victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150329/642d9829/attachment.html>

From pmiscml at gmail.com  Sun Mar 29 20:16:08 2015
From: pmiscml at gmail.com (Paul Sokolovsky)
Date: Sun, 29 Mar 2015 21:16:08 +0300
Subject: [Python-Dev] Builtin functions are magically static methods?
Message-ID: <20150329211608.13d3f2da@x230>

Hello,

I looked into porting Python3 codecs module to MicroPython and saw
rather strange behavior, which is best illustrated with following
testcase:

==========
def foo(a):
    print("func:", a)

import _codecs
fun = _codecs.utf_8_encode
#fun = hash
#fun = str.upper
#fun = foo


class Bar:
    meth = fun

print(fun)
print(fun("foo"))
b = Bar()
print(b.meth("bar"))
==========

Uncommenting either _codecs.utf_8_encode or hash (both builtin
functions) produces 2 similar output lines, which in particular means
that its possible to call a native function as (normal) object method,
which then behaves as if it was a staticmethod - self is not passed to
a native function.

Using native object method in this manner produces error of self type
mismatch (TypeError: descriptor 'upper' for 'str' objects doesn't apply
to 'Bar' object).

And using standard Python function expectedly produces error about
argument number mismatch, because used as a method, function gets extra
self argument.

So the questions are:

1. How so, the native functions exhibit such magic behavior? Is it
documented somewhere - I never read or heard about that (cannot say I
read each and every word in Python reference docs, but read enough. As
an example, https://docs.python.org/3/library/stdtypes.html#functions
is rather short and mentions difference in implementation, not in
meta-behavior).

2. The main question: how to easily and cleanly achieve the same
behavior for standard Python functions? I'd think it's staticmethod(),
but:

>>> staticmethod(lambda:1)()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'staticmethod' object is not callable

Surprise.

(By "easily and cleanly" I mean without meta-programming tricks, like
instead of real arguments accept "*args, **kwargs" and then munge args).


Thanks,
 Paul                          mailto:pmiscml at gmail.com

From jcea at jcea.es  Sun Mar 29 21:03:46 2015
From: jcea at jcea.es (Jesus Cea)
Date: Sun, 29 Mar 2015 21:03:46 +0200
Subject: [Python-Dev] MemoryError and other bugs on AMD64 OpenIndiana 3.x
In-Reply-To: <CAMpsgwY1DN-SzRv7NkakLoaLkUrP+ewsXKUWUk9T7iezZ7F-ng@mail.gmail.com>
References: <CAMpsgwY1DN-SzRv7NkakLoaLkUrP+ewsXKUWUk9T7iezZ7F-ng@mail.gmail.com>
Message-ID: <55184C92.9050103@jcea.es>

Sorry, I am currently traveling. Away from real computer until April 7th.


On 28/03/15 10:48, Victor Stinner wrote:
> Hi,
> 
> The buildbot AMD64 OpenIndiana 3.x is always red since at least 6
> months. Almost always, tests fail because the buildbot has not enough
> memory. Well, it's fun to see how Python handles MemoryError (it's
> pretty good, there is no hard crash! my work on failmalloc was maybe
> useful ;-)), but it doesn't help to detect regressions.
> 
> Would it be possible to enhance this buildbot?

Those buildbots are hosted in OpenIndiana infraestructure. I take care
of the Solaris Zone (similar to docker but better :-)) but I can't
interact with the "real" machine. Sometimes changes in the "real"
machine had impacted us and nobody realized until some time later, when
nobody remembers what was changed or who did it.

I have contacted the machine manager and he has said to me that 16GB
were taken in the "/tmp" directory by buildbot user. He has deleted them
and everything should be fine now. Lets see.

Anyway if buildbot is leaving garbage behind we should take care of it.
Doing a daily cron for manual cleaning up seems a bit hacky :).

I would be interested, too, in getting an email when one of my buildbots
is red for more than, lets say, 4 days in a row. An email per day would
be fine.

BTW, buildbot pages are painfully slow. The manager had to "insist" and
be patient to see
<http://buildbot.python.org/all/waterfall?show=x86%20OpenIndiana%203.x>,
etc.

How can that be improved?.

> I copy this email to the owner of buildbot. But I also sent it to
> python-dev because MemoryError is not the only error, please take care
> of our buildbots!

Thanks for keeping me in the loop. I hope everything is going green for
a while now :)

-- 
Jes?s Cea Avi?n                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
Twitter: @jcea                        _/_/    _/_/          _/_/_/_/_/
jabber / xmpp:jcea at jabber.org  _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150329/d6c6055f/attachment.sig>

From mark at hotpy.org  Sun Mar 29 22:35:53 2015
From: mark at hotpy.org (Mark Shannon)
Date: Sun, 29 Mar 2015 21:35:53 +0100
Subject: [Python-Dev] Builtin functions are magically static methods?
In-Reply-To: <20150329211608.13d3f2da@x230>
References: <20150329211608.13d3f2da@x230>
Message-ID: <55186229.70603@hotpy.org>



On 29/03/15 19:16, Paul Sokolovsky wrote:
> Hello,
>
> I looked into porting Python3 codecs module to MicroPython and saw
> rather strange behavior, which is best illustrated with following
> testcase:
>
> ==========
> def foo(a):
>      print("func:", a)
>
> import _codecs
> fun = _codecs.utf_8_encode
> #fun = hash
> #fun = str.upper
> #fun = foo
>
>
> class Bar:
>      meth = fun
>
> print(fun)
> print(fun("foo"))
> b = Bar()
> print(b.meth("bar"))
> ==========
>
> Uncommenting either _codecs.utf_8_encode or hash (both builtin
> functions) produces 2 similar output lines, which in particular means
> that its possible to call a native function as (normal) object method,
> which then behaves as if it was a staticmethod - self is not passed to
> a native function.
>
> Using native object method in this manner produces error of self type
> mismatch (TypeError: descriptor 'upper' for 'str' objects doesn't apply
> to 'Bar' object).
>
> And using standard Python function expectedly produces error about
> argument number mismatch, because used as a method, function gets extra
> self argument.
>
> So the questions are:
>
> 1. How so, the native functions exhibit such magic behavior? Is it
> documented somewhere - I never read or heard about that (cannot say I
> read each and every word in Python reference docs, but read enough. As
> an example, https://docs.python.org/3/library/stdtypes.html#functions
> is rather short and mentions difference in implementation, not in
> meta-behavior).

In fact the "magic" is exhibited by Python functions, not by builtin 
ones. Python functions are descriptors, builtin functions are not.

>
> 2. The main question: how to easily and cleanly achieve the same
> behavior for standard Python functions? I'd think it's staticmethod(),
> but:
>

Write your own "BuiltinFunction" class which has the desired properties, 
ie. it would be callable, but not a descriptor.
Then write a "builtin_function" decorator to produce such an object from 
a function. The class and decorator could be the same object.

Personally, I think that such a class (plus a builtin function type that 
behaved like a Python function) would be a useful addition to the 
standard library. Modules do get converted from Python to C and vice-versa.

>>>> staticmethod(lambda:1)()
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
> TypeError: 'staticmethod' object is not callable
>
> Surprise.
>
> (By "easily and cleanly" I mean without meta-programming tricks, like
> instead of real arguments accept "*args, **kwargs" and then munge args).
>
>
> Thanks,
>   Paul                          mailto:pmiscml at gmail.com

Cheers,
Mark

From victor.stinner at gmail.com  Mon Mar 30 02:26:07 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 30 Mar 2015 02:26:07 +0200
Subject: [Python-Dev] Sporadic failures of test_subprocess and
	test_multiprocessing_spawn
In-Reply-To: <CAPTjJmqLGfDs8St5FRMm7eMV+w7=WAyJyVdjDheiKMeUCOHqvw@mail.gmail.com>
References: <CAMpsgwbNZJyD_ekNoyxMtSg7oqxzKU-=LGbPkzQW8hsdFwQh7g@mail.gmail.com>
 <CAPTjJmrw0tKpJrksfv3nxayZFB=ZJn4Z=kgayHrCr12ObFLH=w@mail.gmail.com>
 <CAPTjJmqLGfDs8St5FRMm7eMV+w7=WAyJyVdjDheiKMeUCOHqvw@mail.gmail.com>
Message-ID: <CAMpsgwZ7FR0=JKj3O3kq3RckRv5MHKUf_45rE92KY8eKVq56Ww@mail.gmail.com>

Hi,

2015-03-28 12:26 GMT+01:00 Chris Angelico <rosuav at gmail.com>:
> It seems to be stalling out. I'm not sure exactly what's happening
> here. Running just that one file doesn't do it, but running the full
> test suite results in a stall.

Ok, I reproduced the issue on David's buildbot. A (child) process was
stuck in _Py_open(), function called from _close_open_fds_safe() which
is called to run a child process.

Calling _Py_open() is not safe here because the GIL may or may not be
held. After fork, the status of the GIL is unclear.

I fixed the issue by replacing _Py_open() with _Py_open_noraise(),
this version doesn't use the GIL (it doesn't release the GIL to call
open() and it doesn't raise an exception).
https://hg.python.org/cpython/rev/2e1234208bab

Thanks David for the buildbot.

Victor

From victor.stinner at gmail.com  Mon Mar 30 02:31:45 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 30 Mar 2015 02:31:45 +0200
Subject: [Python-Dev] Sporadic failures of test_subprocess and
	test_multiprocessing_spawn
In-Reply-To: <mf6li5$6tq$1@ger.gmane.org>
References: <CAMpsgwbNZJyD_ekNoyxMtSg7oqxzKU-=LGbPkzQW8hsdFwQh7g@mail.gmail.com>
 <mf6li5$6tq$1@ger.gmane.org>
Message-ID: <CAMpsgwbxY7Xh1P4aqcGM_ZJS=N8V6HPFvjGZNCFTqCkWNvJztA@mail.gmail.com>

Hi Serhiy,

2015-03-28 17:40 GMT+01:00 Serhiy Storchaka <storchaka at gmail.com>:
> Just run tests with low memory limit.
>
> (ulimit -v 60000; ./python -m test.regrtest -uall -v
> test_multiprocessing_spawn;)
>
> test_io also hangs.

I confirm that some tests using threads hang under very low memory
condition. At bootstrap, Python doesn't handle correctly all
exceptions when starting a new thread. The "parent thread" waits until
the "child thread" completes, which never occurs because the thread
already completed.

I don't think that it was my regression, it probably exists since
Python 2 and maybe before. I'm not interested to touch this fragile
part of Python. It's maybe easy to fix the issue, I don't know.

Victor

From storchaka at gmail.com  Mon Mar 30 08:06:51 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 30 Mar 2015 09:06:51 +0300
Subject: [Python-Dev] cpython: Issue #23752: When built from an existing
 file descriptor, io.FileIO() now only
In-Reply-To: <20150330012250.1925.16638@psf.io>
References: <20150330012250.1925.16638@psf.io>
Message-ID: <mfap5t$rbq$1@ger.gmane.org>

On 30.03.15 04:22, victor.stinner wrote:
> https://hg.python.org/cpython/rev/bc2a22eaa0af
> changeset:   95269:bc2a22eaa0af
> user:        Victor Stinner <victor.stinner at gmail.com>
> date:        Mon Mar 30 03:21:06 2015 +0200
> summary:
>    Issue #23752: When built from an existing file descriptor, io.FileIO() now only
> calls fstat() once. Before fstat() was called twice, which was not necessary.
>
> files:
>    Misc/NEWS            |  26 ++++++++++++++++++++++++++
>    Modules/_io/fileio.c |  24 ------------------------
>    2 files changed, 26 insertions(+), 24 deletions(-)
>
>
> diff --git a/Misc/NEWS b/Misc/NEWS
> --- a/Misc/NEWS
> +++ b/Misc/NEWS
> @@ -2,6 +2,32 @@
>   Python News
>   +++++++++++
>
> +What's New in Python 3.5.0 alpha 4?
> +===================================

Return time machine back Victor. Current version is 3.5.0a2+.



From victor.stinner at gmail.com  Mon Mar 30 09:30:02 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 30 Mar 2015 09:30:02 +0200
Subject: [Python-Dev] cpython: Issue #23752: When built from an existing
 file descriptor, io.FileIO() now only
In-Reply-To: <mfap5t$rbq$1@ger.gmane.org>
References: <20150330012250.1925.16638@psf.io> <mfap5t$rbq$1@ger.gmane.org>
Message-ID: <CAMpsgwYypBx5+e9mOvPkcNMoQpKd4_QKm7MyiGodB9kpO-sD2A@mail.gmail.com>

Hi,

2015-03-30 8:06 GMT+02:00 Serhiy Storchaka <storchaka at gmail.com>:
>> +What's New in Python 3.5.0 alpha 4?
>> +===================================
>
> Return time machine back Victor. Current version is 3.5.0a2+.

Larry Hastings and Ned Deily told me that Larry is releasing Python
3.5 alpha 3. You can see their repository at:
https://hg.python.org/releasing/3.5/

According to the PEP 478 - Python 3.5 Release Schedule, we are in time:
3.5.0 alpha 3: March 29, 2015

Maybe Misc/NEWS was not updated when the alpha 2 was released? It's
always a mess to maintain this file... We should generate it from
Mercurial changelogs instead.

So is it correct to use "alpha 4" in Misc/NEWS?

Victor

From larry at hastings.org  Mon Mar 30 10:44:10 2015
From: larry at hastings.org (Larry Hastings)
Date: Mon, 30 Mar 2015 01:44:10 -0700
Subject: [Python-Dev] cpython: Issue #23752: When built from an existing
 file descriptor, io.FileIO() now only
In-Reply-To: <CAMpsgwYypBx5+e9mOvPkcNMoQpKd4_QKm7MyiGodB9kpO-sD2A@mail.gmail.com>
References: <20150330012250.1925.16638@psf.io> <mfap5t$rbq$1@ger.gmane.org>
 <CAMpsgwYypBx5+e9mOvPkcNMoQpKd4_QKm7MyiGodB9kpO-sD2A@mail.gmail.com>
Message-ID: <55190CDA.4070203@hastings.org>


On 03/30/2015 12:30 AM, Victor Stinner wrote:
> So is it correct to use "alpha 4" in Misc/NEWS?

It's the release manager's responsibility to update the version number 
in Misc/NEWS.  As I finish up 3.5.0a3, I will be changing it to say 
"What's New in Python 3.5.0 alpha 4?".  And I will change the release 
number in patchlevel.h to say "3.5.0a3+".  I don't know why it is this 
way, but that's the way it is, and I assume there's a good reason for it.

Since you don't have the 3.5.0a3 changes yet, no, it's not correct for 
you to put "alpha 4" into Misc/NEWS.  Doing so would trip me up a little.

The release process is mostly automated, although there are manual steps 
where the RM is guided through the process.  Fixing up the heading in 
Misc/NEWS is one such manual step.  FWIW, I have in the past bungled a 
release so badly that I missed making this change.  I was happy to 
discover someone else had fixed it for me.  So I'm not saying "nobody 
but the RM should ever touch it".  But maybe you should wait a day or 
two after the release comes out before you start up your text editor.

Cheers,


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150330/0bfd1d6c/attachment.html>

From larry at hastings.org  Mon Mar 30 10:46:38 2015
From: larry at hastings.org (Larry Hastings)
Date: Mon, 30 Mar 2015 01:46:38 -0700
Subject: [Python-Dev] [RELEASED] Python 3.5.0a3 is now available
Message-ID: <55190D6E.8030300@hastings.org>



On behalf of the Python development community and the Python 3.5 release 
team, I'm thrilled to announce the availability of Python 3.5.0a3.   
Python 3.5.0a3 is the third alpha release of Python 3.5, which will be 
the next major release of Python.  Python 3.5 is still under heavy 
development, and is far from complete.

This is a preview release, and its use is not recommended for production 
settings.

Two important notes for Windows users about Python 3.5.0a3:

  * If you have previously installed Python 3.5.0a1, you may need to
    manually uninstall it before installing Python 3.5.0a3 (issue23612).
  * If installing Python 3.5.0a3 as a non-privileged user, you may need
    to escalate to administrator privileges to install an update to your
    C runtime libraries.


You can find Python 3.5.0a3 here:

    https://www.python.org/downloads/release/python-350a3/

Happy hacking,


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150330/7a36ec5b/attachment.html>

From victor.stinner at gmail.com  Mon Mar 30 10:57:50 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 30 Mar 2015 10:57:50 +0200
Subject: [Python-Dev] [python-committers] [RELEASED] Python 3.5.0a3 is
	now available
In-Reply-To: <55190D6E.8030300@hastings.org>
References: <55190D6E.8030300@hastings.org>
Message-ID: <CAMpsgwYiYQ_6V0wUbe0TZqLko+VdrdVkM_4EEG0aopJKB6GQUA@mail.gmail.com>

Hi,

2015-03-30 10:46 GMT+02:00 Larry Hastings <larry at hastings.org>:
> On behalf of the Python development community and the Python 3.5 release
> team, I'm thrilled to announce the availability of Python 3.5.0a3.   Python
> 3.5.0a3 is the third alpha release of Python 3.5, which will be the next
> major release of Python.  Python 3.5 is still under heavy development, and
> is far from complete.

Between alpha2 and alpha3, the implementation of the PEP 471
(os.scandir) and 475 (EINTR) made progress. For scandir, os.walk() now
uses os.scandir() internally instead of os.listdir() and so should be
(much) faster.

These PEPs are very dependent on the platform, so I'm interested of
feedback if you get any bug related to these PEPs. For example, Robert
Collins told me that fixtures test started to hang. The reason is that
time.sleep() was modified to restart on EINTR and fixtures test used
an exception handler which expected time.sleep() to return on signal.
The signal handler only modified a flag without raising an exception.
Robert quickly fixed his test.

I started to document EINTR changes in the documentation of each
function, and I added a note in the "Porting to Python 3.5" section of
What's New in Python 3.5:
https://docs.python.org/dev/whatsnew/3.5.html#porting-to-python-3-5

By the way, Python now always requires a 64-bit signed integer type to
work. Python internal functions now store timestamps as a number of
nanoseconds in a 64-bit integer. It looks to be the case on all
buildbots.

Victor

From ncoghlan at gmail.com  Mon Mar 30 15:11:42 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 30 Mar 2015 23:11:42 +1000
Subject: [Python-Dev] RFC: PEP 490 - Chain exceptions at C level
In-Reply-To: <CAMpsgwY23Vze5oGPu97cBMxHumu93Ss-Pi1Gtq95cuZvwgP5Uw@mail.gmail.com>
References: <CAMpsgwY23Vze5oGPu97cBMxHumu93Ss-Pi1Gtq95cuZvwgP5Uw@mail.gmail.com>
Message-ID: <CADiSq7dZt_OD6Fwjfo3pRHqPayb6Xa-gKyuo2OVRXhP_pFY2dw@mail.gmail.com>

On 26 March 2015 at 22:30, Victor Stinner <victor.stinner at gmail.com> wrote:
> Hi,
>
> Here is the first draft of my PEP to chain automatically exceptions at
> C level in Python 3.5. Plain text version of the PEP below. HTML
> version of the PEP:
>
>    https://www.python.org/dev/peps/pep-0490/
>
> It has already an implementation, the pyerr_chain.patch file attached to
> http://bugs.python.org/issue23763
>
> The patch is very short.

I like the idea, but I think it would raise the priority of the
proposed change to the display of chained exceptions in
http://bugs.python.org/issue12535

That proposal suggests adding the line "<truncated: another exception
occurred>" to the start of earlier tracebacks in the chain so you get
an up front hint that a chained exception will be shown after the
initial traceback.

> Backward compatibility
> ======================
>
> A side effect of chaining exceptions is that exceptions store
> traceback objects which store frame objects which store local
> variables.  Local variables are kept alive by exceptions. A common
> issue is a reference cycle between local variables and exceptions: an
> exception is stored in a local variable and the frame indirectly
> stored in the exception. The cycle only impacts applications storing
> exceptions.
>
> The reference cycle can now be fixed with the new
> ``traceback.TracebackException`` object introduced in Python 3.5. It
> stores informations required to format a full textual traceback without
> storing local variables.

Also potentially worth noting here is "traceback.clear_frames()",
introduced in 3.4 to explicitly clear local variables from a
traceback: https://docs.python.org/3/library/traceback.html#traceback.clear_frames

PEP 442 also makes it more likely that cycles will be cleared
properly, even if some of the objects involved have __del__ methods:
https://www.python.org/dev/peps/pep-0442/

However, I'll grant that neither of those can be backported the way
that traceback.TracebackException can be.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From aarmour at cipmail.org  Tue Mar 31 17:13:25 2015
From: aarmour at cipmail.org (Alan Armour)
Date: Tue, 31 Mar 2015 11:13:25 -0400
Subject: [Python-Dev] importante!!
Message-ID: <CAC=V4WdhPHsec84AGzs290xQQuQgCKXQdFvwOY0z1miX6TjmDQ@mail.gmail.com>

its french! lol

I just wanted to see if you could, as well as making python able to have
assembly written, you should totally use blender as your main IDE it would
be so epic

and do pyjackd, and i think pyaudio will write some stuff, and like
pycoreaudio and like directpy for windows direct x

oh having blender as the main ide would just be epic
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150331/c9c448cb/attachment.html>

From marky1991 at gmail.com  Tue Mar 31 17:36:45 2015
From: marky1991 at gmail.com (Mark Young)
Date: Tue, 31 Mar 2015 11:36:45 -0400
Subject: [Python-Dev] importante!!
In-Reply-To: <CAC=V4WdhPHsec84AGzs290xQQuQgCKXQdFvwOY0z1miX6TjmDQ@mail.gmail.com>
References: <CAC=V4WdhPHsec84AGzs290xQQuQgCKXQdFvwOY0z1miX6TjmDQ@mail.gmail.com>
Message-ID: <CAG3cHaZLuVL=Hg+dnYFhu+pYzPC99vVQAvksztEVn=257gx1=Q@mail.gmail.com>

Features of blender are relevant on blender mailing lists, not here . (I
don't understand why you would want a 3d modeling program to be an IDE, but
whatever floats your boat) Also, python-dev isn't really the place for
feature requests. If you want something added, add it yourself. : )
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150331/b5a3d86f/attachment.html>

From rymg19 at gmail.com  Tue Mar 31 17:31:21 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Tue, 31 Mar 2015 10:31:21 -0500
Subject: [Python-Dev] importante!!
In-Reply-To: <CAC=V4WdhPHsec84AGzs290xQQuQgCKXQdFvwOY0z1miX6TjmDQ@mail.gmail.com>
References: <CAC=V4WdhPHsec84AGzs290xQQuQgCKXQdFvwOY0z1miX6TjmDQ@mail.gmail.com>
Message-ID: <CAO41-mMrZOqoxRh0Z2xgF0JmmbuYZkyVtVoGgEFabZyG7+ZiYw@mail.gmail.com>

????????????

I think this was meant for python-list.

On Tue, Mar 31, 2015 at 10:13 AM, Alan Armour <aarmour at cipmail.org> wrote:

> its french! lol
>
> I just wanted to see if you could, as well as making python able to have
> assembly written, you should totally use blender as your main IDE it would
> be so epic
>
> and do pyjackd, and i think pyaudio will write some stuff, and like
> pycoreaudio and like directpy for windows direct x
>
> oh having blender as the main ide would just be epic
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com
>
>


-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150331/8107dc62/attachment.html>

From demianbrecht at gmail.com  Tue Mar 31 18:55:32 2015
From: demianbrecht at gmail.com (Demian Brecht)
Date: Tue, 31 Mar 2015 09:55:32 -0700
Subject: [Python-Dev] http.client: Determining Content-Length
Message-ID: <18A92DFA-B232-405C-B894-C5464BF777D3@gmail.com>

Hi all,

I'm not sure whether this should be python-list or here, but given it's a premature code review for http.client, I figured I'd post here first.

I'm in the process of adding support for chunked transfer encoding to http.client (issue #12319). One of the bits of functionality that I'm working on in is ironing out some of the kinks out in determining the content length of the request bodies. Given the number of data types allowed as bodies it does get a little messy, so I wanted to get some feedback here and see if anyone can shoot holes through it prior to updating the current patch. The tests are passing, but there may be use cases not accounted for in the new implementation.

https://gist.github.com/demianbrecht/f94be5a51e32bb9c81e1

The above is intended to replace _set_content_length (current state of the patch can be seen here: http://bugs.python.org/review/12319/diff/14331/Lib/http/client.py). There is a comprehensive list of data types currently supported can be found here: http://bugs.python.org/issue23740. Comments and feedback are much appreciated.

Thanks,
Demian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150331/b6ebe22e/attachment.sig>